Skip to content

perf(datastore): stream exports and remove redundant event reads - #159

Open
0xbrayo wants to merge 2 commits into
ActivityWatch:masterfrom
0xbrayo:perf/datastore-hotpaths
Open

0xbrayo wants to merge 2 commits into
ActivityWatch:masterfrom
0xbrayo:perf/datastore-hotpaths

Conversation

@0xbrayo

@0xbrayo 0xbrayo commented Sep 24, 2026

Copy link
Copy Markdown
Member

Add datastore operations needed to reduce aw-server's request overhead and stream exports without loading entire event lists:

  • Check bucket existence directly, including cold bucket-handle lookups.
  • Fetch bucket metadata and each bucket's last update in one SELECT on Peewee and SQLite, preserving each backend's existing latest-event ordering and including empty buckets.
  • Iterate disk-backed events through cursors, closing cursors on exhaustion or early cancellation. Memory storage copies events individually.
  • Replace events with a bucket-scoped UPDATE and a boolean result, without a preceding SELECT; cross-bucket IDs cannot move or overwrite another bucket's event.
  • Return isolated memory-bucket metadata so export/listing additions do not mutate stored metadata.

Validation: 253 tests passed, 2 existing skips, using temporary platform directories. New regression tests cover all three storage backends, statement counts, duplicate timestamps, empty buckets, iterator cancellation, and scoped updates. Ruff and mypy passed. Also corrected the existing version-error format arguments exposed by mypy.

Companion server PR: ActivityWatch/aw-server#173. The server branch pins this dependency commit while the coordinated changes are reviewed.

@greptile-apps

greptile-apps Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

The PR should not merge until Peewee cold lookups can discover buckets created by another process.

Findings

  1. P1 Cold lookups miss new buckets ▶

Summary

The PR adds direct bucket checks, single-query bucket listings with last-update values, cursor-backed event iteration, bucket-scoped replacements, and isolated memory metadata.

  • The Peewee existence check relies on a cache that can miss buckets created by another process.

Reviews (1) · Last reviewed commit: "perf(datastore): add streaming reads and..."

Comment thread aw_datastore/storages/peewee.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant