perf: optimize heartbeat writes, queries, bucket listings, and exports

[0xbrayo]

Reduce repeated datastore work on watcher and dashboard requests, and stream large exports with bounded event memory.

Changes:

• Reuse bucket handles and direct existence checks instead of deserializing all bucket metadata per request.
• Merge heartbeats with a bucket-scoped UPDATE by event ID. Preserve IDs returned by inserts, invalidate heartbeat state on other writes, and serialize API writes so event mutations cannot race a merge.
• Fetch bucket metadata and last-update times in one disk-backend query.
• Stream JSON exports through datastore cursors in roughly 64 KiB chunks. Preserve the export schema and event ordering, omit event IDs, return missing-bucket errors before streaming, and close cursors on interrupted responses.
• Cache query results by query text, name, and requested period. Track actual bucket/time-range reads, including queries overriding STARTTIME/ENDTIME. Heartbeats invalidate overlapping reads; imports, upserts, deletes, and metadata changes invalidate affected dependencies. Queries overlapping a write are not cached. Cached values are copied, expire after five minutes, and are bounded by 128 entries and an 8 MiB retained-object budget. HTTP requests cache by default; ?cache=0 bypasses caching.
• Defer event/heartbeat debug formatting, with regression tests for disabled logging.

This PR depends on ActivityWatch/aw-core#159. Both pyproject.toml and poetry.lock pin the tested aw-core commit while these coordinated changes are reviewed. The release dependency can replace that pin after the companion changes ship.

Validation:

• make test: 104 passed, 2 deliberate skips for disk backends in the API-only concurrency test.
• Server mypy, Ruff on changed files, Black, and git diff --check passed. Companion core: 253 passed, 2 existing skips; mypy and Ruff passed.
• Temporary Peewee database measurements: listing 100 buckets took 7.41 → 1.40 ms and 101 → 1 SQL statements; warm heartbeat merge uses one UPDATE.
• For 10,000 events, peak Python allocation during export fell from 18.83 MiB to 0.30 MiB. A repeated 10,000-event aggregate query fell from 143 ms uncached to 0.010 ms cached. These are isolated local measurements, not production HTTP latency.
• Reproducible baseline profiling: python scripts/profile-api-query.py --baseline-ref master --queries 5 measured 151.756 ms median; cProfile attributed 1.728 / 1.760 seconds to Peewee get_events (row hydration and date parsing). --cache measured 0.014 ms median. Both use the same installed backend and disposable 10,000-event data.
• Regression coverage spans all three storage backends, lifecycle changes, failed writes, cache invalidation and bounds, concurrent API heartbeats, export equivalence/cancellation, and lazy logging.
• Nine CORS failures in the broader suite reproduce on untouched master.

The cache tracks writes made through this ServerAPI instance. As with the existing heartbeat cache, independently modifying the database outside the running server is not a supported live-coherency mechanism; restarting clears cached state.

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

@greptile review

[greptile-apps]

Confidence Score: 4/5

The PR is not ready to merge until the build workflow can install from the new lockfile and the repository’s profiling requirement is satisfied.

Findings

1.  Lockfile breaks CI builds ▶
2.  Query caching lacks prior profiling ▶

Summary

The PR reduces datastore work for bucket operations and heartbeats, adds bounded query-result caching, and streams JSON exports. It also adds cross-backend regression tests and pins a companion aw-core commit.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Q[Query request] --> C{Cache lookup}
  C -->|hit| R[Copied result]
  C -->|miss| T[Tracked datastore reads]
  T --> S[Store if revision unchanged]
  W[Bucket mutation] --> I[Invalidate dependent entries]
  I --> C

Loading

_{Reviews (2) · Last reviewed commit: "perf: stream exports and cache queries w..."}

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

@greptile review