sync_one has no stored cursor. It infers where to resume by reading the newest event already in the destination:
let most_recent_events = ds_to.get_events(bucket_to.id.as_str(), None, None, Some(1))?;
let resume_sync_at = most_recent_events
.first()
.map(|e| e.timestamp + e.duration)
.or(sync_spec.start);then fetches only events newer than that. The destination copy is being used as its own progress marker.
Consequence 1: events that arrive late never sync, silently
Events are not always appended in timestamp order. An importer (aw-import-*, screentime imports), a retroactively edited stopwatch entry, or any backfill writes events with timestamps older than the bucket's current newest. Those events are permanently invisible to sync: the cursor is already past them, and nothing ever looks back. No error, no warning — the pass reports ✓ Already up to date!.
Consequence 2: it is the mechanism behind #683
Because the cursor lives in the destination rather than being per-source, two folders for one device write into the same destination bucket and the first one to be imported sets the resume point for the second. That is how a 4,068-event partial slice silently suppressed 1,027,343 events of real history. #686 fixes the trigger (duplicate folders) but the amplifier is this cursor design.
Consequence 3: it forces the fragile pagination
Because the cursor is a timestamp, sync_one has to reason carefully about events sharing an identical timestamp at page boundaries — the boundary_ts tie-handling loop, with a documented pathological case it explicitly declines to handle. A monotonic source-side cursor removes that reasoning entirely.
Proposal
Store an explicit cursor per (source_device_id, source_bucket_id) on the destination, keyed by the source's events.id rowid, which is monotonic in insertion order:
sync_cursor(source_device_id, source_bucket_id, last_source_rowid, updated)
- Fetch
WHERE id > last_source_rowid ORDER BY id LIMIT n — no timestamp ties, no boundary loop, and late-arriving events are picked up because they get a new rowid even though their timestamp is old.
- Initialise from today's resume-from-newest so existing installs migrate with no re-import.
- Per-source keying means two folders for one device cannot contaminate each other's progress.
Caveat worth designing around: rowids are only monotonic within one source database. If a peer's staging db is rebuilt from scratch its rowids restart, so the cursor needs to be invalidated when the source's identity or generation changes — which is another argument for per-device metadata in the folder (ActivityWatch/activitywatch#302, #691). A cheap interim guard is to store the source's event count alongside the cursor and reset if it goes backwards.
Credit: found in a design review of aw-sync.
Related: #683, #686, #691, ActivityWatch/activitywatch#302.
cc @TimeToBuildBob
sync_onehas no stored cursor. It infers where to resume by reading the newest event already in the destination:then fetches only events newer than that. The destination copy is being used as its own progress marker.
Consequence 1: events that arrive late never sync, silently
Events are not always appended in timestamp order. An importer (
aw-import-*, screentime imports), a retroactively edited stopwatch entry, or any backfill writes events with timestamps older than the bucket's current newest. Those events are permanently invisible to sync: the cursor is already past them, and nothing ever looks back. No error, no warning — the pass reports✓ Already up to date!.Consequence 2: it is the mechanism behind #683
Because the cursor lives in the destination rather than being per-source, two folders for one device write into the same destination bucket and the first one to be imported sets the resume point for the second. That is how a 4,068-event partial slice silently suppressed 1,027,343 events of real history. #686 fixes the trigger (duplicate folders) but the amplifier is this cursor design.
Consequence 3: it forces the fragile pagination
Because the cursor is a timestamp,
sync_onehas to reason carefully about events sharing an identical timestamp at page boundaries — theboundary_tstie-handling loop, with a documented pathological case it explicitly declines to handle. A monotonic source-side cursor removes that reasoning entirely.Proposal
Store an explicit cursor per
(source_device_id, source_bucket_id)on the destination, keyed by the source'sevents.idrowid, which is monotonic in insertion order:WHERE id > last_source_rowid ORDER BY id LIMIT n— no timestamp ties, no boundary loop, and late-arriving events are picked up because they get a new rowid even though their timestamp is old.Caveat worth designing around: rowids are only monotonic within one source database. If a peer's staging db is rebuilt from scratch its rowids restart, so the cursor needs to be invalidated when the source's identity or generation changes — which is another argument for per-device metadata in the folder (ActivityWatch/activitywatch#302, #691). A cheap interim guard is to store the source's event count alongside the cursor and reset if it goes backwards.
Credit: found in a design review of aw-sync.
Related: #683, #686, #691, ActivityWatch/activitywatch#302.
cc @TimeToBuildBob