aw-sync: sync failures are silent — add `aw-sync status`, loud diagnostics, per-device manifest, and surface peers in Raw Data

[ErikBjare]

Setting up sync and having it silently do nothing is currently indistinguishable, from the user's side, from setting it up correctly. Two concrete bugs that produce exactly that are filed separately (#682 daemon layout mismatch, #683 duplicate-folder truncation) — this issue is about the fact that neither was diagnosable without reading the source and opening the peer sqlite files by hand.

In the case that prompted this, the daemon ran 102 sync passes over ~19 hours, logged Pulling... 102 times, imported nothing, and reported no problem at any level: not in the log, not in the UI, not in the sync folder.

1. Fail loud

sync_run() logs Pulling... unconditionally and then iterates an empty vec. It only logs when remotes are found:

if !remote_dbfiles.is_empty() {
    info!("Found {} remote db files: {:?}", ...);
}

Finding zero peers in a directory the user explicitly configured for sync is the interesting case, not the boring one. It should warn! with the glob that was tried and the directory entries that were skipped and why. Same for a peer database that fails to open, or whose schema version is unrecognised (there is already a TODO: Check for compatible remote db version before opening).

2. aw-sync status

A doctor subcommand, dumping in one screen:

• resolved sync dir, profile, local hostname + device_id, target server + port
• every directory found in the sync dir, classified: recognised peer / own staging copy / unrecognised
• per peer: device_id, hostname as recorded on its buckets, db size, bucket count, newest event timestamp, and whether it has been imported locally
• warnings for: two folders sharing a device_id, folder name ≠ bucket hostname, staging copy older than the local server's newest event, peers not imported

list_buckets() is already 80% of the data-gathering for this; it just isn't reachable in a form that answers "why is my phone missing".

This one command replaces the entire manual investigation that produced these issues.

3. Per-device manifest.json

Write a small JSON file next to each test.db:

{
  "device_id": "41662faa-...",
  "hostname": "poco_f8_ultra",
  "display_name": "POCO F8 Ultra",
  "aw_version": "v0.14.2b1",
  "last_push": "2026-09-14T19:54:12Z",
  "buckets": [{"id": "aw-watcher-android", "events": 1027343, "last_event": "..."}]
}

Cheap to write, and it buys:

• listing peers without opening multi-hundred-MB sqlite files (the database in this setup is 273 MB; the local staging copy 1.19 GB)
• dedupe by device_id across folders — aw-sync: duplicate folders for one device_id silently truncate history on pull #683 becomes detectable at read time
• a stable place for the display name, decoupled from both the folder name and the bucket hostname column, which is the root of the rename problem (Any way to rename the host? I want to merge my devices aw-android#133)
• status and the UI can show "last pushed 3 days ago" for a peer that has stopped syncing

4. Surface sync peers in the existing Raw Data page

Correction to an earlier draft of this: the "Devices" view already exists. /buckets (Raw Data) groups by device via the bucketsByDevice getter in aw-webui/src/stores/buckets.ts, shows hostname + device ID + "this device" badge + last-updated with a recency highlight + desktop/mobile icon, and already runs consistency checks per device:

runChecks: function (device) {
  // "Device known by several hostnames: ..."
  // "Device known by several IDs: ..."
}

So the structure to extend is there, and runChecks is the natural home for sync warnings. The gap is that the page can only see devices that already have buckets locally — it is built from this.buckets. A device that is present in the sync folder but failing to import is invisible on the one page a user would look at to find it, which is precisely the failure mode above.

Proposed: expose sync-folder peers (from manifest.json, or a /api/0/sync/peers-style endpoint) and render them on the same page as devices in a "in sync folder, not imported" state, with the reason. Plus a runChecks entry for "this device has not published to the sync folder since {date}" — which would have caught the 8-month-stale local staging copy on its own.

5. Unify the two folder walkers

find_remotes() / find_remotes_nonlocal() / get_remotes() are three different directory walks with two different depth assumptions, and the source already carries // TODO: share logic with find_remotes and find_remotes_nonlocal. One walker returning a typed Vec<Peer> — rather than Vec<PathBuf> that callers re-derive meaning from — removes the class of bug entirely.

6. Smaller items

• Name the staging database after the device rather than test.db — it reads as leftover test scaffolding in a directory users are told to inspect.
• Clean up (or at least warn about) staging databases containing …-synced-from-… buckets left over from before fix(sync): never re-sync buckets synced from another host #648; they are re-exported peer data that nothing prunes.
• First-run flow: aw-qt currently has no sync entry in autostart_modules and no way to pick a sync directory, so getting to a working setup means finding the binary inside the .app bundle and knowing that sync and daemon behave differently.

cc @TimeToBuildBob

[TimeToBuildBob]

Opened #687 for items 1–2 of this issue: fail-loud empty-pull warnings, plus aw-sync status.

aw-sync status is read-only (no staging test.db created, sqlite opened READ_ONLY so it does not drop -wal/-shm into a Syncthing folder). It classifies both layouts, inspects peer dbs, and warns on duplicate device_id (#683), hostname mismatch, unpublished own staging, and unimported peers — the investigation that produced this issue, in one command.

Not in that PR, still open here:

• per-device manifest.json (item 3)
• Raw Data / aw-webui peer surfacing (item 4) — different repo
• walker unification (item 5) — fix(aw-sync): make default daemon use host-layout pull/push #685
• first-run / test.db rename (item 6) — status warns about test.db; it does not rename

#685 is the actual pull fix; #687 is the doctor. They touch the same files in different hunks and should merge either order.

[ErikBjare]

@TimeToBuildBob Correction to the last line above — #685 and #687 do not merge in either order:

#685 + #686 => CONFLICT  aw-sync/src/util.rs
#685 + #687 => CONFLICT  aw-sync/src/util.rs  (2 hunks, ~430 and ~180 lines)
#686 + #687 => clean

MERGEABLE on each PR is only against master. Full review on each PR; suggested order is #686 → #687 → #685 rebased, so the walkers consolidate onto RemoteDb instead of item 5 going the wrong way (the three PRs currently add five or six directory walks to util.rs between them).

[TimeToBuildBob]

@ErikBjare You're right — MERGEABLE was only vs master, and the "either order" note was wrong. Reproduced:

#685 + #686 => CONFLICT  aw-sync/src/util.rs
#685 + #687 => CONFLICT  aw-sync/src/util.rs
#686 + #687 => clean

Resequenced as you asked:

1. fix(aw-sync): skip duplicate device_id folders that truncate history on pull #686 first — RemoteDb is the typed-peer walker. 3-level-only is explicit (comment + test). Dead get_remotes / contains_db_file / contains_subdir_with_db_file trio removed here, not on fix(aw-sync): make default daemon use host-layout pull/push #685.
2. feat(aw-sync): add status doctor command and fail-loud empty-pull warnings #687 on top of that — status consumes list_remote_dbs + select_remote_dbs_by_device_id; leftover 2-level / unrecognised entries sit on top of that list.
3. fix(aw-sync): make default daemon use host-layout pull/push #685 rebased last — daemon host-layout switch + Option<Datastore> staging fix only. Walker rewrite dropped.

All three now merge clean in any pairing. Still pull-only on this repo, so they need a maintainer click. Suggested order remains #686 → #687 → #685.

ActivityWatch/aw-android#272 is still the source of the duplicate folders #686 defends against.

Still open on this issue: per-device manifest.json (item 3) and Raw Data peer surfacing (item 4, aw-webui).

[TimeToBuildBob]

Loop closed on the remaining items, now that #687 merged and this issue is closed:

• Items 1–2 (aw-sync status + fail-loud empty-pull): shipped in feat(aw-sync): add status doctor command and fail-loud empty-pull warnings #687 (988e606)
• Item 5 (typed walker / RemoteDb): shipped in fix(aw-sync): skip duplicate device_id folders that truncate history on pull #686
• fix(aw-sync): make default daemon use host-layout pull/push #685 stays parked (DIRTY) per Sync: tracking issue and v0.14.0 release triage activitywatch#1445 — not restacking it
• Items 3–4 (per-device manifest, Raw Data peer surfacing) stay on the post-v0.14.0 sequence already written on Sync: tracking issue and v0.14.0 release triage activitywatch#1445: feat(aw-sync): return and persist a SyncReport from each pass #699 SyncReport → manifest → … Not filing duplicate issues.

Nothing further on this thread.