From 028d7706e7cd2f362717056cac8950f3fad4dde1 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Wed, 23 Sep 2026 16:13:15 -0700 Subject: [PATCH] feat(reconcile): walk Proton once the last walk is a week old The Proton walk ran on the first run of a configured UTC weekday, tying it to when a run is dispatched, and a walk left partial waited for the next such day. lib v2.2.0's toolbox now decides: `due` leaves .run/reconcile once the last complete walk, at .state/reconciled, is RECONCILE_HOURS old, and this mirror sets 168. The run row takes the flag; a complete walk leaves .run/walked, and record-walk then runs the toolbox's `reconciled`, so a partial walk stays due and the next run resumes it. pull and push over the image's s3 are what due and reconciled keep the key through. reconcile.weekday, the RECONCILE environment variable and is_reconcile_run are gone. --- CLAUDE.md | 13 ++++--- README.md | 12 +++--- Taskfile.yml | 41 ++++++++++++++++---- config/mirror.toml | 3 -- render.txt | 21 +++++++++- src/migrator/commands.py | 29 +++----------- src/migrator/config.py | 11 +----- src/migrator/env.py | 2 - src/migrator/paths.py | 10 +++++ src/migrator/phases/p60_reconcile.py | 1 + tests/conftest.py | 3 +- tests/test_commands.py | 57 +++++++++------------------- tests/test_config.py | 3 -- tests/test_env.py | 3 -- tests/test_reconcile.py | 2 + 15 files changed, 105 insertions(+), 106 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b9c265f..618789b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,9 +8,10 @@ this mirror includes. This file is what a change must not break. Nothing in this repo starts a run: an external scheduler dispatches `sync.yml` nightly. This mirror includes the toolbox alone: `Taskfile.yml` owns the order, one -`python -m migrator ` per step, and `src/migrator/` owns every decision. The -infrastructure modules there are from donphi/dropbox_proton at `cfd0e57`, MIT; the phases -under `src/migrator/phases/` and the Taskfile are this repository's own. +`python -m migrator ` per step, and `src/migrator/` owns every decision but when +to walk Proton, which is lib's `due`. The infrastructure modules there are from +donphi/dropbox_proton at `cfd0e57`, MIT; the phases under `src/migrator/phases/` and the +Taskfile are this repository's own. ## Must knows @@ -42,8 +43,10 @@ under `src/migrator/phases/` and the Taskfile are this repository's own. - **The logs are public.** The report is built from the state and carries counts only; errors print as their class unless `MIRROR_VERBOSE=1`; `op run` masks every value. - **Run flags go after the double dash** (`task sync -- RUN_BUDGET_MIN=30 RECONCILE=true`, - or the workflow's `vars` input); the Taskfile maps them to the environment the migrator - reads. `RECONCILE` takes the literal word `true`. + or the workflow's `vars` input). The Taskfile maps `RUN_BUDGET_MIN` to the environment + the migrator reads; `RECONCILE` (`true`, `false` or `auto`) is lib's `due`'s, which + leaves `.run/reconcile` for the migrator when the last complete walk is + `RECONCILE_HOURS` (168) old. - **`config/mirror.toml` is strict** and rejects unknown keys; the three account identifiers come from the environment and override its keys. diff --git a/README.md b/README.md index 347b370..61d2ab9 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ flowchart LR | `confirm` | The upload summary must account for every verified file plus every folder, and every failure must name a file in the batch. Those alone are recorded as failed; the rest confirm | | `checkpoint` | Merges the confirmed rows into `mirror_objects` and pushes the state to the bucket, a dated copy first and then the canonical key. Always the last step of a batch, so a killed run repeats at most one | | `trash` | Only when every planned batch landed. A topmost folder the mirror holds nothing live under goes in one `filesystem trash` call, subtree and all; a folder still holding live files gets its deleted files trashed by name, 50 paths per call. Each unit's `mirror_objects` rows are dropped as it lands. Checkpoints every 50 units, stops at the run budget and chains the next run for the rest | -| `reconcile` | On the first run of the configured weekday, or with `RECONCILE=true`: a full Proton walk compared against `mirror_objects`. Rows Proton lacks or mis-sizes are dropped so they re-upload; nodes neither Dropbox nor the state knows are trashed. A walk that does not fit one run resumes on the next, and a partial walk drops and trashes nothing | +| `reconcile` | Once the last complete walk is `RECONCILE_HOURS` (168, a week) old, by lib's `due`, or with `RECONCILE=true`: a full Proton walk compared against `mirror_objects`. Rows Proton lacks or mis-sizes are dropped so they re-upload; nodes neither Dropbox nor the state knows are trashed. A walk that does not fit one run resumes on the next, and a partial walk drops and trashes nothing | | `report` | Builds the step summary from the state alone, finishes the run row, writes the chain marker, and returns the run's status | Every step is plan-by-default: `batches`, `trash`, `reconcile`, `report` and @@ -107,6 +107,7 @@ The bucket holds the state and the session, nothing of the mirrored tree. .state/state.sqlite.xz.age the state: evidence tables, mirror_objects, runs, batches, deletions .state/history/-