feat(toolbox): reconcile by the age of the last one, not the hour

CLAUDE.md

@@ -33,6 +33,8 @@ each entry is a verified go-task or platform fact that cost a session to find.
   so the container engine's own progress lines never reach `render.txt`.
 - Inside a `sh:` var, `printf -- '-e %s'` prints dashes: task's built-in shell takes
   the `--` as the format. Use `printf '%s %s ' -e "$v"`.
+- Task's built-in shell runs every command under `set -e`, and an assignment takes its
+  command substitution's exit: `x=$(cat missing)` stops the command (verified, 3.53.1).
 - Task's built-in shell has no `umask`. A file that must be born 0600 is
   `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does.
 - Every Proton CLI call goes through `pd`, which pushes the session back whatever the

README.md

@@ -68,7 +68,7 @@ flowchart TB
     cw["sync.yml, check.yml<br/>ten-line callers"]
   end
   subgraph lib["katoptra/lib, at v2"]
-    tb["toolbox.yml<br/>menu, image, run, op, sync, plan, check<br/>clock, report, ping, failed"]
+    tb["toolbox.yml<br/>menu, image, run, op, sync, plan, check<br/>clock, due, reconciled, report, ping, failed"]
     en["engines/rsync.yml, engines/proton.yml<br/>the pipeline vocabulary, one file per transport"]
     im["ghcr.io/katoptra/toolbox:rsync-v2, :proton-v2<br/>docker/, toolchain.lock.toml"]
     wf[".github/workflows/sync.yml, check.yml<br/>.github/actions/toolbox"]
@@ -185,7 +185,9 @@ keeps its own.
 
 | Verb | Does |
 |---|---|
-| `clock` | Write "epoch UTC-hour weekday" to `.run/start.txt` and forget the last run's chain file |
+| `clock` | Write the run's start epoch to `.run/start.txt` and forget the last run's chain file and `.run/reconcile` |
+| `due` | Decide whether this run reconciles, by the age of the last reconcile at `.state/reconciled`; leave `.run/reconcile` when it does. `due-rule` is the rule alone |
+| `reconciled` | Record this run's start at `.state/reconciled`; an engine runs it once a reconcile completes |
 | `report` | Append the run summary to the Actions job page (stdout elsewhere): its own rows, then `report-engine`'s, then `report-mirror`'s |
 | `report-engine`, `report-mirror` | Hooks: no-ops here; the engine's and the mirror's rows |
 | `ping` | GET `HEALTHCHECK_URL`; skipped when unset |
@@ -198,6 +200,27 @@ the toolbox's rows (when the run started and how long it took, the image, whethe
 next run is queued), the engine's, the mirror's. Every row tolerates a missing file,
 because the report also runs after a failed pipeline.
 
+#### Reconcile, by age
+
+A reconcile, the pass that checks the mirror against its destination rather than
+against its own state, is due by the age of the last one and never by when a run starts:
+the schedule lives outside the mirror and can move. `due` runs early in a pipeline that
+reconciles, and the pipeline's reconcile runs only when `.run/reconcile` is there.
+
+| Var | Default | Meaning |
+|---|---|---|
+| `RECONCILE` | `auto` | Set per run: `task sync -- RECONCILE=true`. `true` reconciles, `false` does not, `auto` reconciles once the last is `RECONCILE_HOURS` old or none is on record |
+| `RECONCILE_HOURS` | 24 | Whole hours between reconciles. A mirror that wants another interval sets it in its root vars |
+
+The last reconcile is the start epoch of the run that finished it, at `.state/reconciled`,
+kept through the engine's `pull` and `push`, so under the rsync engine it is
+xz-compressed like the state. The mark is half an hour short of `RECONCILE_HOURS`: a
+daily run starts seconds either side of 24 h after the one before, and a strict 24 h
+would reconcile every other day. A run that fails before its reconcile completes
+records nothing, so the next run is due as well, and so does an rsync run with batches
+still waiting: its reconcile holds for the chained run that lands the last one. A mirror with no engine that
+reconciles defines `pull` and `push` itself.
+
 ### What a mirror gives the toolbox
 
 | Name | Kind | Meaning |
@@ -209,6 +232,8 @@ because the report also runs after a failed pipeline.
 | `MENU` | include var | Extra lines for the menu, one per mirror-specific verb |
 | `report-mirror` | task | The mirror's rows of the run summary. Excluded on the toolbox include |
 | `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib` |
+| `RECONCILE_HOURS` | root var | Hours between reconciles, when not 24. See [Reconcile, by age](#reconcile-by-age) |
+| `pull`, `push` | tasks | What `due` and `reconciled` keep `.state/reconciled` through. An engine supplies both; a mirror with no engine that reconciles defines them |
 | `excludes:` | include key | Library verbs the mirror replaces. See [Changing it](#changing-it) |
 
 ## The engines
@@ -233,12 +258,12 @@ delta, in batches, each committed before the next starts.
 
 ```mermaid
 flowchart LR
-  clock --> list --> state --> rebuild["rebuild<br/>only if the state was missing"] --> diff --> split --> prepare --> batches
+  clock --> due --> list --> state --> rebuild["rebuild<br/>only if the state was missing"] --> diff --> split --> prepare --> batches
   subgraph b["batch, for each of the first MAX_BATCHES"]
     direction LR
     fetch --> verify --> publish --> checkpoint
   end
-  batches --> b --> delete --> reconcile["reconcile<br/>once a day, or RECONCILE=true"] --> index --> smoke --> report --> ping
+  batches --> b --> delete --> reconcile["reconcile<br/>when due"] --> index --> smoke --> report --> ping
   smoke --> sm["smoke-mirror"]
   report --> re["report-engine"] --> rm["report-mirror"]
   classDef hook stroke-dasharray: 5 5
@@ -263,7 +288,7 @@ and `smoke-mirror` do nothing until a mirror fills them.
 | `publish` | `aws s3 cp --recursive` of staging, one PutObject per file, never a destination listing; `timestamp` last |
 | `checkpoint` | `merge` what landed into the state and push it as one PutObject; empty staging |
 | `delete` | Remove the keys upstream dropped, 1,000 per call, once every batch has landed, and drop them from the state |
-| `reconcile` | In the run that starts in hour 03 UTC, or with `RECONCILE=true`: rebuild the state, then delete what neither upstream, `OWN` nor the state's own directories own |
+| `reconcile` | When `due` left `.run/reconcile`: rebuild the state, delete what neither upstream, `OWN` nor the state's own directories own, then `reconciled` |
 | `index` | Hook. Nothing here; a mirror that draws directory pages or a landing page replaces it |
 | `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing; the tlpdb sha512 when `TL` is set; then `smoke-mirror` |
 | `smoke-mirror` | Hook. Nothing here; a mirror with more to read back defines it |
@@ -299,7 +324,8 @@ An hourly run costs one listing of upstream and none of the bucket. Three things
   so a path that vanished upstream between listing and fetch never enters the state, and
   the state never names a key the bucket lacks.
 - **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket on
-  purpose, daily by default, and deletes keys neither upstream nor the state owns.
+  purpose, once every `RECONCILE_HOURS` (24) by default, and deletes keys neither upstream
+  nor the state owns.
 
 What a mirror cannot afford to lose is the bucket. Everything else, the state file and the
 staging tree included, is derived from it and from upstream.
@@ -347,7 +373,7 @@ has an inline default, and a mirror sets only what differs:
 | `BATCH_GB` | 4 | Decimal GB per batch; a larger file is a batch by itself |
 | `MAX_BATCHES` | 4 | Batches per run; the rest chain the next run |
 | `LIST_FLOOR` | 0, no guard | A listing under this many lines is a truncated one, never a deletion list |
-| `RECONCILE` | `auto` | `true`, `false`, or `auto`: the run that started in hour 03 UTC |
+| `RECONCILE`, `RECONCILE_HOURS` | `auto`, 24 | The toolbox's: see [Reconcile, by age](#reconcile-by-age) |
 | `RETRY_BASE` | 15 | Seconds; the retry sleeps are `RETRY_BASE * 2^i` plus jitter |
 | `TL`, `TL_KEY` | empty | A signed TeX Live subtree and the fingerprint that signs it; empty, no signature checks |
 | `FILTER` | empty | rsync filter arguments that narrow the listing, for a mirror of a subtree |
@@ -448,9 +474,10 @@ no history in the bucket**: a stale copy holds a rotated-out token and cannot be
 A mirror whose logic is a program of its own includes the toolbox alone and defines
 `pipeline` and `plan-pipeline` from the toolbox's `clock`, `report` and `ping` and its own
 steps. dropbox is that shape: each step is one `python -m migrator <command>`, the
-Taskfile owns the order, the Python owns every decision, and the `proton` image supplies
-the interpreter, boto3 and the CLI. Such a mirror fills `report-mirror` with its own
-report and nothing else of the toolbox's changes.
+Taskfile owns the order, the Python owns every decision but when to reconcile, which is
+the toolbox's `due`, and the `proton` image supplies the interpreter, boto3 and the CLI.
+Such a mirror fills `report-mirror` with its own report and nothing else of the
+toolbox's changes.
 
 ## Secrets
 
@@ -533,6 +560,7 @@ Every mirror has one S3-compatible bucket. What it holds depends on the engine.
 | rsync, a mirror's own | `.state/indexed.txt.xz`, `index.html` | ctan's record of what its directory pages show; tlnet's landing page, spared by `OWN` |
 | proton | `.state/session.tar.age` | The CLI session, encrypted. The only key |
 | a pipeline of its own | `.state/state.sqlite.xz.age`, `.state/history/<epoch>-<label>...` | dropbox's state and its dated copies; a lifecycle rule expires the history |
+| any that reconciles | `.state/reconciled` | The start epoch of the run that last completed a reconcile, through the engine's `push`: xz under rsync, with no `.xz` suffix, plain under dropbox |
 
 `.state/` is the one reserved prefix, chosen because no upstream in the org has a
 dot-prefixed root entry.

engines/rsync.yml

@@ -2,16 +2,17 @@
 # mirror that includes this file flattened beside the toolbox. A run is a list diff, never
 # a local tree: list upstream, diff it against the state file the last run left in the
 # bucket, fetch the delta in batches of BATCH_GB, upload, checkpoint the state once per
-# batch, delete what left upstream, and once a day reconcile the bucket against the
-# state. The mirror orders these verbs in its pipeline and holds its identity in root vars.
+# batch, delete what left upstream, and reconcile the bucket against the state when the
+# toolbox's due says one is due (RECONCILE and RECONCILE_HOURS are its vars). The mirror
+# orders these verbs in its pipeline and holds its identity in root vars.
 #
 # Read from the mirror: SOURCE, BUCKET, HOST (required); and, each with an inline default,
 # CEILING_GB (0: no ceiling), BATCH_GB (4), MAX_BATCHES (4), LIST_FLOOR (0: no
-# truncated-listing guard), RECONCILE (auto), RETRY_BASE (15 s), TL and TL_KEY (a signed
-# TeX Live subtree and the fingerprint that signs it; empty: no signature checks), FILTER
-# (rsync filter arguments for the listing; empty: the whole tree), OWN (bucket-root keys
-# the mirror owns and reconcile never deletes, space separated; empty: none), INDEX (the
-# directory-page key suffix an index hook writes; empty: none).
+# truncated-listing guard), RETRY_BASE (15 s), TL and TL_KEY (a signed TeX Live subtree
+# and the fingerprint that signs it; empty: no signature checks), FILTER (rsync filter
+# arguments for the listing; empty: the whole tree), OWN (bucket-root keys the mirror owns
+# and reconcile never deletes, space separated; empty: none), INDEX (the directory-page
+# key suffix an index hook writes; empty: none).
 #
 # Hooks: prepare, verify, index, smoke, smoke-mirror and report-engine are the verbs a
 # mirror may replace, by listing each under excludes: on this include and defining its
@@ -70,9 +71,10 @@ vars:
 
 tasks:
   pipeline:
-    desc: clock -> list -> state -> rebuild? -> diff -> split -> prepare -> batches -> delete -> reconcile? -> index -> smoke -> report -> ping
+    desc: clock -> due -> list -> state -> rebuild? -> diff -> split -> prepare -> batches -> delete -> reconcile? -> index -> smoke -> report -> ping
     cmds:
       - {task: clock}
+      - {task: due}
       - {task: list}
       - {task: state}
       - {task: rebuild}
@@ -165,7 +167,7 @@ tasks:
       - aws s3 cp {{.AWS_FLAGS}} {{.FILE}}.xz {{.S3}}/{{.KEY}}
 
   rebuild:
-    desc: List the bucket and make the state exactly what is there (a lost state file, or the daily reconcile)
+    desc: List the bucket and make the state exactly what is there (a lost state file, or a reconcile)
     status: ['test ! -f {{.RUN}}/rebuild-now']
     set: [pipefail]
     cmds:
@@ -281,7 +283,7 @@ tasks:
       # symlink into its file. A path that vanished since the listing is skipped: it never
       # lands in staging, so merge (the batch joined against what staging holds) drops it
       # from the state rather than recording it; any object it once had in the bucket
-      # waits for the daily reconcile to clean up. No --partial: staging is emptied every
+      # waits for the next reconcile to clean up. No --partial: staging is emptied every
       # batch, so there is nothing to resume, and a partial file must never reach the bucket.
       - task: retry
         vars: {CMD: '{{.RSYNC}} -Lt --files-from={{.RUN}}/files.txt --ignore-missing-args {{.SOURCE}} {{.STAGING}}/'}
@@ -407,12 +409,11 @@ tasks:
       - {task: checkpoint, vars: {B: '{{.D}}'}}
 
   reconcile:
-    desc: Rebuild the state from a bucket listing, then delete keys that neither upstream nor .state/ owns
-    # auto is the run that started in hour 03 UTC, one a day. A run held in the dispatch
-    # queue behind a long one can start in 04 and skip the day; the next day's run does it,
-    # and RECONCILE=true asks for it by hand.
-    status:
-      - 'test {{shellQuote (.RECONCILE | default "auto")}} = false || { test {{shellQuote (.RECONCILE | default "auto")}} = auto && test "$(cut -d" " -f2 {{.RUN}}/start.txt)" != 03; }'
+    desc: When due left .run/reconcile, rebuild the state from a bucket listing, delete keys that neither upstream nor .state/ owns, and record the reconcile
+    # Not while a batch waits for a later run: delete would hold the orphans back, so the run
+    # stays due and the chained run that lands the last batch reconciles. A run that fails
+    # before the last line records nothing either, so the next run is due too.
+    status: ['test ! -f {{.RUN}}/reconcile || test "$(ls {{.RUN}}/batch-*.txt 2>/dev/null | wc -l)" -gt {{.BATCHES_MAX}}']
     cmds:
       - touch {{.RUN}}/rebuild-now
       - {task: rebuild}
@@ -432,6 +433,7 @@ tasks:
         | LC_ALL=C comm -23 - {{.RUN}}/index-keys.txt
         | LC_ALL=C comm -23 - {{.RUN}}/paths.txt > {{.RUN}}/orphans.txt
       - {task: delete, vars: {D: '{{.RUN}}/orphans.txt'}}
+      - {task: reconciled}
 
   # ---- after the bytes moved ----
   index:

examples/proton/render.txt

@@ -1,4 +1,4 @@
-task: [clock] mkdir -p /work/examples/proton/.run && rm -f /work/examples/proton/.run/chain && date -u '+%s %H %u' > /work/examples/proton/.run/start.txt
+task: [clock] mkdir -p /work/examples/proton/.run && rm -f /work/examples/proton/.run/chain /work/examples/proton/.run/reconcile && date -u +%s > /work/examples/proton/.run/start.txt
 task: [session] rm -rf /work/examples/proton/.run/session && mkdir -p /work/examples/proton/.run/session && chmod 700 /work/examples/proton/.run/session
 task: [pull] rc=0; s3 get .state/session.tar.age /work/examples/proton/.run/session.tar.age || rc=$?
 case $rc in 0) ;; 3) echo "no Proton session at .state/session.tar.age: log in on a laptop and run task session-seal" >&2; exit 1 ;; *) exit $rc ;; esac

examples/rsync/Taskfile.yml

@@ -67,6 +67,28 @@ tasks:
       - task retry CMD='exit 24' RETRY_BASE=0
       - '! task retry CMD=''exit 5'' RETRY_BASE=0 2>/dev/null'
       - '! task retry CMD=''exit 3'' RETRY_BASE=0 2>/dev/null'
+      # due-rule, a run starting at 1000000: auto reconciles with none on record and at 23 h
+      # 50 min, not at under 3 h, unless RECONCILE_HOURS is 2 or RECONCILE=true; false never
+      # does. A RECONCILE that is none of the three stops the run, and so does an hour count
+      # that is not a whole number from 1 (08 would read as octal).
+      - mkdir -p {{.F}}/due && echo 1000000 > {{.F}}/due/start.txt
+      - {task: due-rule, vars: {RUN: '{{.F}}/due'}}
+      - test -f {{.F}}/due/reconcile
+      - {task: due-rule, vars: {RUN: '{{.F}}/due', RECONCILE: 'false'}}
+      - test ! -f {{.F}}/due/reconcile
+      - echo 914000 > {{.F}}/due/reconciled
+      - {task: due-rule, vars: {RUN: '{{.F}}/due'}}
+      - test -f {{.F}}/due/reconcile
+      - echo 990000 > {{.F}}/due/reconciled
+      - {task: due-rule, vars: {RUN: '{{.F}}/due'}}
+      - test ! -f {{.F}}/due/reconcile
+      - {task: due-rule, vars: {RUN: '{{.F}}/due', RECONCILE_HOURS: 2}}
+      - test -f {{.F}}/due/reconcile
+      - {task: due-rule, vars: {RUN: '{{.F}}/due', RECONCILE: 'true'}}
+      - test -f {{.F}}/due/reconcile
+      - task due-refuses R=yes 2>&1 | grep -q 'RECONCILE is true, false or auto'
+      - task due-refuses H=abc 2>&1 | grep -q 'RECONCILE_HOURS is a whole number'
+      - task due-refuses H=08 2>&1 | grep -q 'RECONCILE_HOURS is a whole number'
       # run-tl: the signed subtree. prepare fetches the control files from the tree and checks
       # the signed sha512 against the pinned key; fetch stages the batch; verify checks the
       # signed file it carries, every container against the tlpdb under both of its names
@@ -81,6 +103,10 @@ tasks:
       # A container the tlpdb does not describe cannot pass, whatever its bytes.
       - '! task verify-refuses 2>/dev/null'
       - 'echo "offline: every check passed"'
+  due-refuses:
+    desc: due-rule over offline's due fixture with RECONCILE R and RECONCILE_HOURS H; offline expects it to refuse
+    cmds:
+      - {task: due-rule, vars: {RUN: '{{.RUN}}/fixtures/due', RECONCILE: '{{.R | default "auto"}}', RECONCILE_HOURS: '{{.H | default 24}}'}}
   verify-refuses:
     desc: verify over a batch naming a container the tlpdb does not describe, after offline staged run-tl; offline expects it to fail
     vars: