diff --git a/.github/workflows/sync.yml b/.github/workflows/sync.yml
index 45ccd1a..0ed63d6 100644
--- a/.github/workflows/sync.yml
+++ b/.github/workflows/sync.yml
@@ -1,5 +1,5 @@
# Reusable: one mirror, one run. The caller is a workflow_dispatch of a few lines that
-# jshvn/dispatch triggers; it passes `vars` through and inherits its repository secrets.
+# an external scheduler triggers; it passes `vars` through and inherits its repository secrets.
# 1Password is read once per run: one `op run` resolves op.env around the whole pipeline,
# failure path included. A mirror without an op.env runs on its repository secrets instead,
# exported into the sync step's environment and crossing into the container by the names
diff --git a/CLAUDE.md b/CLAUDE.md
index 754d0f3..59ed913 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,37 +1,21 @@
# lib
-The toolbox every katoptra mirror includes by URL. Read `README.md` for the contract and
-`docs/superpowers/specs/2026-09-08-toolbox-library-design.md` for the decisions.
+The toolbox every katoptra mirror includes by URL. `README.md` is the manual: the layers,
+the verbs, the engines, secrets, storage, the images, the workflows, and how a mirror
+uses and changes any of it. `docs/superpowers/specs/2026-09-08-toolbox-library-design.md`
+is the record of what was decided when. This file is what a change must not break:
+each entry is a verified go-task or platform fact that cost a session to find.
-## Constraints
+## Hazards
-- `toolbox.yml` declares no `vars:` default for anything a mirror owns. Defaults go
- inline. This is a go-task fact, verified: a library `vars:` default shadows the
- mirror's root value.
-- Verb names are reserved across host and container. `plan` is the host-side read-only
- run; an engine's batch planner is `split`. The hooks, `prepare`, `verify`, `index`,
- `smoke`, `smoke-mirror`, `report-engine` in the rsync engine, `stage` and `prune` in
- the proton engine, and `report-mirror` in the toolbox, are the only verbs a mirror
- redefines, each excluded on the include that defines it; `pipeline` and
- `plan-pipeline` come from the engine and a mirror excludes them only for another
- order. `report-engine` exists in both files, so an engine
- consumer excludes it on the toolbox include. Extension is a hook, never a copy: a
- verb that needs more than the engine does gets a `-mirror` hook the engine calls last.
-- A mirror's root var shadows a command-line `KEY=value` inside an included verb, so an
- engine tunable is an inline default and never a root var of the example. A var of
- another name that reads the tunable, `BATCHES_MAX` from `MAX_BATCHES`, still sees the
- command line (verified), so each default is spelled once.
-- A called task sees none of its caller's call vars. A hook a verb calls with an
- overridable `RUN` or `STAGING` gets them passed on explicitly, as `smoke` does.
- A global var that reads another var is rendered once, from the root's and the command
line's values, never from a call var: `GPGCHECK` takes the fingerprint as an awk `-v`
at each use, where a call var `TL_KEY` is visible.
+- A var of another name that reads a tunable, `BATCHES_MAX` from `MAX_BATCHES`, still
+ sees the command line (verified), so each default is spelled once, inline.
- No `dir:` on an engine verb. In a flattened include read by path, task joins even an
absolute `dir:` onto the include's directory (verified, 3.53.1), so `verify` starts
each command with `cd {{.STAGING}}` instead.
-- `render` mounts an empty directory over `.run`, so a `sh:` var read from it at parse
- time renders the same from any working tree. A verb that reads `.run` at parse time
- must tolerate an empty one.
- `.taskrc.yml` keys are `trusted-hosts` and `cache-expiry`; task ignores a key it does
not know, silently, and refetches on every invocation.
- Every tool in an image comes from `toolchain.lock.toml`, read through `docker/lock.py`,
@@ -51,12 +35,9 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
the `--` as the format. Use `printf '%s %s ' -e "$v"`.
- Task's built-in shell has no `umask`. A file that must be born 0600 is
`install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does.
-- The proton engine keeps no state of the mirror's in the bucket, only the CLI session,
- because `filesystem upload` skips a file whose content Proton already holds and
- `-f create-new-revision` handles one that changed. Every CLI call goes through `pd`,
- which pushes the session back whatever the exit: the refresh token rotates, and a run
- that kept a rotated token to itself leaves the next run unable to log in. Two mirrors
- never share one session for the same reason.
+- Every Proton CLI call goes through `pd`, which pushes the session back whatever the
+ exit: the refresh token rotates, and a run that kept a rotated token to itself leaves
+ the next run unable to log in. Two mirrors never share one session for the same reason.
- Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the
two reusable workflows that way; each checks this repository out at its own commit
(`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is
@@ -78,3 +59,7 @@ example. Regenerate the tree with a new key only to change its shape; the privat
was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an
accepting and a refusing upload summary, and the `age` verb round trip with a throwaway
identity.
+
+A README change is a diff `task check` cannot see; the review is reading it. Every
+anchor a mirror's README links here (`#secrets`, `#storage`, `#the-rsync-engine`,
+`#the-proton-engine`) is a heading in `README.md`; renaming one breaks four repositories.
diff --git a/README.md b/README.md
index 6880fc2..7d1b88b 100644
--- a/README.md
+++ b/README.md
@@ -7,39 +7,73 @@ The toolbox every katoptra mirror includes by URL. The rule it enforces: the cod
starts a run, contains it, resolves its secrets, checks it and reports it lives here,
once. The code that moves bytes for a transport lives here too, once per engine:
`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a
-staging tree into Proton Drive. A mirror holds only its identity, the order of its
-pipeline, and the few verbs no other mirror shares.
+staging tree into Proton Drive. A mirror holds only its identity, the hooks it fills, and
+the few verbs no other mirror shares.
+
+This file is the manual. A mirror's README says what it mirrors and how to fork it, and
+links here for everything the mirrors share. Read it top to bottom once and the system
+should hold no surprises; after that, each section stands on its own.
+
+## The system in one picture
+
+```mermaid
+flowchart LR
+ subgraph up["Upstreams"]
+ u1["CTAN, over rsync"]
+ u2["GitHub, over the API and git"]
+ u3["Dropbox, over the API"]
+ end
+ sched["A scheduler workflow_dispatch on a cron"] --> host
+ subgraph job["One GitHub Actions job per run"]
+ host["The runner: task sync"] --> box["The toolbox image: task pipeline"]
+ end
+ vault["A 1Password vault one item per mirror"] -. "op run: values, by name" .-> host
+ u1 --> box
+ u2 --> box
+ u3 --> box
+ box --> r2["An S3-compatible bucket, R2 a public mirror: the mirror itself a Proton mirror: the session, and state"]
+ box --> pd["Proton Drive the private mirrors' sink"]
+ r2 --> cf["A custom domain on the bucket"] --> cl["tlmgr, browsers, anyone"]
+ box --> hc["healthchecks.io, one ping per run"] --> site["katoptra.org, live status"]
+```
+
+Every mirror is that picture with one upstream and one sink. A public mirror (ctan,
+tlnet) copies an rsync tree into a bucket and serves the bucket; the bucket is the mirror.
+A private mirror (github, dropbox) copies an account into Proton Drive and keeps only
+what the next run needs in the bucket. Both run the same way: a job on GitHub Actions,
+started by a dispatch, that pulls one image, resolves its secrets by name from a vault,
+and runs `task pipeline` inside the image. The same command runs on a laptop.
## The layers
```mermaid
flowchart TB
- subgraph mirror["A mirror repository (katoptra/ctan, say)"]
- tf["Taskfile.yml root vars: SOURCE, BUCKET, HOST pipeline, plan-pipeline overrides, if any"]
- opv["op.env op:// references"]
+ subgraph mirror["A mirror repository"]
+ tf["Taskfile.yml root vars: the identity the hooks it fills, verbs of its own"]
+ opv["op.env op:// references, the vault by UUID"]
rt["render.txt the committed dry run"]
- cw["sync.yml, check.yml callers of lib's"]
+ cw["sync.yml, check.yml ten-line callers"]
end
- subgraph lib["katoptra/lib, pinned to v2"]
- tb["toolbox.yml menu, image, run, op, sync, plan render, check, clock, ping"]
- en["engines/rsync.yml list, diff, split, batches, reconcile ... hooks: prepare, verify, index, smoke"]
- im["ghcr.io/katoptra/toolbox:<variant>-v2 docker/ + toolchain.lock.toml"]
+ subgraph lib["katoptra/lib, at v2"]
+ tb["toolbox.yml menu, image, run, op, sync, plan, check clock, report, ping, failed"]
+ en["engines/rsync.yml, engines/proton.yml the pipeline vocabulary, one file per transport"]
+ im["ghcr.io/katoptra/toolbox:rsync-v2, :proton-v2 docker/, toolchain.lock.toml"]
wf[".github/workflows/sync.yml, check.yml .github/actions/toolbox"]
end
tf -- "includes, flattened" --> tb
tf -- "includes, flattened" --> en
tb -- "runs pipeline inside" --> im
- cw -- "uses @release SHA" --> wf
- wf -- "task sync / task check" --> tb
+ cw -- "uses, at a release SHA" --> wf
+ wf -- "task sync, task check" --> tb
```
| Layer | Lives in | Owns | A mirror touches it when |
|---|---|---|---|
| Toolbox | `toolbox.yml` | How a run starts, is contained, resolves secrets, is rendered, checked and reported | Never; it is included as is |
-| Engine | `engines/.yml` | How bytes move for one transport: rsync to R2, Proton Drive, HTTPS | It lists a verb under `excludes:` and defines its own |
+| Engine | `engines/.yml` | How bytes move for one transport, and the pipeline's order | It lists a hook under `excludes:` and defines its own |
| Image | `docker/`, `toolchain.lock.toml` | Every tool a run needs, pinned by checksum | Never; it names one in `IMAGE` |
| Workflows | `.github/workflows`, `.github/actions/toolbox` | How Actions installs the tools and runs `task sync` and `task check` | Never; its callers are a few lines each |
-| Mirror | The mirror's own repository | Identity, pipeline order, the exceptions | Always; this is all it holds |
+| Mirror | The mirror's own repository | Identity, the hooks, the exceptions | Always; this is all it holds |
## What a run looks like
@@ -47,7 +81,7 @@ The same path on a laptop and in Actions. On a laptop it starts at `task sync`.
```mermaid
sequenceDiagram
- participant D as jshvn/dispatch
+ participant D as A scheduler
participant W as sync.yml (reusable)
participant H as Host: task
participant O as op run
@@ -59,9 +93,9 @@ sequenceDiagram
H->>O: op: op run --env-file=op.env
O->>C: run: engine run --rm -v repo:/work -e NAME ... IMAGE task pipeline
C->>C: clock
- C->>C: engine verbs: list, diff, split, batches, reconcile ...
+ C->>C: the engine's verbs, and the mirror's hooks
C->>C: report, then ping
- alt pipeline failed
+ alt the pipeline failed
C->>C: failed: report STATUS=failed, then ping-fail
else the runner cut the run off
W->>C: task op -- task failed
@@ -70,120 +104,229 @@ sequenceDiagram
end
```
-Three things the diagram hides:
+Four things the diagram hides:
- **Secrets cross by name, never by value.** `op run` resolves `op.env` on the host and
exports the values; `run` passes `-e NAME` for each name in `op.env` and in `PASS`,
plus `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always, so no value
- ever appears on a command line or in a log.
-- **1Password is read once per run.** One `op run` wraps the whole pipeline, and the
+ ever appears on a command line or in a log. The whole path is under [Secrets](#secrets).
+- **The vault is read once per run.** One `op run` wraps the whole pipeline, and the
failure path, `failed`, runs inside the same container, so a failed run costs no
second read. The one exception is a run the runner cut off, by the timeout or a
cancellation, which never reached its own failure path: the workflow runs `failed`
- again from outside, one more read on a rare day. A mirror without an `op.env` runs
- on its repository secrets: the reusable workflow exports every inherited secret into
- the sync step's environment by name, the ones the mirror lists in `PASS` cross into
- the container, and the 1Password CLI is not installed at all.
+ again from outside, one more read on a rare day.
- **Nothing inside the container reaches the network for task itself.** The image sets
`TASK_REMOTE_OFFLINE=1` and the mirror's `.task/remote` cache rides in with the repo.
- **`plan` is `sync` with the read-only half.** It runs `plan-pipeline` instead of
`pipeline`, inside the same image with the same secrets.
-## The verbs
+## The toolbox
-### What a mirror must define
+`toolbox.yml` is every verb a mirror has whatever moves its bytes. Half of its verbs run
+on the host and wrap the container; the other half run inside it and are the first and
+last steps of every pipeline.
-| Name | Kind | Meaning |
-|---|---|---|
-| `SOURCE`, `BUCKET`, `HOST` | root vars | The mirror's identity. Engines read them; the library never defaults them. |
-| `NAME`, `DESC`, `IMAGE` | include vars | The menu's title and line, and the image the run happens inside. |
-| `pipeline`, `plan-pipeline` | tasks | The full run and its read-only half, in order, inside the image; `sync` and `plan` call them. An engine supplies both; a mirror with no engine defines them. |
-
-### What a mirror may define
-
-| Name | Kind | Meaning |
-|---|---|---|
-| `PASS` | include var | Host environment names that cross into the container beside the ones in `op.env`, for a pipeline that reads its environment. Task vars are not environment: they go after `--`, as `task sync -- MAX_BATCHES=8`. |
-| `report-mirror` | task | The mirror's rows of the run summary, after the toolbox's and the engine's. Excluded on the toolbox include. |
-| `smoke-mirror` | task | The mirror's own read-back checks, after the engine's sample. Excluded on the engine include. |
-| `MENU` | include var | Extra lines for the menu, one per mirror-specific verb. |
-| `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib`. |
-| `op.env` | file | `op://` references, one per secret. Absent means the environment is already resolved: on a laptop, whatever is exported; in Actions, the repository's secrets, crossing by the names in `PASS`. |
-| `excludes:` | include key | Library verbs the mirror replaces. See below. |
+```mermaid
+flowchart TB
+ sync["task sync -- K=v"] --> op
+ plan["task plan -- K=v"] --> op
+ op["op: op run --env-file=op.env, when the file exists"] --> run
+ run["run: engine run --rm -v repo:/work -e NAME ... IMAGE"] --> image["image: pull IMAGE unless it is present"]
+ run --> pipeline["task pipeline, inside the image"]
+ run --> pp["task plan-pipeline, inside the image"]
+ pipeline -- "on failure" --> failed["failed: report STATUS=failed, then ping-fail"]
+ check["task check"] --> render["render: task --dry --force pipeline inside the image, over an empty .run"]
+ render --> image
+ render --> d["diff against render.txt"]
+ ru["task render-update"] --> render
+ ru --> cp["accept it as render.txt"]
+```
-### What the toolbox provides
+### Host side
-Host side, and the mirror does not override these. They are the contract the workflows
-call and the menu documents.
+The contract the workflows call and the menu documents. A mirror does not override these.
| Verb | Does |
|---|---|
| `default` | The grouped menu, from `NAME`, `DESC` and `MENU` |
-| `sync` | `op -- task pipeline`: one run |
-| `plan` | `op -- task plan-pipeline`: the read-only half |
+| `sync -- [K=v ...]` | `op -- task pipeline`: one run. The words after `--` are task vars for the pipeline inside the image |
+| `plan -- [K=v ...]` | `op -- task plan-pipeline`: the read-only half |
| `check` | `render`, then diff against `render.txt` |
| `render-update` | `render`, then accept it as `render.txt` |
-| `render` | `task --dry --force pipeline` inside the image, over an empty `.run`, saved to `.run/render.txt` |
-| `run -- ` | Anything inside the image, with the repo at `/work` |
+| `render` | `task --dry --force pipeline` inside the image, over an empty `.run`, saved to `.run/render.txt`; no secrets cross |
+| `run -- ` | Anything inside the image, with the repository at `/work` |
| `op -- ` | `run`, wrapped in `op run --env-file=op.env` when the file exists |
| `image` | Pull `IMAGE`; a no-op while it exists locally |
-| `image-build` | Build `IMAGE` from `LIB_DIR/docker/.Dockerfile` |
+| `image-build` | Build `IMAGE` from `LIB_DIR/docker/.Dockerfile`; `LIB_DIR` defaults to `../lib` |
| `image-clean` | Remove `IMAGE` |
| `clean` | Delete `.run`, `staging` and the Taskfile cache; nothing else |
-Inside the image. A mirror or engine may override these when it keeps its own.
+The container engine is Apple `container` when its daemon is up, else Docker;
+`ENGINE=docker` forces Docker. The host needs go-task, the container engine, and the
+1Password CLI for anything that reads the vault. Nothing else is installed on a host.
+
+### Inside the image
+
+The first and last verbs of every pipeline. A mirror or engine may override these when it
+keeps its own.
| Verb | Does |
|---|---|
| `clock` | Write "epoch UTC-hour weekday" to `.run/start.txt` and forget the last run's chain file |
| `report` | Append the run summary to the Actions job page (stdout elsewhere): its own rows, then `report-engine`'s, then `report-mirror`'s |
-| `report-engine`, `report-mirror` | Hooks: no-ops here, the engine's and the mirror's rows |
+| `report-engine`, `report-mirror` | Hooks: no-ops here; the engine's and the mirror's rows |
| `ping` | GET `HEALTHCHECK_URL`; skipped when unset |
| `ping-fail` | GET `HEALTHCHECK_URL/fail`; skipped when unset |
| `failed` | The failure path: `report STATUS=failed`, then `ping-fail` |
-`sync` runs `pipeline`, and on a failure `failed`, all in the one container; the
-workflow runs `failed` from outside only for a run the runner cut off. One table, three
-layers: the toolbox's rows (when the run started
-and how long it took, the image, whether the next run is queued), the engine's, the
-mirror's, each a `| Label | value |` line appended to the same file. Every row tolerates
-a missing file, because the report also runs after a failed pipeline.
+`sync` runs `pipeline`, and on a failure `failed`, all in the one container. The report
+is one table in three layers, each a `| Label | value |` line appended to the same file:
+the toolbox's rows (when the run started and how long it took, the image, whether the
+next run is queued), the engine's, the mirror's. Every row tolerates a missing file,
+because the report also runs after a failed pipeline.
+
+### What a mirror gives the toolbox
-### What an engine provides
+| Name | Kind | Meaning |
+|---|---|---|
+| `NAME`, `DESC`, `IMAGE` | include vars, required | The menu's title and line, and the image the run happens inside |
+| `pipeline`, `plan-pipeline` | tasks, required | The full run and its read-only half, inside the image. An engine supplies both; a mirror with no engine defines them |
+| `op.env` | file | `op://` references, one per secret. Absent means the environment is already resolved: on a laptop, whatever is exported; in Actions, the repository's secrets, crossing by the names in `PASS` |
+| `PASS` | include var | Host environment names that cross into the container beside the ones in `op.env`. Task vars are not environment: they go after `--` |
+| `MENU` | include var | Extra lines for the menu, one per mirror-specific verb |
+| `report-mirror` | task | The mirror's rows of the run summary. Excluded on the toolbox include |
+| `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib` |
+| `excludes:` | include key | Library verbs the mirror replaces. See [Changing it](#changing-it) |
+
+## The engines
An engine is a second include with the verbs for one transport. Its verbs are the
-pipeline vocabulary, reserved across every engine so a mirror's `pipeline` reads the
-same whichever transport it uses. The rsync engine, `engines/rsync.yml`, is the first:
-an rsync upstream into an S3 bucket, as a list diff and never a local tree.
+pipeline vocabulary, reserved across every engine so a mirror's pipeline reads the same
+whichever transport it uses. An engine reads the mirror's root vars and the environment;
+it declares no `vars:` default for anything a mirror owns, because a value there would
+shadow the mirror's. Every tunable is an inline `{{.X | default N}}`.
+
+The hooks are the verbs an engine leaves empty for a mirror to fill. A mirror lists each
+it defines under `excludes:` on the engine's include. Extension is a hook, never a copy:
+a mirror that needs the engine's `smoke` and more defines `smoke-mirror`, which `smoke`
+runs last.
+
+### The rsync engine
+
+`engines/rsync.yml`: an rsync upstream into an S3 bucket, as a list diff and never a
+local tree. The runner has 14 GB of disk and CTAN has 140; a run lists upstream, diffs
+the listing against the state file the last run left in the bucket, and moves only the
+delta, in batches, each committed before the next starts.
+
+```mermaid
+flowchart LR
+ clock --> list --> state --> rebuild["rebuild only if the state was missing"] --> diff --> split --> prepare --> batches
+ subgraph b["batch, for each of the first MAX_BATCHES"]
+ direction LR
+ fetch --> verify --> publish --> checkpoint
+ end
+ batches --> b --> delete --> reconcile["reconcile once a day, or RECONCILE=true"] --> index --> smoke --> report --> ping
+ smoke --> sm["smoke-mirror"]
+ report --> re["report-engine"] --> rm["report-mirror"]
+ classDef hook stroke-dasharray: 5 5
+ class prepare,verify,index,sm,re,rm hook
+```
+
+Dashed verbs are hooks. `prepare` and `verify` do nothing until `TL_KEY` is set; `index`
+and `smoke-mirror` do nothing until a mirror fills them.
| Verb | Does |
|---|---|
-| `pipeline`, `plan-pipeline` | The verbs below in order, and the read-only half of them: `clock`, `list`, `state`, `diff`, `split` |
-| `list` | `rsync --list-only` of `SOURCE`, through `FILTER`, normalised to `.run/upstream.txt` as `path TAB size TAB mtime`, byte-sorted; a listing under `LIST_FLOOR` lines stops the run |
+| `pipeline`, `plan-pipeline` | The verbs above in order, and the read-only half: `clock`, `list`, `state`, `diff`, `split` |
+| `list` | `rsync -rL --list-only` of `SOURCE`, through `FILTER`, normalised to `.run/upstream.txt` as `path TAB size TAB mtime`, byte-sorted; a listing under `LIST_FLOOR` lines stops the run |
| `state` | Fetch `.state/applied.txt.xz` from the bucket; a missing one asks `rebuild` |
-| `rebuild` | List the bucket and make the state exactly what it holds at upstream's sizes |
-| `diff` | `changed.txt`, `deleted.txt` and `paths.txt`: upstream against the state |
+| `rebuild` | List the bucket and make the state exactly what it holds, at upstream's sizes |
+| `diff` | `changed.txt` (upstream has, the state lacks), `deleted.txt` (the state has, upstream lacks), `paths.txt` |
| `split` | Refuse a tree over `CEILING_GB` or a file the disk cannot hold; split the delta into `batch-NNNN.txt` of `BATCH_GB`, the decision batch last |
| `prepare` | Hook. With `TL_KEY` set and the delta touching `TL`: fetch the tlpdb and check its signature against the pinned key |
| `batches` | Work the first `MAX_BATCHES`, each `fetch`, `verify`, `publish`, `checkpoint`; touch `.run/chain` when batches remain |
+| `fetch` | rsync the batch's files into `staging/`, dereferencing symlinks; a path that vanished since the listing is skipped |
| `verify` | Hook. With `TL_KEY` set: every signed file and every container in the batch against the tlpdb |
+| `publish` | `aws s3 cp --recursive` of staging, one PutObject per file, never a destination listing; `timestamp` last |
+| `checkpoint` | `merge` what landed into the state and push it as one PutObject; empty staging |
| `delete` | Remove the keys upstream dropped, 1,000 per call, once every batch has landed, and drop them from the state |
| `reconcile` | In the run that starts in hour 03 UTC, or with `RECONCILE=true`: rebuild the state, then delete what neither upstream, `OWN` nor the state's own directories own |
| `index` | Hook. Nothing here; a mirror that draws directory pages or a landing page replaces it |
-| `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing, the tlpdb sha512 when `TL` is set, then `smoke-mirror` |
+| `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing; the tlpdb sha512 when `TL` is set; then `smoke-mirror` |
| `smoke-mirror` | Hook. Nothing here; a mirror with more to read back defines it |
| `report-engine` | Hook. The engine's rows of the run summary |
| `retry` | Run a command, retrying rsync's transport exit codes with backoff; 24 is a success |
-`normalise`, `pull`, `push`, `batch`, `fetch`, `publish`, `merge`, `checkpoint` and
-`remove` are the verbs those call; `pull` and `push` move an xz-compressed key between
-the bucket and `.run`, and a mirror's own verb may call them. A mirror replaces a hook by
-listing it under `excludes:` on the engine include and defining its own; `report-engine`
-is defined in the toolbox too, as a no-op, so an engine consumer's toolbox include
-excludes it.
+`normalise`, `pull`, `push`, `batch`, `merge` and `remove` are the verbs those call;
+`pull` and `push` move an xz-compressed key between the bucket and `.run`, and a
+mirror's own verb may call them.
+
+#### The bucket is the mirror; the state is a cache
+
+```mermaid
+flowchart LR
+ up["upstream, rsync --list-only path TAB size TAB mtime"] --> diff
+ st[".state/applied.txt.xz what the bucket holds, at upstream's size and mtime"] --> diff
+ diff --> ch["changed.txt upstream has, the state lacks"] --> split --> bt["batch-NNNN.txt"] --> batches
+ diff --> de["deleted.txt the state has, upstream lacks"] --> delete
+ batches -- "checkpoint, once per batch: merge what landed, one PutObject" --> st
+ delete -- "checkpoint" --> st
+ bucket["the bucket, listed"] -- "rebuild: join to upstream on size" --> st
+ reconcile -- "rebuild, then delete what neither upstream, OWN nor the state's directories own" --> bucket
+```
+
+An hourly run costs one listing of upstream and none of the bucket. Three things follow.
+
+- **There is no seed.** A missing state file is rebuilt from a listing of the bucket. An
+ empty bucket gives an empty state, so the first run's delta is the whole tree, worked
+ `MAX_BATCHES` at a time; when batches remain the run chains the next one, and the fill
+ converges in a few chained runs with no flag. A lost state file costs the same listing
+ and nothing else.
+- **The state records what landed.** `merge` joins the batch against what is in staging,
+ so a path that vanished upstream between listing and fetch never enters the state, and
+ the state never names a key the bucket lacks.
+- **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket on
+ purpose, daily by default, and deletes keys neither upstream nor the state owns.
+
+What a mirror cannot afford to lose is the bucket. Everything else, the state file and the
+staging tree included, is derived from it and from upstream.
+
+#### The decision batch
+
+`split` puts last whatever a client reads to decide what to fetch: the signed subtree's
+`tlpkg/` and every bucket-root file, `timestamp` among them. Containers land before the
+tlpdb that names them, and `delete` waits for the run in which every batch has landed, so
+the live tlpdb never names a removed container and a `tlmgr` run that overlaps a publish
+sees the previous tlpdb, never one naming a file that is not there.
+
+#### A signed subtree
+
+With `TL` and `TL_KEY` set, the engine verifies TeX Live's signatures before anything is
+published. The keyring is fetched from the same mirror as the signature, so the pinned
+fingerprint is the only real check; `GOODSIG` is required too, because gpgv reports an
+expired or revoked key as `VALIDSIG` with exit 0.
+
+```mermaid
+flowchart TB
+ subgraph p["prepare: once a run, when the delta touches TL"]
+ f1["fetch tlpkg/texlive.tlpdb.xz, .sha512, .sha512.asc, gpg/pubring.gpg"] --> f2["shasum -a 512 -c"] --> f3["gpgv: GOODSIG, and VALIDSIG ending in TL_KEY"]
+ end
+ subgraph v["verify: every batch, before publish"]
+ v1["every *.sha512 in the batch: shasum, then gpgv against the same keyring"]
+ v2["every archive/ container: its checksum in the verified tlpdb, under both of its names"]
+ v3["the decision batch: the tlpdb equals prepare's, the .xz decompresses to it, every container it names is in the bucket after this run"]
+ end
+ p --> v
+```
+
+A batch that fails any check stays local; the previous good copy stays live. `smoke`
+reads `texlive.tlpdb.sha512` back through the domain afterwards and compares it with the
+verified copy. `tlmgr` repeats the signature check on the client.
+
+#### The vars
A mirror sets `SOURCE`, `BUCKET` and `HOST` in its root vars, always. Everything else
-has an inline default in the engine, and a mirror sets only what differs:
+has an inline default, and a mirror sets only what differs:
| Var | Default | Meaning |
|---|---|---|
@@ -196,198 +339,351 @@ has an inline default in the engine, and a mirror sets only what differs:
| `TL`, `TL_KEY` | empty | A signed TeX Live subtree and the fingerprint that signs it; empty, no signature checks |
| `FILTER` | empty | rsync filter arguments that narrow the listing, for a mirror of a subtree |
| `OWN` | empty | Bucket-root keys the mirror owns, space separated; `reconcile` never deletes them |
-| `INDEX` | empty | The key suffix of the directory pages a mirror's `index` draws; set, `reconcile` spares those pages and every bare directory of the state, their second key |
+| `INDEX` | empty | The key suffix of the directory pages a mirror's `index` draws; set, `reconcile` spares those pages and every bare directory of the state |
A var the mirror puts in its root `vars:` is fixed for every run: inside an included
verb a root value shadows a `KEY=value` from the command line. One the mirror leaves to
its default is the run's to set, `task sync -- MAX_BATCHES=8 RECONCILE=true`. So a root
var that restates a default is worse than none: it silences the command line.
-A mirror of one signed subtree, shaped like tlnet:
-
-```yaml
-version: '3'
-vars:
- SOURCE: rsync://rsync.dante.ctan.org/CTAN/
- BUCKET: tlnet
- HOST: tlnet.katoptra.org
- TL: systems/texlive/tlnet
- TL_KEY: C78B82D8C79512F79CC0D7C80D5E5D9106BAB6BC
- CEILING_GB: 10
- OWN: index.html
- FILTER: >-
- --exclude=*.r[0-9]*.tar.xz --exclude=/systems/texlive/tlnet/update-tlmgr-r*
- --include=/systems/ --include=/systems/texlive/ --include=/systems/texlive/tlnet/***
- --exclude=*
-env:
- AWS_CONFIG_FILE: '{{.ROOT_DIR}}/aws.config'
-includes:
- toolbox:
- taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml
- flatten: true
- excludes: [report-engine, report-mirror]
- vars:
- NAME: tlnet
- DESC: a daily mirror of TeX Live's tlnet at https://tlnet.katoptra.org/
- IMAGE: ghcr.io/katoptra/toolbox:rsync-v2
- PASS: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ENDPOINT_URL
- rsync:
- taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/rsync.yml
- flatten: true
- excludes: [index]
-tasks:
- index:
- desc: The landing page, dated today, at the bucket root
- cmds:
- - sed "s/DATE/$(date -u +%Y-%m-%d)/" site/index.html | aws s3 cp --content-type text/html --cache-control no-cache - s3://{{.BUCKET}}/index.html
- report-mirror:
- internal: true
- cmds:
- - 'echo "| Landing page | index.html, dated $(date -u +%Y-%m-%d) |" >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}"'
-```
-
-`SOURCE` is the archive root and `FILTER` narrows the listing to the subtree, so every
-key keeps its `systems/texlive/tlnet/` prefix. `OWN` keeps `reconcile` off the landing
-page, `index` is the one hook the mirror fills, and the pipeline is the engine's.
-
-### The bucket is the mirror; the state is a cache
-
-An rsync engine keeps a state file in the bucket, under `.state/`, listing every key the
-bucket holds at upstream's size and mtime. Each run diffs upstream's listing against it,
-so an hourly run costs one listing of upstream and none of the bucket. Two things follow.
-
-- **There is no seed.** A missing state file is rebuilt from a listing of the bucket. An
- empty bucket gives an empty state, so the first run's delta is the whole tree, worked
- `MAX_BATCHES` at a time; when batches remain the run chains the next one, and the fill
- converges in a few chained runs with no flag. A lost state file costs the same listing
- and nothing else.
-- **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket
- on purpose, daily by default, and deletes keys neither upstream nor the state owns.
-
-What a mirror cannot afford to lose is the bucket. Everything else, the state file and the
-staging tree included, is derived from it and from upstream.
-
### The proton engine
-`engines/proton.yml` moves a staging tree into one Proton Drive folder, through the
-official `proton-drive` CLI, for a mirror whose upstream fits in a run: the mirror fills
+`engines/proton.yml`: a staging tree into one Proton Drive folder, through the official
+`proton-drive` CLI, for a mirror whose upstream fits in a run. The mirror fills
`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket
but the CLI session, because the CLI skips a file whose content Proton already holds and
-`-f create-new-revision` makes a revision of one that changed; Proton's version history
+`-f create-new-revision` makes a revision of one that changed: Proton's version history
is the history of the mirror.
+```mermaid
+flowchart LR
+ clock --> session --> destination --> stage --> upload --> confirm --> prune --> report --> ping
+ report --> re["report-engine"] --> rm["report-mirror"]
+ classDef hook stroke-dasharray: 5 5
+ class stage,prune,re,rm hook
+```
+
| Verb | Does |
|---|---|
-| `pipeline`, `plan-pipeline` | `clock`, `session`, `destination`, `stage`, `upload`, `confirm`, `prune`, `report`, `ping`; and the read-only half, through `stage` |
+| `pipeline`, `plan-pipeline` | The verbs above in order; the read-only half runs through `stage` and prints what an upload would carry |
| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` |
-| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit |
| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` |
-| `stage` | Hook. The mirror fills `staging/` with what Proton should hold |
+| `stage` | Hook. Every mirror defines it: fill `staging/` with what Proton should hold |
| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` |
| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` |
| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` |
+| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit |
| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash |
| `session-seal -- ` | Host side: a laptop login's two files, encrypted into the bucket |
| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently |
| `report-engine` | Hook. The engine's rows of the run summary |
The engine reads no root var. Its inputs are the environment, by the names every Proton
-mirror's `op.env` carries: `MIRROR_PROTON_DESTINATION` and its `_UID`, `MIRROR_R2_BUCKET`,
-`MIRROR_AGE_IDENTITY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and
-`AWS_ENDPOINT_URL_S3`. The session is the one thing to set up by hand: the CLI can only be
-seeded by a browser sign-in, so log in once on a laptop into a directory inside the repo
-and run `task session-seal -- `. Two mirrors never share a session; its
-refresh token rotates on every call, and the loser of a race needs a fresh login.
+mirror's `op.env` carries:
-A mirror of every repository under two GitHub owners, shaped like github:
+| Name | What it is |
+|---|---|
+| `MIRROR_PROTON_DESTINATION` | The CLI path of the destination folder, `/my-files/GitHub` say |
+| `MIRROR_PROTON_DESTINATION_UID` | Its UID, compared on every run before any write |
+| `MIRROR_R2_BUCKET` | The bucket that holds the session |
+| `MIRROR_AGE_IDENTITY` | The `AGE-SECRET-KEY-...` line the session is encrypted to |
+| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3` | What boto3 reads; note `_S3` on the endpoint |
-```yaml
-version: '3'
-vars:
- OWNERS: jshvn katoptra
-includes:
- toolbox:
- taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml
- flatten: true
- excludes: [report-engine, report-mirror]
- vars: {NAME: github, DESC: a nightly mirror of every repository under jshvn and katoptra into Proton Drive, IMAGE: ghcr.io/katoptra/toolbox:proton-v2}
- proton:
- taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml
- flatten: true
- excludes: [stage, prune]
-tasks:
- stage: {cmds: ['# list the repositories, clone each as a mirror, bundle it under {{.STAGING}}//']}
- prune: {cmds: ['# list-folder each owner in Proton; trash the bundles no repository has']}
- report-mirror: {cmds: ['# the mirror rows']}
+#### The session
+
+The CLI can only be seeded by a browser sign-in, and fresh sign-ins from datacenter
+addresses are blocked, so the session is made once on a laptop and carried to every run
+encrypted. Its refresh token rotates on use, so every run writes it back.
+
+```mermaid
+flowchart LR
+ subgraph laptop["Once, on a laptop"]
+ l1["proton-drive auth login with the session as plain files under .run/pd"] --> l2["task session-seal -- .run/pd tar, age-encrypt, push"]
+ end
+ l2 --> key[".state/session.tar.age in the bucket"]
+ subgraph run["Every run"]
+ r1["session: pull, decrypt, extract to .run/session"] --> r2["pd: one CLI call"] --> r3{"token rotated?"}
+ r3 -- yes --> r4["session-push: seal it back"]
+ r3 -- no --> r2
+ end
+ key --> r1
+ r4 --> key
```
-## Overriding a verb
+To make one:
-List it under `excludes:` on the include that defines it and define it in the mirror.
-A duplicate without `excludes` is a parse error, on purpose.
+```sh
+export PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file
+proton-drive auth login # a browser opens; .run/pd holds two JSON files after
+proton-drive filesystem create-folder /my-files # /my-files is the drive's own root, not a folder you make
+proton-drive filesystem list -j /my-files # the entry named : its uid is the destination UID
+task session-seal -- .run/pd # once the bucket and the vault exist
+```
-```yaml
-includes:
- rsync:
- taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/rsync.yml
- flatten: true
- excludes: [index] # the mirror draws its own landing page
-tasks:
- index:
- cmds: [...]
+Install the CLI at the version `toolchain.lock.toml` pins: the session file format is
+tied to the version, and the Linux binary in the image must read what the laptop wrote.
+`.run/pd` must be inside the repository, because only the repository is mounted into the
+image; `.run/` is ignored by git and deleted by `task clean`.
+
+Three rules. **Two mirrors never share a session**: its refresh token rotates on every
+call, and the loser of a race needs a fresh login. **Never run `sync` or `plan` from a
+laptop while an Actions run may be in progress**, for the same reason. **The session gets
+no history in the bucket**: a stale copy holds a rotated-out token and cannot be restored.
+
+### A pipeline of your own
+
+A mirror whose logic is a program of its own includes the toolbox alone and defines
+`pipeline` and `plan-pipeline` from the toolbox's `clock`, `report` and `ping` and its own
+steps. dropbox is that shape: each step is one `python -m migrator `, the
+Taskfile owns the order, the Python owns every decision, and the `proton` image supplies
+the interpreter, boto3 and the CLI. Such a mirror fills `report-mirror` with its own
+report and nothing else of the toolbox's changes.
+
+## Secrets
+
+Every credential and every account identifier lives in one vault and reaches a run by
+name. A repository holds `op://` references only.
+
+```mermaid
+flowchart LR
+ item["A vault item, one per mirror sections per service: r2, healthcheck, proton, age, ..."] --> ref["op.env NAME=op://vault-uuid/item/section/field"]
+ ref --> oprun["op run --env-file=op.env, on the host values exported into its child's environment only"]
+ sa["A service account that reads the vault its token: the secret OP_SERVICE_ACCOUNT_TOKEN"] --> oprun
+ oprun --> run["run: -e NAME for every name in op.env and PASS values never on a command line, never in a log"]
+ run --> tool["the tool inside the image reads its environment aws, boto3, curl, git"]
```
-The rules that keep this honest:
+### The vault
-- Keep the name and the meaning. A `verify` that does something other than verify is a
- new verb with its own name, listed in `MENU`.
-- Extend through a hook, never a copy. A mirror that needs the engine's `smoke` and
- more defines `smoke-mirror`, which `smoke` runs last, rather than excluding `smoke`
- and pasting it: the copy stops following releases the day it is made. A mirror with a
- step to add beside `clock` adds a step to its pipeline, not a `clock` of its own.
-- The replacement reads the mirror's root vars, the same as the original did.
-- Override engine verbs and the in-container toolbox verbs. Do not override the host
- side: `sync`, `plan`, `check` and `run` are what the workflows and the menu promise.
-- Everything the mirror does not exclude comes from here, and a release changes it.
+One vault. One item per mirror, named for its repository. One section per service in
+the item (`r2`, `healthcheck`, `proton`, `age`, `github`, `dropbox`), with the field names
+the mirror's README lists. Neither the vault's name nor its UUID is a secret: without the
+service account's token it opens nothing.
-## Plugging in an engine
+The references address the vault by UUID, `op://///`, so a
+renamed vault cannot break a run and a name with a slash in it cannot be written:
-One engine per transport, one image variant per engine. Adding one is five files and
-two matrix entries:
+```sh
+op vault get --format json | jq -r .id
+```
-1. **Tools**: add each tool's version and per-arch checksum to `toolchain.lock.toml`.
-2. **Image**: `docker/.Dockerfile`, base pinned by digest, every tool installed
- from the lock, `TASK_REMOTE_OFFLINE=1`, and a final stage that asserts each tool
- reports the locked version. `docker/rsync.Dockerfile` is the model.
-3. **Verbs**: `engines/.yml` with the pipeline vocabulary above. No `vars:`
- default for anything a mirror owns; defaults go inline as `{{.X | default N}}`.
-4. **Example**: `examples//` with a Taskfile whose `pipeline` is every verb, a
- committed `render.txt`, and an `offline` verb that runs the ones that need no bucket
- over `fixtures/`. `examples/rsync/` is the model. This is the engine's own check.
-5. **CI**: add the variant to the `matrix` in `ci.yml` and `release.yml`.
+One service account that can read that vault and nothing else. Its token is the secret
+`OP_SERVICE_ACCOUNT_TOKEN`: on an organization, one organization secret that every
+repository inherits; on a personal account, one repository secret per mirror. Only a
+dispatched sync run holds it; a pull-request check gets no secret at all.
+
+### On the host
+
+`op run --env-file=op.env` resolves every reference and exports the values into the
+environment of the one command it wraps, `task run -- ...`, masking each value in that
+command's output. On a laptop `op` uses the signed-in desktop app; in Actions it uses the
+service account token the reusable workflow put in its environment. The vault is read
+once per run.
+
+### Into the container
+
+`run` passes `-e NAME` for each name in `op.env` and in the mirror's `PASS`, plus
+`HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always. The container engine
+copies each named variable from the host's environment; no value is on the command line.
+A host variable the pipeline reads and `op.env` does not name arrives empty.
+
+### The names
+
+| Name | Read by | Set by |
+|---|---|---|
+| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | the AWS CLI and boto3 | the `r2` section |
+| `AWS_ENDPOINT_URL` | the AWS CLI, in the `rsync` image | the `r2` section's `endpoint` |
+| `AWS_ENDPOINT_URL_S3` | boto3, in the `proton` image | the same field, under boto3's name |
+| `AWS_REGION` | both | a literal `auto` in the image; R2 has one region, and masking the word `auto` would corrupt output |
+| `MIRROR_R2_BUCKET` | the proton engine's `s3` | the `r2` section's `bucket` |
+| `MIRROR_AGE_IDENTITY` | the proton engine's `age` | the `age` section |
+| `MIRROR_PROTON_DESTINATION`, `_UID` | the proton engine's `destination` | the `proton` section |
+| `HEALTHCHECK_URL` | `ping`, `ping-fail` | the `healthcheck` section; optional |
+| `MIRROR_*` of a mirror's own | that mirror | a section of its own: `github`, `dropbox` |
+
+### Without a vault
+
+A mirror with no `op.env` runs on its repository secrets: the reusable workflow exports
+every inherited secret into the sync step's environment by name, the ones the mirror
+lists in `PASS` cross into the container, and the 1Password CLI is not installed at all.
+On a laptop, whatever is exported. This is the fallback, not the pattern: it works, and
+it puts the values in a second place.
+
+## Storage
+
+Every mirror has one S3-compatible bucket. What it holds depends on the engine.
+
+| Engine | Key | What |
+|---|---|---|
+| rsync | every upstream path, at the root | The mirror. A public domain serves the bucket |
+| rsync | `.state/applied.txt.xz` | The state: what the bucket holds, at upstream's size and mtime |
+| rsync, a mirror's own | `.state/indexed.txt.xz`, `index.html` | ctan's record of what its directory pages show; tlnet's landing page, spared by `OWN` |
+| proton | `.state/session.tar.age` | The CLI session, encrypted. The only key |
+| a pipeline of its own | `.state/state.sqlite.xz.age`, `.state/history/-