diff --git a/.github/workflows/sync.yml b/.github/workflows/sync.yml index 45ccd1a..0ed63d6 100644 --- a/.github/workflows/sync.yml +++ b/.github/workflows/sync.yml @@ -1,5 +1,5 @@ # Reusable: one mirror, one run. The caller is a workflow_dispatch of a few lines that -# jshvn/dispatch triggers; it passes `vars` through and inherits its repository secrets. +# an external scheduler triggers; it passes `vars` through and inherits its repository secrets. # 1Password is read once per run: one `op run` resolves op.env around the whole pipeline, # failure path included. A mirror without an op.env runs on its repository secrets instead, # exported into the sync step's environment and crossing into the container by the names diff --git a/CLAUDE.md b/CLAUDE.md index 754d0f3..59ed913 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,37 +1,21 @@ # lib -The toolbox every katoptra mirror includes by URL. Read `README.md` for the contract and -`docs/superpowers/specs/2026-09-08-toolbox-library-design.md` for the decisions. +The toolbox every katoptra mirror includes by URL. `README.md` is the manual: the layers, +the verbs, the engines, secrets, storage, the images, the workflows, and how a mirror +uses and changes any of it. `docs/superpowers/specs/2026-09-08-toolbox-library-design.md` +is the record of what was decided when. This file is what a change must not break: +each entry is a verified go-task or platform fact that cost a session to find. -## Constraints +## Hazards -- `toolbox.yml` declares no `vars:` default for anything a mirror owns. Defaults go - inline. This is a go-task fact, verified: a library `vars:` default shadows the - mirror's root value. -- Verb names are reserved across host and container. `plan` is the host-side read-only - run; an engine's batch planner is `split`. The hooks, `prepare`, `verify`, `index`, - `smoke`, `smoke-mirror`, `report-engine` in the rsync engine, `stage` and `prune` in - the proton engine, and `report-mirror` in the toolbox, are the only verbs a mirror - redefines, each excluded on the include that defines it; `pipeline` and - `plan-pipeline` come from the engine and a mirror excludes them only for another - order. `report-engine` exists in both files, so an engine - consumer excludes it on the toolbox include. Extension is a hook, never a copy: a - verb that needs more than the engine does gets a `-mirror` hook the engine calls last. -- A mirror's root var shadows a command-line `KEY=value` inside an included verb, so an - engine tunable is an inline default and never a root var of the example. A var of - another name that reads the tunable, `BATCHES_MAX` from `MAX_BATCHES`, still sees the - command line (verified), so each default is spelled once. -- A called task sees none of its caller's call vars. A hook a verb calls with an - overridable `RUN` or `STAGING` gets them passed on explicitly, as `smoke` does. - A global var that reads another var is rendered once, from the root's and the command line's values, never from a call var: `GPGCHECK` takes the fingerprint as an awk `-v` at each use, where a call var `TL_KEY` is visible. +- A var of another name that reads a tunable, `BATCHES_MAX` from `MAX_BATCHES`, still + sees the command line (verified), so each default is spelled once, inline. - No `dir:` on an engine verb. In a flattened include read by path, task joins even an absolute `dir:` onto the include's directory (verified, 3.53.1), so `verify` starts each command with `cd {{.STAGING}}` instead. -- `render` mounts an empty directory over `.run`, so a `sh:` var read from it at parse - time renders the same from any working tree. A verb that reads `.run` at parse time - must tolerate an empty one. - `.taskrc.yml` keys are `trusted-hosts` and `cache-expiry`; task ignores a key it does not know, silently, and refetches on every invocation. - Every tool in an image comes from `toolchain.lock.toml`, read through `docker/lock.py`, @@ -51,12 +35,9 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont the `--` as the format. Use `printf '%s %s ' -e "$v"`. - Task's built-in shell has no `umask`. A file that must be born 0600 is `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does. -- The proton engine keeps no state of the mirror's in the bucket, only the CLI session, - because `filesystem upload` skips a file whose content Proton already holds and - `-f create-new-revision` handles one that changed. Every CLI call goes through `pd`, - which pushes the session back whatever the exit: the refresh token rotates, and a run - that kept a rotated token to itself leaves the next run unable to log in. Two mirrors - never share one session for the same reason. +- Every Proton CLI call goes through `pd`, which pushes the session back whatever the + exit: the refresh token rotates, and a run that kept a rotated token to itself leaves + the next run unable to log in. Two mirrors never share one session for the same reason. - Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the two reusable workflows that way; each checks this repository out at its own commit (`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is @@ -78,3 +59,7 @@ example. Regenerate the tree with a new key only to change its shape; the privat was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an accepting and a refusing upload summary, and the `age` verb round trip with a throwaway identity. + +A README change is a diff `task check` cannot see; the review is reading it. Every +anchor a mirror's README links here (`#secrets`, `#storage`, `#the-rsync-engine`, +`#the-proton-engine`) is a heading in `README.md`; renaming one breaks four repositories. diff --git a/README.md b/README.md index 6880fc2..7d1b88b 100644 --- a/README.md +++ b/README.md @@ -7,39 +7,73 @@ The toolbox every katoptra mirror includes by URL. The rule it enforces: the cod starts a run, contains it, resolves its secrets, checks it and reports it lives here, once. The code that moves bytes for a transport lives here too, once per engine: `engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a -staging tree into Proton Drive. A mirror holds only its identity, the order of its -pipeline, and the few verbs no other mirror shares. +staging tree into Proton Drive. A mirror holds only its identity, the hooks it fills, and +the few verbs no other mirror shares. + +This file is the manual. A mirror's README says what it mirrors and how to fork it, and +links here for everything the mirrors share. Read it top to bottom once and the system +should hold no surprises; after that, each section stands on its own. + +## The system in one picture + +```mermaid +flowchart LR + subgraph up["Upstreams"] + u1["CTAN, over rsync"] + u2["GitHub, over the API and git"] + u3["Dropbox, over the API"] + end + sched["A scheduler
workflow_dispatch on a cron"] --> host + subgraph job["One GitHub Actions job per run"] + host["The runner: task sync"] --> box["The toolbox image: task pipeline"] + end + vault["A 1Password vault
one item per mirror"] -. "op run: values, by name" .-> host + u1 --> box + u2 --> box + u3 --> box + box --> r2["An S3-compatible bucket, R2
a public mirror: the mirror itself
a Proton mirror: the session, and state"] + box --> pd["Proton Drive
the private mirrors' sink"] + r2 --> cf["A custom domain on the bucket"] --> cl["tlmgr, browsers, anyone"] + box --> hc["healthchecks.io, one ping per run"] --> site["katoptra.org, live status"] +``` + +Every mirror is that picture with one upstream and one sink. A public mirror (ctan, +tlnet) copies an rsync tree into a bucket and serves the bucket; the bucket is the mirror. +A private mirror (github, dropbox) copies an account into Proton Drive and keeps only +what the next run needs in the bucket. Both run the same way: a job on GitHub Actions, +started by a dispatch, that pulls one image, resolves its secrets by name from a vault, +and runs `task pipeline` inside the image. The same command runs on a laptop. ## The layers ```mermaid flowchart TB - subgraph mirror["A mirror repository (katoptra/ctan, say)"] - tf["Taskfile.yml
root vars: SOURCE, BUCKET, HOST
pipeline, plan-pipeline
overrides, if any"] - opv["op.env
op:// references"] + subgraph mirror["A mirror repository"] + tf["Taskfile.yml
root vars: the identity
the hooks it fills, verbs of its own"] + opv["op.env
op:// references, the vault by UUID"] rt["render.txt
the committed dry run"] - cw["sync.yml, check.yml
callers of lib's"] + cw["sync.yml, check.yml
ten-line callers"] end - subgraph lib["katoptra/lib, pinned to v2"] - tb["toolbox.yml
menu, image, run, op, sync, plan
render, check, clock, ping"] - en["engines/rsync.yml
list, diff, split, batches, reconcile ...
hooks: prepare, verify, index, smoke"] - im["ghcr.io/katoptra/toolbox:<variant>-v2
docker/ + toolchain.lock.toml"] + subgraph lib["katoptra/lib, at v2"] + tb["toolbox.yml
menu, image, run, op, sync, plan, check
clock, report, ping, failed"] + en["engines/rsync.yml, engines/proton.yml
the pipeline vocabulary, one file per transport"] + im["ghcr.io/katoptra/toolbox:rsync-v2, :proton-v2
docker/, toolchain.lock.toml"] wf[".github/workflows/sync.yml, check.yml
.github/actions/toolbox"] end tf -- "includes, flattened" --> tb tf -- "includes, flattened" --> en tb -- "runs pipeline inside" --> im - cw -- "uses @release SHA" --> wf - wf -- "task sync / task check" --> tb + cw -- "uses, at a release SHA" --> wf + wf -- "task sync, task check" --> tb ``` | Layer | Lives in | Owns | A mirror touches it when | |---|---|---|---| | Toolbox | `toolbox.yml` | How a run starts, is contained, resolves secrets, is rendered, checked and reported | Never; it is included as is | -| Engine | `engines/.yml` | How bytes move for one transport: rsync to R2, Proton Drive, HTTPS | It lists a verb under `excludes:` and defines its own | +| Engine | `engines/.yml` | How bytes move for one transport, and the pipeline's order | It lists a hook under `excludes:` and defines its own | | Image | `docker/`, `toolchain.lock.toml` | Every tool a run needs, pinned by checksum | Never; it names one in `IMAGE` | | Workflows | `.github/workflows`, `.github/actions/toolbox` | How Actions installs the tools and runs `task sync` and `task check` | Never; its callers are a few lines each | -| Mirror | The mirror's own repository | Identity, pipeline order, the exceptions | Always; this is all it holds | +| Mirror | The mirror's own repository | Identity, the hooks, the exceptions | Always; this is all it holds | ## What a run looks like @@ -47,7 +81,7 @@ The same path on a laptop and in Actions. On a laptop it starts at `task sync`. ```mermaid sequenceDiagram - participant D as jshvn/dispatch + participant D as A scheduler participant W as sync.yml (reusable) participant H as Host: task participant O as op run @@ -59,9 +93,9 @@ sequenceDiagram H->>O: op: op run --env-file=op.env O->>C: run: engine run --rm -v repo:/work -e NAME ... IMAGE task pipeline C->>C: clock - C->>C: engine verbs: list, diff, split, batches, reconcile ... + C->>C: the engine's verbs, and the mirror's hooks C->>C: report, then ping - alt pipeline failed + alt the pipeline failed C->>C: failed: report STATUS=failed, then ping-fail else the runner cut the run off W->>C: task op -- task failed @@ -70,120 +104,229 @@ sequenceDiagram end ``` -Three things the diagram hides: +Four things the diagram hides: - **Secrets cross by name, never by value.** `op run` resolves `op.env` on the host and exports the values; `run` passes `-e NAME` for each name in `op.env` and in `PASS`, plus `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always, so no value - ever appears on a command line or in a log. -- **1Password is read once per run.** One `op run` wraps the whole pipeline, and the + ever appears on a command line or in a log. The whole path is under [Secrets](#secrets). +- **The vault is read once per run.** One `op run` wraps the whole pipeline, and the failure path, `failed`, runs inside the same container, so a failed run costs no second read. The one exception is a run the runner cut off, by the timeout or a cancellation, which never reached its own failure path: the workflow runs `failed` - again from outside, one more read on a rare day. A mirror without an `op.env` runs - on its repository secrets: the reusable workflow exports every inherited secret into - the sync step's environment by name, the ones the mirror lists in `PASS` cross into - the container, and the 1Password CLI is not installed at all. + again from outside, one more read on a rare day. - **Nothing inside the container reaches the network for task itself.** The image sets `TASK_REMOTE_OFFLINE=1` and the mirror's `.task/remote` cache rides in with the repo. - **`plan` is `sync` with the read-only half.** It runs `plan-pipeline` instead of `pipeline`, inside the same image with the same secrets. -## The verbs +## The toolbox -### What a mirror must define +`toolbox.yml` is every verb a mirror has whatever moves its bytes. Half of its verbs run +on the host and wrap the container; the other half run inside it and are the first and +last steps of every pipeline. -| Name | Kind | Meaning | -|---|---|---| -| `SOURCE`, `BUCKET`, `HOST` | root vars | The mirror's identity. Engines read them; the library never defaults them. | -| `NAME`, `DESC`, `IMAGE` | include vars | The menu's title and line, and the image the run happens inside. | -| `pipeline`, `plan-pipeline` | tasks | The full run and its read-only half, in order, inside the image; `sync` and `plan` call them. An engine supplies both; a mirror with no engine defines them. | - -### What a mirror may define - -| Name | Kind | Meaning | -|---|---|---| -| `PASS` | include var | Host environment names that cross into the container beside the ones in `op.env`, for a pipeline that reads its environment. Task vars are not environment: they go after `--`, as `task sync -- MAX_BATCHES=8`. | -| `report-mirror` | task | The mirror's rows of the run summary, after the toolbox's and the engine's. Excluded on the toolbox include. | -| `smoke-mirror` | task | The mirror's own read-back checks, after the engine's sample. Excluded on the engine include. | -| `MENU` | include var | Extra lines for the menu, one per mirror-specific verb. | -| `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib`. | -| `op.env` | file | `op://` references, one per secret. Absent means the environment is already resolved: on a laptop, whatever is exported; in Actions, the repository's secrets, crossing by the names in `PASS`. | -| `excludes:` | include key | Library verbs the mirror replaces. See below. | +```mermaid +flowchart TB + sync["task sync -- K=v"] --> op + plan["task plan -- K=v"] --> op + op["op: op run --env-file=op.env, when the file exists"] --> run + run["run: engine run --rm -v repo:/work -e NAME ... IMAGE"] --> image["image: pull IMAGE unless it is present"] + run --> pipeline["task pipeline, inside the image"] + run --> pp["task plan-pipeline, inside the image"] + pipeline -- "on failure" --> failed["failed: report STATUS=failed, then ping-fail"] + check["task check"] --> render["render: task --dry --force pipeline
inside the image, over an empty .run"] + render --> image + render --> d["diff against render.txt"] + ru["task render-update"] --> render + ru --> cp["accept it as render.txt"] +``` -### What the toolbox provides +### Host side -Host side, and the mirror does not override these. They are the contract the workflows -call and the menu documents. +The contract the workflows call and the menu documents. A mirror does not override these. | Verb | Does | |---|---| | `default` | The grouped menu, from `NAME`, `DESC` and `MENU` | -| `sync` | `op -- task pipeline`: one run | -| `plan` | `op -- task plan-pipeline`: the read-only half | +| `sync -- [K=v ...]` | `op -- task pipeline`: one run. The words after `--` are task vars for the pipeline inside the image | +| `plan -- [K=v ...]` | `op -- task plan-pipeline`: the read-only half | | `check` | `render`, then diff against `render.txt` | | `render-update` | `render`, then accept it as `render.txt` | -| `render` | `task --dry --force pipeline` inside the image, over an empty `.run`, saved to `.run/render.txt` | -| `run -- ` | Anything inside the image, with the repo at `/work` | +| `render` | `task --dry --force pipeline` inside the image, over an empty `.run`, saved to `.run/render.txt`; no secrets cross | +| `run -- ` | Anything inside the image, with the repository at `/work` | | `op -- ` | `run`, wrapped in `op run --env-file=op.env` when the file exists | | `image` | Pull `IMAGE`; a no-op while it exists locally | -| `image-build` | Build `IMAGE` from `LIB_DIR/docker/.Dockerfile` | +| `image-build` | Build `IMAGE` from `LIB_DIR/docker/.Dockerfile`; `LIB_DIR` defaults to `../lib` | | `image-clean` | Remove `IMAGE` | | `clean` | Delete `.run`, `staging` and the Taskfile cache; nothing else | -Inside the image. A mirror or engine may override these when it keeps its own. +The container engine is Apple `container` when its daemon is up, else Docker; +`ENGINE=docker` forces Docker. The host needs go-task, the container engine, and the +1Password CLI for anything that reads the vault. Nothing else is installed on a host. + +### Inside the image + +The first and last verbs of every pipeline. A mirror or engine may override these when it +keeps its own. | Verb | Does | |---|---| | `clock` | Write "epoch UTC-hour weekday" to `.run/start.txt` and forget the last run's chain file | | `report` | Append the run summary to the Actions job page (stdout elsewhere): its own rows, then `report-engine`'s, then `report-mirror`'s | -| `report-engine`, `report-mirror` | Hooks: no-ops here, the engine's and the mirror's rows | +| `report-engine`, `report-mirror` | Hooks: no-ops here; the engine's and the mirror's rows | | `ping` | GET `HEALTHCHECK_URL`; skipped when unset | | `ping-fail` | GET `HEALTHCHECK_URL/fail`; skipped when unset | | `failed` | The failure path: `report STATUS=failed`, then `ping-fail` | -`sync` runs `pipeline`, and on a failure `failed`, all in the one container; the -workflow runs `failed` from outside only for a run the runner cut off. One table, three -layers: the toolbox's rows (when the run started -and how long it took, the image, whether the next run is queued), the engine's, the -mirror's, each a `| Label | value |` line appended to the same file. Every row tolerates -a missing file, because the report also runs after a failed pipeline. +`sync` runs `pipeline`, and on a failure `failed`, all in the one container. The report +is one table in three layers, each a `| Label | value |` line appended to the same file: +the toolbox's rows (when the run started and how long it took, the image, whether the +next run is queued), the engine's, the mirror's. Every row tolerates a missing file, +because the report also runs after a failed pipeline. + +### What a mirror gives the toolbox -### What an engine provides +| Name | Kind | Meaning | +|---|---|---| +| `NAME`, `DESC`, `IMAGE` | include vars, required | The menu's title and line, and the image the run happens inside | +| `pipeline`, `plan-pipeline` | tasks, required | The full run and its read-only half, inside the image. An engine supplies both; a mirror with no engine defines them | +| `op.env` | file | `op://` references, one per secret. Absent means the environment is already resolved: on a laptop, whatever is exported; in Actions, the repository's secrets, crossing by the names in `PASS` | +| `PASS` | include var | Host environment names that cross into the container beside the ones in `op.env`. Task vars are not environment: they go after `--` | +| `MENU` | include var | Extra lines for the menu, one per mirror-specific verb | +| `report-mirror` | task | The mirror's rows of the run summary. Excluded on the toolbox include | +| `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib` | +| `excludes:` | include key | Library verbs the mirror replaces. See [Changing it](#changing-it) | + +## The engines An engine is a second include with the verbs for one transport. Its verbs are the -pipeline vocabulary, reserved across every engine so a mirror's `pipeline` reads the -same whichever transport it uses. The rsync engine, `engines/rsync.yml`, is the first: -an rsync upstream into an S3 bucket, as a list diff and never a local tree. +pipeline vocabulary, reserved across every engine so a mirror's pipeline reads the same +whichever transport it uses. An engine reads the mirror's root vars and the environment; +it declares no `vars:` default for anything a mirror owns, because a value there would +shadow the mirror's. Every tunable is an inline `{{.X | default N}}`. + +The hooks are the verbs an engine leaves empty for a mirror to fill. A mirror lists each +it defines under `excludes:` on the engine's include. Extension is a hook, never a copy: +a mirror that needs the engine's `smoke` and more defines `smoke-mirror`, which `smoke` +runs last. + +### The rsync engine + +`engines/rsync.yml`: an rsync upstream into an S3 bucket, as a list diff and never a +local tree. The runner has 14 GB of disk and CTAN has 140; a run lists upstream, diffs +the listing against the state file the last run left in the bucket, and moves only the +delta, in batches, each committed before the next starts. + +```mermaid +flowchart LR + clock --> list --> state --> rebuild["rebuild
only if the state was missing"] --> diff --> split --> prepare --> batches + subgraph b["batch, for each of the first MAX_BATCHES"] + direction LR + fetch --> verify --> publish --> checkpoint + end + batches --> b --> delete --> reconcile["reconcile
once a day, or RECONCILE=true"] --> index --> smoke --> report --> ping + smoke --> sm["smoke-mirror"] + report --> re["report-engine"] --> rm["report-mirror"] + classDef hook stroke-dasharray: 5 5 + class prepare,verify,index,sm,re,rm hook +``` + +Dashed verbs are hooks. `prepare` and `verify` do nothing until `TL_KEY` is set; `index` +and `smoke-mirror` do nothing until a mirror fills them. | Verb | Does | |---|---| -| `pipeline`, `plan-pipeline` | The verbs below in order, and the read-only half of them: `clock`, `list`, `state`, `diff`, `split` | -| `list` | `rsync --list-only` of `SOURCE`, through `FILTER`, normalised to `.run/upstream.txt` as `path TAB size TAB mtime`, byte-sorted; a listing under `LIST_FLOOR` lines stops the run | +| `pipeline`, `plan-pipeline` | The verbs above in order, and the read-only half: `clock`, `list`, `state`, `diff`, `split` | +| `list` | `rsync -rL --list-only` of `SOURCE`, through `FILTER`, normalised to `.run/upstream.txt` as `path TAB size TAB mtime`, byte-sorted; a listing under `LIST_FLOOR` lines stops the run | | `state` | Fetch `.state/applied.txt.xz` from the bucket; a missing one asks `rebuild` | -| `rebuild` | List the bucket and make the state exactly what it holds at upstream's sizes | -| `diff` | `changed.txt`, `deleted.txt` and `paths.txt`: upstream against the state | +| `rebuild` | List the bucket and make the state exactly what it holds, at upstream's sizes | +| `diff` | `changed.txt` (upstream has, the state lacks), `deleted.txt` (the state has, upstream lacks), `paths.txt` | | `split` | Refuse a tree over `CEILING_GB` or a file the disk cannot hold; split the delta into `batch-NNNN.txt` of `BATCH_GB`, the decision batch last | | `prepare` | Hook. With `TL_KEY` set and the delta touching `TL`: fetch the tlpdb and check its signature against the pinned key | | `batches` | Work the first `MAX_BATCHES`, each `fetch`, `verify`, `publish`, `checkpoint`; touch `.run/chain` when batches remain | +| `fetch` | rsync the batch's files into `staging/`, dereferencing symlinks; a path that vanished since the listing is skipped | | `verify` | Hook. With `TL_KEY` set: every signed file and every container in the batch against the tlpdb | +| `publish` | `aws s3 cp --recursive` of staging, one PutObject per file, never a destination listing; `timestamp` last | +| `checkpoint` | `merge` what landed into the state and push it as one PutObject; empty staging | | `delete` | Remove the keys upstream dropped, 1,000 per call, once every batch has landed, and drop them from the state | | `reconcile` | In the run that starts in hour 03 UTC, or with `RECONCILE=true`: rebuild the state, then delete what neither upstream, `OWN` nor the state's own directories own | | `index` | Hook. Nothing here; a mirror that draws directory pages or a landing page replaces it | -| `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing, the tlpdb sha512 when `TL` is set, then `smoke-mirror` | +| `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing; the tlpdb sha512 when `TL` is set; then `smoke-mirror` | | `smoke-mirror` | Hook. Nothing here; a mirror with more to read back defines it | | `report-engine` | Hook. The engine's rows of the run summary | | `retry` | Run a command, retrying rsync's transport exit codes with backoff; 24 is a success | -`normalise`, `pull`, `push`, `batch`, `fetch`, `publish`, `merge`, `checkpoint` and -`remove` are the verbs those call; `pull` and `push` move an xz-compressed key between -the bucket and `.run`, and a mirror's own verb may call them. A mirror replaces a hook by -listing it under `excludes:` on the engine include and defining its own; `report-engine` -is defined in the toolbox too, as a no-op, so an engine consumer's toolbox include -excludes it. +`normalise`, `pull`, `push`, `batch`, `merge` and `remove` are the verbs those call; +`pull` and `push` move an xz-compressed key between the bucket and `.run`, and a +mirror's own verb may call them. + +#### The bucket is the mirror; the state is a cache + +```mermaid +flowchart LR + up["upstream, rsync --list-only
path TAB size TAB mtime"] --> diff + st[".state/applied.txt.xz
what the bucket holds, at upstream's size and mtime"] --> diff + diff --> ch["changed.txt
upstream has, the state lacks"] --> split --> bt["batch-NNNN.txt"] --> batches + diff --> de["deleted.txt
the state has, upstream lacks"] --> delete + batches -- "checkpoint, once per batch:
merge what landed, one PutObject" --> st + delete -- "checkpoint" --> st + bucket["the bucket, listed"] -- "rebuild: join to upstream on size" --> st + reconcile -- "rebuild, then delete what neither
upstream, OWN nor the state's directories own" --> bucket +``` + +An hourly run costs one listing of upstream and none of the bucket. Three things follow. + +- **There is no seed.** A missing state file is rebuilt from a listing of the bucket. An + empty bucket gives an empty state, so the first run's delta is the whole tree, worked + `MAX_BATCHES` at a time; when batches remain the run chains the next one, and the fill + converges in a few chained runs with no flag. A lost state file costs the same listing + and nothing else. +- **The state records what landed.** `merge` joins the batch against what is in staging, + so a path that vanished upstream between listing and fetch never enters the state, and + the state never names a key the bucket lacks. +- **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket on + purpose, daily by default, and deletes keys neither upstream nor the state owns. + +What a mirror cannot afford to lose is the bucket. Everything else, the state file and the +staging tree included, is derived from it and from upstream. + +#### The decision batch + +`split` puts last whatever a client reads to decide what to fetch: the signed subtree's +`tlpkg/` and every bucket-root file, `timestamp` among them. Containers land before the +tlpdb that names them, and `delete` waits for the run in which every batch has landed, so +the live tlpdb never names a removed container and a `tlmgr` run that overlaps a publish +sees the previous tlpdb, never one naming a file that is not there. + +#### A signed subtree + +With `TL` and `TL_KEY` set, the engine verifies TeX Live's signatures before anything is +published. The keyring is fetched from the same mirror as the signature, so the pinned +fingerprint is the only real check; `GOODSIG` is required too, because gpgv reports an +expired or revoked key as `VALIDSIG` with exit 0. + +```mermaid +flowchart TB + subgraph p["prepare: once a run, when the delta touches TL"] + f1["fetch tlpkg/texlive.tlpdb.xz, .sha512, .sha512.asc, gpg/pubring.gpg"] --> f2["shasum -a 512 -c"] --> f3["gpgv: GOODSIG, and VALIDSIG ending in TL_KEY"] + end + subgraph v["verify: every batch, before publish"] + v1["every *.sha512 in the batch: shasum, then gpgv against the same keyring"] + v2["every archive/ container: its checksum in the verified tlpdb, under both of its names"] + v3["the decision batch: the tlpdb equals prepare's, the .xz decompresses to it,
every container it names is in the bucket after this run"] + end + p --> v +``` + +A batch that fails any check stays local; the previous good copy stays live. `smoke` +reads `texlive.tlpdb.sha512` back through the domain afterwards and compares it with the +verified copy. `tlmgr` repeats the signature check on the client. + +#### The vars A mirror sets `SOURCE`, `BUCKET` and `HOST` in its root vars, always. Everything else -has an inline default in the engine, and a mirror sets only what differs: +has an inline default, and a mirror sets only what differs: | Var | Default | Meaning | |---|---|---| @@ -196,198 +339,351 @@ has an inline default in the engine, and a mirror sets only what differs: | `TL`, `TL_KEY` | empty | A signed TeX Live subtree and the fingerprint that signs it; empty, no signature checks | | `FILTER` | empty | rsync filter arguments that narrow the listing, for a mirror of a subtree | | `OWN` | empty | Bucket-root keys the mirror owns, space separated; `reconcile` never deletes them | -| `INDEX` | empty | The key suffix of the directory pages a mirror's `index` draws; set, `reconcile` spares those pages and every bare directory of the state, their second key | +| `INDEX` | empty | The key suffix of the directory pages a mirror's `index` draws; set, `reconcile` spares those pages and every bare directory of the state | A var the mirror puts in its root `vars:` is fixed for every run: inside an included verb a root value shadows a `KEY=value` from the command line. One the mirror leaves to its default is the run's to set, `task sync -- MAX_BATCHES=8 RECONCILE=true`. So a root var that restates a default is worse than none: it silences the command line. -A mirror of one signed subtree, shaped like tlnet: - -```yaml -version: '3' -vars: - SOURCE: rsync://rsync.dante.ctan.org/CTAN/ - BUCKET: tlnet - HOST: tlnet.katoptra.org - TL: systems/texlive/tlnet - TL_KEY: C78B82D8C79512F79CC0D7C80D5E5D9106BAB6BC - CEILING_GB: 10 - OWN: index.html - FILTER: >- - --exclude=*.r[0-9]*.tar.xz --exclude=/systems/texlive/tlnet/update-tlmgr-r* - --include=/systems/ --include=/systems/texlive/ --include=/systems/texlive/tlnet/*** - --exclude=* -env: - AWS_CONFIG_FILE: '{{.ROOT_DIR}}/aws.config' -includes: - toolbox: - taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml - flatten: true - excludes: [report-engine, report-mirror] - vars: - NAME: tlnet - DESC: a daily mirror of TeX Live's tlnet at https://tlnet.katoptra.org/ - IMAGE: ghcr.io/katoptra/toolbox:rsync-v2 - PASS: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ENDPOINT_URL - rsync: - taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/rsync.yml - flatten: true - excludes: [index] -tasks: - index: - desc: The landing page, dated today, at the bucket root - cmds: - - sed "s/DATE/$(date -u +%Y-%m-%d)/" site/index.html | aws s3 cp --content-type text/html --cache-control no-cache - s3://{{.BUCKET}}/index.html - report-mirror: - internal: true - cmds: - - 'echo "| Landing page | index.html, dated $(date -u +%Y-%m-%d) |" >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}"' -``` - -`SOURCE` is the archive root and `FILTER` narrows the listing to the subtree, so every -key keeps its `systems/texlive/tlnet/` prefix. `OWN` keeps `reconcile` off the landing -page, `index` is the one hook the mirror fills, and the pipeline is the engine's. - -### The bucket is the mirror; the state is a cache - -An rsync engine keeps a state file in the bucket, under `.state/`, listing every key the -bucket holds at upstream's size and mtime. Each run diffs upstream's listing against it, -so an hourly run costs one listing of upstream and none of the bucket. Two things follow. - -- **There is no seed.** A missing state file is rebuilt from a listing of the bucket. An - empty bucket gives an empty state, so the first run's delta is the whole tree, worked - `MAX_BATCHES` at a time; when batches remain the run chains the next one, and the fill - converges in a few chained runs with no flag. A lost state file costs the same listing - and nothing else. -- **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket - on purpose, daily by default, and deletes keys neither upstream nor the state owns. - -What a mirror cannot afford to lose is the bucket. Everything else, the state file and the -staging tree included, is derived from it and from upstream. - ### The proton engine -`engines/proton.yml` moves a staging tree into one Proton Drive folder, through the -official `proton-drive` CLI, for a mirror whose upstream fits in a run: the mirror fills +`engines/proton.yml`: a staging tree into one Proton Drive folder, through the official +`proton-drive` CLI, for a mirror whose upstream fits in a run. The mirror fills `staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket but the CLI session, because the CLI skips a file whose content Proton already holds and -`-f create-new-revision` makes a revision of one that changed; Proton's version history +`-f create-new-revision` makes a revision of one that changed: Proton's version history is the history of the mirror. +```mermaid +flowchart LR + clock --> session --> destination --> stage --> upload --> confirm --> prune --> report --> ping + report --> re["report-engine"] --> rm["report-mirror"] + classDef hook stroke-dasharray: 5 5 + class stage,prune,re,rm hook +``` + | Verb | Does | |---|---| -| `pipeline`, `plan-pipeline` | `clock`, `session`, `destination`, `stage`, `upload`, `confirm`, `prune`, `report`, `ping`; and the read-only half, through `stage` | +| `pipeline`, `plan-pipeline` | The verbs above in order; the read-only half runs through `stage` and prints what an upload would carry | | `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` | -| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit | | `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` | -| `stage` | Hook. The mirror fills `staging/` with what Proton should hold | +| `stage` | Hook. Every mirror defines it: fill `staging/` with what Proton should hold | | `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` | | `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` | | `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` | +| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit | | `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash | | `session-seal -- ` | Host side: a laptop login's two files, encrypted into the bucket | | `empty-trash` | Host side, asks first: everything in Proton's trash, permanently | | `report-engine` | Hook. The engine's rows of the run summary | The engine reads no root var. Its inputs are the environment, by the names every Proton -mirror's `op.env` carries: `MIRROR_PROTON_DESTINATION` and its `_UID`, `MIRROR_R2_BUCKET`, -`MIRROR_AGE_IDENTITY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and -`AWS_ENDPOINT_URL_S3`. The session is the one thing to set up by hand: the CLI can only be -seeded by a browser sign-in, so log in once on a laptop into a directory inside the repo -and run `task session-seal -- `. Two mirrors never share a session; its -refresh token rotates on every call, and the loser of a race needs a fresh login. +mirror's `op.env` carries: -A mirror of every repository under two GitHub owners, shaped like github: +| Name | What it is | +|---|---| +| `MIRROR_PROTON_DESTINATION` | The CLI path of the destination folder, `/my-files/GitHub` say | +| `MIRROR_PROTON_DESTINATION_UID` | Its UID, compared on every run before any write | +| `MIRROR_R2_BUCKET` | The bucket that holds the session | +| `MIRROR_AGE_IDENTITY` | The `AGE-SECRET-KEY-...` line the session is encrypted to | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3` | What boto3 reads; note `_S3` on the endpoint | -```yaml -version: '3' -vars: - OWNERS: jshvn katoptra -includes: - toolbox: - taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml - flatten: true - excludes: [report-engine, report-mirror] - vars: {NAME: github, DESC: a nightly mirror of every repository under jshvn and katoptra into Proton Drive, IMAGE: ghcr.io/katoptra/toolbox:proton-v2} - proton: - taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml - flatten: true - excludes: [stage, prune] -tasks: - stage: {cmds: ['# list the repositories, clone each as a mirror, bundle it under {{.STAGING}}//']} - prune: {cmds: ['# list-folder each owner in Proton; trash the bundles no repository has']} - report-mirror: {cmds: ['# the mirror rows']} +#### The session + +The CLI can only be seeded by a browser sign-in, and fresh sign-ins from datacenter +addresses are blocked, so the session is made once on a laptop and carried to every run +encrypted. Its refresh token rotates on use, so every run writes it back. + +```mermaid +flowchart LR + subgraph laptop["Once, on a laptop"] + l1["proton-drive auth login
with the session as plain files under .run/pd"] --> l2["task session-seal -- .run/pd
tar, age-encrypt, push"] + end + l2 --> key[".state/session.tar.age
in the bucket"] + subgraph run["Every run"] + r1["session: pull, decrypt, extract to .run/session"] --> r2["pd: one CLI call"] --> r3{"token rotated?"} + r3 -- yes --> r4["session-push: seal it back"] + r3 -- no --> r2 + end + key --> r1 + r4 --> key ``` -## Overriding a verb +To make one: -List it under `excludes:` on the include that defines it and define it in the mirror. -A duplicate without `excludes` is a parse error, on purpose. +```sh +export PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file +proton-drive auth login # a browser opens; .run/pd holds two JSON files after +proton-drive filesystem create-folder /my-files # /my-files is the drive's own root, not a folder you make +proton-drive filesystem list -j /my-files # the entry named : its uid is the destination UID +task session-seal -- .run/pd # once the bucket and the vault exist +``` -```yaml -includes: - rsync: - taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/rsync.yml - flatten: true - excludes: [index] # the mirror draws its own landing page -tasks: - index: - cmds: [...] +Install the CLI at the version `toolchain.lock.toml` pins: the session file format is +tied to the version, and the Linux binary in the image must read what the laptop wrote. +`.run/pd` must be inside the repository, because only the repository is mounted into the +image; `.run/` is ignored by git and deleted by `task clean`. + +Three rules. **Two mirrors never share a session**: its refresh token rotates on every +call, and the loser of a race needs a fresh login. **Never run `sync` or `plan` from a +laptop while an Actions run may be in progress**, for the same reason. **The session gets +no history in the bucket**: a stale copy holds a rotated-out token and cannot be restored. + +### A pipeline of your own + +A mirror whose logic is a program of its own includes the toolbox alone and defines +`pipeline` and `plan-pipeline` from the toolbox's `clock`, `report` and `ping` and its own +steps. dropbox is that shape: each step is one `python -m migrator `, the +Taskfile owns the order, the Python owns every decision, and the `proton` image supplies +the interpreter, boto3 and the CLI. Such a mirror fills `report-mirror` with its own +report and nothing else of the toolbox's changes. + +## Secrets + +Every credential and every account identifier lives in one vault and reaches a run by +name. A repository holds `op://` references only. + +```mermaid +flowchart LR + item["A vault item, one per mirror
sections per service: r2, healthcheck, proton, age, ..."] --> ref["op.env
NAME=op://vault-uuid/item/section/field"] + ref --> oprun["op run --env-file=op.env, on the host
values exported into its child's environment only"] + sa["A service account that reads the vault
its token: the secret OP_SERVICE_ACCOUNT_TOKEN"] --> oprun + oprun --> run["run: -e NAME for every name in op.env and PASS
values never on a command line, never in a log"] + run --> tool["the tool inside the image reads its environment
aws, boto3, curl, git"] ``` -The rules that keep this honest: +### The vault -- Keep the name and the meaning. A `verify` that does something other than verify is a - new verb with its own name, listed in `MENU`. -- Extend through a hook, never a copy. A mirror that needs the engine's `smoke` and - more defines `smoke-mirror`, which `smoke` runs last, rather than excluding `smoke` - and pasting it: the copy stops following releases the day it is made. A mirror with a - step to add beside `clock` adds a step to its pipeline, not a `clock` of its own. -- The replacement reads the mirror's root vars, the same as the original did. -- Override engine verbs and the in-container toolbox verbs. Do not override the host - side: `sync`, `plan`, `check` and `run` are what the workflows and the menu promise. -- Everything the mirror does not exclude comes from here, and a release changes it. +One vault. One item per mirror, named for its repository. One section per service in +the item (`r2`, `healthcheck`, `proton`, `age`, `github`, `dropbox`), with the field names +the mirror's README lists. Neither the vault's name nor its UUID is a secret: without the +service account's token it opens nothing. -## Plugging in an engine +The references address the vault by UUID, `op:////
/`, so a +renamed vault cannot break a run and a name with a slash in it cannot be written: -One engine per transport, one image variant per engine. Adding one is five files and -two matrix entries: +```sh +op vault get --format json | jq -r .id +``` -1. **Tools**: add each tool's version and per-arch checksum to `toolchain.lock.toml`. -2. **Image**: `docker/.Dockerfile`, base pinned by digest, every tool installed - from the lock, `TASK_REMOTE_OFFLINE=1`, and a final stage that asserts each tool - reports the locked version. `docker/rsync.Dockerfile` is the model. -3. **Verbs**: `engines/.yml` with the pipeline vocabulary above. No `vars:` - default for anything a mirror owns; defaults go inline as `{{.X | default N}}`. -4. **Example**: `examples//` with a Taskfile whose `pipeline` is every verb, a - committed `render.txt`, and an `offline` verb that runs the ones that need no bucket - over `fixtures/`. `examples/rsync/` is the model. This is the engine's own check. -5. **CI**: add the variant to the `matrix` in `ci.yml` and `release.yml`. +One service account that can read that vault and nothing else. Its token is the secret +`OP_SERVICE_ACCOUNT_TOKEN`: on an organization, one organization secret that every +repository inherits; on a personal account, one repository secret per mirror. Only a +dispatched sync run holds it; a pull-request check gets no secret at all. + +### On the host + +`op run --env-file=op.env` resolves every reference and exports the values into the +environment of the one command it wraps, `task run -- ...`, masking each value in that +command's output. On a laptop `op` uses the signed-in desktop app; in Actions it uses the +service account token the reusable workflow put in its environment. The vault is read +once per run. + +### Into the container + +`run` passes `-e NAME` for each name in `op.env` and in the mirror's `PASS`, plus +`HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always. The container engine +copies each named variable from the host's environment; no value is on the command line. +A host variable the pipeline reads and `op.env` does not name arrives empty. + +### The names + +| Name | Read by | Set by | +|---|---|---| +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | the AWS CLI and boto3 | the `r2` section | +| `AWS_ENDPOINT_URL` | the AWS CLI, in the `rsync` image | the `r2` section's `endpoint` | +| `AWS_ENDPOINT_URL_S3` | boto3, in the `proton` image | the same field, under boto3's name | +| `AWS_REGION` | both | a literal `auto` in the image; R2 has one region, and masking the word `auto` would corrupt output | +| `MIRROR_R2_BUCKET` | the proton engine's `s3` | the `r2` section's `bucket` | +| `MIRROR_AGE_IDENTITY` | the proton engine's `age` | the `age` section | +| `MIRROR_PROTON_DESTINATION`, `_UID` | the proton engine's `destination` | the `proton` section | +| `HEALTHCHECK_URL` | `ping`, `ping-fail` | the `healthcheck` section; optional | +| `MIRROR_*` of a mirror's own | that mirror | a section of its own: `github`, `dropbox` | + +### Without a vault + +A mirror with no `op.env` runs on its repository secrets: the reusable workflow exports +every inherited secret into the sync step's environment by name, the ones the mirror +lists in `PASS` cross into the container, and the 1Password CLI is not installed at all. +On a laptop, whatever is exported. This is the fallback, not the pattern: it works, and +it puts the values in a second place. + +## Storage + +Every mirror has one S3-compatible bucket. What it holds depends on the engine. + +| Engine | Key | What | +|---|---|---| +| rsync | every upstream path, at the root | The mirror. A public domain serves the bucket | +| rsync | `.state/applied.txt.xz` | The state: what the bucket holds, at upstream's size and mtime | +| rsync, a mirror's own | `.state/indexed.txt.xz`, `index.html` | ctan's record of what its directory pages show; tlnet's landing page, spared by `OWN` | +| proton | `.state/session.tar.age` | The CLI session, encrypted. The only key | +| a pipeline of its own | `.state/state.sqlite.xz.age`, `.state/history/-