diff --git a/CLAUDE.md b/CLAUDE.md index b1bcc32..dddfcd7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,7 @@ # dispatch Starts the katoptra mirrors' workflows on UTC slots, from a systemd timer on a NixOS host. -`README.md` is for users; this file is the design. The spec and the reasoning behind each -choice: `docs/superpowers/specs/2026-09-21-katoptra-dispatch-design.md`. +`README.md` is for users; this file is the design. ## The files diff --git a/README.md b/README.md index 4cd747b..bc3cbed 100644 --- a/README.md +++ b/README.md @@ -180,8 +180,7 @@ sudo STATE_DIRECTORY=/var/lib/private/katoptra-dispatch katoptra-dispatch --dry- ## Reference [`CLAUDE.md`](CLAUDE.md) is the design: the files, the constraints, and what breaks if a -choice is undone. The spec behind it, with the options weighed and rejected, is under -[`docs/superpowers/specs/`](docs/superpowers/specs). +choice is undone. Pull requests are welcome. diff --git a/docs/superpowers/plans/2026-09-21-katoptra-dispatch.md b/docs/superpowers/plans/2026-09-21-katoptra-dispatch.md deleted file mode 100644 index bdab0b5..0000000 --- a/docs/superpowers/plans/2026-09-21-katoptra-dispatch.md +++ /dev/null @@ -1,119 +0,0 @@ -# katoptra/dispatch Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** A Go binary plus NixOS module that dispatches the katoptra mirrors' workflows on -UTC slots from a 5-minute systemd timer, at most once per slot, with catch-up. - -**Architecture:** `schedules/` holds slot types and the job list. The root `main` package is -a oneshot: lock, read state, plan what is due, write state ahead, dispatch through GitHub, -ping healthchecks. `module.nix` wraps it in a timer and a hardened service; `flake.nix` -exports both. - -**Tech Stack:** Go 1.26 standard library, Nix flake (nixos-26.05), go-task, Apple -`container`/Docker with `golang:1.26-alpine`. - -**Spec:** `docs/superpowers/specs/2026-09-21-katoptra-dispatch-design.md` - -## Global Constraints - -- Standard library only; `vendorHash = null`. -- Go 1.26: nixos-26.05's `buildGoModule` is `buildGo126Module`; the image is - `golang:1.26-alpine@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c`. -- Slots `HH:42` UTC. `Evening = S5`, `Overnight = S11`, `Morning = S17`, `Afternoon = S23`. -- Timer `OnCalendar=*:02/5 UTC`. Service oneshot, `DynamicUser`, `StateDirectory`, - `LoadCredential` files `app-id`, `private-key`, `healthcheck-url`, `TimeoutStartSec=4min`. -- Dispatch body `{"ref":"main","inputs":{}}`; any 2xx is success. -- Retries: 5xx, 408, 429, network errors; three retries after 10 s, 20 s, 40 s. -- Ping body truncated to 10 KB; three attempts. -- No AI attribution; commit format `(): `. - -## File map - -| file | responsibility | -|---|---| -| `go.mod` | module `github.com/katoptra/dispatch`, `go 1.26` | -| `schedules/schedules.go` | `Slot` bitmask, names, `Latest`, `Job`, `Jobs()` | -| `schedules/{ctan,tlnet,dropbox,github}.go` | one repo each | -| `schedules/schedules_test.go` | slot math table; registry sanity | -| `tick.go` | `State`, load/save, `Plan`, `Tick` | -| `tick_test.go` | catch-up, no repeat, write-ahead, corrupt state | -| `github.go` | key parse, JWT, installation token, dispatch, retry | -| `github_test.go` | httptest: 204, 500 retried, 404 fatal, JWT verifies | -| `main.go` | flags, credentials, logging, ping, exit code | -| `module.nix`, `flake.nix`, `flake.lock` | NixOS module and flake | -| `Dockerfile`, `Taskfile.yml`, `.github/workflows/check.yml` | tooling, CI | -| `README.md`, `CLAUDE.md`, `.gitignore` | docs | - -### Task 1: Slots and schedules - -**Interfaces — Produces:** -- `type Slot uint32`; `S0`..`S23`; `Evening, Overnight, Morning, Afternoon, Hourly Slot` -- `func (s Slot) Latest(now time.Time) (time.Time, bool)` — latest `HH:42` UTC at or before - `now` among the set bits; false when `s == 0` -- `type Job struct{ Repo, File string; Slots Slot }`; `func (j Job) ID() string` = `Repo+"/"+File` -- `func Jobs() []Job` - -- [ ] Test: table for `Latest` — `Hourly` at 12:41:59 -> 11:42; at 12:42:00 -> 12:42; - `Afternoon` at 00:10 -> yesterday 23:42; `Evening` at 05:41 -> yesterday 05:42; - `Morning|Evening` at 18:00 -> 17:42; `0` -> false. Registry: all under `katoptra/`, - non-zero slots, unique IDs. -- [ ] Run `go test ./schedules/` — fails (undefined). -- [ ] Implement; run — passes. Commit `feat(schedules): add utc slots and the katoptra jobs`. - -### Task 2: State and the tick - -**Interfaces — Consumes:** `schedules.Job`, `Slot.Latest`. -**Produces:** -- `type State map[string]time.Time` -- `func LoadState(dir string) (State, error)` — missing file is empty state; unreadable or - corrupt is an error -- `func SaveState(dir string, s State) error` — temp, fsync, rename, fsync dir -- `type Fire struct{ Job schedules.Job; Slot time.Time }` -- `func Plan(jobs []schedules.Job, s State, now time.Time) (fires []Fire, warnings []string)` -- `type Dispatcher interface{ Dispatch(ctx context.Context, repo, file string) error }` -- `func Tick(ctx context.Context, dir string, jobs []schedules.Job, now time.Time, d Dispatcher, dryRun bool, log *Log) error` -- `type Log struct` with `Info`, `Warn`, `Error(format, ...)` and `Errors() []string` - -- [ ] Tests: ten missed hourly slots fire once with the latest slot; second tick in the same - slot fires nothing; dispatcher reading `state.json` during `Dispatch` sees its slot - (write-ahead); dispatch failure still records the slot and `Tick` returns an error; - corrupt `state.json` dispatches nothing and errors; clock behind state warns, no fire; - removed job's entry is dropped; dry run writes nothing and dispatches nothing. -- [ ] Run — fails. Implement with `syscall.Flock` on the directory. Run — passes. - Commit `feat(tick): plan due slots and record them before dispatch`. - -### Task 3: GitHub - -**Produces:** -- `func ParseKey(pem []byte) (*rsa.PrivateKey, error)` — PKCS#1, then PKCS#8 -- `func AppJWT(appID string, key *rsa.PrivateKey, now time.Time) (string, error)` -- `type GitHub struct{ API, AppID, Org string; Key *rsa.PrivateKey; HTTP *http.Client; Sleep func(time.Duration) }` implementing `Dispatcher`; mints one token lazily per process -- `type StatusError struct{ Status int; Msg string }`; `func fatal(err error) bool` - -- [ ] Tests (httptest): full happy path hits installation, token, dispatch with the body - above; a 500 then 204 dispatch succeeds after one retry (Sleep recorded 10 s); 404 fails - with no retry; JWT signature verifies with `rsa.VerifyPKCS1v15`; PKCS#8 key parses. -- [ ] Run — fails. Implement. Run — passes. Commit `feat(github): mint app tokens and dispatch workflows`. - -### Task 4: main, ping - -- [ ] `main.go`: `--dry-run`; reads `$STATE_DIRECTORY` and, unless dry run, - `$CREDENTIALS_DIRECTORY/{app-id,private-key,healthcheck-url}`; runs `Tick`; pings - `` or `/fail` with `Log.Errors()` joined (10 KB cap, three attempts); exit 1 on - any error. Journal priority prefixes `<3>`/`<4>` only when `JOURNAL_STREAM` is set. -- [ ] Test: `ping` against httptest — clean posts to `/`, failure posts to `/fail` with body - capped at 10 KB. Commit `feat: wire the tick, credentials and healthcheck ping`. - -### Task 5: Nix - -- [ ] `flake.nix`: packages (buildGoModule, `preCheck = "go vet ./..."`), `nixosModules.default`, - `checks` building the package and the two generated unit files from a minimal - `nixosSystem`. `module.nix` per the spec. Generate `flake.lock` and run `nix flake check` - inside `nixos/nix`. Commit `feat(nix): add the flake and the nixos module`. - -### Task 6: Tooling and docs - -- [ ] `Dockerfile` (golang 1.26 alpine, digest), `Taskfile.yml` (banner; `check`, `targets`, - `runs`, `run`, `image`, `clean`), `check.yml`, `README.md`, `CLAUDE.md`, `.gitignore`. - `task check` green. Commit `chore: add toolbox image, taskfile, ci and docs`. diff --git a/docs/superpowers/specs/2026-09-21-katoptra-dispatch-design.md b/docs/superpowers/specs/2026-09-21-katoptra-dispatch-design.md deleted file mode 100644 index e0de44b..0000000 --- a/docs/superpowers/specs/2026-09-21-katoptra-dispatch-design.md +++ /dev/null @@ -1,211 +0,0 @@ -# katoptra/dispatch design - -2026-09-21. Starts the katoptra mirrors' GitHub Actions workflows on a schedule, from a -systemd timer on one NixOS host. - -## Purpose and scope - -- Dispatches `workflow_dispatch` to katoptra repositories on fixed daily slots. It does not - run work, poll run outcomes, or monitor the mirrors; each mirror pings its own healthcheck. -- katoptra repositories only. `jshvn/terraform`'s `drift.yml` stays on `jshvn/dispatch`. -- Code lives here. The host (`bendalloy.jgrid.net`) is configured in `jshvn/jgrid.net`. - -## Requirements - -1. systemd is the clock: a timer starts a oneshot, there is no long-running process. -2. 24 slots a day, `S0`..`S23`, at `HH:42` UTC. No daylight-saving handling of any kind. -3. Named slots are aliases of hourly slots: `Evening = S5`, `Overnight = S11`, - `Morning = S17`, `Afternoon = S23` (Pacific 21, 03, 09, 15 in winter, an hour later in - summer). `Hourly` is all 24. -4. At-most-once per slot per job, and catch-up after any outage: however many slots a job - missed, it fires once, for its latest one. No lookback limit. -5. Errors go to the journal. One healthcheck ping per tick reports whether the tick was clean. - -## Repositories - -### katoptra/dispatch (this repo) - -- Go module, standard library only. -- `flake.nix` outputs: - - `packages.x86_64-linux.default`: `buildGoModule`, `vendorHash = null`; `checkPhase` - runs `go vet` and `go test`, so a failing test fails the host's build. - - `nixosModules.default`: the service below. - - `checks`: the package, and the module evaluated in a minimal `nixosSystem`. -- `schedules/`: one Go file per repo, compiled into the binary. A schedule change is a - commit here, then a `flake.lock` bump in `jshvn/jgrid.net`, then a deploy (the host's - nightly pull-and-switch, or by hand). - -### jshvn/jgrid.net (the host) - -- Flake input `katoptra-dispatch`. -- `nix/fleet.nix`: `bendalloy = { roles = [ "dispatcher" ]; tunnels = [ "jgrid-net" ]; }`. - No `lanIp`, no `publicTcpPorts`: the firewall is closed to the outside, as on iron and - pewter; ssh arrives only through the tunnel. Portainer stays on, as on every host. -- `nix/roles/dispatcher.nix`: imports the module, enables it, adds three - `jgrid.secretTemplates` entries from the jgrid.net vault item `bendalloy.jgrid.net` - (`GITHUB_APP/id`, `GITHUB_APP/private_key`, `HEALTHCHECK/url`; field names settled when the - item is made), and orders the service `after`/`wants` `jgrid-secrets.service` with a - `ConditionPathExists` per secret file. -- Disk is 10 GB. Set `nix.gc` and a small `boot.loader.*.configurationLimit` on bendalloy. - The host builds this package itself (not in cache.nixos.org), fetching the Go toolchain - (~250 MB) until the next GC. - -## The NixOS module - -`services.katoptra-dispatch`: - -| option | type | meaning | -|----------------------|------|-------------------------------------------------| -| `enable` | bool | | -| `appIdFile` | path | the GitHub App's id | -| `privateKeyFile` | path | the App's private key, PKCS#1 or PKCS#8 PEM | -| `healthcheckUrlFile` | path | the healthchecks.io ping URL | - -It knows nothing of 1Password or jgrid: it takes file paths. - -Units: - -- `katoptra-dispatch.timer`: `OnCalendar=*:02/5` with `UTC` (fires :02, :07 .. :42 .. :57, - so the slot minute is always a tick). No `Persistent=`: every tick reconciles from state. -- `katoptra-dispatch.service`: `Type=oneshot`, `DynamicUser=yes`, - `StateDirectory=katoptra-dispatch`, `LoadCredential=` for the three files (root-owned 0400 - files stay unreadable to anyone else), `TimeoutStartSec=4min`, and the standard hardening - (`ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `NoNewPrivileges`, empty - `CapabilityBoundingSet`, `RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6`). - -## Schedules - -```go -// schedules/ctan.go -var _ = register(Job{Repo: "katoptra/ctan", File: "sync.yml", Slots: Hourly}) -``` - -`Slot` is a bitmask of UTC hours with named constants, so two slots are `Morning | Evening` -and a misspelt slot fails to compile. A job's identity -is `/`, e.g. `katoptra/ctan/sync.yml`. Every dispatch is `ref: "main"` with empty -inputs; `ref` and `inputs` fields come back when a job needs them. - -Starting set, carried over from `jshvn/dispatch`: - -| job | slot | UTC | -|------------------------------|-----------|-----------| -| `katoptra/ctan/sync.yml` | Hourly | every :42 | -| `katoptra/tlnet/sync.yml` | Evening | 05:42 | -| `katoptra/dropbox/sync.yml` | Overnight | 11:42 | -| `katoptra/github/sync.yml` | Morning | 17:42 | - -A workflow added here must declare `workflow_dispatch`, a `concurrency` group with -`cancel-in-progress: false`, and ping its own healthcheck. Nothing here can check these. - -## A tick - -1. `flock` the state directory. systemd already refuses to start a running oneshot; the lock - covers a hand-run binary. -2. Read `state.json`: job id to the last slot fired, an RFC 3339 UTC timestamp. -3. For each job, `due` is its latest registered slot time at or before now (today's and - yesterday's slots are always enough). -4. A job fires when it has no entry, or when `due` is after its entry. An entry after `due` - (clock went backwards) is a journal warning and no fire. -5. Write-ahead: record `due` for every job about to fire in one atomic write (temp file, - fsync, rename, fsync the directory). Entries for jobs no longer in `schedules/` are - dropped in the same write. -6. Dispatch each (below). Jobs are independent; one failing does not stop the rest. -7. Ping (below). Exit non-zero if anything failed. - -Consequences: - -- After any outage each job fires once, for its latest missed slot. -- A crash between the write and the POST loses that slot; it never repeats one. -- A new job fires once at the next tick. -- A tick with nothing due makes no GitHub calls; it only pings. - -## Dispatching - -Only when something is due: - -1. Sign an App JWT (RS256, `iat` backdated 60 s, `exp` 9 min). The key parses as PKCS#1, - falling back to PKCS#8. -2. `GET /orgs/katoptra/installation`, then `POST /app/installations//access_tokens`. - The token lives in memory for the tick and is never written anywhere. -3. `POST /repos//actions/workflows//dispatches` with - `{"ref":"main","inputs":{}}`. Any 2xx is success. - -Retries: 5xx, 408, 429 and network errors get three retries, after 10 s, 20 s, 40 s. Any other -4xx is fatal at once (missing workflow file, App not installed, missing permission) and the -error names the job. - -## Errors, journal, ping - -- One journal line per fire (`fired katoptra/ctan/sync.yml for slot 2026-09-21T17:42Z`) and - per skip warning. Errors on stderr carry the sd-daemon `<3>` prefix, so - `journalctl -p err -u katoptra-dispatch` finds them. -- The unit fails on any dispatch failure or state read/write failure. The next tick runs - anyway: a timer starts a failed oneshot again. -- Ping, always last: a clean tick pings ``; any error pings `/fail` with the error - lines as the body, truncated to 10 KB. Three quick attempts; a ping that never gets through - is logged and does not change the exit code. -- Silence covers the rest: a dead host, a stopped timer, a hung tick. The healthchecks.io - check is period 5 min, grace 10 min. -- A corrupt or unreadable `state.json` fails the tick loudly and fires nothing. It never - falls back to empty state, which would repeat slots already fired. The fix is by hand; - deleting the file means every job fires once, knowingly. - -## The binary - -`katoptra-dispatch` reads its secrets from `$CREDENTIALS_DIRECTORY` (files `app-id`, -`private-key`, `healthcheck-url`) and its state from `$STATE_DIRECTORY`. Two flags: -`--dry-run` prints what is due and why, writes nothing, POSTs nothing, pings nothing; -`--list` prints every job and its UTC slot times, for `task targets` and `task runs`. - -## Testing - -`go test`, run in the Nix `checkPhase` and in CI: - -- Slot math, table-driven: :41:59 against :42:00, midnight crossing, yesterday's S23 seen at - 00:10, Hourly against a single named slot. -- The tick, with an injected clock, a fake dispatcher and a temp state directory: ten missed - slots fire once; the same slot never fires twice; a dispatcher failing mid-POST leaves the - slot recorded; corrupt state fires nothing and exits non-zero. -- GitHub, against `httptest`: 204 succeeds; 500 is retried; 404 is fatal without retry; the - JWT verifies against the key's public half. -- Schedules: every job is under `katoptra/`, has at least one slot, and ids are unique. -- The module: `nix flake check` evaluates it. `ponytail:` no `nixosTest` VM; add one if a - unit bug gets past evaluation. - -## Repo tooling - -- Taskfile with a banner menu: `task check` (vet, gofmt, test), `task targets` (schedules and - their UTC times), `task runs` (recent `gh run list` per target). -- Tooling runs in a digest-pinned official `golang` image. -- `.github/workflows/check.yml` runs `task check` on pull requests. -- `README.md` for users, `CLAUDE.md` for the design. - -## Cutover - -Catch-up makes the order safe; there is no double-firing window. - -1. Create a GitHub App owned by the `katoptra` org: Actions read and write, no webhook, - installed on all repositories. Put its id, its key and a new healthchecks.io check's URL - (period 5 min, grace 10 min) in the jgrid.net vault, item `bendalloy.jgrid.net`. -2. Provision bendalloy: tunnel in `jshvn/terraform`, fleet entry, `nix/roles/dispatcher.nix`, - flake input, install. -3. Remove the four katoptra schedules from `jshvn/dispatch`; terraform stays. -4. Enable the service. Its first tick fires each job once, covering any slot missed between - steps 3 and 4. -5. Watch each mirror's healthcheck and `task runs` through a day of slots. - -## Decisions - -- **Go over TypeScript on Node.** The host has 10 GB of disk; Node adds ~100-200 MB per - generation kept, Go a few MB, and Go's compile is the type check inside the Nix build. -- **UTC, not Pacific.** No DST edge cases. Named slots drift an hour against Pacific in - summer, as they do in `jshvn/dispatch` today. -- **One 5-minute timer, not a timer per slot.** A timer per slot with `Persistent=true` - would fire each missed slot on recovery, not once per job. -- **Write-ahead state.** At-most-once over at-least-once: a lost slot is caught up by the - next one; a duplicate would be a second run. -- **A katoptra-owned App, secrets in the jgrid.net vault.** Retiring either scheduler never - touches the other's credentials, and the host's service account token gains no access to - the mirrors' secrets. -- **`/fail` on a failed dispatch.** A typo in a schedule alerts within one tick instead of - when a daily mirror's grace runs out.