Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
860ed7d
loops: minimal-path gate at plan time (R127 slice 1)
iselur Aug 9, 2026
c7650bb
loops: make the minimal-path gate actually gate (diff review round 1)
iselur Aug 9, 2026
0b6dc45
loops: drop the backlog hunk from this diff (review round 2)
iselur Aug 9, 2026
a4cc77c
Merge pull request #312 from iselur/loops/self-correction
iselur Aug 9, 2026
330b7ec
review: relax self-review refusal to instance level (rule 7; R128)
iselur Aug 9, 2026
a594e64
review: docs stop overstating the guarantee (r128-diff review round 1)
iselur Aug 9, 2026
8ddaf95
chore: R102 Claude slots move to claude-opus-5 (owner 2026-08-10)
iselur Aug 10, 2026
7f3774c
test: follow the row config to claude-opus-5
iselur Aug 10, 2026
4a625f2
Merge pull request #313 from iselur/loops/self-correction
iselur Aug 11, 2026
8b04d00
Merge branch 'ready-for-main' into chore/r102-claude-opus-5
iselur Aug 11, 2026
5edcfcd
Merge pull request #314 from iselur/chore/r102-claude-opus-5
iselur Aug 11, 2026
950a5ec
docs: rewrite the README in plain language
iselur Aug 11, 2026
4a86e5d
Merge pull request #315 from iselur/docs/readme-human
iselur Aug 11, 2026
701b405
docs: README describes the machinery, not just the shape
iselur Aug 11, 2026
e888a8a
Merge pull request #317 from iselur/docs/readme-human
iselur Aug 11, 2026
b103630
docs: README stops overclaiming three guarantees
iselur Aug 11, 2026
0c99a55
Merge pull request #318 from iselur/docs/readme-honesty
iselur Aug 11, 2026
88e108a
docs: README gets the review rules right (promotion review round 4)
iselur Aug 11, 2026
32e3207
docs: the isolation claim matches what the test asserts
iselur Aug 11, 2026
2abc665
Merge pull request #319 from iselur/docs/readme-honesty-2
iselur Aug 11, 2026
7e7d1d3
docs: answer promotion review round 5 (blocking + nonblocking)
iselur Aug 11, 2026
c302a32
Merge pull request #320 from iselur/docs/readme-round5
iselur Aug 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .orchestrator/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,5 +64,5 @@ knows them, accepts them, and does not want them built (2026-08-06). Do not re-a
`R102_BENCHMARK=1`. PLAN-014, PLAN-015 and SPEC-054 are retired unmerged.
- Test the rendered reviewer EVIDENCE section (incl. spec-declared commands) directly instead of
relying partly on a source-marker grep (harness-spec-command-evidence round 1, PASS backlog note).
- Relax scripts/review's model-level self-review refusal to instance/context level to match
CLAUDE.md rule 7 (owner, 2026-08-06) — stricter-than-rule today, safe to keep until needed.
- Reviewer-pairing comparison (Sol-reviews-Sol vs cross-model) as its own experiment — owner interest
2026-08-09; not representable in R102's worker-row matrix; first datapoint free in plan-001 rounds.
1 change: 1 addition & 0 deletions .orchestrator/REVIEW-FRAMING.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,4 @@ the simpler version; a vague "could be simpler" goes to the backlog. Every block
must name the affected code, a credible consequence, and the needed change.
Backlog: stylistic preferences, optional refactors, speculative peer behavior, defense
against implausible conditions, low-impact edge cases.
For plan reviews: the Minimal existing path must name a command and paste its argv, installed commit, input digests, exit status and complete output — any one of those missing is a FAIL.
10 changes: 5 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ An orchestrator that dispatches worker jobs from schema-validated specs, checks

## Codex on this box

- Model split (`scripts/models.json`): worker BUILD `gpt-5.6-luna`; plans `gpt-5.6-sol`; artifact reviews `gpt-5.6-luna` — never the plan author's model, or every plan is refused.
- Model split (`scripts/models.json`): worker BUILD `gpt-5.6-luna`; plans `gpt-5.6-sol`; artifact reviews `gpt-5.6-sol` — same model as the plan author is allowed: a review is always a fresh instance (rule 7; owner 2026-08-09).
- Invocation: `codex exec -m <model per split above> -c model_reasoning_effort=high
--sandbox read-only --skip-git-repo-check - <prompt.txt` — prompt on stdin always (argv dies
over 130KB). Web search: `-c tools.web_search=true`. Standard tier: never set `service_tier` (owner cost decision 2026-07-16).
Expand All @@ -50,10 +50,10 @@ An orchestrator that dispatches worker jobs from schema-validated specs, checks
`bwrap: loopback: Failed RTM_NEWADDR`; proof: `tests/worker_userns.sh`). Inlining context is a
choice now, not a requirement — the bound reviewer still gets spec + diff + evidence only, never
a live checkout. The final answer is recoverable from the `--json` stream (last `agent_message`).
- **Orchestrator artifacts** go through `scripts/review`: it refuses the reviewer's own recorded author
model, and the whole vendor where none is recorded — there `--author` IS that unauthenticated vendor,
elsewhere only a cross-check (SECURITY.md gap 8). Worker diffs go to the bound reviewer in
`scripts/models.json`; both are model-level. Five rounds each; a sixth is refused.
- **Orchestrator artifacts** go through `scripts/review`: a recorded author model may match the
reviewer's (fresh instance, rule 7); the whole vendor is refused where none is recorded — there
`--author` IS that unauthenticated vendor, elsewhere only a cross-check (SECURITY.md gap 8). Worker
diffs go to the bound reviewer in `scripts/models.json` (model-level). Five rounds each; a sixth is refused.
- Plans go through `scripts/codex-plan --brief` (cap 400; refuses a brief missing any required
section); the no-flag standard tier remains usable. Trigger: CLAUDE.md rule 5.

Expand Down
8 changes: 4 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@ Run `./scripts/dispatch reconcile`; resume from state files, never ask the owner
7. **Maximal delegation:** the orchestrator delegates every delegable task to a worker by default; what a worker cannot take for architectural reasons (it needs the
orchestrator's harness) goes to parallel subagents in isolated worktrees, several at once when the pieces are independent; the orchestrator works directly only on
its own tasks (dispatch, review, the trust boundary). Nothing reviews its own context's work — separate instances, even of the same model, may review each other (owner, 2026-08-06); the owner sets worker and reviewer models and vendors in scripts/models.json.
8. **Code discipline:** the simplest, cleanest solution that works, held to a deletion test at brief and
diff review: anything the approved outcome, existing external contracts, and named safety invariants
can be met at least as simply without is omitted — tests, symmetry, or hypothetical future consumers
never establish need. Diffs are surgical: touch no adjacent code, comments, or formatting; match existing style; remove only what your change orphaned.
8. **Code discipline:** the simplest, cleanest solution that works, held to a deletion test before a spec exists — name what the definition of done
needs that installed code cannot do; nothing means no spec — and again at brief and diff review: anything the approved outcome, existing external
contracts, and named safety invariants can be met at least as simply without is omitted — tests, symmetry, or hypothetical future consumers never
establish need. Diffs are surgical: touch no adjacent code, comments, or formatting; match existing style; remove only what your change orphaned.
9. **Failure discipline:** on a failure, search the web with the literal error text — secrets,
tokens, and personal data redacted first — before the first retry; read the whole error and
log it under the same redaction; change one variable, and never queue work behind a blocked
Expand Down
144 changes: 95 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,99 @@
# Relay

Relay is a reusable oversight approach for coding agents. It gives an agent a checked path from
a request to a pull request: a worker produces the change, the harness verifies it, and a
configured separate reviewer checks the exact diff before it moves forward.

The approach can be reused in any repository where coding agents need oversight. This repository
is a ready-to-run reference implementation for Linux, GitHub, and subscription CLIs; those
implementation choices belong to this reference, not to the approach itself.

## The workflow

`request` → `approved spec` → `worker build` → `harness checks` → `bound review` → `pull request`

The harness owns the authoritative commit, evidence, and release decision. Worker prose alone is
not proof: tests and checks must run against the exact candidate. Passing work targets
`ready-for-main`; promotion to `main` remains separately protected.

## What Relay provides

- Structured owner, orchestrator, worker, and reviewer roles.
- Exact-candidate scope, test, and review checks.
- A repeatable path from an approved request to a reviewable pull request.

Relay has been used across more than 500 production pull requests. Its worker/reviewer loop
repeatedly surfaced concrete issues and improvements before merge.

## See the system

[How Relay works](how-it-works.html) gives a visual overview of the flow and roles.

[BOOTSTRAP.md](BOOTSTRAP.md) is the setup path for making this repository yours. It walks through
the toolchain, GitHub, CLI access, worker isolation, and the first job.

[SECURITY.md](SECURITY.md) describes what repository tests prove, what depends on deployment
configuration, and the known gaps. Relay's guarantees are deliberately scoped.

[CLAUDE.md](CLAUDE.md) is the operating rulebook. [AGENTS.md](AGENTS.md) records the role
assignments and repository commands.

## Roles

The owner approves specs and protects the final promotion. The orchestrator coordinates the work
and applies the harness gates. The worker implements the approved spec. The bound reviewer checks
the exact candidate diff.

Worker and reviewer configuration lives in `scripts/models.json`; the orchestrator is whichever
supported CLI runs the process.

Relay keeps the implementation focused on evidence, scope, tests, and review so a repository can
use the same oversight pattern repeatedly.
Relay is a harness for running coding agents against a real repository. It is built around one
assumption: an agent's account of its own work is not evidence. Everything below exists to
replace that account with something checkable.

It runs two loops.

**The outer loop is delivery.** A request becomes a one-line goal and a checkable definition of
done. Anything larger than a single reversible change gets one written brief — what exists at the
end that does not now, what is deliberately not being done, the decisions already made, the
smallest run that would prove the approach wrong, and the slices it ships in. The brief is
cross-reviewed, then the program runs end to end and you step in only at its checkpoints. Work is
dispatched as approved specs, results are reviewed, pull requests open, and passing work is
promoted.

**The inner loop is the build.** A worker agent gets an isolated checkout and implements one
spec. The harness — never the worker — runs the installed tests against the exact candidate
commit. A second agent that did not write the code reads the exact diff and returns a structured
verdict. If that verdict says revise, the work goes back, for a bounded number of rounds, each
answered by exactly one revision.

## The machinery that makes it hold

**Specs bind.** A spec is schema-validated and digest-bound, and high-risk work needs an approval
file the orchestrator is forbidden to write for itself. Editing the spec voids the approval. That
prohibition is a rule with an audit trail, not something the filesystem enforces: software running
in the owner's own context can write one, and `SECURITY.md` says so.

**Workers are isolated, and how much depends on which kind.** Every worker builds in its own git
worktree, never your working tree. A worker driven through an external CLI runs as a separate
operating-system identity, and a test asserts it cannot traverse the owner's home directory or
read the credential files there. That is narrower than "workers have no credentials", which is
not true and is not claimed: one vendor's setup stages a copy of a login inside the worker's own
home. A subagent worker is outside all of this by design — it runs inside the orchestrator's own
session and shares its trust domain. `SECURITY.md` says which guarantee applies where, and what
is still open.

**The grader restores the installed tests.** A worker cannot pass by rewriting the assertion it
failed: grading runs the tests as the repository has them, against the worker's exact commit, and
refuses to grade at all if the tree it is grading has drifted.

**The review that gates a worker's diff is structured and narrow.** It returns JSON checked
against a pinned schema, so a reviewer that emits prose and no verdict cannot pass anything, and
the verdict binds only the exact code it was shown — moved code means a fresh review. Reviews of
plans and of a promotion are prose and carry no such validation, even though the rules do gate on
them: a plan leaves plan mode only after its review is answered, and promotion to `main` requires
a binding PASS on that exact diff.

**Nothing reviews its own work.** A review always runs in a fresh instance, never the one that
produced the work. Where the author's model is on record, that fresh instance may share it — the
separation is between instances, not between models. What the tool refuses outright is the case it
cannot decide: an artefact whose provenance shows only the reviewer's own vendor with no author
model recorded. Whether reviewer and worker are different vendors at all is your configuration
choice in `scripts/models.json`, not a property of the harness.

**Failure has a budget.** Attempts against one spec are capped. A spec that fails structurally
stops rather than looping, and an escalation carries the finding rather than the symptom. Review
rounds are capped in code, because a cap written only in prose already lost once to a ten-round
loop.

**Every attempt leaves replayable evidence.** Its launch and result records and raw evidence stay
on disk; attempts that reach review also retain that binding review. What happened can be checked
afterwards rather than believed.

**Autonomy is a file, not a mood.** A grant names its scope, its gates and its risk classes, and
deleting the file revokes it. A watchdog notices a session that has died or is blocked on your
decision, and either resumes it or tells you.

**The repository caps its own prose.** Standing documentation is allowlisted and line-capped by a
test, because this repo once held roughly 39,000 lines of process prose against 4,000 lines of
code and its owner stopped understanding his own system.

## What reaches `main`

Passing work opens a pull request against `ready-for-main`, which itself only changes through a
pull request with CI green. Promotion to `main` is the owner's act, or the orchestrator's under a
recorded grant — and that grant's gates are the promotion's own green CI plus a binding review
PASS on that exact diff. No path merges to `main` on an agent's say-so.

## What it does not claim

This is a working reference implementation, not a product: Linux, GitHub, and the vendor CLIs you
already have. The isolation is real but bounded, and the boundaries are written down rather than
implied. [SECURITY.md](SECURITY.md) states what the tests actually prove, what depends on how you
deploy it, and the known gaps.

## Where to look next

[How Relay works](how-it-works.html) — the same two loops, drawn.
[BOOTSTRAP.md](BOOTSTRAP.md) — setup, from toolchain and GitHub access through worker isolation to
the first job.
[CLAUDE.md](CLAUDE.md) — the operating rulebook the agents follow, and its safety invariants.
[AGENTS.md](AGENTS.md) — the commands and the exact role assignments.

Model and role configuration lives in `scripts/models.json`; the orchestrator is whichever
supported CLI is running the session.

MIT — see [LICENSE](LICENSE).
14 changes: 7 additions & 7 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ proves them), **configured assumptions** (set up outside this repo, verified man
| Worker changes outside the spec's declared scope are rejected | `tests/dispatch_gate4.sh`, `tests/scope_glob.sh` |
| A verdict is bound to the exact diff and base; a stale base is refused | `tests/dispatch_gate4.sh` |
| The rulebook and repo prose cannot silently grow back | `tests/rulebook_cap.sh`, `tests/prose_cap.sh`, `tests/plain_language.sh` |
| Review rounds are capped at five per topic, in code (and only round-N.md files count as rounds), and the reviewer refuses any artifact its own MODEL is recorded as authoring — and the whole author vendor wherever no author model is on record | `tests/review_cap.sh`, `tests/review_authorship.sh` |
| Review rounds are capped at five per topic, in code (and only round-N.md files count as rounds), and the reviewer refuses the whole author vendor wherever no author model is on record; a RECORDED author model runs even when it matches the reviewer (instance-level rule 7, owner 2026-08-09) — that record is an unauthenticated stamp, see gap 8 | `tests/review_cap.sh`, `tests/review_authorship.sh` |

## Configured assumptions (outside this repo; verify during bootstrap and after any GitHub change)

Expand Down Expand Up @@ -97,12 +97,12 @@ proves them), **configured assumptions** (set up outside this repo, verified man
mount is accepted for sources proven writable only by root/operator.

8. **`scripts/review`'s self-review refusal rests on a DERIVED author, not an authenticated one.**
Three fail-OPEN holes: a worker worktree is classified `codex` by path alone, so a Claude or Kimi
reviewer can be handed an artifact of its own vendor; the refusal is model-level, so `codex-plan`'s
`author_model:` stamp — the author's own word — is what lets one vendor hold both roles; and an
artifact with no recorded provenance falls back to the `--author` flag, so an orchestrator of any
vendor can understate its own draft (a fallback carries no model, so the asserted vendor is
refused whole). All three close with a digest-bound receipt written at dispatch, not by the author.
The rule is instance-level (rule 7; owner 2026-08-09), and no artifact carries instance identity,
so rule 7 is HELD BY CONSTRUCTION (every invocation spawns a fresh reviewer), never proven per
artifact. Two consequences: any known `author_model:` stamp — the author's own unauthenticated
word — flips the derivation off `-` and suppresses the vendor refusal entirely; and an unstamped
artifact asserting `--author` of another vendor routes to this vendor's reviewer unrefused. Both
close with a digest-bound receipt written at dispatch, not by the author.

## Scope

Expand Down
14 changes: 11 additions & 3 deletions scripts/codex-plan
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ no flag is given.
trust-boundary/high-risk work, a migration, or
an architecture decision across components
A full brief must contain every anatomy section (checked below): outcome, scope
and non-goals, frozen decisions, assumptions, the earliest falsifiable proof,
slices, gates, verification, rollback, deferred, definition of done.
and non-goals, frozen decisions, assumptions, the minimal existing path, the
earliest falsifiable proof, slices, gates, verification, rollback, deferred,
definition of done.

Supply TASK as arguments or, when no task argument is given, on standard input.
EOF
Expand Down Expand Up @@ -152,6 +153,7 @@ and put real content under each — a heading with filler is worse than none:
## Scope and non-goals
## Frozen decisions
## Assumptions
## Minimal existing path
## Earliest falsifiable proof
## Slices
## Gates
Expand All @@ -167,6 +169,12 @@ Rules for the content:
condition under which it may be reopened. A technical assumption is never
frozen — it goes under Assumptions with the evidence for it, or into a step
that verifies it.
- Minimal existing path: the command built from installed code that comes closest
to the Definition of done, run for real, recorded as its exact argv, the
installed commit it ran at, a digest of every input it consumed, its exit status,
and its COMPLETE output — then the one thing that output does not give you. If it
gives you everything, no build is authorized: say so and stop. An authentication
or environment failure is not an insufficiency; fix it and re-run.
- Earliest falsifiable proof: the smallest end-to-end run that would prove this
whole approach wrong, done FIRST, before anything is built around it.
- Slices: an ordered list of independently shippable increments — one PR each —
Expand Down Expand Up @@ -263,7 +271,7 @@ fi
if [[ "$tier" == "brief" ]]; then
if ! awk -v out="$stdout_path" '
BEGIN {
n = split("Outcome|Scope and non-goals|Frozen decisions|Assumptions|Earliest falsifiable proof|Slices|Gates|Verification|Rollback|Deferred|Definition of done", want, "|")
n = split("Outcome|Scope and non-goals|Frozen decisions|Assumptions|Minimal existing path|Earliest falsifiable proof|Slices|Gates|Verification|Rollback|Deferred|Definition of done", want, "|")
for (i = 1; i <= n; i++) required[tolower(want[i])] = 1
next_want = 1; fence = 0
}
Expand Down
2 changes: 1 addition & 1 deletion scripts/models.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"utility_subagent": { "model": "claude-sonnet-4-6", "effort": "default" },
"worker": { "model": "gpt-5.6-luna", "effort": "high" },
"bound_reviewer": { "model": "gpt-5.6-sol", "effort": "high" },
"orchestrator_artifact_reviewer": { "model": "gpt-5.6-luna", "effort": "high" }
"orchestrator_artifact_reviewer": { "model": "gpt-5.6-sol", "effort": "high" }
},
"cli_aliases": {
"claude-fable-5": "fable",
Expand Down
Loading
Loading