Answer mentions with Codex, and give each task its own worktree - #740
jorgemanrubia wants to merge 96 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Repository filters can execute outside containment, and migration ordering plus forced-prune and policy-verification paths have correctness gaps.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds Codex workers and isolated Git worktrees so connector tasks can run concurrently without sharing checkout state.
Changes:
- Adds the sandboxed Codex spawn driver.
- Adds ledger-backed worktree creation, recovery, retention, and pruning.
- Exposes worktree management through new CLI commands and tests.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
File summaries
| File | Description |
|---|---|
.surface |
Records the new CLI surface. |
e2e/smoke/smoke_lifecycle.bats |
Marks worktree commands out of E2E scope. |
internal/commands/commands.go |
Registers the worktrees action. |
internal/commands/connect.go |
Adds the worktrees command group. |
internal/commands/connect_run.go |
Enables per-task workspaces during dispatch. |
internal/commands/connect_worktrees.go |
Implements list and prune commands. |
internal/commands/connect_worktrees_test.go |
Tests worktree command behavior. |
internal/connector/driver/codex/codex.go |
Implements the Codex driver. |
internal/connector/driver/codex/codex_test.go |
Tests Codex policy and lifecycle behavior. |
internal/connector/driver/codex/fake_test.go |
Provides a fake Codex executable. |
internal/connector/driver/spawn/spawn.go |
Registers the Codex constructor. |
internal/connector/ledger.go |
Adds the worktree migration. |
internal/connector/ledger_worktrees.go |
Implements worktree persistence. |
internal/connector/setup/file.go |
Allows Codex worker configuration. |
internal/connector/worktrees.go |
Implements worktree lifecycle management. |
internal/connector/worktrees_test.go |
Tests worktree safety and recovery. |
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 4
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
e0cda3d to
9428454
Compare
07ead08 to
2d8ae66
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Cancellation can bypass unsafe-policy reporting, and automatic removal can race with creation of ignored work.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
internal/connector/driver/codex/codex.go:846
- This canceled branch returns before waiting for the rollout check. When cancellation races a
turn.failedevent afterthread.started, an absent or unsafe policy is therefore recorded as a normal cancellation, contrary to the requirement that every started turn's applied policy be verified. CheckfailedVerification()first, as the completed-turn path does.
if canceled {
s.finish(t, driver.PromptResult{Stop: driver.TurnCanceled, Refusals: refusals}, nil)
return
}
if err := s.failedVerification(); err != nil {
- Files reviewed: 16/16 changed files
- Comments generated: 2
- Review effort level: Balanced
9428454 to
7866973
Compare
2d8ae66 to
49f4b34
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Worktree-enabled runs remain blocked, and several races can violate the stated preservation and cancellation guarantees.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 5
- Review effort level: Balanced
7866973 to
b5efc24
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Project scoping, failed settlement handling, and worktree branch recovery contain unresolved correctness issues.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
internal/commands/connect_run.go:259
--projectis parsed and applied to intake/admission, but it is not passed to the dispatcher. A scoped run can therefore dispatch older admitted ledger records from projects outside the requested scope;TestTheProjectScopeNarrowsDispatchonly exercisesDispatcherOptions.Bucketsdirectly. Passbucketshere as well.
internal/connector/worktrees.go:487- A crash or transient failure after creating this anchor leaves
<task>-headbehind. On the next forced prune, the create-onlyupdate-reffails because that same anchor already exists, so the retained worktree can never be forced again even when the existing ref already points at thisHEAD. Treat an existing ref at exactlyheadas a valid prior anchor (and still refuse if it points elsewhere).
- Files reviewed: 41/41 changed files
- Comments generated: 2
- Review effort level: Balanced
f742eb7 to
494e920
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Worktree reconciliation can lose retained records, while several Codex completion and cancellation races can misreport turns or panic the connector.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
internal/connector/driver/codex/codex.go:532
- The cancellation check is still not atomic with finishing the turn:
Cancelcan sett.canceledafter this snapshot is read as false but beforefinishclearss.turn. That schedule returnsErrSessionEndedfor a requested cancellation. Recheck and claim the turn under the same lock, then route the claimed completion through the cancellation-aware path.
s.mu.Lock()
canceled := t.canceled
s.mu.Unlock()
- Files reviewed: 16/16 changed files
- Comments generated: 4
- Review effort level: Balanced
494e920 to
30f1da5
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The review found token exposure and worktree safety/reporting defects that must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 5
- Review effort level: Balanced
…tree never checked out as such Also: the branch is deleted before the directory, so a crash between them leaves nothing unreachable; worktrees list shows what a removal left mid-flight; and per-worktree refs are counted by their tips, since git keeps no reflog for them.
…d redactor Its case covers every path in drivertest.RedactionPaths, and the worktrees' git errors go through a redactor too.
…d redactor Its case covers every path in drivertest.RedactionPaths, and the worktrees' git errors go through a redactor too.
Each refusal is recorded once as it is read — by tool call id for the ones Codex puts on its stream, by position for the ones it only logs — and the stderr tail is read before a canceled, failed or lost turn is finished.
A worktree was removed while only one of the refs its judgment leaned on was checked again: the branch tip's. A commit the worktree reached through its reflogs, held by another branch, could lose that holder between the check and the removal, and the removal took it with the record. The judgment now carries every ref it leaned on and where each one stood, and the one ref transaction that ends the task branch verifies all of them immediately before the frozen copy is deleted. Anything moved since makes git refuse the transaction, and the worktree is kept instead of removed. Three smaller things from the same review: a record whose HEAD names a deleted branch is still judged, by reading the reflog file git refuses to read for it, so a crash between the two deletes leaves no row an operator cannot clear; a Finish that cannot take the lock says the worktree is kept instead of leaving a live row nothing lists; and a Codex refusal logged after the worker closed its stdout is read, because the reader now waits for the process rather than for its output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two reviews of the last head demonstrated work lost by removing a worktree automatically: a commit reachable only through the record's reflog when its holder was deleted while the deletion ran, a commit only ORIG_HEAD or an unreadable reflog reached, and the commit a worktree was made from when the route's branch had moved since. Each was closable, and each was the same shape: a judgment about what may be lost, made by a machine, acted on without anyone being asked. So the default goes instead of being hardened again. A task's end and a start's recovery now only ever keep the worktree, whatever is in it, and record it; `worktrees list` shows it with the task it was for and its size on disk; `worktrees prune` is the only thing that removes one. There is no flag and no second behaviour. What the removal itself learned from those reviews carries over to the prune: every commit a worktree reaches is held under a ref of the connector's own while its directory and record are deleted, so a holder someone deletes in the middle takes nothing with it; a worktree whose reflog cannot be read is not judged clean; the record's pseudo-refs are judged with everything else; and the commit the worktree was made from is judged like any other. Also here: a Codex refusal logged after the turn it belonged to has ended still goes through the shared recorder, because the reader reads stderr whether or not a turn is left to hang it on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A record left behind when a worktree's directory is deleted by hand was judged by its HEAD, its HEAD reflog and its per-worktree refs, but not by the pseudo-refs living in it (ORIG_HEAD after a reset, among others) or by the task branch's own reflog. A commit only one of those reached could have its last ref deleted with the row, leaving it for git to discard. Both are read now, as the frozen judgment already reads them. The refs a removal holds its commits under are also made after the judgment is proven to still stand, not before, so a removal that stops there leaves nothing of the connector's own behind for a later judgment to lean on. Also: the commands hand a profile name back in a pasteable command through the repository's shell quoting, and the connector setup skill carries the worker field and its flag, so an agent setting a connector up can find Codex. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A task whose worktree directory was already gone had its branch deleted at the end of the task, on a judgment that never read that branch's reflog: a commit only the reflog reached went with it, without anyone asking. The rule the worktree itself now follows applies to the branch too — the row is kept with its branch, and only an operator's prune decides. Two places were answering "what does this worktree still reach?": the frozen judgment and the judgment of a record whose directory is gone. They are one list now, so what one reads the other reads: the branch and its reflog, the record's HEAD and reflog, its per-worktree refs, its pseudo-refs — including MERGE_AUTOSTASH — what a rebase stashed away in a file, and a missing reflog as no evidence rather than as nothing to lose. The refs a removal holds commits under move to refs/basecamp-connect/removing/ and are never counted as holding a commit for anybody, so one a crash leaves behind cannot make a later judgment think someone else holds a commit; the refs a force keeps for the operator stay where they were. A force on a worktree whose directory is gone now keeps what nothing holds and clears the row instead of refusing forever. Also: the size walk in `worktrees list` runs apart from its answer, measures a frozen worktree under its removing name and reports no size for one that is not there; a ledger failure while keeping a worktree is returned by Finish and fails Recover rather than starting dispatch with worktrees nothing lists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared worker now hands out every line of a worker's stderr it kept, redacted, so a sandbox refusal Codex logged before it wrote anything else is read and recorded like the rest. Before this it could see only the last line, and a refusal followed by any other output was lost to the ledger. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The class of defect this card kept finding — a judgment about what a commit still reaches, made by a machine, acted on by deleting something — had one place left: a row whose directory something outside the connector removed. Every round hardened that judgment; this one deletes it. What is left of such a worktree is git's record of it and the task branch. The connector now judges neither and deletes neither. The row is kept, says it is orphaned, and is listed with the record, so an operator can see what is there. Naming its path in a force deletes the branch, having said so; git's own `worktree prune` is what clears the record. Nothing about reachability is decided on that path at all, so nothing on it can be wrong. The judgment stays where a force still needs it: a worktree that is on disk. Three things it was missing, each found by review: a force can now clear a worktree whose reflog cannot be read (a repository with core.logAllRefUpdates off would otherwise leave rows nothing could ever clear, the force being refused forever); per-worktree refs' own reflogs are read where the repository keeps them; and a bare repository a worker made inside its worktree is git data like any other, so a force refuses it rather than discarding its commits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A record's pseudo-refs can name more than one commit — FETCH_HEAD does, and MERGE_HEAD in an octopus merge — and `rev-parse` reduced them to the first, so the rest were judged as if they were not there. They are read as the files they are now, every object name in them. The help said a force never loses commits. That is true of a worktree still on disk, whose commits are kept under refs first; it is not true of one whose directory something else removed, where the force deletes the task branch and whatever only that branch reached goes with git's record when `git worktree prune` runs. The help now says which is which, and a plain prune is no longer described as removing such a worktree, because it leaves it alone. A force whose branch deletion fails keeps the row rather than closing it, and a removed worktree no longer reports the reason it was kept for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A repository a worker made inside its worktree was only recognised where git tracks nothing: one at the directory the task worked in, or at the worktree's own root, read as ordinary files and discarded by a force along with its commits. Every directory the walk enters is now looked at for what it is. Three rows that could never be cleared, each an operator left holding something no command would take: a worktree whose repository is gone at all (a force now closes the row, because there is nothing anywhere left to delete), a row that never stored where git's record is and was closed on the strength of not knowing (the repository is asked, and a record still there keeps the row), and — from the round before — one whose reflog cannot be read. And two things a force now says: the commit the task branch stood at when a force on an orphaned worktree deleted it (branch_deleted_at, which is what puts it back), and, in the help, that the promise of keeping every commit is the on-disk path's, not the orphan's. A ref found at two different commits while the judgment ran is a ref that moved, so the worktree is kept, and refs a half-finished hold made are let go again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
349651a to
17d5971
Compare
…left Three endings of a canceled turn each decided for themselves whether to read the refusals Codex only logs: the reader's, a completed turn's and a failed turn's. They go through one place now, which waits for the worker to go and reads its stderr before the turn ends, so the result carries what the ledger carries. Also from review: a removal that could not delete the frozen directory puts the task branch back before the worktree comes back, so what returns is what was there; one that could not delete git's record leaves the row removing for the next start to restore, rather than recording it removed and leaving a locked record nothing lists; `connect show` says which coding agent workers are, not only how they are run; and the list help names "orphaned". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unsafe Codex exits omit refusals from results, and forced orphan pruning hides its recovery commit.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 4
- Review effort level: Balanced
…it a force took A session ended because Codex ran under a policy other than the one it was asked to run under finished its turn with an empty result: the refusals it had already made were in the ledger and not in the answer. Every ending of a turn now goes through a place that ends the worker, reads its last word and reports what the ledger has. And the commit a force on an orphaned worktree deletes the task branch at — the one thing that puts it back — was carried by the result and dropped by the command that prints it. It is in the JSON now (branch_deleted_at), as the help says. Prune's own doc comment said it removes worktrees whose directory is gone, which is the one thing it does not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Forced pruning can delete a task branch currently checked out by another linked worktree.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 1
- Review effort level: Balanced
| if r.AdminDir != "" { | ||
| switch at, err := os.ReadFile(filepath.Join(r.AdminDir, "gitdir")); { | ||
| case err == nil: | ||
| // The record is the worktree's .git file, absolute or — with | ||
| // worktree.useRelativePaths — relative to the admin directory. | ||
| path := strings.TrimSpace(string(at)) | ||
| if !filepath.IsAbs(path) { | ||
| path = filepath.Join(r.AdminDir, path) | ||
| } | ||
| path = filepath.Dir(path) | ||
| return !samePath(path, r.Path) && exists(path) | ||
| case !errors.Is(err, os.ErrNotExist): | ||
| // The record cannot be read: assume it is still somewhere. | ||
| return true | ||
| } | ||
| return false | ||
| } |
0fc79b0 to
caab8f1
Compare
… tables are migration 8 #736 squash-merged into main, so the 33 commits this branch carries of it are one commit there. Merging the parent branch in settles that at the tip instead of replaying them onto a history that already holds their content. Six files conflicted; five were files both histories hold from 736, and took the parent's version with this card's own change applied back on top. The sixth is the migration list. The worktrees tables are migration 8. They were 8 before as well, but behind card 20's outbox at 7, which has not landed; they follow the dispatcher's tasks and attempts at 7 now. The number stayed and the reason for it moved, and the comment says so, because a ledger that has applied a migration never sees it renumbered.
A conflicted merge leaves AUTO_MERGE naming the tree ort merged to, and a per-worktree ref names whatever a worker put under it. Asking git what contains a tree is a question it answers with an error, so the judgment came back unverified and the worktree could never be forced away. Every object name a judgment collects is now resolved to the commit it reaches — itself, or what an annotated tag points at — and what reaches no commit is dropped, because there is no history in it to lose. A HEAD that names no commit was the same dead end: rev-parse and reflog show both refuse it, so a worker's `checkout --orphan` left a worktree nothing could remove in place. An unborn HEAD is now an answer rather than doubt — `--quiet`'s exit 1 says it, and any other failure still leaves the worktree unjudged — and HEAD's reflog is read from the file, so the commits it stood at before are still kept.
|
🤖 Landed in #748, Combining them is what found the defects none of us could see alone — a |
Stacked on Dispatch admitted mentions to Claude Code workers; must not merge before it.
The connector can only hand work to Claude Code, and it runs one task per repository at a time: a second mention on another card in the same project waits for the first to finish, because both would edit the same checkout.
This adds Codex as a second worker under the same driver boundary, which is the evidence that the boundary holds, and gives each task its own git worktree so tasks on one repository run side by side without losing each other's uncommitted work.
Originally tracked in Codex spawn driver and worktrees.
Codex
connect setup --worker codexmakes workerscodex exec --json. A Codex process takes one prompt and exits, so it advertises no follow-up prompts: a follow-up starts a task of its own once the first ends, as the spec says for such drivers.Findings from running Codex 0.153.4 that shaped the driver:
env_vars) puts them in Codex's environment, and the model's shell could read them. So the task token never goes through the driver. The worker's MCP server is the connector's ownbasecamp connect worker-mcpbridge, which takes the token over a one-use socket. The driver passes each server only its declared, non-secret environment, as Codex'smcp_serversenv table, and writes nothing to disk.codex execreports no permission mode, and silently ignores a-coverride it does not recognise. So the driver runs with--strict-config, and reads the policy Codex actually applied from the turn'sturn_contextrecord in its rollout file. It ends the session as unsafe when approvals are notnever, when the sandbox is notworkspace-writewithout network or/tmp, when the filesystem policy lets anything but the working directory be written, or when the directory is another.--ignore-user-configalone still let the host's skills in: the worker read and followed the connector's own front-thread skill. Skills, connected apps, plugins, hooks, and browser and computer use are switched off. Every MCP server isrequired, so Codex refuses to start a turn without its Basecamp server.Codex's reach differs from Claude Code's. It reads through shell commands, so its shell is not removed; it runs inside Codex's own sandbox (writes only in the working directory, no network) with approvals set to never. That is still policy, not containment. One consequence: a worktree's git data lives in the main repository's
.git, outside the sandbox, so a Codex worker cannot commit. A Codex task that edits files always ends with its worktree kept as dirty. Codex's sandbox can read the whole filesystem. A model in one session can therefore read the connector's state directory, including another session's MCP environment file in the moment between its writing and its server's start. Whether--ignore-user-configalso keeps out a user-level$CODEX_HOME/AGENTS.mdis unverified: this machine has none, and no flag for it was found.Worktrees
With
--worktrees, each task gets a worktree on abasecamp-connect/branch at the route's HEAD. When the task ends, the worktree is kept — the connector removes none of its own accord — and listed bybasecamp connect worktrees listwith the task it was for, its size on disk, and why it is kept.basecamp connect worktrees pruneis the only thing that removes one: it removes those that hold no work and keeps the rest, with their reason (dirty, unpushed, locked, moved, unverified). With--force <path>it removes one that still holds work, after keeping every commit it reaches underrefs/basecamp-connect/retained/.The rule for removal is written once, as "One worktree, one removal" on the
Worktreestype. It says when a worktree may go, what counts as work, what happens to that work, and who may force. One function carries it out, reached only from prune. That function freezes the worktree before judging it: it renames git's record of the worktree, then the directory. From that moment nothing can land a commit in it or write to it by path. Every commit the worktree reaches is then held under a ref of the connector's own for as long as the deleting takes, so a branch someone deletes in the middle takes nothing with it. It then deletes the frozen copy, or restores both names and keeps the worktree. Nothing in the connector runsgit worktree remove. A crash while a worktree is frozen is restored on the next start.Worktrees live under the connector's state directory, not "under the repository" as the card says. Inside the checkout, a task working in the route itself could edit other tasks' kept work, and
git add -Ain the checkout would pick it up. Placement is one replaceable function, the seam the sandbox launcher's broker-owned scopes will use.Invariants
Codex driver (3 also covers turns that fail or lose their process; 5 also covers a cancel sent before the prompt):
CODEX_HOME/CODEX_API_KEY; host config, rules, skills, apps, plugins and hooks are off; the model's shell gets Codex's core variables only.Worktrees:
TestTheWorktreeRuletries each case against the operator's discard: clean, modified, untracked, ignored, unpushed, a commit only a reflog reaches, a per-worktree ref, a stash, a lock, a commit tried between the check and the removal, a file written by path in that window, and two forced cases. Other tests cover a crash while frozen, a holder deleted while the deleting runs, a repository that keeps no reflogs, and a commit onlyORIG_HEADreaches.git worktree addand before a worktree is frozen.GIT_CONFIG_KEY_n, so no filter name escapes).Each rule has a test that was watched fail with the rule reverted.
Two paths, one answer
Rounds of review kept finding the same shape of defect: two pieces of code answering one question, and only one of them kept up to date. They are unified here, and where a second answer was not needed at all it is gone. "What does this worktree still reach?" was asked by the frozen judgment and, separately, by the judgment of a record whose directory is already gone; they read one list now — the branch and its reflog, the record's HEAD and reflog, its per-worktree refs, its pseudo-refs, and what a rebase stashed away in a file — so what one reads the other reads. "What holds this commit?" was asked when judging and again when removing; the removal now verifies, in one ref transaction, every ref the judgment leaned on.
The removal default: the connector never removes a worktree
This flipped late, on the operator's standing rule and a review finding: "Base commits can become unreachable during automatic removal." A default that loses work in a case the rule says it holds is not a default worth keeping, so automatic removal is gone rather than hardened again — there is no flag, and no second behaviour to reason about.
A task's end and a start's recovery now only ever keep the worktree, whatever is in it, and record it. The same goes for the task branch: a row whose directory is gone keeps its branch, because the branch reaches commits of its own.
A worktree whose directory something outside the connector removed goes further still. What is left of it — git's record and the task branch — reaches whatever it reaches, and the connector judges none of that and deletes none of it. The row is kept, says it is
orphaned, and is listed with the record, so an operator can see what is there; naming its path in a force deletes the branch, having said so, and git's ownworktree pruneclears the record. Reachability is decided only where a force still needs it: a worktree that is on disk. The accepted cost is that refs accumulate for worktrees someone deleted by hand; the listing makes them visible and the force clears them. Reclaiming disk is an operator's move:basecamp connect worktrees listshows every kept worktree with its task, its size and why it is kept, andbasecamp connect worktrees pruneremoves the ones that hold no work —--force <path>removes one that does, after keeping its commits underrefs/basecamp-connect/retained/.The ref verification built for the old default carries over to that discard: a prune removes a worktree only while every ref its judgment leaned on is still where the judgment found it, and refuses otherwise unless the operator forces it.
End-to-end
On the scratch project, as the agent profile pinned to that project, with
worker: codexandworktrees: true(the runs below predate the flip: what they showed removed at the task's end is now kept until a prune): card 19: codex answers, and again after each round of review fixes, most recently on card 19: codex answers (head 5678ddef), with the token carried by the connector's bridge and the removal verifying every holder. Codex acknowledged in 32 s and replied from its own worktree.complete_dispatchwas recorded as succeeded with its reply id, and the attemptfinished. The file it was asked to leave uncommitted kept the worktree (dirty), andconnect worktrees listshowed it.prunekept it, then removed it and its branch once the file was deleted. The session directory was empty afterwards.The ledger does not get a session id for a Codex attempt: Codex reports its thread id only after the prompt is written, while the dispatcher records the session at
running. Nothing resumes a Codex session yet, so this is noted rather than worked around.