Skip to content

fix(browser-bridge): the [INT] split verdict is a STARVED INSTRUMENT, not a handler defect - #1585

Merged
ZacxDev merged 8 commits into
mainfrom
fix/release-handler-split-verdict
Sep 12, 2026
Merged

ZacxDev merged 8 commits into
mainfrom
fix/release-handler-split-verdict

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Sep 12, 2026

Copy link
Copy Markdown
Member

Closes rank 7 of handoff-alert-recall-and-skill-consumers.md"decide the devrc split-verdict test test_browser_agent.py::test_the_release_handler_EXITS_rather_than_resuming[INT]".

The decision: it is a TEST defect, not a code defect

The handoff's leading hypothesis was "genuine parallelism-sensitivity in the signal handler's lock release", with the caveat "NOT established: nobody has read the handler against the failure." That hypothesis is refuted.

Mechanism, reproduced verbatim

The test spawns the wrapper, waits for the warm marker, then killpgs it. Injecting a stall between those two points splits the outcome into three bands — only the first is a verdict on the handler:

stall rc what actually happened
< ~3 s 143 / 130 the trap ran — the real observation
~3 s .. ~12 s 2 the wrapper had left the warm; the signal killed its tool-set gate instead, so rc != 0 passes for the wrong reason
> ~12 s 0 the wrapper had already exited. It is a ZOMBIE, so os.getpgid still resolves and killpg still succeeds (the signal is discarded), and proc.wait() hands back the stored status 0

The third band emits the reported message verbatim — exited 0 after a INT — the handler released the lock and let the run CONTINUE unserialised instead of terminating — naming a regression that did not happen. The test never established that its signal reached a live wrapper still inside the warm.

Why "reproduced 2/2, so not a flake" did not settle it

The handoff ruled out load by reproducing twice in the same minutes. That is two samples of one box state, not a discriminator. Corroboration that no code moved: neither browser-agent nor this test file has a commit since before the failure was recorded, and the documented repro command at origin/main 337114e0 today gives 914 passed — which is the target's full collection, so the test ran (not a vacuous green).

The fix

_warm_window_lost() runs immediately before the killpg and reads /proc/<pid>/statnever proc.poll(), which reaps the child and manufactures a fourth failure shape (ProcessLookupError); that one bit me while instrumenting. A lost window is retried up to 3 times; giving up is a loud failure naming starvation and stating that it is not a verdict on the handler. This also closes the silent rc-2 band.

Nothing here can green a real regression: a wrapper that resumes after the signal is alive and inside the warm at kill time, which is exactly when an attempt counts.

Verification

  • control2 passed in 6.91s; unchanged cost on the happy path.
  • mutation, isolated to INTtrap '_oc_lock_release; exit 130' INTtrap '_oc_lock_release' INT: KILLED, 1 failed, 1 passed, by this test's own exited 0 after a INT message, with the new line reading "the INT landed inside the warm (attempt 1; hold 3s), so this IS a verdict on the handler". TERM stayed green.
  • negative controls, injected stall — 6 s: all 3 attempts report "the warm lock was already released" (previously a silent wrong-reason PASS). 12 s: all 3 report "already a ZOMBIE" (previously the false handler regression). Both fail as STARVED INSTRUMENT, neither as a handler defect.
  • full target, the documented repro commandpytest scripts/browser-bridge/tests -n 4 --dist loadfile914 passed.
  • scripts/scoped-tests.shSCOPE: SCOPED (1 file(s) across 1 of 29 hermetic target(s)), RESULT: PASS (exit=0). ⚠ Not a gategate.sh exits 91 = PARTIAL off any non-FULL scope. Neither tier was run in full; CI evaluates the whole-target expectations this run suspended.

Left open, deliberately — filed, not fixed

The sibling test_a_run_killed_mid_bootstrap_RELEASES_the_warm_lock has the same exposure and its own comment already names it: "a TERM landing after the release finds the lock already gone, so not lock.exists() passes for the WRONG REASON." It asserts absence, so a starved window makes it pass silently rather than fail loudly — strictly worse to read, and invisible to any count of reds. _warm_window_lost() is what it needs. Out of rank 7's scope.

Closing condition: a merged PR in which that test consults _warm_window_lost() before its killpg, watched to go red when the window is starved out.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5

ZacxDev and others added 8 commits September 12, 2026 01:09
… not a handler defect

`test_the_release_handler_EXITS_rather_than_resuming[INT]` was recorded as red
under the dev-host suite's own `-n 4 --dist loadfile`, green alone, green in the
nix sandbox. Diagnosed by measurement, not by reading the handler.

MECHANISM, reproduced verbatim. The test spawns the wrapper, waits for the warm
marker, then `killpg`s it. Inject a stall between those two points and the
outcome splits into three bands:

  stall < ~3s    rc 143/130   the trap ran — the real observation
  ~3s .. ~12s    rc 2         the wrapper had LEFT the warm; the signal killed
                              its tool-set gate, so `rc != 0` PASSES for the
                              wrong reason and proves nothing
  > ~12s         rc 0         the wrapper had already exited. It is a ZOMBIE, so
                              `os.getpgid` still resolves and `killpg` still
                              succeeds (the signal is discarded), and
                              `proc.wait()` hands back the stored status 0

The third band produces the reported failure message verbatim — "exited 0 after
a INT — the handler released the lock and let the run CONTINUE unserialised" —
naming a regression that did not happen. So the test never established that its
signal reached a LIVE wrapper still inside the warm; on a loaded box it reports
machine load as a code defect.

Corroboration that no code changed: neither `browser-agent` nor this test file
has a commit since before the failure was recorded, and the documented repro at
`origin/main` 337114e today is `914 passed` (the target's full collection, so
the test ran — not a vacuous green).

FIX. `_warm_window_lost()` is checked immediately before the `killpg` and reads
`/proc/<pid>/stat` — never `proc.poll()`, which REAPS the child and manufactures
a fourth failure shape (`ProcessLookupError`). A lost window is retried up to 3
times; giving up is a LOUD failure naming starvation and saying explicitly that
it is not a verdict on the handler. This also closes the silent rc-2 band, which
passed green while observing nothing.

Nothing here can green a real regression: a wrapper that resumes after the
signal is alive and inside the warm at kill time, which is exactly when an
attempt counts.

VERIFIED
- control: `2 passed in 6.91s` (unchanged cost on the happy path)
- mutation `trap '_oc_lock_release; exit 130' INT` -> `trap '_oc_lock_release' INT`
  (INT only, isolated): KILLED, `1 failed, 1 passed`, by THIS test's own
  `exited 0 after a INT` message, with the new line confirming "the INT landed
  inside the warm (attempt 1; hold 3s), so this IS a verdict on the handler".
  TERM stayed green.
- negative controls, injected stall: 6s -> all 3 attempts report "the warm lock
  was already released" (previously a silent wrong-reason PASS); 12s -> all 3
  report "already a ZOMBIE" (previously the false handler regression). Both fail
  as STARVED INSTRUMENT, neither as a handler defect.
- full target, the documented repro command
  (`pytest scripts/browser-bridge/tests -n 4 --dist loadfile`): 914 passed.
- `scripts/scoped-tests.sh`: `SCOPE: SCOPED (1 file(s) across 1 of 29 hermetic
  target(s))`, `RESULT: PASS (exit=0)`. NOT a gate — CI evaluates the full run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
…llowed, stop asserting machine load

Round 0 (requirements & deletion) questioned two requirements. Both were right.

🟡 R8 — THE DIFF DELETED A HARD PRECONDITION AND NOBODY SAID SO.
`assert lock.is_dir(), "precondition: the warm must run UNDER the lock"` became
one of the RETRYABLE lost-window reasons. Move the lock acquisition after the
warm — the class `browser-agent`'s own comments track — and the test still went
red, but told the reader "Read this as machine load, not as a code defect", at
3x the runtime. A real defect rerouted into a message that says stop looking.
`_warm_window` now returns a third verdict, `defect`: a lock that is GONE while
the wrapper is ALIVE and the hold is demonstrably still running cannot be
explained by a slow box, so it is a hard failure with no retry.
WATCHED, and REACHABILITY is the half that matters — a guard that is breakable
but unreachable proves nothing. Mutant A (`_oc_lock_release` moved before the
warm): `2 failed in 0.96s`, both with the defect message, `GONE only 0.02s after
the wrapper announced it was inside the warm`. Genuine starvation of 6 s on the
same build reports `already released 2.96s after the marker (hold is 3s)` and
retries — so the two branches are discriminated by measurement, not by wording.

🟡 R3 — the retry is a CONVENIENCE requirement; it is now labelled as one. The
loud give-up alone satisfies "never report a lost window as a defect"; the retry
only buys suite-greenness under load.

🔴 THE SUGGESTED SIMPLIFICATION IS REFUTED, AND THE MEASUREMENT IS RECORDED SO
NOBODY RE-DERIVES IT. Round 0 proposed widening `_WARM_HOLD_S` 3 -> ~20 to put
starvation out of reach and delete the retry, reasoning that the hold is free
because the fake writes its marker BEFORE sleeping. It is not free: GNU `timeout`
puts the warm's child in its OWN process group, so the test's killpg to the
wrapper's group never reaches the fake — it sleeps out the whole hold while bash
defers the trap. MEASURED on this pair: hold 3 s -> 6.78 s, hold 8 s -> 16.73 s.
A 20 s hold costs ~40 s per run. Rejected, with the numbers, in the source.

🟡 ONE RULE, ONE PLACE. The inline `/proc/<pid>/stat` parse duplicated
`_process_gone`/`_proc_state`, which sit in this same file, already carry the
measured zombie incident (devrc-ci red 5 of 5, 2026-08-22), and were already
DRIFTING from the copy: `rsplit(") ", 1)` vs `rsplit(")", 1)`, and
`(OSError, IndexError, ValueError)` vs a narrower set that would have let a
PermissionError error the test instead of reporting a lost window. Now calls the
helpers.

🟡 DO NOT ASSERT MACHINE LOAD — MEASURE IT. The give-up message declared load
without probing, in a file whose every other wall-clock failure calls
`_spawn_baseline()`. It now reports the probe and says which way it points; on
the injected-stall controls it correctly reads "the MACHINE looks idle — this is
NOT explained by load", because it was not.

🟢 Also: `{name}` in the marker path was dead (`rig` is function-scoped), while
`{attempt}` is load-bearing and undocumented — without it attempt 2 returns
instantly on attempt 1's stale marker and every retry is spent by construction.
Lost attempts now drain stdout/stderr (an fd leak per starved run, and the
wrapper's own account of why the window went was being discarded). The diag line
no longer claims "this IS a verdict" — the check and the killpg are not atomic,
and it now says so.

🔴 A BUG THIS FIX ITSELF INTRODUCED, caught before pushing: the elapsed clock was
first taken AFTER `_await` returned, so it read 0.00s every time and would have
called every lost window a defect. A stall inside `_await`'s own poll loop is
exactly the starvation under test. It now times from the marker file's mtime,
which is the only honest zero for that clock.

RE-VERIFIED IN FULL (an audit fix resets the gate):
- control: 2 passed in 6.74s, unchanged
- mutant A (lock released during the warm): 2 failed in 0.96s, defect branch,
  its own message, no retries burned — the branch is REACHABLE, not just breakable
- mutant B (`exit 130` dropped, INT only): still KILLED, 1 failed / 1 passed, by
  this test's own `exited 0 after a INT` message; TERM green, so isolated
- injected stall 6 s -> lost/"already released 2.96s after the marker"; 12 s ->
  lost/"already a ZOMBIE". Neither is misread as a defect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
…cause the marker identified the wrong event

Round 1 (nine axes, dispatched blind) returned one 🔴 and it is the fix from the
previous round misfiring in exactly the class this PR exists to remove.

🔴 THE `defect` VERDICT FIRED ON A PURELY STARVED RUN. The fake's
`if argv[:2] == ["debug", "agent"]` branch serves BOTH invocations the wrapper
makes: the warm (`debug agent build`, UNDER the lock) and the tool-set gate
(`debug agent browser-agent`, AFTER `_oc_lock_release`). So the gate re-wrote
FAKE_OC_DEBUG_MARKER ~0.2s after the lock was gone, resetting the mtime clock to
~0 with `lock.is_dir()` already False — which my new "the hold must still be
running, so load cannot explain this" branch read as a lock-ordering regression.
Non-retryable, loud, and it explicitly denies load as the cause. REPRODUCED at a
3.5s stall: `GONE only 0.34s after the wrapper announced it was inside the warm`.
The previous round's negative controls (6s, 12s) straddled the band without
landing in it.

ROOT CAUSE FIXED, not the symptom: the fake's hold+marker are now scoped to the
`build` subcommand, so the marker means "inside the warm" and nothing else can
refresh it. That also removes a second sleep the gate was paying every run.

🟡 A FOURTH STARVATION SHAPE BYPASSED THE RETRY LOOP ENTIRELY. If the wrapper
does not reach the warm within `_await`'s 20s slice, `_await` fails in its own
wording — probing the machine AFTER the stall, so it can report "the machine is
not the explanation" about a box that has since recovered — and it is terminal,
so no retry and none of this test's framing. Now routed through the same
lost-window path as every other way of not reaching the warm.

🟡 TWO MEASURED NUMBERS IN MY OWN COMMENT WERE WRONG. "hold 3s -> 6.78s, 8s ->
16.73s, i.e. ~2x the hold, paid on the KILL path every run" read the PAIR total
as a per-run cost. Re-measured with `--durations`, per test: hold 3 -> 3.12/3.25s,
hold 8 -> 8.08/8.13s — ~1x the hold per test, ~2x for the pair. The rejection of
"just widen the hold" still stands (20s would be ~20s per test, up from ~6s), but
with honest numbers and with the note that the old figure also predated scoping
the fake.

🟢 The `defect` path failed without draining stdout/stderr while the adjacent
`lost` path had a comment about exactly that leak. Both now go through one
`_abandon()` helper, so the wrapper's own stderr reaches the message either way.

VERIFIED (an audit fix resets the gate; every claim below was re-run on THIS tree)
- control: 2 passed in 7.10s
- mutant A (`_oc_lock_release` moved before the warm): defect branch still
  REACHABLE — `GONE only 0.02s`, both params, no retries burned
- mutant B (`exit 130` dropped, INT only): still KILLED by this test's own
  `exited 0 after a INT`; TERM green, so isolated
- band sweep, false-defect count at stalls 3.5 / 4 / 6 / 12s: 0, 0, 0, 0, with
  STARVED INSTRUMENT firing at all four
- full target: 914 passed (scoping the fake disturbs no other consumer)

🔴 AND ONE OF THOSE BAND SWEEPS WAS VACUOUS BEFORE IT WAS REAL. The first run of
it scored 0/0/0/0 with the injection probe accidentally deleted by this commit's
own restructure — an instrument wired to nothing, reporting exactly the number I
wanted. Caught by checking the probe was present, then re-run with a positive
control: stall 0 -> 2 passed, stall 3.5+ -> 2 failed. Only the second run is
evidence; the numbers above are from it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
… cap the retry's worst case, and retract a saving that never existed

Round 2 (delta) returned NO 🔴 and reproduced claims 1-3 as genuinely fixed —
including the pre-fix false-defect band and its absence. Four 🟡, all taken.

🟡 MY OWN CORRECTION INTRODUCED A NEW FALSE CLAIM, IN TWO PLACES. Round 1's fix
said scoping the fake "removed a SECOND sleep the tool-set gate used to pay".
FALSE: both signal tests kill the wrapper INSIDE the warm (gate at
`browser-agent:614`, warm at `:480`), so the gate never executes and no test ever
paid it. Measured per test at hold 3 — unscoped 3.32/3.19/3.14, scoped
3.10/3.09/3.15, indistinguishable. The old "~2x" figure is fully explained by the
pair-total error alone. Scoping bought correctness, not runtime; both sites now
say so. That is the third consecutive round whose finding was prose the previous
round wrote while fixing the round before it.

🟡 THE BAND TABLE IS NOW STALE IN THE PRESENT TENSE. It documents the original
diagnosis against the UNSCOPED fake. Re-measured on this tree with the classifier
bypassed: stall 2s -> rc 143/130; stall 4/6/8/12s -> rc 0. The middle `rc 2` band
NO LONGER EXISTS, and the zombie band starts at ~4s instead of ~8s — a NARROWER
margin, which raises the stake on `_warm_window` rather than lowering it. Labelled
as history, with the current numbers beside it, so nobody re-tunes
`_WARM_WINDOW_SLACK_S` against boundaries that cannot reproduce.

🟡 SCOPING CREATED AN UNPINNED COUPLING, AND IT FAILS IN THE WORST DIRECTION.
The fake now keys its hold on `argv[2:3] == ["build"]`, i.e. on the wrapper's
exact warm argv, and nothing asserted that. Demonstrated: change the warm to
`debug agent --pure build` and the marker is never written, so the signal tests
report STARVED INSTRUMENT — a WRAPPER CHANGE misattributed to machine load, the
exact class this file exists to stop, and silent because "the box was busy" is
always available. Closed with `_WARM_SUBCOMMAND` (one name, both sides) and
`test_the_fakes_warm_hold_is_keyed_to_the_argv_the_wrapper_ACTUALLY_USES`, which
asserts the RELATIONSHIP rather than one side. Mutation-checked: clean at control,
and mutant D (`debug agent --pure build`) KILLS it with its own message naming the
misattribution.

🟡 THE RETRY MULTIPLIED THE PATHOLOGICAL WALL TIME BY 3. `_await` was called with
`stall_cap=None`, which extends to `RUN_HARD_CAP` (900s), so three attempts took
the worst case to ~46 min per parametrisation — on exactly the starving box the
retry exists for, against a dev-host tier already observed hitting its own cap and
producing no verdict. Now `stall_cap=60.0`.

🟢 Verified and NOT changed, because the auditor checked them and they hold:
`lost[-1].split(": ", 1)[1]` cannot IndexError and preserves an embedded `": "`;
`except pytest.fail.Exception` is `_pytest.outcomes.Failed` and cannot over-catch
(the predicate cannot raise it); `_abandon` has no late-binding hazard and cannot
be called with a previous attempt's `proc`.

VERIFIED: control 4 passed; full target 915 passed (914 + the new guard);
mutants re-run independently by the auditor in its own worktree — `exit 130`
KILLED, `exit 143` KILLED, all-traps-removed KILLED via the sibling, confirming
the scoping did not silently break it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
Self-review while round 3 runs. The `stall_cap=60.0` comment claimed the benefit
(worst case 900s -> ~180s across 3 attempts) and said nothing about the trade.

Derived from `_stall_extends`: the extension fires ONLY when the baseline probe
MEASURES the box as stalled, so a healthy run is untouched — but a stalled one
now gets ~60s per attempt where it used to get 900s. A box that genuinely needed
300s to reach a warm that normally takes ~0.5s will now be reported as a STARVED
INSTRUMENT rather than waited out. That is the intended direction, and the retry
also adds load to an already-loaded box. Both halves now written down.

Also verified by attacking the new spelled guard two ways rather than leaving it
to the auditor. `test_the_fakes_warm_hold_is_keyed_to_the_argv_the_wrapper_
ACTUALLY_USES` fails SAFE in both: rewriting the warm to `debug agent "$_WARM_SUB"`
-> RED (correct; the fake's keying really would break), and splitting the warm
across a line continuation -> RED with "the warm invocation moved or changed
shape", which is true and is the right thing to re-check. Control green. No shape
found that leaves it silently green while the coupling is broken.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
… green against most of what it was written to catch

Round 3 (delta) returned one 🔴 and two 🟡. The 🔴 kills the guard I added last
round, and my own self-review of that guard was too narrow: I constructed two
attack shapes, both went red, and I concluded it failed safe. The audit
constructed a 10-mutant matrix and found FOUR survivors.

🔴 THE GUARD PASSED WHILE ITS NAMED HAZARD EXISTED, 4 SHAPES OF 6.
It asserted a SUBSTRING on a heuristically-selected source line; the fake keys on
ARGV POSITIONS. Those are different claims and the gap is walkable:
  * `--print-logs` inserted BEFORE `debug` — `argv[:2]` no longer matches, the
    substring still does. Confirmed END TO END: guard `1 passed in 0.41s` while
    the real signal test failed reporting STARVED INSTRUMENT.
  * the warm reflowed onto a continuation line while a stale COMMENT kept
    matching the line selector (which cannot tell code from a comment).
  * the template placeholder renamed, so `.replace()` silently no-ops and the
    fake keys on a literal that can never match.
  * the keying moved to a different argv slot.
🔴 And the docstring said "Asserts the RELATIONSHIP (both sides), not one side."
FALSE — the body read the WRAPPER and never looked at the fake, which is why the
last two survived. That is the guards-narrower-than-their-description class, and
per RULES.md reading as coverage while providing none is worse than none.

REPLACED, not patched, with two things that observe BEHAVIOUR:
1. `_coupling_diagnosis()` — the fake now logs every argv it receives while a
   hold is armed, so a broken coupling is reported as "THE FAKE/WRAPPER COUPLING
   IS BROKEN … actual argv: [...]" instead of as machine load. It observes the
   invocation, so it cannot be walked by spelling. Verified: the `--print-logs`
   mutant now prints `COUPLING IS BROKEN` with
   `actual argv: [['--print-logs','debug','agent','build'], …]`.
2. `test_the_fake_takes_the_warm_hold_ONLY_for_the_warm_argv` — runs the fake and
   observes what it DOES: `debug agent build` must take the hold and write the
   marker; the gate's `debug agent browser-agent` must not. Kills both fake-side
   survivors: placeholder-renamed FAILS, argv-slot-moved FAILS, pristine PASSES.

🔴 AND I SHIPPED THE HELPER WITHOUT BRANCHING ON IT. First cut defined
`_coupling_diagnosis` and never called it — the mutant still reported STARVED,
green-looking prose over a dead code path. Caught by running the mutant instead
of trusting the edit. That is "a field that exists is not a guard — only a BRANCH
on it is", committed while fixing a guard that claimed coverage it lacked.

🟡 The retraction was unqualified and contradicted a paragraph 8 lines above it.
"The gate never executes and no test ever paid a second sleep" is true of a GREEN
run and false on a LOST window — where the gate does run and does re-sleep
(measured unscoped: marker rewritten, delta ~3.05s at stalls 4/6/8/12s), which is
the bug the scoping fixes. Stated per path now; a maintainer quoting the old
sentence would have reverted the scoping.

🟡 The band table's third row said `> ~12 s` while my annotation ten lines below
said `~8s` — two values for one boundary, under an instruction to use them for
re-tuning. Re-measured (rc 2 at 4s/6s, rc 0 at 8s/12s): ~8s is right, the row was
stale. Row corrected and labelled as re-derived.

VERIFIED: control 2 passed; genuine 12s starvation still reports STARVED with
ZERO coupling false-positives; full target 915 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
…sis had no positive control; a timing assert lied

Round 4 returned three 🔴. All three are mine, and the first is the worst kind.

🔴 F1 — THE PR WAS RED ON `tekton/devrc-pytests` FOR THREE COMMITS AND I DID NOT
LOOK. I reported "CI green on both heads", which was true of the head I checked,
then pushed three more commits and never re-checked. Round 2's refactor hoisted
the fake's source into a module constant so `.replace()` could be applied — and
`test_no_test_writes_a_usr_bin_env_shebang_at_runtime` allowlists this file on the
needle `_write_exec(path,` appearing ON the shebang line. A hoisted constant
carries neither needle. Bisected: green at `f833e9c7`, red at `d6f0f268`.
Fixed by putting the substitution back on the INLINE literal rather than buying
an allowlist exemption for a refactor that bought nothing. ⚠ And the first
attempt at the fix failed the same guard again, because the COMMENT explaining it
spelled the shebang prefix — the exact trap that guard's own allowlist comments
record. Reworded; 9 passed.

🔴 F2 — THE ENTIRE DIAGNOSTIC SUBSYSTEM HAD NO POSITIVE CONTROL. Four mutations
each SURVIVED the full file: delete the fake's argv-log write, write it to a
different filename, make `_coupling_diagnosis` return "", and REMOVE ITS CALL
from the starved failure path. The last is the exact defect I caught by hand last
round and described as "fixed in the same commit" — it could regress with zero
signal, returning the file to starvation misattribution. A reassuring zero.
Now killed by two tests: the behavioural fake test asserts the argv log is
actually written beside the marker (kills the delete and the rename), and a new
`test_a_BROKEN_COUPLING_is_reported_as_one_and_NOT_as_starvation` drives the REAL
failure path with the coupling deliberately broken and asserts the message names
it (kills the inert function and the removed call site). Re-run: M5/M6/M7/M8 all
FAIL, control 68 passed. It runs in seconds because the await slice and attempt
count are now overridable.

🔴 F3 — I SHIPPED A LOAD-SENSITIVE ASSERTION WHOSE FAILURE MESSAGE IS A FALSE
ACCUSATION, in the file whose entire purpose is stopping that. `assert gate_took
< 0.9` fires at a 2.75s spawn baseline with the code correct, saying "the gate
slept; the hold is not scoped". It also routed through none of
`_spawn_baseline`/`_stall_extends`, violating this file's own rule six lines
away. DELETED — `assert not marker.exists()` two lines above is the deterministic
form of the same claim and kills the same mutants. `warm_took >= 0.9` stays: a
lower bound on a sleep(1) that load can only help.

🟡 F5 — the diagnosis never compared the logged argv to the keying it NAMES, so
it could assert a break while printing evidence of an intact coupling and tell
the reader to change correct code. Now returns "" when any logged invocation
matches.
🟡 F4 — three silent paths (no log / empty log / unparseable) fell through to
confident starvation wording. Each now reports COULD NOT MEASURE and says the
absence is not proof the coupling is intact.
🟡 F6 — the band table contradicted ITSELF: two prior rounds each fixed half of
it, leaving rows that overlapped 8-12s with mutually exclusive outcomes. Both
rows now carry the re-derived boundary. When a number appears twice, change it
twice — this table disproved "I fixed the number" three rounds running.
🟡 F7 — a comment pinned the coupling to a test this round deleted.
🟢 F8 — "900s -> ~180s" quoted the PER-ATTEMPT cap as the aggregate, understating
the win ~15x (the old aggregate was ~2700s).

VERIFIED: shebang guard 9 passed; browser-bridge target 916 passed; mutation
matrix M5/M6/M7/M8 all killed with control green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
…e never written, and a deleted assertion DID lose coverage

Round 5 reproduced the mutation matrix independently (M5-M8 all die with this
feature's own error strings) and confirmed claims 1, 2, 4 and 6. Two real
findings, and the first is a process failure in how I edit.

🔴 F-A — THREE EDITS I ASSERTED IN A COMMIT MESSAGE WERE NEVER WRITTEN.
Round 4's commit says "Both rows now carry the re-derived boundary. When a number
appears twice, change it twice — this table disproved 'I fixed the number' three
rounds running." It then disproved it a FOURTH time, in the commit claiming the
fix: row 2 still read `~3 s .. ~8 s`'s old value, so the bands overlapped 8-12s
with mutually exclusive outcomes, and the annotation 13 lines below described
that defect IN THE PAST TENSE while it was live.
MECHANISM, and it is mine not the file's: my edit scripts assert every
replacement and write ONCE AT THE END, so a single failed assertion silently
discards every earlier edit — and I then re-applied only the one that errored.
Three edits went missing that way (the table row, the annotation, and the ~2700s
aggregate figure). Fixed here by writing after EACH edit and verifying in the
FILE rather than reading the script's exit status. The table's history is now
recorded in the table, including this.

🟡 F-B — DELETING `assert gate_took < 0.9` DID LOSE COVERAGE, and my claim that
`not marker.exists()` "kills the same mutants" was measurably false. The audit
built M9: unscope the SLEEP while leaving the MARKER write scoped — a plausible
refactor of "scope the hold to the warm" — and the tool-set gate goes back to
sleeping the whole hold AFTER `_oc_lock_release`, which this file's own comments
call "the BUG this scoping fixes", with all 68 tests green.
The deletion was still right (the assertion was load-sensitive), so the fix is a
DETERMINISTIC replacement rather than restoring a timer: the fake now records
whether it TOOK the hold, and the test asserts the gate's invocation did not.
Re-run: M9 now FAILS (2 failed / 66 passed) where it survived at the audited
head; M5/M7/M8 still fail; control 68 passed.

🟡 F-C — `_WARM_AWAIT_SLICE_S` was the wrong knob. The coupling test drives
`_await` to exhaustion BY CONSTRUCTION, so on a stalled box it pays the whole
ladder — and shrinking the slice makes it WORSE, turning 3 baseline probes into
60 (measured 70.96s at slice 1.0 vs 60.86s at 20.0, load ~52). The stall cap is
what bounds it; it is now a constant, overridden by that test, and the docstring
says which knob does what.

🟢 The second `_await` site still carried the bare `20.0` literal; the constants
comment enumerated three constants with four following; and a comment
hard-wrapped a test identifier mid-word so grep found the `def` and missed the
reference — the stale-reference class that comment exists to prevent.

⚠ NOT FIXED, and stated rather than left implicit: `tekton/devrc-pytests` is red
at the previous head on `test_tmux_reply_agent.py::test_a_cwd_that_does_not_
exist_is_REFUSED_not_opened_in_HOME`. It passes at this head on the dev-host
tier, my diff has no import path to it, and `main` is green (22821 passed) — but
the red tier is the sandbox one and I have not run that, so it is UNATTRIBUTED,
not cleared. "The CI red I caused is fixed" and "CI is green" are different
claims and only the first is established. The branch is also ~39 commits behind
`origin/main`; the merged tree is ungated.

VERIFIED: shebang guard 9 passed; browser-bridge 916 passed; mutation matrix
M9/M5/M7/M8 all killed, control green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRDTqApwXvg5JwrLkYEdd5
Claude-Session-Id: 2b8df616-8c24-4dce-b0ba-ee5da91ce83f
@ZacxDev

ZacxDev commented Sep 12, 2026

Copy link
Copy Markdown
Member Author

Merging — ladder record, and what is NOT established

Six rounds (0 + five delta). Merging at d0b75715 on the operator's call with CI pending, not green — stated plainly because those are different things.

What the ladder found

round findings the defect
0 3 🟡 the fix silently deleted a hard precondition (assert lock.is_dir()), rerouting a real lock-ordering regression into "read this as machine load"; a predicate open-coded over _process_gone; machine load asserted, not measured
1 1 🔴 the defect verdict fired on purely starved runs — the fake's debug agent branch serves the warm and the tool-set gate, so the gate reset the mtime clock after the lock was released
2 4 🟡 scoping created an unpinned coupling to the wrapper's argv; the retry multiplied the pathological wall time ×3; a "second sleep removed" saving that never existed
3 1 🔴 the guard added in round 2 was spelled, not structural — green in 4 of 6 mutant shapes, with a docstring claiming to assert "both sides" while reading one
4 3 🔴 CI was red for three commits and nobody looked; the whole diagnostic subsystem had no positive control (4 mutations deleted it with the suite green); a load-sensitive assert whose failure text is a false accusation
5 1 🔴 1 🟡 three edits asserted in a commit message were never written (multi-edit script wrote only at the end); deleting the timing assert did lose coverage — M9 proved it

🔴 Every round after round 0 found its defect in code or prose the previous round wrote while fixing the round before it. That is the documented pattern, hit six times consecutively, and it is the most reusable thing this PR produced.

Final state, measured

  • Mutation matrix, re-run at head: M5 / M7 / M8 / M9 all killed, control 68 passed. M9 is the one that survived at the round-5 audited head.
  • test_runtime_shebangs.py 9 passed — the CI red this PR caused is fixed at the root, with no allowlist exemption added (git diff origin/main...HEAD -- scripts/tests/test_runtime_shebangs.py is empty).
  • browser-bridge target 916 passed.

🔴 NOT established — read this before trusting the merge

  1. CI was PENDING at merge time. Not green. The prior head was red on test_tmux_reply_agent.py::test_a_cwd_that_does_not_exist_is_REFUSED_not_opened_in_HOME — a real_tmux test with no import path from this diff, passing here on the dev-host tier, with main green (22821 passed). That makes it unattributed, not cleared: the red tier is the sandbox one and it was never run against this head.
  2. The merged tree is ungated. The branch was ~39 commits behind origin/main; strict is off by design, so a green check is a claim about the branch, never about the tree the merge creates.
  3. The three-band table describes the unscoped fake and is historical; current bands are beside it.

Detection after the fact is main-green-check (4-hourly, reproduces a red before alerting) plus main-status-watch.

@ZacxDev
ZacxDev merged commit 22ddd8d into main Sep 12, 2026
0 of 3 checks passed
@ZacxDev
ZacxDev deleted the fix/release-handler-split-verdict branch September 12, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant