Skip to content

Log every stop/nudge intervention (CL-6938) - #561

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-6938-salvage-false-positive-rate-is-unmeasurable-log-every
Aug 23, 2026
Merged

Log every stop/nudge intervention (CL-6938)#561
TheGreatAxios merged 1 commit into
mainfrom
cl-6938-salvage-false-positive-rate-is-unmeasurable-log-every

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Closes CL-6938. Stacked on #559 (which is stacked on #558) — same files.

This is the gate the project's standing rule points at: no threshold retuning ships before CL-6938 lands.

What it records

One JSONL record per intervention, in the firing worker's own trace dir (interventions.jsonl):

  • intervention id + class (stop / nudge / block / clear)
  • the trigger's measured value beside its thresholdconsecutiveIdentical=5 threshold=5, silenceMs=…, turnsCompleted=…
  • provider, model, resolved model family, role (leaf / orchestrator / parent), intent
  • run state at the decision: turns used vs budget, total tool calls, read count, edited-path count

Wired at: no-progress, turn-budget, never-acted, never-edited, no-ship, incomplete-report (nudge and stop), report-forced, tool-failure-recovery, stall-nudge, stalled, and the run-loop-owned repetition / deadline / cancelled salvages. Refused parent re-dispatches are recorded from task-tool.ts, since no leaf run exists there to record them.

Writes are fire-and-forget and swallow their own errors — a diagnostic must not be able to fail a run — and the sink defaults to a no-op, so nothing depends on logging being wired.

Reading it back

bun run scripts/intervention-forensics.ts aggregates across local sessions:

  • per-intervention counts, split by model family (the axis model-family-policy.ts has no data for today — kimi is a verbatim copy of default)
  • measured-value distribution (p50/p90/max) against the thresholds seen
  • two false-positive proxies: edited = stops that fired on a run which had already edited files; early = stops that fired before half the turn budget was spent

Neither proxy is proof alone; both are cheap and directional, which is what the tuning history has been missing. It lstats and skips symlinks, so the latest session link cannot double-count.

Verification

bun run check green: 5277 pass / 0 fail. New tests cover the record shape, append ordering, failure-swallowing, and that the director actually emits a nudge and a stop with value ≥ threshold.

Review

needs-sawyer-review — Guardrail 4: touches director/stop-policy internals. Not for auto-merge.


Review fix applied: two changes. (a) Relabeled editedWork/earlyBudget in scripts/intervention-forensics.ts and the ARCHITECTURE.md paragraph as neutral context columns, not false-positive proxies — either is equally consistent with a correct stop or a wrong one. (b) Added a real outcome signal: the parent now appends one outcome intervention record per completed dispatch (salvage kind or clean-complete, plus dispatch count), added outcome to InterventionClass, and taught intervention-forensics.ts to report outcome counts by kind. CHANGELOG and ARCHITECTURE updated to describe the outcome records.

@linear-code

linear-code Bot commented Aug 23, 2026

Copy link
Copy Markdown

CL-6938

@TheGreatAxios
TheGreatAxios force-pushed the cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real branch from 5c8466a to bdb9ced Compare August 23, 2026 19:49
@TheGreatAxios
TheGreatAxios force-pushed the cl-6938-salvage-false-positive-rate-is-unmeasurable-log-every branch from 7390f7e to ee51560 Compare August 23, 2026 19:50
Fourteen stop reasons and ~20 injected-text interventions decide when a
run is stuck, and there was no way to tell how often any of them was
wrong. Four threshold judgments in this tree were later reverted, one on
a justification the file itself retracts.

interventions.jsonl in the firing worker's trace dir now records each
intervention with its measured value beside the threshold it crossed, the
provider/model/family, and the run state at that moment — turns used vs
budget, tool calls, read and edit counts. A refused parent re-dispatch is
recorded on the parent side, where no leaf run exists to record it.

Writes are fire-and-forget and swallow their own errors: a diagnostic
must not be able to fail a run. The sink defaults to a no-op, so nothing
depends on logging being wired.

scripts/intervention-forensics.ts aggregates across local sessions: per
intervention counts by family, measured-value distribution against
threshold, and two false-positive proxies — stops that fired on runs
which had already edited files, and stops that fired before half the turn
budget was spent. It lstats and skips symlinks so the `latest` session
link cannot double-count.

Stacked on cl-6937.
@TheGreatAxios
TheGreatAxios force-pushed the cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real branch from bdb9ced to 4fd8c94 Compare August 23, 2026 19:57
@TheGreatAxios
TheGreatAxios force-pushed the cl-6938-salvage-false-positive-rate-is-unmeasurable-log-every branch from ee51560 to 7b8d247 Compare August 23, 2026 19:57
@TheGreatAxios
TheGreatAxios changed the base branch from cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real to main August 23, 2026 20:00
@TheGreatAxios
TheGreatAxios merged commit 60a8e7a into main Aug 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant