Skip to content

Tag dispatch outcome records with model identity - #590

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-6968-dispatch-outcome-records-carry-no-model-tag-so-no-intervention-rate
Aug 24, 2026
Merged

Tag dispatch outcome records with model identity#590
TheGreatAxios merged 1 commit into
mainfrom
cl-6968-dispatch-outcome-records-carry-no-model-tag-so-no-intervention-rate

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Closes CL-6968. Dispatch outcome records in interventions.jsonl were written parent-side (task-tool.ts) with no provider/model/family tag, so every per-model intervention count had no dispatch denominator: "grok aborted 40 times" was unusable without knowing whether that was out of 50 dispatches or 5,000.

Record shape change

Each class: "outcome" record now carries provider / model / family, taken from the resolved provider the dispatch actually ran under. task-tool.ts already resolves the final provider/model before calling run() (profile/agent inference resolution happens up front), and that is the same value run.ts uses for the leaf's own intervention-log context — so the two stay in sync by construction.

One wrinkle found while confirming this: a dispatch can fail over mid-run to a different configured source (the reactor retries the next entry in the priority-ordered source list on a retryable error). The pre-dispatch resolved provider can therefore diverge from what actually served inference. The outcome record now prefers the last inference.done event's source (forwarded via the task tool's existing onEvent hook) over the pre-dispatch value, falling back to the resolved provider when no inference.done was observed.

{"id":"dispatch-outcome","class":"outcome","outcome":{"kind":"clean-complete","dispatchCount":1},"provider":"xai","model":"grok-4.6","family":"grok"}

No new free-text fields — provider/model/family are the same short identifiers the leaf side already logs, and writes stay fire-and-forget/best-effort.

Forensics output

scripts/intervention-forensics.ts now reports interventions-per-dispatch per model as an explicit rate, with both the numerator and the denominator visible on the same line:

interventions per dispatch by model (stop+nudge count / dispatch count = rate)
claude-sonnet-5                   interventions=   1 dispatches=    2 rate=0.500
grok-4.6                          interventions=   1 dispatches=    2 rate=0.500

Outcome records written before this change carry no model tag; they're counted separately (N outcome record(s) predate model tagging...) rather than silently folded into a misleading total.

The existing "repetition aborts by model" table is left as a raw count (it always was, and says so), now with a comment pointing at the new rate table instead of claiming the denominator isn't tracked. No column anywhere reports a ratio without a tracked denominator behind it — the edited/early context columns keep their existing "not a false-positive rate" disclaimer.

Test plan

  • bun run check (lint, typecheck, build, test) — green
  • Manually verified the new forensics table renders a correct rate against a synthetic interventions.jsonl fixture (2 dispatches / 1 intervention per model → rate 0.500 for both)

Outcome records in interventions.jsonl were written parent-side with no
provider/model/family tag, so per-model intervention counts had no
dispatch denominator to divide by. Tag each outcome with the provider/
model that actually served inference (preferring the last inference.done
source over the pre-dispatch resolved provider, since a mid-run failover
can diverge from it), and teach intervention-forensics.ts to report
interventions-per-dispatch per model as an explicit rate alongside the
existing raw counts.
@linear-code

linear-code Bot commented Aug 24, 2026

Copy link
Copy Markdown

CL-6968

@TheGreatAxios
TheGreatAxios merged commit 91639eb into main Aug 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant