Tag dispatch outcome records with model identity - #590
Merged
TheGreatAxios merged 1 commit intoAug 24, 2026
Merged
Conversation
Outcome records in interventions.jsonl were written parent-side with no provider/model/family tag, so per-model intervention counts had no dispatch denominator to divide by. Tag each outcome with the provider/ model that actually served inference (preferring the last inference.done source over the pre-dispatch resolved provider, since a mid-run failover can diverge from it), and teach intervention-forensics.ts to report interventions-per-dispatch per model as an explicit rate alongside the existing raw counts.
TheGreatAxios
enabled auto-merge
August 24, 2026 00:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes CL-6968. Dispatch outcome records in interventions.jsonl were written parent-side (task-tool.ts) with no provider/model/family tag, so every per-model intervention count had no dispatch denominator: "grok aborted 40 times" was unusable without knowing whether that was out of 50 dispatches or 5,000.
Record shape change
Each
class: "outcome"record now carriesprovider/model/family, taken from the resolved provider the dispatch actually ran under. task-tool.ts already resolves the final provider/model before callingrun()(profile/agent inference resolution happens up front), and that is the same valuerun.tsuses for the leaf's own intervention-log context — so the two stay in sync by construction.One wrinkle found while confirming this: a dispatch can fail over mid-run to a different configured source (the reactor retries the next entry in the priority-ordered source list on a retryable error). The pre-dispatch resolved provider can therefore diverge from what actually served inference. The outcome record now prefers the last
inference.doneevent's source (forwarded via the task tool's existingonEventhook) over the pre-dispatch value, falling back to the resolved provider when no inference.done was observed.{"id":"dispatch-outcome","class":"outcome","outcome":{"kind":"clean-complete","dispatchCount":1},"provider":"xai","model":"grok-4.6","family":"grok"}No new free-text fields — provider/model/family are the same short identifiers the leaf side already logs, and writes stay fire-and-forget/best-effort.
Forensics output
scripts/intervention-forensics.tsnow reports interventions-per-dispatch per model as an explicit rate, with both the numerator and the denominator visible on the same line:Outcome records written before this change carry no model tag; they're counted separately (
N outcome record(s) predate model tagging...) rather than silently folded into a misleading total.The existing "repetition aborts by model" table is left as a raw count (it always was, and says so), now with a comment pointing at the new rate table instead of claiming the denominator isn't tracked. No column anywhere reports a ratio without a tracked denominator behind it — the
edited/earlycontext columns keep their existing "not a false-positive rate" disclaimer.Test plan
bun run check(lint, typecheck, build, test) — greeninterventions.jsonlfixture (2 dispatches / 1 intervention per model → rate 0.500 for both)