feat(eval): record provider-resolved model id separate from requested alias - #8
Merged
Merged
Conversation
… alias Record the provider-returned model id as `resolvedModel` alongside the requested matrix alias as `requestedModel`, so evaluation provenance no longer conflates the two. `model` keeps its documented meaning (`resolvedModel ?? requestedModel`), and every legacy run row stays valid and readable. `resolvedModel` is established only by the SPEC 9.1 J3 rules from the provider stream: one identical model named by every main-loop assistant message, or a sole `modelUsage` key when no such message exists, otherwise null. Contradictory producing evidence never falls back to a usage key, so a known disagreement is not hidden, and no alias substring or cost heuristic is used. Each call keeps its raw `providerEvidence`. The real and session adapters move to `--output-format stream-json --verbose` and read usage, cost, and text from the final `result` line. Rows also gain `harnessCommit`, a one-shot `artifact` (output plus locked inputs for later rescoring), and, for trajectory rows, a row-level `referenceLockCommit`. One-shot report sections are keyed by prompt hash, model, requested model, resolved model, harness commit, and reference-lock commit, so different model provenance never shares a comparison table. No provider evaluation call was made; stream metadata is exercised only against fixtures. The trajectory partition, both trajectory reports, and the verify gates are left to T4 and untouched here.
Remove two comments that restated the adjacent statement and the SPEC 9.1 J3 rule restatement, keeping only the non-obvious contradiction guard. Aligns the provenance change with the change-history comment policy without touching behavior.
Per the latest explicit comment policy (only change-history-context comments are kept), remove the generic invariant and spec-restatement comments this change had added across the eval provenance code. Behavior is unchanged; SPEC 9.1 and 9.6 remain the owning contract for the rules these comments described.
… NUL byte The one-shot section key joined its parts with a literal NUL byte in the source, which made git treat stats.ts as binary and its diffs unreviewable. Use the same Unicode U+0000 escape that groupKey uses, for the identical runtime separator and a text-clean source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Evaluation run records stored only the requested model alias (for example
opus), so a report could not tell which concrete provider model actually produced the scored output. That made cross-model comparison and later rescoring unreliable. This change records the provider-returned resolved model ID separately from the requested alias, keeps every legacy row valid and readable, and never lets rows with different model provenance share a comparison partition.What changed
--output-format stream-json --verboseand parses the finalresultline for usage, cost, and text.LlmResultandEvaluationRungain additive provenance fields:requestedModel,resolvedModel,modelResolution, per-callproviderEvidence, andharnessCommit. One-shot rows also embed anartifact(output plus locked inputs) so a later scorer change can rescore from stored output without a new model call.modelUsagekey when no producing message exists, andnullotherwise. Contradictory producing evidence resolves tonullrather than falling back to a usage key, so a known contradiction is never hidden. No alias substring or cost heuristic is used as identity; raw provider evidence is preserved verbatim.modelkeeps its documented meaning (resolvedModel ?? requestedModel), so legacy rows continue to parse and group as before.nullnever merges with a value. Legacy rows lack the added parts and keep their existing sections.docs/reference/spec.mdsections 9.1, 9.2, and 9.6 describe the new fields, the resolution order, the stream-json invocation, and the one-shot section key.The trajectory partition, both trajectory reports, and the verify gate are intentionally untouched; this change records the row-level provenance fields only.
Follow-up commits on this branch: the explanatory comments this change had added were removed to match the change-history-only comment policy (behavior unchanged); and the one-shot section-key separator, which had been written as a literal NUL byte in the source (making the file diff as binary), now uses the same Unicode U+0000 escape as
groupKeyfor the identical runtime separator and a text-clean source.How I checked
Results: lint, typecheck, and build clean;
reference-lock --checkreports no changed reference output;scope,scoring, andtypesgates pass;pnpm testpasses 1335 tests across 75 files (net +18 provenance tests over the 1317 baseline). New tests assert resolved-ID persistence, the alias/resolved-ID distinction, partition separation for differing IDs, and legacy-row readability.Full
pnpm verify(all gates, includes--selftest) was then run from the branch tip: 12 of 13 gates pass (types,patterns,tests,reference,determinism,properties,mutations,rules,benchmarks,scope,scoring,encoding). Theselftestgate failed once on a transient host-load timing collateral inside its own benchmark probe (allocation 1.842x vs the 1.5x limit, self-labelledenvironment: probe is slower than baseline); the top-levelbenchmarksgate itself passed. An isolated re-run ofpnpm verify --selftestthen returned PASS with 27 of 27 samples as-expected and zero failures, confirming the failure was an environment-timing flake, not a code defect. The provenance change does not affect allocation performance.No incidental verify or benchmark artifacts are included: the run's appended
bench/results.jsonland itsverify/*.jsonproof were reverted so the branch is the three provenance commits only.What I left alone
No gate threshold, forbidden pattern, dependency allowlist, or self-test expectation was weakened. No committed run record, report, or benchmark result was rewritten, and no
vitest -uwas run. No real evaluation calls were made; all stream-json behavior is exercised against fixtures only. The trajectory partition, reports, and verify gate are unchanged.