Skip to content

feat(eval): record provider-resolved model id separate from requested alias - #8

Merged
hsskey merged 4 commits into
mainfrom
fm/tokenloom-eval-provenance
Sep 17, 2026
Merged

hsskey merged 4 commits into
mainfrom
fm/tokenloom-eval-provenance

Conversation

@hsskey

@hsskey hsskey commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Why

Evaluation run records stored only the requested model alias (for example opus), so a report could not tell which concrete provider model actually produced the scored output. That made cross-model comparison and later rescoring unreliable. This change records the provider-returned resolved model ID separately from the requested alias, keeps every legacy row valid and readable, and never lets rows with different model provenance share a comparison partition.

What changed

  • The real Claude adapter runs with --output-format stream-json --verbose and parses the final result line for usage, cost, and text.
  • LlmResult and EvaluationRun gain additive provenance fields: requestedModel, resolvedModel, modelResolution, per-call providerEvidence, and harnessCommit. One-shot rows also embed an artifact (output plus locked inputs) so a later scorer change can rescore from stored output without a new model call.
  • Model resolution follows the SPEC 9.1 J3 rules: a producing-message ID when every main-loop assistant message names one identical model, a sole modelUsage key when no producing message exists, and null otherwise. Contradictory producing evidence resolves to null rather than falling back to a usage key, so a known contradiction is never hidden. No alias substring or cost heuristic is used as identity; raw provider evidence is preserved verbatim.
  • model keeps its documented meaning (resolvedModel ?? requestedModel), so legacy rows continue to parse and group as before.
  • One-shot report sections partition by prompt hash, model, requested model, resolved model, harness commit and dirty state, and reference-lock commit. null never merges with a value. Legacy rows lack the added parts and keep their existing sections.
  • docs/reference/spec.md sections 9.1, 9.2, and 9.6 describe the new fields, the resolution order, the stream-json invocation, and the one-shot section key.

The trajectory partition, both trajectory reports, and the verify gate are intentionally untouched; this change records the row-level provenance fields only.

Follow-up commits on this branch: the explanatory comments this change had added were removed to match the change-history-only comment policy (behavior unchanged); and the one-shot section-key separator, which had been written as a literal NUL byte in the source (making the file diff as binary), now uses the same Unicode U+0000 escape as groupKey for the identical runtime separator and a text-clean source.

How I checked

pnpm lint
pnpm typecheck
pnpm build
pnpm tsx scripts/reference-lock.ts --check
pnpm verify --gate scope
pnpm verify --gate scoring
pnpm verify --gate types
pnpm exec vitest run packages/eval packages/verify scripts apps/cli
pnpm test

Results: lint, typecheck, and build clean; reference-lock --check reports no changed reference output; scope, scoring, and types gates pass; pnpm test passes 1335 tests across 75 files (net +18 provenance tests over the 1317 baseline). New tests assert resolved-ID persistence, the alias/resolved-ID distinction, partition separation for differing IDs, and legacy-row readability.

Full pnpm verify (all gates, includes --selftest) was then run from the branch tip: 12 of 13 gates pass (types, patterns, tests, reference, determinism, properties, mutations, rules, benchmarks, scope, scoring, encoding). The selftest gate failed once on a transient host-load timing collateral inside its own benchmark probe (allocation 1.842x vs the 1.5x limit, self-labelled environment: probe is slower than baseline); the top-level benchmarks gate itself passed. An isolated re-run of pnpm verify --selftest then returned PASS with 27 of 27 samples as-expected and zero failures, confirming the failure was an environment-timing flake, not a code defect. The provenance change does not affect allocation performance.

No incidental verify or benchmark artifacts are included: the run's appended bench/results.jsonl and its verify/*.json proof were reverted so the branch is the three provenance commits only.

What I left alone

No gate threshold, forbidden pattern, dependency allowlist, or self-test expectation was weakened. No committed run record, report, or benchmark result was rewritten, and no vitest -u was run. No real evaluation calls were made; all stream-json behavior is exercised against fixtures only. The trajectory partition, reports, and verify gate are unchanged.

… alias

Record the provider-returned model id as `resolvedModel` alongside the
requested matrix alias as `requestedModel`, so evaluation provenance no
longer conflates the two. `model` keeps its documented meaning
(`resolvedModel ?? requestedModel`), and every legacy run row stays valid
and readable.

`resolvedModel` is established only by the SPEC 9.1 J3 rules from the
provider stream: one identical model named by every main-loop assistant
message, or a sole `modelUsage` key when no such message exists, otherwise
null. Contradictory producing evidence never falls back to a usage key, so a
known disagreement is not hidden, and no alias substring or cost heuristic is
used. Each call keeps its raw `providerEvidence`.

The real and session adapters move to `--output-format stream-json --verbose`
and read usage, cost, and text from the final `result` line. Rows also gain
`harnessCommit`, a one-shot `artifact` (output plus locked inputs for later
rescoring), and, for trajectory rows, a row-level `referenceLockCommit`.
One-shot report sections are keyed by prompt hash, model, requested model,
resolved model, harness commit, and reference-lock commit, so different model
provenance never shares a comparison table.

No provider evaluation call was made; stream metadata is exercised only
against fixtures. The trajectory partition, both trajectory reports, and the
verify gates are left to T4 and untouched here.
Remove two comments that restated the adjacent statement and the SPEC
9.1 J3 rule restatement, keeping only the non-obvious contradiction
guard. Aligns the provenance change with the change-history comment
policy without touching behavior.
Per the latest explicit comment policy (only change-history-context
comments are kept), remove the generic invariant and spec-restatement
comments this change had added across the eval provenance code. Behavior
is unchanged; SPEC 9.1 and 9.6 remain the owning contract for the rules
these comments described.
… NUL byte

The one-shot section key joined its parts with a literal NUL byte in the
source, which made git treat stats.ts as binary and its diffs
unreviewable. Use the same Unicode U+0000 escape that groupKey uses, for
the identical runtime separator and a text-clean source.
@hsskey
hsskey merged commit 2dcb57f into main Sep 17, 2026
2 checks passed
@hsskey
hsskey deleted the fm/tokenloom-eval-provenance branch September 17, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant