Add Claude Fable 5.1 (#2 of 33), publish dashboard-data-20260901b, refreeze the manuscript - #160
Open
MaxGhenis wants to merge 4 commits into
Open
Add Claude Fable 5.1 (#2 of 33), publish dashboard-data-20260901b, refreeze the manuscript#160MaxGhenis wants to merge 4 commits into
MaxGhenis wants to merge 4 commits into
Conversation
Fable 5.1 (released 2026-09-01) rejects forced tool use: tool_choice
type "tool" or "any" returns a 400, because adaptive thinking is always
on and a forced call would skip it. The gauntlet confirmed the 400 and
passed the JSON contract 3/3 and 16/16 whole-scenario, so the card runs
the JSON contract like Kimi K3 and Qwen 3.8 Max. Under litellm the JSON
contract sends the plain prompt (response_format json_object maps to no
Anthropic parameter), so this board row reasons at the API default,
unlike Fable 5's forced-tool row. The card notes say so.
Register claude-fable-5-1 with litellm locally, the way claude-fable-5
already is, so routing and cost reconstruction do not depend on the
remote price-map fetch (PyPI's litellm 1.99.0 predates the upstream
entry).
Two harness changes make the thinking sensitivity run possible on a
JSON-contract model and correct for every Claude:
- POLICYBENCH_CONTRACT_OVERRIDE=tool declares the answer tool on a
JSON-contract model so POLICYBENCH_TOOL_CHOICE=auto has a tool to
leave to the model. The supervisor fingerprints the contract it sent.
- The explanation-repair builder now honors POLICYBENCH_TOOL_CHOICE.
It forced the tool regardless, which would 400 on Fable 5.1 and
silently drop thinking on repair calls for other Claudes.
Tests: lock row for claude-fable-5-1 ("json", None, 600, 16_384);
registry cross-checks (every card belongs to a roster id under its own
key, every price override names a roster model); contract override and
repair-builder tests; fingerprint test; Fable line resolves its provider
without the remote map. App: label, order, release date, frontier group.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…0901) Fable 5.1 debuts at exact 86.3, #2 of 33, behind GPT-5.6 Sol (88.7) and a tenth ahead of Kimi K3 (86.2): 1,984/1,984 parsed, $0.257 per household, $25.70 total, median 49s, no budget escalations. Incumbent modelStats are byte-identical to the 20260822 publish. The row runs the JSON contract because the API rejects forced tool use on this model; under litellm that request is the plain prompt, so the row reasons at the API default. Its thinking sensitivity run (tool declared, tool_choice auto) scores 87.5, +1.2 over the JSON row, isolating request shape under thinking; predictions and per-variable rates are attached to the release and written up in sensitivity/claude-thinking-2026-08.md. Audit: 146 new wrong rows judged (all llm_error; every case already had another wrong model, so no new cases), round one decisive, zero reference suspects, strict gate zero errors. Re-rendering those 146 cases' prompts re-judged their incumbent rows too: failure_source unchanged on all 7,694, failure_subtype changed on 808 (10.5%), diagnosis text on 4,015 — the case-level invalidation already documented for earlier adds. Pointer, manifest live entry, annotation re-pin, versions.json count and hero label move together; the frozen 20260822 pin does not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Snapshot 2026-09-01: Claude Fable 5.1 joins at exact 86.3, second of 33.
Frozen run exports, analysis CSVs, serving-config table, and annotations
re-derived from results/local/fable51/publish; the frozen data.json
recombines byte-exact to the dashboard-data-20260901 artifact
(1b2de096…f3dea, 87,392,615 bytes). Annotation expectations move to
7,840 wrong rows (7,211 llm_error, 629 parse) for the 33-model roster.
Prose: the tool_choice paragraph notes that Fable 5.1 rejects forced
tool use, so its row answers through the JSON transport and reasons at
the provider default, with the auto sensitivity 1.2 points above it. The
open-weight lag sentence is now count-aware ("trails two closed models;
the earliest released of them, GPT-5.6 Sol") because "trails only Sol"
became false once Fable 5.1 scored above Kimi K3; the lag numbers are
unchanged (7 days by API, 18 days by weights; 128 days pre-July).
freeze_snapshot.py resolves the source run from the main clone whether
it runs there or from a worktree beside it, and appends the case-note
supplement only when the row is absent (the repo copy already carries
it). The /paper page labels, which had drifted to 2026-08-17, follow the
snapshot date.
Rendered HTML and PDF (33 pages) re-pinned with --rendered-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, coherent snapshot Peer review (Sol, gate round 2) found three things wrong with the publish, all real: 1. The site still claimed an identical forced-tool request for every model. The versions.json 1.1 description, the leaderboard's serving-sensitivity callout, the Methodology task card, and the sensitivity doc's opener now state the actual condition: identical prompt for every model; forced answer-schema tool call where the provider accepts one; a JSON object where it does not (Kimi K3, Qwen 3.8 Max, DeepSeek, the older Gemini models, Claude Fable 5.1). The callout names Fable 5.1's exception and recomputes the would-rank positions against 33 models. 2. Case notes were copied, not regenerated, so wrong_model_count was stale for every case Fable 5.1 joined (13 vs 14 rows on scenario_039/head_medicare_eligible). They are rebuilt from the same collect_audit pass as the row annotations, and a new test asserts one case per wrong (scenario, output) pair with wrong_model_count equal to the annotated rows. 3. The published payload embeds case-level labels from the case notes, so dashboard-data-20260901 carried stale ones (6,354 case-annotation, 1,220 case-subtype, and 4 case-source leaves; scores identical). The payload is re-exported and republished as dashboard-data-20260901b (sha256 1ff522e0...7e6a, 87,410,551 bytes); the pointer, the manifest's live entry and published pin, the frozen data.json, and the rendered paper follow it. A new test asserts every row annotation is a wrong cell of the frozen roster with counts equal, so the annotation pin cannot drift ahead of the snapshot it certifies. The refreeze is folded into this branch so main never holds a 33-model live pointer against a 32-model frozen snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Claude Fable 5.1 (
claude-fable-5-1, released 2026-09-01) to the board, publishes the 33-model release, and refreezes the manuscript on it — one PR somainnever holds a 33-model live pointer against a 32-model frozen snapshot (peer review round 2). Two harness gaps the add exposed are fixed.Result
tool_choice: auto(sensitivity)Sol stays #1 at 88.7; Kimi K3 (86.2) moves to #3. Incumbent
modelStatsare byte-identical to the 20260822 publish.Release
dashboard-data-20260901b:dashboard-data.jsonsha2561ff522e0…7e6a, 87,410,551 bytes;predictions.csv.gz(33 models); two Fable 5.1 sensitivity assets.dashboard-data-20260901is superseded: its payload embedded case-level labels from case notes that had not been regenerated after the judge sweep (scores identical; 6,354 case-annotation, 1,220 case-subtype, and 4 case-source leaves differ).Fable 5.1 runs the JSON contract. The API rejects forced tool use on this model:
tool_choicetypetooloranyreturns a 400, because adaptive thinking is always on and a forced call would skip it (what's new). The gauntlet reproduced the 400 and passed the JSON contract 3/3 and 16/16 whole-scenario — the accommodation Kimi K3 and Qwen 3.8 Max already have. Under litellm 1.88.1 the JSON contract sends only the prompt (response_format: json_objectmaps to no Anthropic parameter; checked offline against the transformation), so this row reasons at the API default, unlike Fable 5's forced-tool row. Card notes, the sensitivity doc's new Fable 5.1 section, and the paper'stool_choiceparagraph record this.Site copy now states the actual condition (peer review): identical prompt for every model; forced answer-schema tool call where the provider accepts one; a JSON object where it does not (Kimi K3, Qwen 3.8 Max, DeepSeek, the older Gemini models, Claude Fable 5.1). versions.json, the leaderboard's serving-sensitivity callout (with would-rank positions recomputed against 33 models), the Methodology task card, and the sensitivity doc's opener.
Audit. 146 new wrong rows, all
llm_error(top subtype: taxable income or deductions, 53). Every one was already a case another model missed, so no new cases; one judge round was decisive, zero reference suspects, strict gate zero errors. Re-rendering those cases' prompts re-judged their incumbent rows:failure_sourceunchanged on all 7,694,failure_subtypechanged on 808, diagnosis text on 4,015 — the case-level invalidation documented on earlier adds. Case notes are regenerated from the same collect pass (peer review), sowrong_model_countmatches the rows for all 936 cases.Refreeze
paper/snapshot/20260501is the 33-model snapshot dated 2026-09-01: frozen run exports, analysis CSVs, serving-config table, and annotations re-derived from the publish run; frozendata.jsonrecombines byte-exact to the 20260901b artifact; expectations 7,840 wrong rows (7,211llm_error, 629 parse). Table 2 gains the Fable 5.1 row from the harness registry. The open-weight lag sentence is count-aware ("trails two closed models; the earliest released of them, GPT-5.6 Sol…") because "trails only Sol" became false once Fable 5.1 scored above Kimi K3; the lag numbers are unchanged (7 days by API, 18 by weights, 128 pre-July). The/paperpage labels, which had drifted to 2026-08-17, follow the snapshot date. Rendered PDF (33 pages) and HTML re-pinned.Harness changes
claude-fable-5-1, as forclaude-fable-5.POLICYBENCH_CONTRACT_OVERRIDE(toolorjson): lets a JSON-contract model declare the answer tool soPOLICYBENCH_TOOL_CHOICE=autohas a tool to leave to the model; the supervisor fingerprint records the contract sent.POLICYBENCH_TOOL_CHOICEthrough the shared_tool_choice_for; it forced the tool regardless before (a 400 on Fable 5.1; silently dropped thinking on repair calls for other Claudes underauto). The Responses-API builder still forces (gpt-5 family only; tracked in v2 board condition: move every model to tool_choice auto #139).freeze_snapshot.pyresolves the source run from the main clone or a worktree beside it, and appends the Ox Alpha case-note supplement only when absent.Tests
claude-fable-5-1= (json, None, 600, 16_384); registry cross-checks (card keys are live roster ids equal to theirlitellm_id; price overrides name roster models); contract override; repair builder under forced andauto; supervisor fingerprint; Fable-line provider resolution.wrong_model_countequals annotated rows).ruff checkclean; fullpytest596 passed, 5 skipped; appbun test86/86; eslint clean.🤖 Generated with Claude Code