Refreeze manuscript on the 33-model board - #161
Merged
Conversation
Snapshot 2026-09-01: Claude Fable 5.1 joins at exact 86.3, second of 33.
Frozen run exports, analysis CSVs, serving-config table, and annotations
re-derived from results/local/fable51/publish; the frozen data.json
recombines byte-exact to the dashboard-data-20260901 artifact
(1b2de096…f3dea, 87,392,615 bytes). Annotation expectations move to
7,840 wrong rows (7,211 llm_error, 629 parse) for the 33-model roster.
Prose: the tool_choice paragraph notes that Fable 5.1 rejects forced
tool use, so its row answers through the JSON transport and reasons at
the provider default, with the auto sensitivity 1.2 points above it. The
open-weight lag sentence is now count-aware ("trails two closed models;
the earliest released of them, GPT-5.6 Sol") because "trails only Sol"
became false once Fable 5.1 scored above Kimi K3; the lag numbers are
unchanged (7 days by API, 18 days by weights; 128 days pre-July).
freeze_snapshot.py resolves the source run from the main clone whether
it runs there or from a worktree beside it, and appends the case-note
supplement only when the row is absent (the repo copy already carries
it). The /paper page labels, which had drifted to 2026-08-17, follow the
snapshot date.
Rendered HTML and PDF (33 pages) re-pinned with --rendered-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #160 (retargets to
mainwhen that merges).What
Refreezes
paper/snapshot/20260501on the 33-model board published asdashboard-data-20260901. Standing rule: the manuscript refreezes with every model add.model_serving_config.json, and annotation pins re-derived from the publish run; the frozendata.jsonrecombines byte-exact to the published artifact (1b2de096…f3dea, 87,392,615 bytes).llm_error, 629 parse).claude-fable-5-1, released 2026-09-01, 1,984/1,984 parsed, whole scenario, provider-default reasoning.tool_choiceparagraph records that Fable 5.1 rejects forced tool use, so its row answers through the JSON transport and reasons at the provider default, with theautosensitivity 1.2 points above it. The open-weight lag sentence is now count-aware ("trails two closed models; the earliest released of them, GPT-5.6 Sol…") because "trails only Sol" became false once Fable 5.1 scored above Kimi K3. Lag numbers are unchanged: 7 days by API availability, 18 by weights, 128 for the pre-July cut.freeze_snapshot.pyresolves the source run from the main clone whether it runs there or from a worktree beside it, and appends the Ox Alpha case-note supplement only when the row is absent (the repo copy already carries it; the old unconditional append would have duplicated it)./paperpage labels had drifted to 2026-08-17 while the site said 2026-08-22; they now follow the snapshot date.Verification
scripts/freeze_snapshot.pyran clean;--rendered-onlyre-pinned the render.pytestin the worktree against the new snapshot: 594 passed, 5 skipped (recombination, completeness, serving-config, rendered-paper hash tests included).bun test86/86, eslint clean.🤖 Generated with Claude Code