Add Claude Fable 5.1 (#2 of 33), publish dashboard-data-20260901b, refreeze the manuscript

app/src/app/paper/page.tsx

@@ -4,10 +4,10 @@ import Link from "next/link";
 
 import SiteHeader from "../../components/SiteHeader";
 
-const SNAPSHOT_DATE_LABEL = "Snapshot 2026-08-17";
+const SNAPSHOT_DATE_LABEL = "Snapshot 2026-09-01";
 
 const PAPER_DESCRIPTION =
-  "PolicyBench paper — the 2026-08-17 household-equal scored manuscript snapshot, with June 12–August 17 model responses and PolicyEngine reference outputs on the populace US microdata.";
+  "PolicyBench paper — the 2026-09-01 household-equal scored manuscript snapshot, with June 12–September 1 model responses and PolicyEngine reference outputs on the populace US microdata.";
 
 export const metadata: Metadata = {
   title: "Paper",
@@ -40,7 +40,7 @@ export const metadata: Metadata = {
 
 const manuscriptPaths = {
   pdf: "/paper/policybench.pdf",
-  web: "/paper/web/index.html?v=20260817-refreeze",
+  web: "/paper/web/index.html?v=20260901-refreeze",
 };
 const ssrnUrl = process.env.NEXT_PUBLIC_POLICYBENCH_SSRN_URL;
 
@@ -49,7 +49,7 @@ export default function PaperPage() {
     <>
       <p className="max-w-2xl text-sm leading-relaxed text-text-secondary sm:text-base">
         Benchmarking no-tool tax-and-benefit estimation in frontier language
-        models. This page embeds the 2026-08-17 scored manuscript snapshot: a
+        models. This page embeds the 2026-09-01 scored manuscript snapshot: a
         100-household public preview using household-equal impact scores against
         PolicyEngine reference outputs.
       </p>

app/src/components/Hero.tsx

@@ -2,7 +2,7 @@ import { DEFAULT_VERSION_ID } from "../lib/dataVersionsRuntime";
 import type { BenchData, CountryCode } from "../types";
 import SiteHeader, { type HeaderNavItem } from "./SiteHeader";
 
-const SNAPSHOT_DATE_LABEL = "Snapshot 2026-08-22";
+const SNAPSHOT_DATE_LABEL = "Snapshot 2026-09-01";
 
 export default function Hero({
   selectedView,

app/src/components/Methodology.tsx

@@ -132,11 +132,14 @@ export default function Methodology({
           Each model sees the same household description and must return all
           scored outputs plus a short explanation for each output, with no
           external tools — no calculator, search, or PolicyEngine access.
-          Answers come back through a forced answer-schema tool call; a few
-          models are served one output per request for parse reliability, per
-          the repo's model cards. The exact provider-specific prompts are
-          visible in the scenario explorer, so you can inspect the contract
-          instead of inferring it.
+          Answers come back through each provider's structured-output
+          transport: a forced answer-schema tool call where the provider accepts
+          one, and a JSON object where it does not (Kimi K3, Qwen 3.8 Max,
+          DeepSeek, the older Gemini models, and Claude Fable 5.1, which rejects
+          forced tool calls). A few models are served one output per request for
+          parse reliability, per the repo's model cards. The exact
+          provider-specific prompts are visible in the scenario explorer, so you
+          can inspect the contract instead of inferring it.
         </SectionCard>
 
         <SectionCard title="Open-set status">
@@ -200,9 +203,9 @@ export default function Methodology({
           multiplier. The leaderboard reports the exact-match rate as the
           headline deployability bar, with within-1% as a near-miss-tolerant
           companion and bounded score, amount accuracy, and participation
-          accuracy as further diagnostics. Equal-weight and
-          budget-weighted variants are reported alongside for transparency. The
-          leaderboard is a point estimate on this fixed test set.
+          accuracy as further diagnostics. Equal-weight and budget-weighted
+          variants are reported alongside for transparency. The leaderboard is a
+          point estimate on this fixed test set.
         </SectionCard>
 
         <SectionCard title="Sensitivity checks">

app/src/components/ModelLeaderboard.tsx

@@ -283,12 +283,15 @@ export default function ModelLeaderboard({
           </div>
           <p className="mt-2 text-sm leading-relaxed text-text-secondary">
             Claude models skip extended thinking when the answer tool call is
-            forced, as it is in the identical request this board holds every
-            model to; other reasoning-by-default providers reason regardless.
-            Re-run with <code>tool_choice: auto</code>, Claude Fable 5 scores
-            86.9 (would rank #2), Claude Opus 5 85.6 (#3), and Claude Sonnet 5
-            80.2 (#8). The board below is unchanged — those runs sit beside it
-            as a{" "}
+            forced, as it is in the request this board sends every model whose
+            provider accepts a forced tool; other reasoning-by-default providers
+            reason regardless. Re-run with <code>tool_choice: auto</code>,
+            Claude Fable 5 scores 86.9 (would rank #3), Claude Opus 5 85.6 (#4),
+            and Claude Sonnet 5 80.2 (#9). Claude Fable 5.1 rejects forced tool
+            calls outright, so its board row answers as a JSON object and
+            reasons at the provider default; with the tool declared under{" "}
+            <code>auto</code> it scores 87.5 (still #2). The board below is
+            unchanged — those runs sit beside it as a{" "}
             <a
               href="https://github.com/PolicyEngine/policybench/blob/main/sensitivity/claude-thinking-2026-08.md"
               className="text-primary hover:underline"

app/src/data.artifact.json

@@ -1,9 +1,9 @@
 {
   "version": 1,
   "repo": "PolicyEngine/policybench",
-  "tag": "dashboard-data-20260822",
+  "tag": "dashboard-data-20260901b",
   "asset": "dashboard-data.json",
-  "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260822/dashboard-data.json",
-  "sha256": "b883ec669d510ea29c9c18f18c30030c5bbd29f770bcd90d257779940929a895",
-  "bytes": 85042876
-}
+  "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260901b/dashboard-data.json",
+  "sha256": "1ff522e022bbca91732639acdd34e97f48c1e1e6378819fb648b075e721a7e6a",
+  "bytes": 87410551
+}

app/src/data.versions.json

@@ -5,7 +5,7 @@
     {
       "id": "1.1",
       "label": "1.1",
-      "description": "Corrected ground truth (policyengine-us 1.755.4); identical forced-tool request for every model - 32 models",
+      "description": "Corrected ground truth (policyengine-us 1.755.4); identical prompt for every model, answered through each provider's structured-output transport (forced answer tool where the provider accepts one, JSON otherwise) - 33 models",
       "snapshotLabel": null,
       "artifact": {
         "pointer": "live"

app/src/modelMeta.ts

@@ -4,6 +4,7 @@
 // can keep the runtime values out of the JS bundle.
 
 export const MODEL_ORDER = [
+  "claude-fable-5.1",
   "claude-fable-5",
   "claude-opus-5",
   "claude-opus-4.8",
@@ -40,6 +41,7 @@ export const MODEL_ORDER = [
 ] as const;
 
 export const MODEL_LABELS: Record<string, string> = {
+  "claude-fable-5.1": "Claude Fable 5.1",
   "claude-fable-5": "Claude Fable 5",
   "claude-opus-5": "Claude Opus 5",
   "claude-opus-4.8": "Claude Opus 4.8",
@@ -91,6 +93,7 @@ export type ProviderKey =
 // not). Mirrors policybench/paper_results.py MODEL_RELEASE_DATES, where each
 // date carries its source; update both together.
 export const MODEL_RELEASE_DATES: Record<string, string> = {
+  "claude-fable-5.1": "2026-09-01",
   "claude-fable-5": "2026-06-09",
   "claude-opus-5": "2026-07-24",
   "claude-sonnet-5": "2026-06-30",
@@ -169,7 +172,7 @@ export function orderModels(models: Iterable<string>): string[] {
 // can list fallbacks after its current flagship so an older frozen data bundle
 // still retains one visible model (for example GPT-5.5 before Sol is folded).
 const FRONTIER_MODEL_GROUPS = [
-  ["claude-fable-5", "claude-opus-5", "claude-opus-4.8"],
+  ["claude-fable-5.1", "claude-fable-5", "claude-opus-5", "claude-opus-4.8"],
   ["gpt-5.6-sol", "gpt-5.5"],
   ["grok-4.5", "grok-4.3"],
   ["gemini-3.1-pro-preview"],

paper/index.qmd

@@ -1080,7 +1080,7 @@ The benchmark requires each model response to include numeric answers and one ex
 
 Because explanations are required, the canonical task measures policy estimation under a public-facing structured-response contract, not isolated arithmetic accuracy. Prompt fairness is part of the benchmark contract. The current release uses one prompt template per country, with no model-specific tuning. Models receive the same household facts and requested outputs. They receive no web or external tool access. The prompt sets unlisted numeric inputs to `0`, unlisted boolean or status facts to false, and household characteristics as constant over the tax-benefit year. Provider-specific differences are limited to structured-output transport, request shape, and the reasoning setup in @tbl-model-runs. The harness sends no reasoning-control or sampling parameters. It leaves reasoning effort, temperature, and related decoding controls at provider defaults. The frozen GPT-5.5 wave uses provider-default reasoning effort.
 
-The forced `tool_choice` suppresses Claude's extended thinking in this snapshot ([`sensitivity/claude-thinking-2026-08.md`](https://github.com/PolicyEngine/policybench/blob/main/sensitivity/claude-thinking-2026-08.md)). A labeled sensitivity with `tool_choice: "auto"` is published beside the board. The next board version moves every model to `tool_choice: "auto"` so each provider's default reasoning posture can engage.
+The forced `tool_choice` suppresses Claude's extended thinking in this snapshot ([`sensitivity/claude-thinking-2026-08.md`](https://github.com/PolicyEngine/policybench/blob/main/sensitivity/claude-thinking-2026-08.md)). A labeled sensitivity with `tool_choice: "auto"` is published beside the board. Claude Fable 5.1, released September 1, 2026, closes the interaction from the API side: it rejects forced tool use with an error, so its row answers through the JSON transport and reasons at the provider default; a sensitivity run with the tool declared under `tool_choice: "auto"` scores 1.2 points above that row. The next board version moves every model to `tool_choice: "auto"` so each provider's default reasoning posture can engage.
 
 Request shape is the one dimension on which requests were not identical across models. `{python} n_chunked_models` of the `{python} r.n_models_fmt` models could not reliably complete the whole-scenario request — their serving stacks rejected the structured-output call, exhausted completion budgets mid-response, or timed out — and instead answered the same prompt template over subsets of the requested outputs, one to three outputs per request (@tbl-model-runs). Subsetting gives each response fewer requested outputs and a fresh completion budget over the same household facts, so the accommodation, if it moves scores at all, should favor the chunked models; their scores are not strictly comparable to whole-scenario scores. This accommodation is closed going forward: a model added after this snapshot either answers the canonical whole-scenario request or is listed as not scorable rather than accommodated.
 
@@ -1243,14 +1243,19 @@ def _lag(frame):
     ]
     anchor = min(closed_above["model"], key=_rel)
     days = (_rel(best_open.model) - _rel(anchor)).days
-    return best_open, anchor, days
+    return best_open, anchor, days, len(closed_above)
+
+
+def _closed_models_phrase(n: int) -> str:
+    word = {1: "one", 2: "two", 3: "three", 4: "four"}.get(n, str(n))
+    return f"{word} closed model{'' if n == 1 else 's'}"
 
 
 _pre_july = us_model[
     us_model["model"].map(_rel) < _date(2026, 7, 1)
 ]
-june_open, june_anchor, june_days = _lag(_pre_july)
-full_open, full_anchor, full_days = _lag(us_model)
+june_open, june_anchor, june_days, _ = _lag(_pre_july)
+full_open, full_anchor, full_days, full_n_closed_above = _lag(us_model)
 june_months = f"{june_days / 30.44:.1f}"
 full_weights_days = (_date(2026, 7, 27) - _rel(full_anchor)).days
 
@@ -1260,7 +1265,7 @@ def _prose_date(model: str) -> str:
     return f"{d.strftime('%B')} {d.day}, {d.year}"
 ```
 
-The leaderboard also measures time. Following the lag construction the UK AI Security Institute applied to cyber capabilities [@aisi2026openweight], the open-weight lag is the time between the closed frontier first reaching a score and the leading open-weight model matching it. Two cuts of the frozen roster give the trajectory. Among models released before July 2026, the leading open-weight model is `{python} model_display_name(june_open.model)` at `{python} f"{june_open.exact:.1f}"`% exact, whose weights shipped on `{python} _prose_date(june_open.model)` [@deepseek2026v4]; the earliest-released closed model above that score is `{python} model_display_name(june_anchor)`, released `{python} _prose_date(june_anchor)` [@techcrunch2025gemini3flash] — a lag of `{python} str(june_days)` days, about `{python} june_months` months. On the full roster, `{python} model_display_name(full_open.model)` at `{python} f"{full_open.exact:.1f}"`% exact trails only `{python} model_display_name(full_anchor)`, generally available `{python} _prose_date(full_anchor)` [@techcrunch2026gpt56], putting the lag at `{python} str(full_days)` days by API availability, or `{python} str(full_weights_days)` days dated to Kimi K3's July 27 weight release on Hugging Face [@techtimes2026kimik3weights].
+The leaderboard also measures time. Following the lag construction the UK AI Security Institute applied to cyber capabilities [@aisi2026openweight], the open-weight lag is the time between the closed frontier first reaching a score and the leading open-weight model matching it. Two cuts of the frozen roster give the trajectory. Among models released before July 2026, the leading open-weight model is `{python} model_display_name(june_open.model)` at `{python} f"{june_open.exact:.1f}"`% exact, whose weights shipped on `{python} _prose_date(june_open.model)` [@deepseek2026v4]; the earliest-released closed model above that score is `{python} model_display_name(june_anchor)`, released `{python} _prose_date(june_anchor)` [@techcrunch2025gemini3flash] — a lag of `{python} str(june_days)` days, about `{python} june_months` months. On the full roster, `{python} model_display_name(full_open.model)` at `{python} f"{full_open.exact:.1f}"`% exact trails `{python} _closed_models_phrase(full_n_closed_above)`; the earliest released of them, `{python} model_display_name(full_anchor)`, generally available `{python} _prose_date(full_anchor)` [@techcrunch2026gpt56], puts the lag at `{python} str(full_days)` days by API availability, or `{python} str(full_weights_days)` days dated to Kimi K3's July 27 weight release on Hugging Face [@techtimes2026kimik3weights].
 
 Both figures are lower bounds in one direction: a closed model outside the evaluated roster that crossed either score earlier would lengthen the corresponding lag. They also carry the uncertainty of any single-benchmark, single-domain estimate — the AISI analysis reports substantially longer lags on cyber tasks — and a lag measured at a moving frontier compresses mechanically when a strong open-weight release lands days after a frontier release.