Skip to content

feat: add Composer Arena public preview#4

Draft
seanphan wants to merge 5 commits into
mainfrom
codex/composer-arena-public-surface
Draft

feat: add Composer Arena public preview#4
seanphan wants to merge 5 commits into
mainfrom
codex/composer-arena-public-surface

Conversation

@seanphan

Copy link
Copy Markdown
Contributor

Summary

  • add a publication-candidate Composer Archive-to-Output Arena route under /bench/composer-arena/
  • keep controlled-agent and end-to-end-system standings separate
  • add confidence intervals, battle counts, deterministic success/failure, latency, token, cost, and task-slice fields
  • add an anonymous A/B replay with swapped order, tie/both-bad choices, reveal-after-judgment, and browser-local votes excluded from rankings
  • publish schema-gated synthetic fixtures, methodology, checksums, FineVideo provenance, and the exact real-data handoff contract

Safety and evidence boundary

  • every current score/output is prominently marked synthetic and non-rankable
  • the verified 24-cell FineVideo release is cited only as source/provenance evidence
  • official ranking remains disabled until successful comparable runner records and governed human votes exist
  • no customer/GMA data, credentials, signed URLs, hidden annotations, raw reasoning, deployment, or production-default change

Verification

  • npm run check:bench
  • npm run lint
  • npm run build -- --webpack
  • local production server: /bench/composer-arena/ returned HTTP 200
  • Chrome visual smoke: controlled/end-to-end switch, A/B swap, local tie/reveal, and zero browser warnings/errors

@vercel

vercel Bot commented Jul 19, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
av-web Ready Ready Preview, Comment Jul 19, 2026 3:59pm

Request Review

@seanphan

Copy link
Copy Markdown
Contributor Author

Founder decision — HOLD

Sean adopted the 2026-07-19 independent Composer review with amendments. This PR is intentionally returned to draft and must remain open, unmerged, and undeployed.

Reason:

  • the v1 seven-family result is now classified as an internal reliability and integration gate, not an end-to-end intelligence benchmark;
  • the unchanged mechanical control accepted 5/7 complete outputs, while the best model lane accepted 4/7;
  • the pack contains no transcript, OCR, or semantic-event evidence, so it cannot establish that the tasks require agentic intelligence; and
  • the most likely public reading would be “deterministic chronology beat the agents,” which does not advance the paid Composer diagnostic.

The branch, checks, hashes, and zero-rank page remain preserved. Do not add models, rerun cases, or revise the evidence on this PR.

Any later publication request requires a separately approved v2 construct-validity packet using only the same seven families, with rights-safe timestamp-bound semantic evidence, mechanical/plan-only/naive-scaffold/Composer-native controls, operator-grade acceptance, native tool compliance before any logged translation arm, and a reviewable before/after reliability result. That packet is design-only today; no execution is authorized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant