Add external evidence on the ceiling of topology-only repair

[yushuosun]

1 Research question

The Explicit route is topology-only: it defaults to KEEP, keeps text frozen, and
proposes minimal split/merge changes. How much can a metric move when only
topology is corrected and cell content is left exactly as the parser produced it?

2 Hypothesis and changed variable

Hypothesis: correcting the first diverging structure token, then letting the
model continue, recovers most of the loss attributed to structural divergence.

Changed variable: the number of oracle single-token structural corrections
allowed, k from 0 to 16. Everything else is held fixed, including the parser,
decoding parameters, denominator, and GPU model.

Result: the hypothesis is false. The curve saturates at +3.97 pp, and the
k=1 gain has a document-cluster bootstrap 95% CI of [-1.64, +3.85] pp, which
contains zero.

The useful part is the three-way split: 34 of 118 divergent tables reach a cell
score of exactly 1.0000 once structure is repaired, and 59 of 118 become
structurally exact after a single correction, matching the single minimal edit category in this repository's corpus design. The residual is concentrated
elsewhere: 66% of it sits in 20 tables that stay at 0.5766 even after 16 oracle
corrections, 14 of which end up structurally identical to ground truth.

3 Affected CalVer release

Adds 2026.08.14.1 documents under a new docs/external-evidence/ directory.
No existing release, schema, config, or module is modified.

4 Data roles, sources, licenses, isolation

• Source: OmniDocBench table crops, a public benchmark. Parser:
  MinerU2.5-Pro-1.2B.
• No benchmark payload is committed. The manifest contains derived
  measurements only: benchmark sample identifiers plus our own scores. No
  images, no ground-truth text, no annotations.
• Isolation: this data has no relationship to this repository's shared corpus,
  its splits, or its holdout. It is not used for training or selection here, so
  it cannot leak into any role.
• Some sample identifiers contain non-English characters. These are upstream
  OmniDocBench identifiers, which CONTRIBUTING asks not to rename, so they are
  kept verbatim. All project-authored text in this PR is English.

5 Metrics and acceptance criteria

• Metric: official exact-cell match, fixed denominator
  compare(GT, GT)["total_cells"] = 43231 cells over 653 tables. A scoring
  exception zeroes the whole table, per the official semantics.
• Pre-registered control: the 535 already-structurally-exact tables receive no
  repair, so their per-table score must be identical across every k.
  Observed: identical for all 535, 0 violations.
• Pre-registered decision rule: k=1 recovering less than 20% of the gap means
  single-point repair is insufficient. Observed 13.4%.
• Reported alongside the measurement: document-cluster bootstrap, saturation
  check, and a single-table dominance warning (one table is 21% of the residual,
  so leave-one-out is mandatory on this data).

6 Validation commands

The measurement itself ran on the collaborating project's cluster and is not
reproducible from this repository. What can be checked here:

sha256sum docs/external-evidence/TOPOLOGY_REPAIR_CEILING_2026.08.14.1.manifest.json
# must match the accompanying .sha256 file

python -c "import json;d=json.load(open('docs/external-evidence/TOPOLOGY_REPAIR_CEILING_2026.08.14.1.manifest.json',encoding='utf-8'));print(d['totals'])"
# {'n': 118, 'by_category': {'catastrophic': 20, 'struct_ok_content_lossy': 64, 'struct_ok_full_recover': 34}, 'total_gt_cells': 12893, 'total_lost_final': 2499}

pytest

Docs-only change; no module, schema, or config is touched.

Why this is offered

The practical suggestion is in section 4 of the document: report structural
metrics and content-inclusive metrics as separate numbers rather than one
combined score. A topology-only route that freezes text should score clearly
on TEDS-Struct and GriTS-Topology, while the oracle ceiling measured here on an
exact-cell metric is +3.97 pp. Separating them shows what the method actually
achieves.

Section 5 of the document lists the limits: one parser, one corpus, one metric,
oracle conditions, and single-table dominance. This is external evidence, not a
claim about this repository, which is why it is filed under
docs/external-evidence/.

Section 6 records a caliber error we made ourselves: a "+9.63 pp" upper bound we
cited for a long time turned out to be an arithmetic construction rather than a
measurement. The rule we adopted afterwards is offered as a suggestion.

[Copilot]

Pull request overview

Adds a new “external evidence” document and accompanying per-table measurement manifest that quantify how much an exact-cell metric can improve when only table topology is oracle-corrected (text frozen), to inform how the Explicit route should report structural vs content-inclusive metrics.

Changes:

• Add an external evidence write-up describing the topology-only repair ceiling study and its reporting implications.
• Add a derived-measurements manifest JSON enumerating the divergent tables and key per-table fields.
• Add a .sha256 checksum file for integrity verification of the manifest.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
docs/external-evidence/TOPOLOGY_REPAIR_CEILING_2026.08.14.1.md	External evidence narrative, results tables, and reporting implications.
docs/external-evidence/TOPOLOGY_REPAIR_CEILING_2026.08.14.1.manifest.json	Derived per-table measurements and field semantics for the divergent subset.
docs/external-evidence/TOPOLOGY_REPAIR_CEILING_2026.08.14.1.manifest.json.sha256	Checksum for validating the manifest JSON contents.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.