Add external evidence on a zero-cost self-confidence signal for KEEP routing

[yushuosun]

1 Research question

Before any candidate is considered, the safety layer must decide whether the Raw
parse is already good enough to keep. Can that decision be informed by a signal
that costs nothing, or does it require external evidence such as a second model,
a re-render comparison, or repeated sampling?

2 Hypothesis and changed variable

Hypothesis, pre-registered as expected to fail: the parser's own token
probabilities on its own output can identify which tables it got structurally
wrong, and where.

Changed variable: the scoring signal. Instead of sampling repeatedly or adding a
second model, run one extra forward pass over the already-generated sequence and
read the per-position distribution. Nothing else changes.

Result: the hypothesis holds, contradicting our own prior. Table-level AUROC
is 0.8621 for separating "structure will diverge" from "structure will be
correct", and within a table the single lowest-margin position is the true first
error 47.4% of the time.

3 Affected CalVer release

Adds one 2026.08.14.1 document under docs/external-evidence/. No existing
release, schema, config, or module is modified.

4 Data roles, sources, licenses, isolation

• Source: OmniDocBench table crops, a public benchmark. Parser:
  MinerU2.5-Pro-1.2B.
• No data payload is added by this PR. It is a document only.
• Isolation: the measurement has no relationship to this repository's shared
  corpus, splits, or holdout, and is not used for training or selection here.
• Per-table minimum margins for the 118 divergent tables are already present in
  the manifest of the companion PR, in the selfconf_min_margin field.

5 Metrics and acceptance criteria

Pre-registered before the run:

• Position-level recall at a 10% budget of at least 30% means the signal is
  usable; at most 15% means external evidence is required. Observed 0.922
  pooled, 0.793 within-table.
• Table-level AUROC of at least 0.70 is required for table-level use.
  Observed 0.8621, corroborated by min p_top1 (0.8595) and max entropy
  (0.8522).
• Guard against pooled inflation: the pooled position-level metric can be
  inflated by between-table separation, so the same test was repeated ranking
  only each table's own positions. The within-table numbers are the ones to
  trust.

Explicitly not claimed: any end-to-end effect. Every number here is
diagnostic, measured with no router and no safety layer in the loop.

6 Validation commands

The measurement ran on the collaborating project's cluster and is not
reproducible from this repository. This PR adds documentation only.

pytest

Why this is offered

Section 4 of the document states the two caveats that decide whether the signal
is usable here: it predicts structural divergence rather than "Raw is good"
on a content-inclusive metric, and it is computed from one specific parser's
logits, so it must be recalibrated for any other parser. Within those limits it
is a KEEP-side prior that costs one forward pass and can be ablated cleanly as
one feature among others.

Section 5 records a scoping result that may save time elsewhere: signals based
on self-consistency across samples carry no information when the failure is
deterministic, because repeated decoding reproduces the same wrong answer and
the variance is zero. Signals based on single-forward distribution sharpness
still work in exactly that case. We had generalised the first finding into a
belief that confidence signals in general would not work, and that
over-generalisation nearly cost us this result.

[Copilot]

Pull request overview

Adds a new external-evidence document capturing results from a collaborating project on using a parser’s self-confidence (logit-derived) signal to inform the safety layer’s KEEP vs EDIT decision, without introducing any code or data into this repository.

Changes:

• Add a CalVer’d external-evidence writeup describing a “self-confidence” signal computed from one extra forward pass over the parser’s own output.
• Record reported table-level discrimination (AUROC) and position-level first-error localization metrics, plus stated limits of validity.

Suppressed comments (1)

docs/external-evidence/SELF_CONFIDENCE_KEEP_SIGNAL_2026.08.14.1.md:19

• This paragraph calls the signal "free", but it still requires an additional forward pass over the generated sequence (Section 3.3). Rewording would make the cost model consistent throughout the document.

The usual instinct is to add evidence from outside the parser: a second model, a
re-render comparison, or repeated sampling. All of them cost inference. This
document reports that, on the parser we tested, a **free** signal already
carries most of that information.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.