Beyond M1 — deeper LLM-driven metadata & experiment-design resolution + a metadata-resolution accuracy benchmark

[lhqing]

Post-Milestone-1 tracking issue — directions, not yet scheduled. Milestone 1 (#56) makes
seqforge modality-general and gives us the fingerprint-package + benchmark substrate. This issue
collects the next arc, which leans much harder on the LLM: reading papers and metadata deeply to
recover the experiment, recording what the authors originally did, spotting dataset-specific
special handling — and measuring all of it with a metadata-resolution accuracy benchmark.

These stay inside seqforge's discipline. "Lean harder on the LLM" means propose more, always
span-verified, with code deciding — never make the model an authority. We can only widen LLM
scope safely once we can measure metadata accuracy (thread 4), so the benchmark gates the rest.

Why this comes after M1

M1 answers "what IS this library, across any technology?" — from bytes. This arc answers a different,
harder question: "what was the experiment, what did the authors do, and what's special about
this dataset?" — mostly from prose and records. It depends on M1's fingerprint packages, its benchmark
harness, and its modality-general architecture already being in place.

The worm compile audit (#42) is the evidence this arc is needed: seqforge got the chemistry right
almost everywhere, but the experiment layer was thin — covariates buried in GSM titles went uncaptured
(#50), design factors were under-labeled, and dataset-specific breakage (swapped reads, a missing
barcode read, nuclei-vs-cells) was real. That layer is where the LLM earns its keep.

---

Thread 1 · Deeper sample-metadata & experiment-design resolution (LLM-heavy)

Today resolve_metadata maps archive records → per-sample attributes (NCBI's 960 harmonized BioSample
names), each Evidenced with a basis, disagreements left null-over-wrong. That is correct but shallow
relative to what a paper actually contains.

Goal: recover the experiment design, not just flat attributes — sample groupings, conditions /
treatments, genotypes, timepoints, replicate and batch structure, the full factor × level matrix. Much of
this lives in GSM free-text titles and in the paper's prose and supplementary tables, not in structured
fields.

Discipline unchanged: every claim is a span-verified Assertion that greps back and entails its
value; code arbitrates; the model simply reads more. Connects to #50 (deterministic GSM-title covariate
parser) and the under-labeling gap noted in #42 (F5).

Thread 2 · Record the paper's original pipeline (provenance of "what the authors did")

A new class of extracted fact: what pipeline the authors originally ran — aligner + version, genome
build + annotation release, key parameters, and custom steps (3′-UTR extensions, custom references,
non-standard filtering). Why it matters:

• It's the baseline to reconcile against our recipe — it explains why our matrix differs from the
  published one (cf. the WS298-vs-paper cross-cutting note in Compile audit (13 worm datasets): equal-length reads silently mis-seat the barcode role — 2 datasets broken — + 4 smaller gaps #42 and the 3′-UTR work in 3'UTR extension for WS298: regenerate from pooled single-cell data + benchmark necessity (research/design) #52).
• It's a rich signal the model can use to propose our processing.yaml.
• It documents divergence honestly, in the manifest's provenance rather than in a reviewer's head.

Hard line: the authors' original pipeline is evidence — a recorded fact about what they did — not an
instruction to our compiler. It stays distinct from processing.yaml (our recipe). It informs; it never
commands. Same two-artifact separation seqforge already enforces, extended to a third category:
what-they-did vs what-the-data-is vs what-we-choose-to-do.

Thread 3 · Spot dataset-specific special handling

Detect when a dataset needs non-default treatment — swapped R1/R2, a missing technical read, nuclei vs
cells, chimeric / multi-species libraries, custom references, unusual sample structure. These are exactly
the classes the worm audit surfaced (#42: equal-length role mis-seat, the 18 samples missing a barcode
read; #51 wrong-PDF / provenance; #55 low-confidence winners).

How it stays honest: the model flags candidates (proposer); bytes / code confirm; the result
surfaces as a Conflict / ValidationWarning / Question — never a silent default. This is the
"agents propose, code decides, refusal is an exit code" line applied to dataset gotchas.

Thread 4 · A metadata-resolution accuracy benchmark

M1's benchmark grades the compiler's decisions (chemistry, roles) plus a few sample attributes. This
thread adds a benchmark aimed squarely at metadata-resolution accuracy: given the records + prose,
how accurately do we recover the experiment design — per attribute, per sample, and the groupings /
conditions in threads 1–2?

• Built on the same fingerprint-package substrate + pre-registered expected.yaml as M1, but the
  graded surface is the experiment / metadata fields (experiment.samples.*.<attr>, groupings,
  conditions, and the recorded original-pipeline facts).
• Scored for accuracy and false-accept — a verified-but-wrong metadata claim is the cardinal sin,
  same as elsewhere.
• This is the gate. "Fully utilize the LLM" is only safe to the extent we can measure metadata
  accuracy; this benchmark is the precondition for trusting (and widening) deeper resolution.
• Later, a true held-out metadata test set — mirroring M1's held-out plan — for an unbiased accuracy
  number.

---

Guardrails (carried over, restated because this arc is LLM-heavy)

• Two-job line holds. Bytes → score, prose → assertion, code decides. Deeper reading is more
  proposing, never more authority.
• Every claim span-verified + entailed; equal-authority disagreements go null-over-wrong; a
  verified-but-wrong claim is a hard failure.
• The authors' original pipeline is evidence, not an instruction — recorded, never acted on directly.
• Nothing silent. Special-handling needs are surfaced as conflicts / warnings / questions.

Relationship to Milestone 1 (#56)

Depends on M1's fingerprint packages, benchmark harness, and modality-general architecture. M1 is what
the data is, across technologies; this is what the experiment was, and what the authors did — measured
for accuracy. Related: #50, #52, #42, #51, #55.

[lhqing]

Absorbing #50 (F5 — deterministic GSM-title covariate parser)

Closing #50 into this milestone so the design isn't lost. The decision (2026-07-22): stay simple for now — no deterministic title tokenizer in the near term. Whether deterministic parsing earns its keep over span-verified LLM harvest is exactly the deeper-metadata question this issue (Thread 1) exists to answer, so it belongs here, not as a standalone quick fix.

Preserved design notes from #50 for when Thread 1 tackles GSM-title covariates:

• The under-labeling gap (audit Compile audit (13 worm datasets): equal-length reads silently mis-seat the barcode role — 2 datasets broken — + 4 smaller gaps #42, F5). Design covariates live in the GSM title, not in harmonized BioSample fields: GSE208229 treatment×generation×sort (30/31 blank), GSE229022 diet (25/28 blank), PRJNA1027859 treatment (3/6 blank). The title reaches the model today only as free text on the experiment document; only experiment.samples.treatment is asked of it (harvest/fields.py).

• The null-over-wrong hazard (the reason to be careful). A deterministic title token and a BioSample's own typed field would both arrive as basis="asserted" — equal authority. resolve/records.py::_decide leaves an equal-authority disagreement null (permanent-manifest safety), so a sloppy parser doesn't just add a wrong fact, it can erase a right one. A parser must map only tokens it can attribute confidently.

• Safe-subset candidates if/when we do parse deterministically: dose \d+(\.\d+)?\s?(%|mM|nM|uM), explicit rep\d+/replicate\d+, known sort words (sorted/unsorted). Ambiguous tokens (F1 = filial-1 generation? replicate? female-1?; Batch5) stay unparsed.

• R4 constraint. Any parsed key must pass io/attributes.py::is_attribute (one of NCBI's 960 harmonized names) and ideally go through harmonize() first — never an invented key.

• Structural note. The GSM title lives on the experiment record (io/archive.py experiment_title), not the <SAMPLE> element that _sample_attributes receives, and _sample_attributes output is overwritten by merge_biosample_attributes when a BioSample is reachable. So a parser must run where the experiment→sample join exists (the resolve/records.py layer), not inside per-element transcription.

Original issue: #50.

[lhqing]

Figure extraction produces fragments, not figures + legends (future thread)

Surfaced reviewing an M1 fingerprint deliverable (#63): the PDF image extraction that feeds the harvest
carries raw embedded image XObjects, which for a typical paper are figure fragments — not the
composed figures the LLM would need — and the legends aren't associated at all.

Evidence — one deliverable, Smith et al. 2024 (GSE234962), 70 PNGs from the paper:

• 62% (44/70) are fragments (a side < 200 px); 25 are slivers (< 40 px — thin bars/rules); 5 are degenerate 0×0.
• Only 3/70 are plausibly whole figures (≥ 600×400), and those are full-page raster scans, not the panel-composed main figures.
• Page 1 alone → 26 separate fragments (blurry gradient clips, black rectangles, thin bars). The composed Figure 1 and its legend appear nowhere as a usable unit.

Why: extracting every embedded raster XObject decomposes a composed figure into sub-elements
(panels, gradient fills, masks); vector-drawn figures aren't rasterized/extracted at all; and nothing
ties a figure to its caption.

Direction (not scheduled — a thread under this issue): meaningful figure + legend extraction. Options to weigh:

• render each page to an image and detect figure regions, instead of harvesting embedded XObjects;
• a layout/figure-detection model to bound figure + caption regions;
• caption parsing to hand the LLM (figure, legend) pairs;
• possibly a vision model over page rasters for "what does this figure show".

Prior art / research first. Before building anything, do a proper survey of how people already
extract meaningful figures + legends from scientific-manuscript PDFs — start from PaperQA
(Future House) and its document-ingest pipeline, and branch out to the broader figure/caption-extraction
literature and tools (e.g. layout-analysis and figure-detection systems). This is a solved-enough problem
elsewhere that we should learn from it rather than reinvent; the deliverable of this thread's first step
is a short deep-research writeup of the state of the art, then a choice.

Connects to Thread 1 (reading prose and figures/supplementary tables deeply). Out of scope for
M1 / this session — the current extraction is fine for the text harvest (paper .txt + xlsx .txt
carry the claims); only the images are weak, and they're not on M1's critical path.

[lhqing]

#56 (M1) is closed — this is now the live milestone. Close-out audit and verdicts: #56 (comment)

M1's dependencies for this arc are all in place: the fingerprint-package substrate (seqforge preflight), the benchmark harness (kind: fingerprint + benchmark.yml + 11 cases), and the modality-general architecture (resolve_pipeline + three map/ modules). Threads 1–4 are unblocked.

Two issues carry forward alongside this one, at the top of the queue and not gating anything:

• Benchmark growth: seeded-case review, BD-Enhanced coverage, heavy datasets, held-out test set #81 — benchmark growth. Its first box matters most to Thread 4: seven evals/benchmark/*/expected.yaml still carry a PENDING MAINTAINER REVIEW header, meaning they were seeded from a run rather than pre-registered before one. Thread 4 grades metadata accuracy against exactly those files, so the review is the precondition for that number meaning anything.
• Stress the abstractions: the refusals our docs claim and nothing exercises #153 — claims the docs make that nothing exercises, including one interface decision this arc would inherit: is a 10x Multiome pair one biological sample with two libraries, or two samples?

Struck from M1 and not carried here: the breadth guardrail (#79) and the post-run report (#35).