feat: add public OTSL and synthetic-data tooling

.gitignore

@@ -41,6 +41,10 @@ runs/
 *.pth
 *.safetensors
 
+# Generated Raw-preserving smoke metadata.
+smoke_raw_preserving_v1.json
+raw_preserving_smoke_*.jsonl
+
 # Credentials and local access configuration.
 *.pem
 *.key

README.md

@@ -1,5 +1,72 @@
 # Borderless Table Structuring Lab
 
+Research infrastructure for auditable borderless-table structure recognition,
+Canonical Table supervision, safe Raw MinerU refinement, and independent
+Explicit and LoRA candidate routes.
+
+This repository is designed as the long-lived project home. The initial
+revision contains the data-engineering and safety-integration layers only. It
+does **not** contain model weights, training payloads, terminal benchmark pages,
+Customer50 artifacts, or per-sample terminal predictions.
+
+## Table of contents
+
+- [Research objective](#research-objective)
+- [Current repository scope](#current-repository-scope)
+- [Repository layout](#repository-layout)
+- [System design](#system-design)
+- [Data strategy](#data-strategy)
+- [Installation](#installation)
+- [Tests](#tests)
+- [Reproducibility and evidence](#reproducibility-and-evidence)
+- [Collaboration workflow](#collaboration-workflow)
+- [Roadmap](#roadmap)
+- [Governance and licensing](#governance-and-licensing)
+
+## Research objective
+
+The project targets table-quality improvement under the OmniDocBench document
+parsing protocol while preserving the Raw MinerU document baseline. The core
+engineering principle is selective, auditable table correction:
+
+1. Raw MinerU remains the default output.
+2. The Explicit route may propose minimal topology-only corrections with Raw
+   OCR text frozen.
+3. The LoRA route may propose one complete, table-only Canonical Table state.
+4. Both routes pass through the same legality, token-preservation, geometry,
+   provenance, expected-gain, assembly, and exact-Raw-rollback controls.
+5. Unsafe or unsupported candidates are rejected without modifying Raw.
+
+The target of Table TEDS above 95 is an engineering objective, not a guaranteed
+unobserved result. Public benchmark-aware development and independent terminal
+generalization must be reported separately.
+
+## Current repository scope
+
+Included in the first revision:
+
+- Canonical Table normalization and structural label utilities.
+- Direct-state and order-invariant target compilation.
+- Candidate-integrity checks.
+- Shared fail-closed validation and deterministic Raw rollback.
+- Explicit topology-only and LoRA complete-table candidate interfaces.
+- Synthetic unit fixtures and regression tests.
+- Canonical record schema.
+- Evidence Cards and the active execution contract.
+- Dataset governance, storage, reproducibility, and collaborator handoff
+  documentation.
+- Public OTSL normalization and fixed-denominator paired-metric utilities.
+- Synthetic-data provenance guidance and manifest validation.
+
+Explicitly excluded:
+
+- Model implementations, adapters, checkpoints, or weights.
+- Full training corpora or rendered sample payloads.
+- Formal20k source records and compiled record payloads.
+- Customer50 content.
+- OmniDocBench pages, crops, annotations, recognized strings, coordinates,
+  HTML, LaTeX, page identifiers, or Gold records.
+- Per-sample terminal predictions or case-selection artifacts.
 Research on recovering table structure from weak or absent visual boundaries.
 The repository brings together canonical table representations, controlled
 data generation, explicit topology modeling, and parameter-efficient
@@ -171,6 +238,8 @@ python -m venv .venv
 source .venv/bin/activate
 python -m pip install --upgrade pip
 python -m pip install -e '.[dev]'
+# Add Pillow-backed terminal-blind rendering and perceptual-overlap audits.
+python -m pip install -e '.[dev,synthesis]'
 ```
 
 Run the data-free test suite:
@@ -208,7 +277,11 @@ documentation needed to reproduce them.
 For details, see [Reproducibility](docs/REPRODUCIBILITY_2026.08.12.1.md) and
 [Dataset Storage and Sharing](docs/corpus/DATASET_STORAGE_AND_SHARING_2026.08.12.1.md).
 
-## Collaboration
+Detailed requirements are documented in
+[REPRODUCIBILITY.md](docs/REPRODUCIBILITY.md). The bounded KEEP-majority pair
+contract, exact-Raw baseline, streaming family audit, and fail-closed perceptual
+overlap gate are described in
+[RAW_PRESERVING_SMOKE.md](docs/RAW_PRESERVING_SMOKE.md).
 
 The two model tracks share representations and evaluation but keep model code
 and ablations independent. Suggested branch prefixes are:

docs/RAW_PRESERVING_SMOKE.md

@@ -0,0 +1,55 @@
+# Raw-preserving bounded smoke
+
+The first smoke is intentionally small and deterministic:
+
+- 256 base table groups;
+- 10 counterfactual records per group;
+- all authorized phenomenon families represented;
+- 90% `KEEP_RAW` and 10% `ACCEPT_EDIT` oracle labels after Gold-only labeling;
+- no document, template, content, renderer, or seed family crosses roles.
+
+The smoke must report requested, generated, passed, quarantined, and failed
+counts. It must also check canonical legality, complete grid coverage, OCR token
+ownership, geometry, direct Gold recompilation, deterministic replay, and exact
+/perceptual/structure/text/geometry overlap. Family identifiers are not proof of
+visual isolation: cross-role images are screened with 9-by-8 dHash and confirmed
+with 32-by-32 pHash. A confirmed perceptual hit blocks the corpus rather than
+silently removing records or retuning the threshold after holdout inspection.
+Large-corpus audits must stream records or retain only compact group summaries;
+they must not require all Raw, Gold, and candidate objects in memory.
+
+## Oracle action contract
+
+The offline labeler receives `raw_record`, `candidate_record`, and
+`gold_record`. It emits `KEEP_RAW` unless all of the following hold:
+
+1. the candidate passes the shared safety validator;
+2. the candidate has strictly more exact Gold cells than Raw;
+3. candidate text agreement is not lower than Raw;
+4. candidate geometry coverage is not lower than Raw;
+5. non-table page state is unchanged.
+
+A Raw state that exactly matches Gold is always `KEEP_RAW`, even when a
+candidate is also valid. Ties and partial improvements are `KEEP_RAW`.
+
+Gold and all Gold-derived values are offline-only fields. The runtime selector
+view contains only the image, Raw record, candidate record, and observable
+candidate-vs-Raw differences.
+
+## Initial distribution
+
+The frozen first distribution is 90% KEEP and 10% ACCEPT:
+
+- 35% Raw-correct with a legal harmful over-edit candidate;
+- 25% Raw-near-correct with inflation, over-split, or over-merge candidates;
+- 15% Raw-bad with a tied, partial, or trade-off candidate;
+- 15% identity candidates;
+- 5% Raw-bad with a clearly better local candidate;
+- 5% Raw-bad with a clearly better complete candidate.
+
+The first two KEEP-heavy curricula are 95/5 and then 90/10. A K0 all-KEEP
+execution baseline must pass before any selector training. Selector-only audits
+use a frozen controlled candidate bank and report routing separately from
+candidate generation. In particular, an offline Gold-derived positive candidate
+may measure takeover logic, but it is never runtime evidence and cannot support
+a claim about a deployable candidate generator.

docs/SYNTHETIC_DATA_PROVENANCE.md

@@ -0,0 +1,41 @@
+# Synthetic corpus provenance
+
+This repository does not store rendered data, benchmark pages, model outputs, or
+training payloads. The approved synthetic packages are shared separately.
+
+## SynthFin v3.4
+
+SynthFin v3.4 is a locally authored renderer for financial-report-style table
+pages. It does not copy pixels or annotations from FinTabNet, PubTables,
+OmniDocBench, CNInfo, or customer documents. Its vocabulary, company-like
+names, financial values, page prose, table topology, and metadata are generated
+from local templates and seeded random draws.
+
+The renderer supports Chinese and English financial statements, borderless and
+ruled tables, multi-row headers, row and column spans, dense pages, multiple
+tables per page, narrative context, optional formulas, resolution ladders, and
+light JPEG capture effects. The v3.4 correction avoids drawing ruled segments
+through the interior of merged cells. The ground-truth structure is generated
+from the same table state used by the renderer and is validated by round-trip
+checks before use.
+
+The shared package contains only synthetic images and its manifest. It does not
+contain the upstream public datasets used as external references, model
+weights, terminal benchmark content, or prediction files.
+
+## invoice-synthetic-v1
+
+The invoice package is fully self-generated. It uses local templates for
+invoice-like layouts, random company names, dates, identifiers, addresses,
+items, tax rates, totals, and Chinese amount-in-words strings. Each text draw is
+recorded at generation time, so the visible-text annotations and rendered
+images share one source of truth. The package does not use external invoice
+images or scanned documents as backgrounds.
+
+## Scope and limitations
+
+These packages are research synthetic data, not official benchmark data and
+not a claim of real-domain performance. They are intended for debugging,
+controlled ablations, and reproducibility of rendering/label contracts. Users
+should inspect the applicable license and sharing terms before redistributing
+modified versions.

pyproject.toml

@@ -20,6 +20,9 @@ dependencies = [
 dev = [
   "pytest>=7.4,<9",
 ]
+synthesis = [
+  "Pillow>=10,<13",
+]
 
 [tool.setuptools]
 package-dir = {"" = "src"}

schemas/raw_preserving_pair_v1_schema.json

@@ -0,0 +1,134 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "mpr-tsr/raw-preserving-pair-v1",
+  "title": "Raw-Preserving Candidate Pair",
+  "type": "object",
+  "required": [
+    "schema_version",
+    "sample_id",
+    "pair_group_id",
+    "role",
+    "image",
+    "raw_record",
+    "gold_record",
+    "candidate_record",
+    "oracle_decision",
+    "provenance"
+  ],
+  "properties": {
+    "schema_version": {"const": "mpr-tsr/raw-preserving-pair-v1"},
+    "sample_id": {"type": "string", "minLength": 1},
+    "pair_group_id": {"type": "string", "minLength": 1},
+    "role": {"enum": ["train", "development", "holdout", "terminal"]},
+    "image": {
+      "type": "object",
+      "required": [
+        "relative_path",
+        "sha256",
+        "source_document_hash",
+        "table_index",
+        "image_size"
+      ],
+      "properties": {
+        "relative_path": {
+          "type": "string",
+          "minLength": 1,
+          "pattern": "^(?!/)(?![A-Za-z]:)(?!.*(?:^|/)\\.\\.(?:/|$))[^\\\\]+$"
+        },
+        "sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
+        "source_document_hash": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
+        "table_index": {"type": "integer", "minimum": 0},
+        "image_size": {
+          "type": "array",
+          "prefixItems": [
+            {"type": "integer", "minimum": 1},
+            {"type": "integer", "minimum": 1}
+          ],
+          "items": false
+        }
+      },
+      "additionalProperties": false
+    },
+    "raw_record": {"type": "object"},
+    "gold_record": {"type": "object"},
+    "candidate_record": {"type": ["object", "null"]},
+    "oracle_decision": {
+      "type": "object",
+      "required": ["action", "reason", "raw_good"],
+      "properties": {
+        "action": {"enum": ["KEEP_RAW", "ACCEPT_EDIT"]},
+        "reason": {"type": "string", "minLength": 1},
+        "raw_good": {"type": "boolean"}
+      },
+      "additionalProperties": true
+    },
+    "phenomenon_tags": {"type": "array", "items": {"type": "string"}},
+    "operator": {"type": "object"},
+    "provenance": {
+      "type": "object",
+      "required": [
+        "generator",
+        "generator_version",
+        "generation_seed",
+        "terminal_data_used",
+        "document_cluster_id",
+        "template_family_id",
+        "content_family_id",
+        "renderer_family_id"
+      ],
+      "properties": {
+        "generator": {"type": "string", "minLength": 1},
+        "generator_version": {"type": "string", "minLength": 1},
+        "generator_policy": {"const": "terminal_blind_v1"},
+        "generation_seed": {"type": "integer", "minimum": 0},
+        "terminal_data_used": {"const": false},
+        "document_cluster_id": {"type": "string", "minLength": 1},
+        "template_family_id": {"type": "string", "minLength": 1},
+        "content_family_id": {"type": "string", "minLength": 1},
+        "renderer_family_id": {"type": "string", "minLength": 1},
+        "family_plan_entry_sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
+        "source_dataset": {"type": "string", "minLength": 1},
+        "license": {"type": "string", "minLength": 1},
+        "font": {
+          "type": "object",
+          "required": ["source", "file_name", "sha256", "redistributed"],
+          "properties": {
+            "source": {"type": "string", "minLength": 1},
+            "file_name": {"type": "string", "minLength": 1},
+            "sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
+            "redistributed": {"const": false}
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": true
+    }
+  },
+  "allOf": [
+    {
+      "if": {
+        "properties": {
+          "provenance": {
+            "required": ["generator_policy"],
+            "properties": {
+              "generator_policy": {"const": "terminal_blind_v1"}
+            }
+          }
+        }
+      },
+      "then": {
+        "properties": {
+          "provenance": {
+            "required": [
+              "family_plan_entry_sha256",
+              "source_dataset",
+              "license",
+              "font"
+            ]
+          }
+        }
+      }
+    }
+  ],
+  "additionalProperties": false
+}