feat: add public OTSL and synthetic-data tooling - #1
Open
yushuosun wants to merge 3 commits into
Open
Conversation
Expose reproducible OTSL normalization, fixed-denominator paired metrics, and synthetic manifest validation while documenting the provenance and scope of the privately shared synthetic corpora. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds public-facing research utilities to the mpr_tsr_splitmerge_v2 package, including OTSL post-decode normalization helpers, paired fixed-denominator metric summaries, and lightweight synthetic-manifest validation plus provenance documentation.
Changes:
- Introduces OTSL normalization/validation utilities and a small paired-metrics summarizer module.
- Adds a CLI script to validate synthetic-data manifests against an asset root.
- Adds documentation updates (README scope bullets + synthetic provenance doc) and tests for the new public utilities.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_public_research_tools.py | Adds tests covering OTSL normalization and paired-metrics summary behavior. |
| src/mpr_tsr_splitmerge_v2/paired_metrics.py | Implements paired fixed-denominator summary metrics including LOO summaries. |
| src/mpr_tsr_splitmerge_v2/otsl.py | Adds OTSL normalization, chat-control stripping, and structure-token validation helpers. |
| scripts/validate_synthetic_manifest.py | Adds a standalone manifest validator for synthetic assets and path safety. |
| README.md | Updates the “Current repository scope” list to mention new public utilities and provenance tooling. |
| docs/SYNTHETIC_DATA_PROVENANCE.md | Adds provenance documentation for synthetic corpora (SynthFin v3.4 and invoice-synthetic-v1). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+42
to
+59
| largest_index = max(range(len(deltas)), key=lambda index: abs(deltas[index])) | ||
| largest = deltas[largest_index] | ||
| loo_denominator = denominator - baseline[largest_index][1] | ||
| if loo_denominator <= 0: | ||
| raise ValueError("leave-one-out denominator must be positive") | ||
| loo_delta = candidate_correct - baseline_correct - largest | ||
| return PairedSummary( | ||
| baseline_correct=baseline_correct, | ||
| candidate_correct=candidate_correct, | ||
| fixed_denominator=denominator, | ||
| delta_correct=candidate_correct - baseline_correct, | ||
| delta_pp=100.0 * (candidate_correct - baseline_correct) / denominator, | ||
| help_count=sum(delta > 0 for delta in deltas), | ||
| hurt_count=sum(delta < 0 for delta in deltas), | ||
| same_count=sum(delta == 0 for delta in deltas), | ||
| largest_absolute_delta=largest, | ||
| leave_one_out_delta_pp=100.0 * loo_delta / loo_denominator, | ||
| ) |
Comment on lines
+25
to
+28
| if not any(token in text for token in STRUCTURE_TOKENS): | ||
| raise ValueError("decoded output contains no OTSL structure token") | ||
| if "<nl>\n" not in text and "<nl>" in text: | ||
| raise ValueError("OTSL newline token is not followed by a literal newline") |
Comment on lines
+30
to
+36
| def test_paired_metrics_uses_the_removed_item_denominator_for_loo() -> None: | ||
| baseline = [(8, 10), (90, 100)] | ||
| candidate = [(10, 10), (89, 100)] | ||
| summary = summarize_paired(baseline, candidate) | ||
| assert summary.delta_correct == 1 | ||
| assert summary.largest_absolute_delta == 2 | ||
| assert summary.leave_one_out_delta_pp == -1.0 |
Comment on lines
+11
to
+18
| def test_native_otsl_preserves_one_literal_newline_per_row_boundary() -> None: | ||
| assert to_native_otsl("<fcel>A<nl>\n<fcel>B<nl>") == "<fcel>A<nl>\n<fcel>B<nl>" | ||
|
|
||
|
|
||
| def test_decoded_otsl_removes_only_chat_controls() -> None: | ||
| text = normalize_decoded_otsl("<|im_end|><fcel>A<nl><fcel>B<nl>") | ||
| assert text == "<fcel>A<nl>\n<fcel>B<nl>" | ||
| assert count_structure_tokens(text)["<fcel>"] == 2 |
Introduce terminal-blind pair generation, fail-closed isolation audits, exact Raw rollback checks, and token-geometry fixtures so candidate routing can be evaluated without benchmark leakage. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Safety and claim boundaries
candidate_generation_evaluated=false; they do not claim a deployable candidate generator.Validation
git diff --checkpassed.py_compile.Notes
synthesisdependency group.