diff --git a/docs/design/dsh5-10-replay-preference-rows.md b/docs/design/dsh5-10-replay-preference-rows.md index af4df9ac2..005797dfa 100644 --- a/docs/design/dsh5-10-replay-preference-rows.md +++ b/docs/design/dsh5-10-replay-preference-rows.md @@ -1,6 +1,6 @@ # DSH5-10: replay-grounded preference rows from undo/redo history (SLM-418) -**Status:** partial slice, in progress (sixth increment). +**Status:** partial slice, in progress (eighth increment). **Claim class:** `wiring`. **Honest verdict:** not yet dispositioned -- this PR extends a scoped subset, not the full issue. @@ -14,16 +14,22 @@ held-out measurement of action/operator/argument/reference/branch accuracy, calibration, and CAP0/CAP1/CAP2 retention. The fifth slice added the seventh and final named pattern, bringing -extraction coverage to 7 of 7 (see "Fifth slice (v6)" below). This slice -starts on the issue's remaining training/measurement scope: it adds the -first (and, until now, entirely missing) converter from an extracted row to -the `PreferencePair` shape `scripts/train_preference.py` actually consumes --- see "Sixth slice" below. It still does not run any training or measure -any held-out benefit. What remains after this slice is: wiring the rendered -pairs (or a purpose-built successor shape) into an actual training run -against the DSH3-selected policy/control heads, the four-baseline -comparison, held-out benefit measurement, and turn-depth/context-view -ablations. +extraction coverage to 7 of 7 (see "Fifth slice (v6)" below). The sixth +slice added the first (and, until then, entirely missing) converter from an +extracted row to the `PreferencePair` shape `scripts/train_preference.py` +actually consumes. The seventh slice ran the first real +(`fixture_or_scratch`) end-to-end pass: a scratch SFT checkpoint, a demo +replay-preference pairs corpus covering 3 of the 7 named patterns, and one +bounded `scripts/train_preference.py train` call against it. This slice +extends the demo corpus to a fourth pattern, `merge_success` -- the one +pattern the seventh slice's trace-scan corpus could not reach -- and reruns +the same training chain against the now-richer 3-pair corpus; see "Eighth +slice" below. This is still not the issue's actual training/measurement +claim: what remains is a *real* pairs corpus (no captured +conversation-trace data exists anywhere in this repo -- see below), +training against the DSH3-selected policy/control heads specifically (not +the generic TwoTower pair format used here), the four-baseline comparison, +held-out benefit measurement, and turn-depth/context-view ablations. ## What this PR delivers @@ -352,12 +358,128 @@ training/evaluation work enumerated above. component's behavior changed (`dsl.operators.replay_preference` stays at `v6` -- this slice only *consumes* its existing public API). +## Seventh slice + +* New script `scripts/build_replay_preference_pairs.py`: builds one small, + deterministic, honestly-labeled scratch conversation (`build_demo_trace`) + exercising three of the seven named patterns (edit-then-undo, + undo-then-redo, checkout-another-state) with a toy zero-argument cycling + operator, extracts rows, renders them via the sixth slice's + `render_replay_preference_pairs`, and writes a real `pairs.jsonl` via the + existing `write_pairs`. This is the **first real, on-disk pairs corpus + this feature line has ever produced** -- everything before this slice was + either an in-memory row/pair in a unit test, or a function that could + render one but had never been run outside `pytest`. +* **No real corpus exists to build from.** Confirmed again this slice (grep + for `"conversation_trace"` / `"schema": "conversation_trace` across + `src/slm_training/resources/`): zero persisted `ConversationTraceV1` + records anywhere in this repo, and no harness ingests captured + conversation history (`build_symbolic_operator_corpus` in + `harnesses/train_data/operator_corpus.py` *synthesizes* traces + combinatorially from existing gold DSL records; it does not read + real/captured usage). The demo trace here is explicitly scratch, not a + stand-in for that missing corpus. +* **2 of 3 rows render, honestly.** `edit_then_undo` and + `checkout_another_state` render real, non-degenerate pairs. + `undo_then_redo` never does, for a structural reason, not a bug: for + *any* deterministic, zero-argument operator, `redo` and "reapply the same + operator at the same input state" are, by construction, the identical + resulting text, so the renderer's own dedup guard + (`render_replay_preference_pair`) correctly declines rather than emitting + a self-contradictory pair. The script's own printed report says so + (`pairs_dropped: 1`) rather than silently hiding it. +* **First real training run using DSH5-10 rows, full pipeline, one command + chain:** + ```bash + python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \ + --model twotower --context-backend scratch --steps 8 \ + --run-id replay_pref_sft_ckpt --no-sync-checkpoints --device cpu --seed 0 + python -m scripts.build_replay_preference_pairs \ + --out outputs/data/preference/replay_demo_pairs.jsonl + python -m scripts.train_preference train \ + --checkpoint outputs/runs/replay_pref_sft_ckpt/checkpoints/last.pt \ + --pairs outputs/data/preference/replay_demo_pairs.jsonl \ + --out-dir outputs/runs/replay_pref_dpo --steps 6 --device cpu + ``` + SFT step: `last_loss=32.610084533691406` -- identical to every prior + `wf_smoke_v2`/seed-0/8-step row in + `docs/design/autotrain-loop-ledger-20260725.md` (16+ prior independent + reproductions), confirming this checkpoint is the same deterministic + artifact those rows already verified, not a new unverified path. + Preference step: `{"steps": 6, "last_loss": 1.0767018795013428, + "mean_loss": 0.9917331635951996, "n_pairs": 2, "reference_free": true}` + (`outputs/runs/replay_pref_dpo/preference_summary.json`, not committed -- + `outputs/` is gitignored). Both commands completed in well under + `MAX_RUN_MINUTES=3` (SFT ~10s per the ledger's own prior timings for this + exact recipe; the 6-step preference pass over 2 pairs on CPU is + comparably fast). +* **Still not a training or held-out-benefit claim.** `n_pairs=2` on a + scratch fixture with no held-out split is `fixture_or_scratch` wiring + evidence that the pipeline *runs end to end for real* -- SFT checkpoint + in, DSH5-10-extracted-and-rendered pairs in, a real + `train_preference.py train` loss trajectory out. It says nothing about + whether this signal helps the model, generalizes, or should train the + DSH3-selected policy head the issue actually asks about. +* `harness.preference.replay_pairs` bumped `v1` -> `v2` in + `src/slm_training/resources/versions.json` (adds the new script + test to + its watched paths). + +## Eighth slice + +* Adds `build_demo_merge_scenario` to `scripts/build_replay_preference_pairs.py`: + a second scratch fixture -- two branches forked from a shared base editing + disjoint node refs (title vs body) -- mirroring the exact disjoint-target + shape `tests/test_dsl/test_operator_merge.py` already verifies merges + cleanly, replayably, and order-invariantly. This reaches `merge_success`, + the one named pattern the seventh slice's single-trace corpus structurally + cannot: `extract_merge_preference_row` never operates on a shared + `ConversationTraceV1` (see the sixth slice's `merge_node_resolver`), so it + needs its own two-branch construction rather than another turn in the + same trace. +* `main()` now combines both sources into one report and one `pairs.jsonl`: + 4 rows total (3 from the trace-scan corpus + 1 `merge_success`), 3 render. + `undo_then_redo` is still the only drop, for the same structural reason + the seventh slice documented (never a bug to fix on this fixture family). +* Reran the full training chain against the now-richer corpus: + ```bash + python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \ + --model twotower --context-backend scratch --steps 8 \ + --run-id replay_pref_sft_ckpt2 --no-sync-checkpoints --device cpu --seed 0 + python -m scripts.build_replay_preference_pairs \ + --out outputs/data/preference/replay_demo_pairs_v2.jsonl + python -m scripts.train_preference train \ + --checkpoint outputs/runs/replay_pref_sft_ckpt2/checkpoints/last.pt \ + --pairs outputs/data/preference/replay_demo_pairs_v2.jsonl \ + --out-dir outputs/runs/replay_pref_dpo2 --steps 9 --device cpu + ``` + SFT step: `last_loss=32.610084533691406` again -- the same deterministic + artifact every prior `wf_smoke_v2`/seed-0/8-step row in the smoke-loop + ledger reproduces. Preference step, now over 3 pairs instead of 2: + `{"steps": 9, "last_loss": 0.5314897894859314, "mean_loss": + 0.7201318964362144, "n_pairs": 3, "reference_free": true}` + (`outputs/runs/replay_pref_dpo2/preference_summary.json`, not committed). + Both commands again well under `MAX_RUN_MINUTES=3`. +* **Still not a training or held-out-benefit claim** -- `n_pairs=3` on a + scratch fixture is a larger, more structurally diverse smoke corpus (now + covering 4 of the 7 named patterns instead of 3), not evidence the signal + helps the model or generalizes. The three remaining un-exercised-in-a-script + patterns (`partial_rollback`, `fork_then_choose_one_branch`, and + `pronoun_focus_followup`) are left for a future slice rather than piling + more scratch fixtures onto this one; see the doc's still-open + training/measurement scope above. +* `harness.preference.replay_pairs` bumped `v2` -> `v3` in + `src/slm_training/resources/versions.json`. + ## Reproducibility ```bash -NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_harnesses/preference/test_replay_pairs.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py +NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_harnesses/preference/test_replay_pairs.py tests/test_scripts/test_build_replay_preference_pairs.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py ``` Result (fifth-slice PR, real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. -Result (this PR, sixth slice, real run in a fresh `.venv-dsh510` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `env -u NODE_OPTIONS npm ci` in `src/apps/openui_bridge` -- the ambient `NODE_OPTIONS="--import tsx" --max-old-space-size=8192` is rejected outright by Node for both `npm ci` and `pytest` in this environment, so it has to be unset, not just locally overridden, unlike the fifth slice's note above): `69 passed` against `main` HEAD `5f94b92` (includes the fifth slice, already merged). Also verified: `ruff check` clean on both new files; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. No training run in this slice; `outputs/` untouched. +Result (sixth-slice PR #1124, real run in a fresh `.venv-dsh510` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `env -u NODE_OPTIONS npm ci` in `src/apps/openui_bridge` -- the ambient `NODE_OPTIONS="--import tsx" --max-old-space-size=8192` is rejected outright by Node for both `npm ci` and `pytest` in this environment, so it has to be unset, not just locally overridden, unlike the fifth slice's note above): `69 passed` against `main` HEAD `5f94b92` (includes the fifth slice, already merged). Also verified: `ruff check` clean on both new files; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. No training run in this slice; `outputs/` untouched. + +Result (seventh-slice PR #1125, real run in a fresh `.venv-dsh510`, same environment recipe as the sixth slice above, stacked on top of PR #1124 which was still unmerged when this slice started): `71 passed` (69 from the sixth slice + 2 new). Also verified: `ruff check` clean on both new files; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`. Plus the real training run described above (SFT checkpoint + demo pairs + preference-training pass, both commands well under `MAX_RUN_MINUTES=3`); its `outputs/runs/replay_pref_sft_ckpt/` and `outputs/runs/replay_pref_dpo/` are not committed (`outputs/` is gitignored) per this repo's checked-not-committed convention for scratch run artifacts. + +Result (this PR, eighth slice, real run in a fresh `.venv-dsh510`, same environment recipe as above, stacked on top of PR #1125 which was still unmerged when this slice started): `72 passed` (71 from the seventh slice + 1 new). Also verified: `ruff check` clean; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. Plus the reran training chain described above; `outputs/runs/replay_pref_sft_ckpt2/` and `outputs/runs/replay_pref_dpo2/` are not committed (`outputs/` is gitignored). diff --git a/scripts/build_replay_preference_pairs.py b/scripts/build_replay_preference_pairs.py new file mode 100644 index 000000000..557c8d1c8 --- /dev/null +++ b/scripts/build_replay_preference_pairs.py @@ -0,0 +1,432 @@ +#!/usr/bin/env python3 +"""Build a first replay-preference-pairs corpus and report it honestly. + +SLM-418 (DSH5-10)'s extraction (7/7 patterns) and rendering +(``slm_training.harnesses.preference.replay_pairs``, sixth slice) both +exist, but no real captured multi-turn ``ConversationTraceV1`` corpus does +-- confirmed while scoping this slice: `resources/data/` has no persisted +``conversation_trace/v1`` records anywhere, and no harness ingests one (see +``docs/design/dsh5-10-replay-preference-rows.md``). This script does not +invent one. It builds a single small, deterministic, self-contained +conversation exercising several of the seven named patterns (edit-then- +undo, undo-then-redo, checkout-another-state) with the same toy +zero-argument cycling operator this repo's own DSH5-10 test suite uses, so +the pipeline (extract -> render -> write pairs.jsonl) can be run and +measured end to end for real. This is ``fixture_or_scratch`` corpus, the +same honesty class as the smoke-loop recipes in +``docs/design/autotrain-loop-ledger-20260725.md`` -- not a claim about real +usage patterns, model quality, or held-out benefit. + +One of the four rows this produces (``undo_then_redo``) never renders: for +any deterministic, zero-argument operator, ``redo`` and "reapply the same +operator at the same input state" are, by definition, the identical text -- +no cycle-length trick avoids this; only a genuinely non-deterministic or +argument-bearing operator would. The renderer's own dedup guard +(``render_replay_preference_pair`` in +``slm_training.harnesses.preference.replay_pairs``) correctly declines +rather than emitting a self-contradictory pair -- see this script's own +``pairs_dropped`` in its printed report, and +``test_undo_then_redo_row_declines_a_degenerate_collapse`` in +``tests/test_harnesses/preference/test_replay_pairs.py``. + +A second, separate scenario (``build_demo_merge_scenario``) covers +``merge_success``, the one named pattern ``build_demo_trace`` cannot reach: +``extract_merge_preference_row`` never operates on a shared +``ConversationTraceV1`` the way the other six patterns do (see +``merge_node_resolver`` in ``slm_training.harnesses.preference.replay_pairs``), +so it needs its own two-branch fixture with disjoint-target edits, mirroring +the shape ``tests/test_dsl/test_operator_merge.py`` already verifies is +valid, replayable, and order-invariant. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from dataclasses import replace +from pathlib import Path + +from slm_training.dsl.operators import ( + ActionEffectV1, + ApplicationProvenanceV1, + AstOperatorV1, + BranchEditV1, + CompilerCoverage, + ConversationStateNodeV1, + EffectDeltaKind, + EffectDeltaV1, + OperatorLibraryV1, + OperatorMutationV1, + OperatorReplayPreferenceRowV1, + OperatorStateV1, + RefKind, + ReferenceDescriptorV1, + RegisteredOperatorV1, + append_operator_turn, + branch_fingerprint, + build_reference_table, + checkout_conversation_state, + clone_reference_table_for_branch, + create_conversation_trace, + extract_merge_preference_row, + extract_replay_preference_rows, + merge_conversation_branches, + redo_conversation, + undo_conversation, +) +from slm_training.dsl.operators.conversation import ConversationTraceV1 +from slm_training.dsl.pack import DslPack, get_pack +from slm_training.harnesses.preference import PreferencePair, write_pairs +from slm_training.harnesses.preference.replay_pairs import ( + merge_node_resolver, + render_replay_preference_pair, + render_replay_preference_pairs, +) + +_MERGE_SOURCE = 'root = Card([TextContent(":hero.title"), TextContent(":hero.body")], "clear")' + +_SOURCE = 'root = TextContent(":hero.title")' +_OPERATOR_ID = "openui.demo_cycle_text" +_REQUEST_ID = "replay-pairs-demo" + + +def _sha(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def _provenance(state: OperatorStateV1) -> ApplicationProvenanceV1: + return ApplicationProvenanceV1( + pack_id="openui", + compiler_id="scripts.build_replay_preference_pairs", + compiler_version="v1", + source_artifact_digest=_sha(state.source), + request_id=_REQUEST_ID, + ) + + +def _table(state: OperatorStateV1, branch: str, *, seed: int): + return build_reference_table( + request_id=_REQUEST_ID, + state_digest=state.state_digest, + branch_digest=branch, + descriptors=( + ReferenceDescriptorV1( + ref_kind=RefKind.VALUE, + semantic_fingerprint=_sha("demo-value"), + value_type="openui.string", + ), + ), + seed=seed, + ) + + +def _execute(state: OperatorStateV1, _arguments) -> OperatorMutationV1: + # A 4-state cycle, not 3: long enough that the checkout-back-to-root + # target in build_demo_trace() below doesn't coincidentally equal what + # one more deterministic reapplication of this same operator would + # produce at the checkout decision state (which would make the + # rendered pair degenerate -- see the module's own + # undo_then_redo note for why that's unavoidable for a *different* + # reason on that pattern, but avoidable here by construction). + replacements = ( + (":hero.title", ":hero.body"), + (":hero.body", ":hero.caption"), + (":hero.caption", ":hero.footer"), + (":hero.footer", ":hero.title"), + ) + for before, after in replacements: + if before in state.source: + return OperatorMutationV1( + source=state.source.replace(before, after), + effect=ActionEffectV1(compiler_coverage=CompilerCoverage.EXACT), + ) + raise ValueError("demo.no_transition") + + +def build_demo_trace() -> tuple[DslPack, OperatorLibraryV1, ConversationTraceV1]: + """A single scratch trace exercising three of the seven named patterns. + + root -> edit -> undo (edit_then_undo) -> redo (undo_then_redo) -> edit + -> checkout(root) (checkout_another_state). Deterministic (fixed seeds, + no randomness); see the module docstring for why this is not real + captured conversation data. + """ + base_pack = get_pack("openui") + root_state = OperatorStateV1.from_source(base_pack, _SOURCE) + declaration = AstOperatorV1( + operator_id=_OPERATOR_ID, + version="v1", + domain="openui.ast", + codomain="openui.ast", + argument_slots=(), + preconditions=(), + effect_signature=(), + locality="node", + cost=1.0, + ) + library = OperatorLibraryV1((RegisteredOperatorV1(declaration, _execute),)) + pack = replace(base_pack, operator_library=library) + branch = branch_fingerprint(root_state.state_digest, _sha("demo-branch")) + root_table = _table(root_state, branch, seed=1) + trace = create_conversation_trace( + pack=pack, + root_state=root_state, + root_reference_table=root_table, + provenance=_provenance(root_state), + ) + + def apply_once(current: ConversationTraceV1, *, seed: int) -> ConversationTraceV1: + result = library.apply( + pack, current.current.state, _OPERATOR_ID, (), _provenance(current.current.state) + ) + assert result.succeeded and result.state is not None + return append_operator_turn( + current, + pack=pack, + library=library, + application=result.application, + output_reference_table=_table(result.state, current.current.branch_digest, seed=seed), + ) + + edited = apply_once(trace, seed=2) + original_child_id = edited.current_state_id + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + redone = redo_conversation( + undone, target_state_id=original_child_id, provenance=_provenance(undone.current.state) + ) + edited_again = apply_once(redone, seed=3) + checked_out = checkout_conversation_state( + edited_again, + target_state_id=edited_again.root_state_id, + provenance=_provenance(edited_again.current.state), + ) + return pack, library, checked_out + + +def _merge_branch_edit( + *, + base_pack: DslPack, + base_state: OperatorStateV1, + base_node: ConversationStateNodeV1, + root_table, + name: str, + target_name: str, + replacement: str, + seed: int, + authorities: dict, +) -> BranchEditV1: + branch = branch_fingerprint(base_state.state_digest, _sha(f"demo-merge-{name}")) + table = clone_reference_table_for_branch(root_table, branch_digest=branch, seed=seed) + input_node = ConversationStateNodeV1( + parent_state_id=base_node.state_id, + branch_digest=branch, + state=base_state, + reference_table=table, + ) + target = next( + entry.ref for entry in table.entries if entry.descriptor.value_type == f"openui.{target_name}" + ) + operator_id = f"openui.demo_merge_{name}" + declaration = AstOperatorV1( + operator_id=operator_id, + version="v1", + domain="openui.ast", + codomain="openui.ast", + argument_slots=(), + preconditions=(), + effect_signature=(EffectDeltaKind("property"),), + locality="node", + cost=1.0, + ) + before = f":hero.{target_name}" + + def execute(state: OperatorStateV1, _arguments) -> OperatorMutationV1: + if before not in state.source: + raise ValueError("demo.no_transition") + return OperatorMutationV1( + source=state.source.replace(before, replacement), + effect=ActionEffectV1( + property_deltas=( + EffectDeltaV1( + kind=EffectDeltaKind("property"), target=target, before="before", after="after" + ), + ), + compiler_coverage=CompilerCoverage.EXACT, + ), + ) + + branch_library = OperatorLibraryV1((RegisteredOperatorV1(declaration, execute),)) + branch_pack = replace(base_pack, operator_library=branch_library) + applied = branch_library.apply( + branch_pack, input_node.state, operator_id, (), _provenance(input_node.state) + ) + assert applied.succeeded and applied.state is not None + output_table = build_reference_table( + request_id=table.request_id, + state_digest=applied.state.state_digest, + branch_digest=branch, + descriptors=tuple(entry.descriptor for entry in table.entries), + seed=seed + 10, + ) + output_node = ConversationStateNodeV1( + parent_state_id=input_node.state_id, + branch_digest=branch, + state=applied.state, + reference_table=output_table, + ) + authorities[input_node.state_id] = (branch_pack, branch_library) + return BranchEditV1(input_node, output_node, applied.application) + + +def _rebuild_merged_table(_pack, merged_state: OperatorStateV1, branch_digest: str, seed: int): + """Fixture-only compiler facts, derived anew from the canonical merged source.""" + descriptors = tuple( + ReferenceDescriptorV1( + ref_kind=RefKind.NODE, + semantic_fingerprint=_sha(f"canonical:{marker}"), + value_type=f"openui.{marker.removeprefix(':hero.')}", + ) + for marker in (":hero.heading", ":hero.copy") + if marker in merged_state.source + ) + return build_reference_table( + request_id=_REQUEST_ID, + state_digest=merged_state.state_digest, + branch_digest=branch_digest, + descriptors=descriptors, + seed=seed, + ) + + +def build_demo_merge_scenario() -> tuple[list[OperatorReplayPreferenceRowV1], list[PreferencePair]]: + """A second scratch scenario covering ``merge_success``, the one pattern + ``build_demo_trace`` cannot reach (see module docstring). + + Two branches fork from a shared base and edit disjoint node refs + (title vs body) -- the same disjoint-target shape + ``tests/test_dsl/test_operator_merge.py`` verifies merges cleanly, + replayably, and order-invariantly. + """ + base_pack = get_pack("openui") + base_state = OperatorStateV1.from_source(base_pack, _MERGE_SOURCE) + root_branch = branch_fingerprint(base_state.state_digest, _sha("demo-merge-root")) + descriptors = tuple( + ReferenceDescriptorV1( + ref_kind=RefKind.NODE, semantic_fingerprint=_sha(name), value_type=f"openui.{name}" + ) + for name in ("title", "body") + ) + root_table = build_reference_table( + request_id=_REQUEST_ID, + state_digest=base_state.state_digest, + branch_digest=root_branch, + descriptors=descriptors, + seed=21, + ) + base_node = ConversationStateNodeV1( + parent_state_id=None, branch_digest=root_branch, state=base_state, reference_table=root_table + ) + + authorities: dict = {} + left = _merge_branch_edit( + base_pack=base_pack, + base_state=base_state, + base_node=base_node, + root_table=root_table, + name="left", + target_name="title", + replacement=":hero.heading", + seed=22, + authorities=authorities, + ) + right = _merge_branch_edit( + base_pack=base_pack, + base_state=base_state, + base_node=base_node, + root_table=root_table, + name="right", + target_name="body", + replacement=":hero.copy", + seed=23, + authorities=authorities, + ) + + def resolve(node: ConversationStateNodeV1): + return authorities[node.state_id] + + decision = merge_conversation_branches( + pack=base_pack, + base=base_node, + left=left, + right=right, + authority_resolver=resolve, + reference_table_builder=_rebuild_merged_table, + ) + if not decision.succeeded or decision.continuation is None: + raise RuntimeError("demo merge scenario did not produce a mergeable pair") + + row = extract_merge_preference_row( + left=left, right=right, decision=decision, authority_resolver=resolve, provenance_for=_provenance + ) + if row is None: + return [], [] + + left_pack, left_library = resolve(left.input_node) + pair = render_replay_preference_pair( + row, + resolve_node=merge_node_resolver(left, right, decision), + pack=left_pack, + library=left_library, + provenance_for=_provenance, + ) + return [row], ([pair] if pair is not None else []) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--out", type=Path, default=Path("outputs/data/preference/replay_demo_pairs.jsonl") + ) + args = parser.parse_args(argv) + + pack, library, trace = build_demo_trace() + trace_report = extract_replay_preference_rows( + trace, pack=pack, library=library, provenance_for=_provenance + ) + trace_pairs = render_replay_preference_pairs( + trace_report.rows, + resolve_node=trace.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + merge_rows, merge_pairs = build_demo_merge_scenario() + + all_rows = list(trace_report.rows) + merge_rows + all_pairs = trace_pairs + merge_pairs + counts_by_relation = dict(trace_report.counts_by_relation) + for row in merge_rows: + key = row.semantic_relation.value + counts_by_relation[key] = counts_by_relation.get(key, 0) + 1 + + n = write_pairs(args.out, all_pairs) + print( + json.dumps( + { + "rows": len(all_rows), + "counts_by_relation": counts_by_relation, + "pairs_rendered": n, + "pairs_dropped": len(all_rows) - n, + "out": str(args.out), + "corpus_kind": "fixture_or_scratch", + }, + indent=2, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/slm_training/resources/versions.json b/src/slm_training/resources/versions.json index 2de883482..b6ffb2ea2 100644 --- a/src/slm_training/resources/versions.json +++ b/src/slm_training/resources/versions.json @@ -7942,13 +7942,25 @@ ] }, "harness.preference.replay_pairs": { - "version": "v1", + "version": "v3", "kind": "harness", "paths": [ "src/slm_training/harnesses/preference/replay_pairs.py", - "tests/test_harnesses/preference/test_replay_pairs.py" + "tests/test_harnesses/preference/test_replay_pairs.py", + "scripts/build_replay_preference_pairs.py", + "tests/test_scripts/test_build_replay_preference_pairs.py" ], "history": [ + { + "version": "v3", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) eighth slice: adds build_demo_merge_scenario, a second scratch fixture (two branches with disjoint-target edits, mirroring tests/test_dsl/test_operator_merge.py's own verified-mergeable shape) covering merge_success -- the one named pattern the original build_demo_trace could not reach, since extract_merge_preference_row never operates on a shared ConversationTraceV1. main() now combines both sources: 4 rows (3 from the trace + 1 merge), 3 render (still only undo_then_redo drops, for the structural reason documented in the seventh slice), 1 combined pairs.jsonl. No change to the sixth slice's renderer itself" + }, + { + "version": "v2", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) seventh slice: scripts/build_replay_preference_pairs.py builds a single small, deterministic, honestly-labeled fixture_or_scratch conversation (no real captured ConversationTraceV1 corpus exists anywhere in the repo -- confirmed this slice) exercising three of the seven named patterns, extracts rows, renders them via render_replay_preference_pairs, and writes a real pairs.jsonl. 2 of 3 rows render (undo_then_redo never does -- for any deterministic zero-argument operator, redo and reapplying the same operator at the same input state are, by definition, textually identical, so the renderer's dedup guard correctly declines it rather than emitting a self-contradictory pair). First real, on-disk pairs corpus this feature line has ever produced; still wiring/demo, no training run in this component" + }, { "version": "v1", "date": "2026-07-27", diff --git a/tests/test_scripts/test_build_replay_preference_pairs.py b/tests/test_scripts/test_build_replay_preference_pairs.py new file mode 100644 index 000000000..e139b9b65 --- /dev/null +++ b/tests/test_scripts/test_build_replay_preference_pairs.py @@ -0,0 +1,73 @@ +"""SLM-418 (DSH5-10): the demo replay-preference-pairs corpus builder script.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from scripts.build_replay_preference_pairs import ( + _provenance, + build_demo_merge_scenario, + build_demo_trace, + main, +) +from slm_training.dsl.operators import ReplayPreferenceRelation, extract_replay_preference_rows +from slm_training.harnesses.preference import load_pairs + + +def test_build_demo_trace_exercises_three_named_patterns() -> None: + pack, library, trace = build_demo_trace() + + report = extract_replay_preference_rows( + trace, pack=pack, library=library, provenance_for=_provenance + ) + assert set(report.counts_by_relation) == { + ReplayPreferenceRelation.EDIT_THEN_UNDO.value, + ReplayPreferenceRelation.UNDO_THEN_REDO.value, + ReplayPreferenceRelation.CHECKOUT_ANOTHER_STATE.value, + } + + +def test_build_demo_merge_scenario_yields_one_real_merge_success_pair() -> None: + rows, pairs = build_demo_merge_scenario() + + assert len(rows) == 1 + assert rows[0].semantic_relation is ReplayPreferenceRelation.MERGE_SUCCESS + assert len(pairs) == 1 + pair = pairs[0] + assert pair.meta["pair_corpus"] == "replay_preference" + assert pair.meta["semantic_relation"] == "merge_success" + assert pair.chosen != pair.rejected + + +def test_main_writes_a_real_pairs_file_and_reports_honest_counts( + tmp_path: Path, capsys +) -> None: + out_path = tmp_path / "replay_demo_pairs.jsonl" + + exit_code = main(["--out", str(out_path)]) + + assert exit_code == 0 + report = json.loads(capsys.readouterr().out) + assert report["corpus_kind"] == "fixture_or_scratch" + assert report["rows"] == 4 + assert report["counts_by_relation"] == { + "edit_then_undo": 1, + "undo_then_redo": 1, + "checkout_another_state": 1, + "merge_success": 1, + } + # undo_then_redo never renders here: redo and "reapply the same + # deterministic zero-arg operator" are the identical text by + # construction, so the renderer's dedup guard correctly declines it. + assert report["pairs_rendered"] == 3 + assert report["pairs_dropped"] == 1 + + pairs = load_pairs(out_path) + assert len(pairs) == report["pairs_rendered"] + relations = {pair.meta["semantic_relation"] for pair in pairs} + assert relations == {"edit_then_undo", "checkout_another_state", "merge_success"} + for pair in pairs: + assert pair.meta["pair_corpus"] == "replay_preference" + assert pair.chosen != pair.rejected + assert pair.prompt