Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/design/iter-slm250-lot1-01-not-authorized-20260725.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"contract_hash": "2fad1057fe77352daebf2b8b792d599cf62494ca023169da0570bab9e133d3f7",
"contract_id": "lotus-openui-model-contract-v1",
"gate_1_fidelity_authorization": {
"actual_verdict": "needs_target_trace_contract",
"gate_name": "fidelity_contract_authorization",
"met": false,
"rationale": "Differentiation from every existing repo mechanism (masked recursive denoiser, SemanticPlanV1 predictors/consumers, causal FTPO adapters, explicit compiler traces, valid-state tree-edit/lattice search) is clean and survives line-by-line comparison -- see non_duplication_map, all rows no_duplication. The blocker is narrower: LOTUS's explicit-to-latent curriculum (mechanism row 5) and its causal_latent_use claim class both presuppose a step-decomposable 'explicit trace' analogous to GSM8K CoT steps that this repo does not yet define for OpenUI program synthesis. Defining that target-trace contract honestly -- without assuming math-payload equivalence to GSM8K, and without silently reusing causal_trace.py's visible-token decode capture or SemanticPlanV1's plan IR as if either already were that trace -- is out of scope for this docs/spec-only issue (no corpus generation beyond bounded target-contract probes). Authorizing bounded implementation now would force either an undefined curriculum or a quietly borrowed non-LOTUS trace source; neither is honest. This is not a 'differentiation is weak' close and not a semantic-floor block -- it is a scoped prerequisite.",
"required_verdict": "authorize_bounded_implementation",
"source_contract_hash": "801ce267b64f52b88e6e80fa091084c5f1a6628de60658d2d161db82f5117af2",
"source_contract_id": "lotus-openui-fidelity-contract-v1",
"source_linear_issue": "SLM-248"
},
"gate_2_trace_oracle_ceiling": {
"actual_verdict": "inconclusive",
"gate_name": "trace_oracle_ceiling",
"met": false,
"rationale": "The trace contract, typed step schema, deterministic extractor, and lossless visible serialization are implemented and tested against the repository's existing SemanticPlanV1/production_codec owners (no new compiler/parser/evaluator). A bounded n=16 fixture probe (not a production corpus) confirms every probe record decomposes into all six declared stages with zero truncation, giving a provisional K=6/c=479-chars budget. The oracle ceiling experiment itself -- the only evidence that could support an oracle_ceiling_positive, no_downstream_ceiling, or explicit_trace_equivalent_to_existing_plan verdict -- is specified as a matched multi-arm, multi-seed plan but deliberately not run: the LOT0-01 authorization scopes this issue to a fixture-scale plan only, with GPU training and corpus generation explicitly blocked. Declaring any positive or negative ceiling verdict without running that plan would be exactly the 'no claim that explicit trace benefit implies latent benefit' failure this issue's acceptance criteria forbid. leakage_or_ambiguity_blocked and target_support_insufficient are also not honest: no leakage was found and the bounded probe supports the stage set within its declared scope. inconclusive -- pending the oracle-ceiling campaign -- is the only accurate verdict.",
"required_verdict": "oracle_ceiling_positive",
"source_contract_hash": "6cdd695bf5b32036519f1ee2e787a44e4afbb045fc059f75a92a2d069f0ee7c5",
"source_contract_id": "compiler-reasoning-trace-v1",
"source_linear_issue": "SLM-249"
},
"linear_issue": "SLM-250",
"schema_version": "lotus_openui_model_contract/v1",
"verdict": "not_authorized",
"verdict_rationale": "Hard activation gate(s) unmet: fidelity_contract_authorization, trace_oracle_ceiling. Gate 1 (SLM-248 fidelity authorization) reports verdict 'needs_target_trace_contract' (requires 'authorize_bounded_implementation'). Gate 2 (SLM-249 trace oracle ceiling) reports verdict 'inconclusive' (requires 'oracle_ceiling_positive'). Closing not_authorized in plan-only mode; no K x c model or training code is added by this disposition.",
"version_stamp": {
"code_commit": "79810740906c5eeae601b0379fa083f4c48f0d68",
"code_dirty": true,
"components": {
"harness.experiments": "v111"
},
"stamp_schema": "version_stamp/v1",
"stamped_at": "2026-07-25T15:39:13.771319+00:00"
}
}
93 changes: 93 additions & 0 deletions docs/design/iter-slm250-lot1-01-not-authorized-20260725.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# SLM-250 LOT1-01 — not_authorized disposition

## What

LOT1-01 asks to implement the faithful causal K×c looped-latent model path,
gated on two hard activation gates against its own upstream contracts:

1. SLM-248 (LOT0-01) `LotusOpenUIFidelityContractV1.authorization.verdict`
must be `authorize_bounded_implementation`.
2. SLM-249 (LOT0-02) `CompilerReasoningTraceGateV1.gate.verdict` must be
`oracle_ceiling_positive` (or another explicit authorization supplying
K/c/stage targets).

"Otherwise close `not_authorized` in plan-only mode without production model
code."

## Evaluated result

Both gates were evaluated against the real, committed upstream artifacts:

- **Gate 1** (`docs/design/lotus-openui-fidelity-contract-v1.json`): actual
verdict is `needs_target_trace_contract`, not `authorize_bounded_implementation`.
**Unmet.**
- **Gate 2** (`docs/design/compiler-reasoning-trace-v1.json`): actual verdict
is `inconclusive`, not `oracle_ceiling_positive`; the gate's own
`allowed_lot1_implementation` field explicitly says
`"none: ... not authorized by this issue"`. **Unmet.**

Both gates unmet ⇒ **verdict: `not_authorized`**. No K×c latent-workspace
model, loop driver, curriculum hooks, or training code is added.

## What was built instead

Since evaluating this gate honestly *is* the LOT1-01 deliverable when the
gate is unmet, this issue adds a small, reusable, tested activation-gate
evaluator rather than a bare Linear comment:

- `src/slm_training/harnesses/experiments/lot1_01_activation_gate.py` —
`LotusOpenUIModelContractV1` schema, `GateEvaluation`, and
`evaluate_activation_gates()`, a pure function that reads the two upstream
contract dicts and derives the verdict from their real published fields
(never hardcoded to always fail — synthetic contracts reporting both
required verdicts flip the result to `authorized_wiring_only`, tested).
- `scripts/evaluate_lot1_01_activation_gate.py` — plan-only CLI; loads the
two real committed JSON artifacts and emits the disposition. No model
import, no training, no GPU path.
- Tests proving: the real current contracts yield `not_authorized`; a
synthetic both-gates-met case yields `authorized_wiring_only`; a
mixed/missing-fields case fails closed to `not_authorized`; the contract
hash is stable and changes with the verdict.

This evaluator is reusable: once SLM-249 is rerun with a real oracle-ceiling
campaign (or SLM-248's verdict changes), re-running the same CLI against the
updated artifacts will honestly reflect the new disposition without any
narrative rewrite.

## Files added

- `src/slm_training/harnesses/experiments/lot1_01_activation_gate.py`
- `scripts/evaluate_lot1_01_activation_gate.py`
- `tests/test_harnesses/experiments/test_lot1_01_activation_gate.py`
- `tests/test_scripts/test_evaluate_lot1_01_activation_gate.py`
- `docs/design/iter-slm250-lot1-01-not-authorized-20260725.md`
- `docs/design/iter-slm250-lot1-01-not-authorized-20260725.json`

## Commands

```bash
python -m scripts.evaluate_lot1_01_activation_gate \
--fidelity-contract docs/design/lotus-openui-fidelity-contract-v1.json \
--trace-gate-contract docs/design/compiler-reasoning-trace-v1.json \
--out outputs/runs/slm250_activation_gate
```

## Verification

- `pytest tests/test_harnesses/experiments/test_lot1_01_activation_gate.py tests/test_scripts/test_evaluate_lot1_01_activation_gate.py -q` → 9 passed
- `python -m scripts.verify_version_stamps --check` → ok

## Acceptance criteria mapping

- "close `not_authorized` in plan-only mode without production model code" —
satisfied exactly; no model/training file is touched.
- "no RSC/TwoTower path is reused as the treatment" — no model path is
touched at all.
- "no production default change" — satisfied.
- LOT1-02/LOT2/LOT3/LOT4 remain gated behind LOT1-01, unchanged.

## Non-goals honored

No K×c latent-workspace model code, no learned plan predictor, no causal
intervention campaign, no large training run or quality claim, no adaptive
depth/halting, no production default change.
64 changes: 64 additions & 0 deletions scripts/evaluate_lot1_01_activation_gate.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""Evaluate SLM-250 (LOT1-01)'s hard activation gates in plan-only mode.

No model, training, or K x c workspace code is loaded or executed. This
reads the two real committed upstream contract artifacts and emits the
required ``LotusOpenUIModelContractV1`` disposition.

Example:
python -m scripts.evaluate_lot1_01_activation_gate \
--fidelity-contract docs/design/lotus-openui-fidelity-contract-v1.json \
--trace-gate-contract docs/design/compiler-reasoning-trace-v1.json \
--out outputs/runs/slm250_activation_gate
"""

from __future__ import annotations

import argparse
from pathlib import Path

from slm_training.harnesses.experiments.lot1_01_activation_gate import (
evaluate_activation_gates,
load_upstream_contract,
render_markdown,
)


def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description="SLM-250 LOT1-01 hard-activation-gate evaluator (plan-only, no model code)"
)
parser.add_argument(
"--fidelity-contract",
type=Path,
default=Path("docs/design/lotus-openui-fidelity-contract-v1.json"),
help="SLM-248 LotusOpenUIFidelityContractV1 JSON artifact",
)
parser.add_argument(
"--trace-gate-contract",
type=Path,
default=Path("docs/design/compiler-reasoning-trace-v1.json"),
help="SLM-249 CompilerReasoningTraceGateV1 JSON artifact",
)
parser.add_argument(
"--out",
type=Path,
default=Path("outputs/runs/slm250_activation_gate"),
)
args = parser.parse_args(argv)

fidelity_contract = load_upstream_contract(args.fidelity_contract)
trace_gate_contract = load_upstream_contract(args.trace_gate_contract)

contract = evaluate_activation_gates(fidelity_contract, trace_gate_contract)

args.out.mkdir(parents=True, exist_ok=True)
contract.to_json(args.out / "lotus_openui_model_contract.json")
markdown = render_markdown(contract)
(args.out / "lotus_openui_model_contract.md").write_text(markdown, encoding="utf-8")
print(markdown)
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading