test(predict-rlm): consolidate core regression suite

.github/workflows/tests.yml

@@ -7,9 +7,8 @@ on:
     branches: [main]
 
 jobs:
-  # Core package, no optional extras installed. Proves predict_rlm and its core
-  # logic import and pass standalone. Optional-subsystem modules (sbx/gepa/codex_lm)
-  # are deselected here, so a green run is a real "works without extras" signal.
+  # Core package plus local CPython subprocess contracts, without optional
+  # extras, Deno execution, or external sandbox services.
   unit-core:
     runs-on: ubuntu-latest
     strategy:
@@ -157,7 +156,12 @@ jobs:
         run: uv sync --python 3.12
 
       - name: Run JSPI integration tests
-        run: uv run pytest tests/ -m "integration and not sbx and not local" -q --cov=predict_rlm --cov-report=xml
+        env:
+          PYTHONUNBUFFERED: "1"
+        run: >-
+          uv run pytest tests/ -m "integration and not sbx and not local"
+          -o addopts= -v --tb=short -o faulthandler_timeout=60
+          --cov=predict_rlm --cov-report=xml
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v5

ARCHITECTURE.md

@@ -256,8 +256,11 @@ Python error.
 ### JSPI / Deno / Pyodide
 
 - Successful execute: same live Pyodide VM; full globals persist.
-- Cooperative timeout: Pyodide is interrupted by trace deadline and JS interrupt
-  buffer; stdout/stderr are returned and the VM remains live.
+- Cooperative timeout: deadline tracing is scoped to generated code. The JS
+  interrupt buffer stops active user code; an interrupt in an asyncio scheduler
+  callback cancels the suspended execution task instead of raising through the
+  scheduler. stdout/stderr are returned and the VM remains live. Cleanup waits
+  for the watchdog to disarm and restores the previous Python SIGINT handler.
 - Hard timeout or crash: the host watchdog kills Deno and raises
   `SandboxFatalError`; the interpreter is dead and no state is recovered.
 

CHANGELOG.md

@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Reduced the package regression suite to behavioral contracts; consolidated
+  shared backend execution tests and removed duplicate inherited cases,
+  static/presentation checks, and one-off benchmark/example checks.
+- Direct CPython contracts now run in the core tier without the SBX extra.
+
+### Fixed
+
+- JSPI timeout cleanup now waits for the interrupt worker to disarm before
+  entering Python, and preserves the timeout response if tracing interrupts
+  cleanup instead of leaving the host waiting for an uncorrelated error.
+- JSPI deadlines no longer interrupt asyncio scheduler callbacks; suspended
+  executions are cancelled without orphaning their result promises, and the
+  previous Python SIGINT handler is restored after bounded execution.
+
 ### Breaking Changes
 
 - `SbxBackend.shutdown()` can no longer be called from the event loop that owns

Makefile

@@ -22,8 +22,8 @@ test-integration:
 
 # --- Per-suite targets (one per CI job; each installs only what it needs) ---
 
-# Core: no extras. Pure host-side logic; proves the package works standalone
-# (passes with neither Deno nor websockets present).
+# Core: no extras. Host logic and local CPython subprocess contracts;
+# no Deno execution, websockets, or external sandbox service.
 test-core:
 	uv run pytest -m "not integration and not sbx and not gepa and not codex_lm" $(ARGS)
 

README.md

@@ -354,6 +354,39 @@ result = await rlm.acall(query="...")
 Register globally instead with `dspy.configure(callbacks=[...])` and the
 same handlers fire for every `PredictRLM` instance.
 
+## Tests
+
+The default package regression suite is `tests/`; example-local suites are
+separate. Tests protect observable behavior, not exports, defaults, prompt wording,
+docstrings, source text, or mocked argument forwarding.
+
+- `tests/runtime_contracts/`: shared execution, state, submission, files, tools,
+  and recovery contracts across Direct, JSPI, and SBX. Backend-specific gaps are
+  explicit in `backends.py`; native Direct callbacks are serial, and deferred
+  submission is a Direct-only interpreter API.
+- RLM, adapter, file, and workspace tests: generated-code execution, typed
+  predictions, input/output handling, cancellation ownership, and data-loss boundaries.
+- Trace and telemetry tests: evidence projection, accounting, redaction, and
+  failure classification.
+- GEPA tests: candidate acceptance, merge guardrails, evaluation artifacts,
+  resume, and spend.
+- `tests/codex_lm/`: auth, transport completion/recovery, caching, and usage.
+
+```bash
+make test-unit              # all extras; no Deno or external SBX execution
+make test-integration-jspi  # real Deno/Pyodide contracts; no LM credentials
+make test-core             # no extras; includes local CPython processes
+make test-sbx              # local WebSocket supervisor and pool contracts
+make test-gepa
+make test-codex-lm
+```
+
+Real SBX tests require the CLI, login, and `make test-integration-sbx`.
+Bootstrap image tests additionally require Docker and
+`PREDICT_RLM_RUN_BOOTSTRAP_DOCKER_TESTS=1`. Timing-sensitive `local` tests run
+locally but are excluded from CI. Add a case only for a distinct failure or
+invariant; extend the shared contract instead of copying it into backend suites.
+
 ## Next steps
 
 - [Custom path inputs](docs/custom-path-inputs.md) — add file-, workspace-, or