Skip to content

test(predict-rlm): consolidate core regression suite - #63

Open
eriberdy022 wants to merge 5 commits into
mainfrom
refactor/core-regression-suite
Open

test(predict-rlm): consolidate core regression suite#63
eriberdy022 wants to merge 5 commits into
mainfrom
refactor/core-regression-suite

Conversation

@eriberdy022

@eriberdy022 eriberdy022 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Rationale

The default package suite had accumulated overlapping backend scenarios, repeated inherited tests, static/default/export checks, presentation assertions, and mock-only plumbing. This refactor concentrates coverage on observable regression contracts. CI also exposed a JSPI deadline bug: interrupts could abort asyncio scheduler callbacks and orphan execution results instead of returning recoverable timeouts. The runtime fixes preserve public APIs while correcting that failure.

Summary

  • Reduce collected cases from 1,475 to 389 (74%) and Python test lines from 37,418 to 12,206 (about two thirds).
  • Consolidate shared backend contracts and remove duplicate inherited SBX cases, obsolete fixtures, static/presentation checks, and one-off benchmark/example checks from the default suite.
  • Preserve RLM execution, typed predictions, cancellation/resource ownership, file/workspace safety, trace/accounting, Codex auth/transport/cache, and GEPA acceptance/evidence/resume regressions.
  • Synchronize SBX callbacks across worker event loops with a thread-safe barrier. Spawn isolated timeout-reproduction processes instead of forking multithreaded pytest; separate startup from execution budgets.
  • Scope JSPI deadline tracing to generated code; cancel suspended execution tasks without raising through scheduler callbacks; acknowledge watchdog disarm before cleanup; restore the previous SIGINT handler.
  • Add a red-before-green awaited-sleep timeout regression with state and signal-handler preservation. Keep prediction preparation outside deliberately short loop deadlines and disable unrelated eager package installation in lightweight adapter/deadline tests.
  • Preserve CI selections while adding unbuffered per-case JSPI logs and stack dumps. Update architecture, changelog, and test-policy documentation. Example-local suites remain separate.

Test Plan

  • Latest HEAD a96ac237254edb711fa6fc8aedde8db0058c96bf: workflow succeeded, with all eight jobs green: core on Python 3.11/3.12/3.13, SBX, Codex LM, GEPA, JSPI integration, and lint.
  • Reproduce the SBX cross-loop failure with PYTHONASYNCIODEBUG=1; verify synchronous and asynchronous overlap after correction.
  • Demonstrate the new awaited-sleep regression failing before the scheduler fix and passing afterward.
  • Verify final synchronous-sleep, RLM state/history/predict, and asynchronous-sleep deadline scenarios under Python 3.12, fresh dependencies, Deno 2.9.6, and coverage: 3 passed.
  • Initial full-suite refactor validation, before CI follow-up fixes: uv run --all-extras pytest -o addopts= -v --tb=short --junitxml=/tmp/predict-rlm-refactor-final.xml — 358 passed, 30 skipped.
  • Ruff passes for src/ and tests/; changed Python tests are formatted.
  • Run opt-in real SBX and bootstrap Docker scenarios in provisioned environments. These prerequisites were not enabled locally; other skips reflect unsupported backend capabilities.

The initial local full run reported DSPy deprecations and async-stream cleanup warnings. No warning suppression, retries, or test deselection were added to obtain a green workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant