test(environments): pin the suite scoring rule and document a red suite

[SomSamantray]

Summary

The scoring rule behind the frozen acceptance suites is now pinned by tests, and its documentation states what a red suite reports. Issue #121 asked whether the six suites are meant to be green today; neither the package nor CI answered that, and the prior documentation described the rule without its consequence.

The rule — at least round(0.9 * len(TEST_CASES)) cases passed with zero critical failures, plus a min_confidence gate that treats a sub-threshold call as a refusal — was documented in detail but never exercised. tests/test_environments.py covered schemas, case shape and tool execution, and its one engine-gated smoke test skips wherever no engine is installed, so a change to the rule would have shipped silently. The new tests drive the harness through a stubbed engine that picks its reply from the query, which needs no native engine and therefore runs in CI.

They pin the accepted floor, the exact threshold (28 of 32 must fail, which separates round from a floor variant), the critical override, the strict min_confidence comparison, and the aggregate that ANDs all six.

Rebased onto main (2026-09-20) to resolve conflicts a reviewer flagged, caused by the "Needle 3" restructure landing after this PR was opened:

• main deleted doc/environments.md (and doc/apis.md, doc/finetuning.md), folding a terser environments summary into llms.txt, and rewrote the README sections this PR had touched. Those now-stale README/doc edits were dropped rather than reintroduced.
• The scoring-rule documentation this PR originally added to doc/environments.md is instead folded into llms.txt's ## Environments section, so the pinned contract (the round(0.9 * n) threshold, the min_confidence refusal behavior, the red-suite caveat) stays documented in the file that replaced it.
• main also changed _harness.run_tests scoring to be case-insensitive (commit 10fbeb5) and to fold integral floats to ints; this PR's tests are unaffected since they compare distinct values, not case or numeric-type variants — verified by re-running the full suite post-rebase (see Validation).

Related: #121

Validation

• NEEDLE_TELEMETRY=0 python3 -m pytest tests/test_environments.py -q — 31 passed, 1 skipped (engine-gated smoke test, expected without a native engine).
• CI-equivalent, run against both this branch and a clean main checkout in the same environment for comparison: NEEDLE_TELEMETRY=0 python3 -m pytest -q -m "not slow" --continue-on-collection-errors. main baseline: 14 failed / 75 passed / 6 skipped / 12 deselected / 4 errors (all 14 failures are pre-existing ModuleNotFoundErrors for optional jax/pydantic, unrelated to this change). This branch: the same 14 failed / 81 passed / 6 skipped / 12 deselected / 4 errors — the 6 new tests pass, nothing regresses.
• Three mutation checks confirm the tests are load-bearing. Changing round to int in _harness.run_tests fails the 28-of-32 case. Making environments.run_tests inspect only the first module fails the aggregate case. Changing the confidence gate's < to <= fails the threshold-equality case. The production files were restored after each and verified unchanged against main.

The measurement was taken by running all six suites against the shipped base model on the cached engine 2.0.4 at both gates, and the per-category counts behind the new prose were re-measured directly rather than inferred.

Session-settled decisions carried from planning: correcting the documented contract rather than the suites (user-directed — the expectations are the product spec, and relaxing them would hide the acceptance gap); delivery as a fork pull request (user-directed — the contributing account has pull-only access upstream); a DCO sign-off on every commit (user-directed); folding the scoring-rule documentation into llms.txt rather than recreating doc/environments.md (user-directed, made necessary by the upstream restructure).

Unapplied review findings

• P1 — tests/test_environments.py — No engine-free test asserts the reply contract the harness consumes (function_calls present, arguments a dict, a numeric confidence). The stub fabricates that contract, so a drift between the engine and the harness would leave every new test green while the suites went red. A gate that compares a None confidence raises TypeError on that path. Not applied: the None path is unreachable through _harness.agent_for, which builds an agent without weights, and changing the gate would alter untouched production behavior outside this plan's scope.
  Suggested fix: add cases whose envelope omits confidence and whose confidence is null, then make the gate total.
• P2 — tests/test_environments.py — The category-to-critical mapping and the order-insensitive parallel comparison are unpinned. Not applied: the page no longer claims the whole rule is pinned, having been narrowed to the threshold and the critical effect, so this is a coverage gap rather than a false claim. Every synthetic case carries a single-element calls list, so the sorted-comparison branch never runs.
• P2 — llms.txt — Calling min_confidence=0.4 "the production contract" is pre-existing wording, but no shipped entry point applies it: every default is 0.0, including python -m needle.environments.<name>, so a reader who adopts 0.4 gets a different result from the one the page records. Not applied: shipping a way to select the gate is an API decision for the maintainers.
• P3 — tests/test_environments.py — The 32-case synthetic suite is not the minimum that pins round against floor; a 2-case suite diverges the same way. Kept deliberately: 32 mirrors the shipped suite size so the tests pin the documented 29-of-32 boundary, and with telemetry neutered each suite costs about a millisecond.
• P3 — llms.txt — The measurement paragraph is prose no test can falsify, because CI has no engine. Mitigated by naming the engine version and the gate; it drifts on an engine or model bump. A guard failing when needle/agent/fetch.py's pinned engine version changes would prompt a re-measurement.
• Residual — tests/test_environments.py — Stubbing needle._lib with a plain Python object bypasses ctypes argtypes coercion, so an argument-order or argument-type regression between Needle._complete and needle_complete would not be caught here.
• Skipped during simplification — consolidating the three fake-engine fixtures (test_grounding.py, test_weights.py, and this file) into tests/conftest.py. Not applied: it would edit tests/test_grounding.py, which PR fix(grounding): stop rejecting thousands-separated numbers #123 modifies, and would exceed this plan's declared scope.

Review run context: run_id 20260911-221145-7caa3fca, artifacts at /tmp/compound-engineering-501/ce-code-review/20260911-221145-7caa3fca/.

[HenryNdubuaku]

The tests are good and pass with the case-insensitive scoring now on main. README and doc/environments.md conflict with main; rebase and I'll merge.

[SomSamantray]

Rebased onto main — the conflicts were the "Needle 3" restructure deleting doc/environments.md and rewriting the README sections this PR touched. Dropped the now-stale README/doc edits and folded the scoring-rule documentation into llms.txt's ## Environments section instead, since that's where the file's content now lives. Tests are unaffected by the case-insensitive scoring change in 10fbeb5 (re-verified: 31 passed / 1 skipped, and the full suite matches main's baseline plus these 6 new tests, no regressions — details in the updated PR description). Should be clean to merge now.