G4: a search-quality regression suite that is proven to fail

[brentrager]

Closes the deterministic half of feature gap G4 (answer- & search-quality regression suite) and formalizes the judged half into a scored regression layer with a nightly job. docs/Planning/Feature Gaps.md §G4 is updated in this PR.

Half 1 — deterministic retrieval quality (gates every PR)

rust/evals/tests/retrieval_quality.rs seeds a frozen 20-document corpus through the real ingest→chunk→embed→store pipeline (MockConnector → ingest() → Chunker → DeterministicEmbedder → InMemoryKnowledge), runs a frozen 20-query labeled set through the real KnowledgeSearchTool, and asserts recall@3 / recall@5 / MRR. No LLM, no key, no network, no Docker, no clock.

It is deliberately ungated — no SMOOTH_AGENT_E2E, no #[cfg(feature)], no #[ignore]. A gated suite that prints ok. 0 passed is a suite that did not run.

metric	measured baseline	threshold	headroom
recall@3	0.975	0.90	~1.5 of 20 queries may lose their answer
recall@5	1.000	0.95	1 of 20 queries may lose its answer
MRR	0.975	0.90	~1.5 queries may fall from rank 1 to rank 2

Thresholds are hand-written constants, never computed from the run. The suite has zero run-to-run variance (eval_is_deterministic_across_runs asserts it), so the headroom is a budget for benign ranking churn, not noise insurance.

The corpus is shaped by a failure

The first draft — 13 unrelated documents — scored a perfect recall@3 with every degradation still passing. It detected nothing. It is now built around near-duplicate distractors (exchanges/cancellations vs returns, atlas-r5 vs atlas-r7, diagnostics vs error-codes), and half the queries target a fact in a document's second or third paragraph, so it measures fact retrieval rather than topic matching.

Proof it can fail

Four permanent degradation tests keep the gate's own sensitivity under test:

pipeline	recall@3	recall@5	MRR	breaches gate
shipped config (baseline)	0.975	1.000	0.975	—
+ LexicalReranker	0.975	0.975	1.000	—
half the corpus never ingested	0.500	0.500	0.500	yes
48-char chunks, no overlap	0.975	0.975	0.842	yes (MRR)
first paragraph only	0.775	0.800	0.717	yes
reranker comparator reversed	0.325	0.450	0.250	yes

And two production-code regressions were introduced by hand, confirmed red, then restored:

production change	effect	result
Chunker defaults 500/64 → 60/0	MRR 0.975 → 0.792	gate FAILED on MRR
LexicalReranker comparator flipped (one char)	recall@3 0.975 → 0.300, MRR → 0.213	rerank guard FAILED

Two honest findings recorded in the docs: an earlier "truncate every paragraph to 40 chars" degradation improved the numbers (the in-memory scorer divides by chunk length, so shorter chunks rank higher) — which is why a degradation has to be measured rather than assumed. And the LexicalReranker trades coverage for ordering (MRR up, recall@5 down), so the suite asserts MRR and recall@3 do not regress and deliberately does not pin recall@5.

Not covered: InMemoryKnowledge ranks lexically, so an embedder swap is not scored. Scoring dense retrieval means running this same corpus against the pgvector adapter under testcontainers; the corpus, labels, and metrics are backend-agnostic and move over unchanged.

Half 2 — judged regression layer + nightly

Every Scenario now declares a typed Competency — a required field, not a name→competency lookup table that would drift the moment someone adds a scenario. tests/regression.rs runs all 15 scenarios from both suites and rolls them into a per-competency Scorecard with its own floor, so a drop in grounding no longer averages away against a rise in tone:

competency	floor
anti_hallucination, safety	4.0
grounding, tool_use, tone	3.5
multi_turn_reasoning	2.5 — a floor that describes the known cross-turn-memory gap rather than hiding it

.github/workflows/nightly-evals.yml sweeps a matrix of agent models (new SMOOTH_AGENT_EVAL_MODEL), appends each night's scorecard to a cached eval-history.jsonl rendered as a trend table in the job summary, and refuses two failure modes:

1. Silently not running — SMOOTH_AGENT_EVALS_REQUIRED=1 makes the suite fail rather than skip without credentials, and a preflight step fails first with an actionable message.
2. Being fooled by log output — nothing greps, tallies, or ^-anchors a test log; the gate is the cargo test exit code. CARGO_TERM_COLOR: never is set regardless.

Prerequisite (unverified): the workflow reads a SMOOAI_GATEWAY_KEY repository secret, which does not exist yet. Until it is added the nightly job fails at its preflight step — loudly, which is intended, but it must be added before the first useful night.

Verification

• cargo test --workspace — all green (no failures across the workspace).
• cargo fmt --all -- --check — clean.
• cargo clippy --workspace --all-targets -- -D warnings — clean.
• Judged half is unverified against the live gateway in this PR (no key was used); its credential-free half (every_scenario_declares_a_competency_with_a_floor) does run.

[changeset-bot]

🦋 Changeset detected

Latest commit: cc504c9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@smooai/smooth-operator	Patch
@smooai/smooth-operator-web-chat-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR