G4: a search-quality regression suite that is proven to fail

.changeset/g4-retrieval-quality-evals.md

@@ -0,0 +1,13 @@
+---
+'@smooai/smooth-operator': patch
+---
+
+Add a deterministic search-quality regression suite and formalize the judged evals into a scored regression layer (feature gap G4).
+
+**The half that gates CI.** `rust/evals` now ships a retrieval-quality eval that needs no LLM, no key, and no network: a frozen 20-document corpus is seeded through the real ingest→chunk→embed→store pipeline, a frozen 20-query labeled set runs through the real `knowledge_search` tool, and the ranked results are scored with recall@3, recall@5, and MRR against hand-written thresholds. It is deliberately ungated — no `SMOOTH_AGENT_E2E`, no feature flag, no `#[ignore]` — so it runs on every PR and catches a chunker change, an embedder swap, or a rerank bug the day it lands.
+
+Four permanent degradation tests prove the suite can actually go red: half the corpus dropped, 48-char chunking, first-paragraph-only extraction, and a reranker with its comparator reversed each breach the thresholds the gate enforces.
+
+**The judged half.** Every eval scenario now declares a typed `Competency` (grounding, anti-hallucination, tool use, multi-turn reasoning, safety, tone), and a new `regression` suite rolls all 15 scenarios up into a per-competency `Scorecard` with its own floor — so a drop in grounding no longer averages away against a rise in tone. `SMOOTH_AGENT_EVAL_MODEL` lets the agent model be swept, and `SMOOTH_AGENT_EVALS_REQUIRED=1` turns "skipped for want of credentials" into a hard failure.
+
+**Nightly CI.** `.github/workflows/nightly-evals.yml` runs the judged suite across a model matrix, appends each night's scorecard to a cached score history, and renders the trend into the job summary. It fails loudly when the gateway key is missing rather than reporting a green no-op, and nothing in it parses a test log.

.github/workflows/nightly-evals.yml

@@ -0,0 +1,164 @@
+# Nightly LLM-judged eval regression across models (feature gap G4).
+#
+# The deterministic half of the eval layer — `rust/evals/tests/retrieval_quality.rs`,
+# recall@k / MRR over a frozen corpus — needs no credentials and already runs on
+# every PR inside `rust.yml`. It is NOT duplicated here: it cannot drift between
+# nightly runs, only between commits, and the PR job already covers that.
+#
+# This workflow is for the half that can't gate a PR: the judged scenarios, which
+# need a live gateway, cost money, and have real run-to-run variance. It sweeps a
+# matrix of agent models over the same frozen scenario set so provider/model drift
+# on ANY model shows up, not just on the default one, and appends each night's
+# scorecard to a cached history file so a slow slide reads as a trend.
+#
+# ## Two failure modes this workflow refuses to have
+#
+# 1. **Silently not running.** A gated suite that prints "ok. 0 passed" is a suite
+#    that did not run. The `judged` job sets `SMOOTH_AGENT_EVALS_REQUIRED=1`, which
+#    makes the eval suite FAIL rather than skip when credentials are missing, and a
+#    preflight step fails first with an actionable message. There is no path where
+#    a missing key produces a green night.
+# 2. **Being fooled by log output.** Nothing here greps, tallies, or `^`-anchors a
+#    test log. The gate is the `cargo test` exit code. `CARGO_TERM_COLOR: never`
+#    is set anyway, so nothing downstream can be confused by ANSI escapes.
+name: Nightly evals
+
+on:
+    schedule:
+        # 07:17 UTC — off the hour, so it doesn't queue behind the cron stampede.
+        - cron: '17 7 * * *'
+    workflow_dispatch:
+
+env:
+    CARGO_TERM_COLOR: never
+
+jobs:
+    judged:
+        name: Judged evals (${{ matrix.agent_model }})
+        runs-on: ubuntu-latest
+        timeout-minutes: 45
+        strategy:
+            fail-fast: false
+            matrix:
+                include:
+                    # The default/production model, graded by a stronger, different
+                    # family — the most trustworthy grade we can get.
+                    - agent_model: claude-haiku-4-5
+                      judge_model: claude-sonnet-4-5
+                    # Sonnet grading Sonnet is same-family judging and reads lenient;
+                    # it is here to catch drift in the stronger model over time, and
+                    # its absolute score should not be compared against the row above.
+                    - agent_model: claude-sonnet-4-5
+                      judge_model: claude-sonnet-4-5
+
+        steps:
+            - uses: actions/checkout@v4
+
+            - name: Require the gateway key
+              env:
+                  SMOOAI_GATEWAY_KEY: ${{ secrets.SMOOAI_GATEWAY_KEY }}
+              run: |
+                  if [ -z "${SMOOAI_GATEWAY_KEY}" ]; then
+                      echo "::error title=Nightly evals cannot run::The SMOOAI_GATEWAY_KEY repository secret is unset or empty. The judged evals need a live gateway key; without it this job would skip every scenario and report success, which is the exact failure this workflow exists to prevent. Add the secret (the smooai org LLM virtual key, same value scripts/run-evals.sh fetches from @smooai/config) and re-run."
+                      exit 1
+                  fi
+                  echo "gateway key present"
+
+            - name: Install Rust toolchain
+              uses: dtolnay/rust-toolchain@stable
+
+            - name: Cache cargo
+              uses: Swatinem/rust-cache@v2
+              with:
+                  workspaces: rust
+
+            - name: Judged regression suite
+              working-directory: rust
+              env:
+                  SMOOTH_AGENT_E2E: '1'
+                  # Turns "skipped for want of credentials" into a hard failure.
+                  SMOOTH_AGENT_EVALS_REQUIRED: '1'
+                  SMOOAI_GATEWAY_KEY: ${{ secrets.SMOOAI_GATEWAY_KEY }}
+                  SMOOTH_AGENT_EVAL_MODEL: ${{ matrix.agent_model }}
+                  SMOOTH_AGENT_JUDGE_MODEL: ${{ matrix.judge_model }}
+              run: |
+                  cargo test -p smooai-smooth-operator-evals --test regression \
+                      -- --nocapture --test-threads=1
+
+            # The suite writes its scorecard BEFORE asserting, so a failing night
+            # still produces the row that explains what it failed on.
+            - name: Upload scorecard
+              if: always()
+              uses: actions/upload-artifact@v4
+              with:
+                  name: eval-scorecard-${{ matrix.agent_model }}
+                  path: rust/target/eval-scorecard.json
+                  if-no-files-found: warn
+
+    history:
+        name: Append score history
+        needs: judged
+        if: always()
+        runs-on: ubuntu-latest
+        timeout-minutes: 10
+        steps:
+            - name: Download tonight's scorecards
+              uses: actions/download-artifact@v4
+              with:
+                  pattern: eval-scorecard-*
+                  path: scorecards
+
+            # The history lives in the Actions cache rather than in the repo: a
+            # committed history file would need push rights on a protected branch
+            # and would put a bot commit on main every night. The rotating key +
+            # prefix restore-key means each run reads the newest saved history and
+            # writes a new immutable entry.
+            - name: Restore score history
+              uses: actions/cache/restore@v4
+              with:
+                  path: eval-history.jsonl
+                  key: smooth-operator-eval-history-${{ github.run_id }}
+                  restore-keys: |
+                      smooth-operator-eval-history-
+
+            - name: Append tonight's rows
+              run: |
+                  set -euo pipefail
+                  touch eval-history.jsonl
+                  found=0
+                  while IFS= read -r f; do
+                      found=$((found + 1))
+                      jq -c \
+                          --arg date "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+                          --arg run "${GITHUB_RUN_ID}" \
+                          --arg sha "${GITHUB_SHA}" \
+                          '. + {date: $date, run_id: $run, commit: $sha}' \
+                          "$f" >> eval-history.jsonl
+                  done < <(find scorecards -name 'eval-scorecard.json')
+                  echo "appended ${found} scorecard row(s)"
+
+            - name: Write the trend to the job summary
+              run: |
+                  set -euo pipefail
+                  {
+                      echo "### Eval score history — most recent 20 runs"
+                      echo ""
+                      echo "| date | agent | judge | overall | grounding | anti-halluc | tool use | multi-turn | safety | tone |"
+                      echo "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"
+                      tail -n 20 eval-history.jsonl | jq -r '
+                          def m(k): (.competencies[k].mean // null) | if . == null then "–" else (.*100|round/100|tostring) end;
+                          "| \(.date) | \(.agent_model) | \(.judge_model) | \((.overall_mean*100|round)/100) | \(m("grounding")) | \(m("anti_hallucination")) | \(m("tool_use")) | \(m("multi_turn_reasoning")) | \(m("safety")) | \(m("tone")) |"
+                      '
+                  } >> "$GITHUB_STEP_SUMMARY"
+
+            - name: Save score history
+              uses: actions/cache/save@v4
+              with:
+                  path: eval-history.jsonl
+                  key: smooth-operator-eval-history-${{ github.run_id }}
+
+            - name: Upload score history
+              uses: actions/upload-artifact@v4
+              with:
+                  name: eval-history
+                  path: eval-history.jsonl

docs/Operations/Evals.md

@@ -1,4 +1,116 @@
-# LLM-as-Judge Evaluation Harness
+# Evaluation Harness
+
+The `evals` crate (`smooai-smooth-operator-evals`, at `rust/evals/`) holds the
+repo's **quality regression layer** (feature gap G4). It has two halves, and the
+split is the whole design:
+
+| Half | Needs | Runs | Gates a PR |
+| --- | --- | --- | --- |
+| **Retrieval quality** — recall@k / MRR over a frozen corpus | nothing | every PR | **yes** |
+| **LLM-as-judge** — rubric scoring per competency | gateway key + `SMOOTH_AGENT_E2E` | nightly | no |
+
+The deterministic half is described first because it is the one that can be a
+hard gate. The judged half follows.
+
+---
+
+## Part 1 — Deterministic retrieval quality (`tests/retrieval_quality.rs`)
+
+Seeds a **frozen 20-document corpus** (`src/corpus.rs`) through the *real*
+ingest→chunk→embed→store pipeline (`MockConnector` → `ingest()` → `Chunker` →
+`DeterministicEmbedder` → `InMemoryKnowledge`), runs a **frozen 20-query labeled
+set** through the *real* `KnowledgeSearchTool`, and scores the ranked results.
+Every box on that path is production code; the eval owns only the corpus, the
+labels, and the arithmetic.
+
+### Why it is not gated
+
+No env var, no `#[cfg(feature = …)]`, no `#[ignore]`. A gated suite that prints
+`ok. 0 passed` is a suite that did not run, and this repo has shipped that
+mistake. Anything here that ever needs a credential belongs in the judged half
+instead.
+
+### The corpus and why those queries
+
+The first draft was 13 unrelated documents and scored a **perfect recall@3 with
+every degradation still passing** — it detected nothing. The corpus is now built
+around *near misses*: `policies/exchanges.md` and `policies/cancellations.md`
+compete with `policies/returns.md`; `product/atlas-r5-specs.md` and
+`support/battery-care.md` compete with `product/atlas-r7-specs.md`;
+`support/diagnostics.md` competes with `support/error-codes.md`. Half the
+queries target a fact stated in a document's **second or third** paragraph, so
+the eval measures fact retrieval rather than topic matching and is sensitive to
+anything that loses document tails.
+
+Labels name **document sources**, not chunk ids, so a chunker change does not
+invalidate the ground truth.
+
+### Thresholds and the headroom
+
+Thresholds are hand-written constants. A threshold computed from the code it
+guards can never fail. The suite has **zero run-to-run variance**
+(`eval_is_deterministic_across_runs` asserts it), so headroom is not noise
+insurance — it is the budget for benign ranking churn:
+
+| metric | measured baseline | threshold | headroom |
+| --- | --- | --- | --- |
+| recall@3 | 0.975 | 0.90 | ~1.5 of 20 queries may lose their answer |
+| recall@5 | 1.000 | 0.95 | 1 of 20 queries may lose its answer |
+| MRR | 0.975 | 0.90 | ~1.5 queries may fall from rank 1 to rank 2 |
+
+### Proof it can fail
+
+An eval that has never been shown to go red is theater. Four degradation tests
+break one real pipeline stage each and assert the metrics fall through the gate:
+
+| pipeline | recall@3 | recall@5 | MRR | breaches gate |
+| --- | --- | --- | --- | --- |
+| shipped config (baseline) | 0.975 | 1.000 | 0.975 | — |
+| + `LexicalReranker` | 0.975 | 0.975 | 1.000 | — |
+| half the corpus never ingested | 0.500 | 0.500 | 0.500 | yes |
+| 48-char chunks, no overlap | 0.975 | 0.975 | 0.842 | yes (MRR) |
+| first paragraph only | 0.775 | 0.800 | 0.717 | yes |
+| reranker comparator reversed | 0.325 | 0.450 | 0.250 | yes |
+
+Two findings worth carrying forward:
+
+- **An earlier "truncate every paragraph to 40 chars" degradation *improved* the
+  numbers.** The in-memory scorer divides match count by chunk length, so shorter
+  chunks rank higher. That is why a degradation has to be *measured* rather than
+  assumed to hurt — it was replaced with first-paragraph-only.
+- **The `LexicalReranker` trades coverage for ordering**: MRR 0.975 → 1.000 while
+  recall@5 goes 1.000 → 0.975, because it over-fetches 4×`limit` and truncates
+  back. The suite asserts MRR and recall@3 do not regress and deliberately does
+  *not* pin recall@5 — asserting an improvement that isn't real is how a suite
+  starts lying.
+
+### What it does not cover
+
+The backend is `InMemoryKnowledge`, which ranks **lexically**. The
+`DeterministicEmbedder` on the ingest path really runs (batch shape and vector
+dimension are validated) but its vectors do not influence ranking, so **an
+embedder swap is not scored by this eval** (pearl `th-15a147`). Scoring dense retrieval means running
+this same corpus and query set against the pgvector adapter under testcontainers
+— the corpus, labels, and metrics are backend-agnostic and move over unchanged.
+Until that exists, a green run here is not evidence that dense retrieval is fine.
+
+### Verified end to end
+
+Two production-code regressions were introduced deliberately and the suite went
+red on both, then was restored:
+
+| production change | effect | result |
+| --- | --- | --- |
+| `Chunker` defaults 500/64 → 60/0 | MRR 0.975 → **0.792** | gate FAILED on MRR |
+| `LexicalReranker` sort comparator flipped (one character) | recall@3 0.975 → **0.300**, MRR → 0.213 | rerank guard FAILED |
+
+```sh
+cargo test -p smooai-smooth-operator-evals --test retrieval_quality -- --nocapture
+```
+
+---
+
+## Part 2 — LLM-as-Judge
 
 The `evals` crate (`smooai-smooth-operator-evals`, at `rust/evals/`) is a
 quality-scoring harness for the reference agent. Where the core crate's
@@ -16,8 +128,13 @@ cheap `claude-haiku-4-5` model — no mocks on the agent path.
 
 | File | Purpose |
 | --- | --- |
-| `rust/evals/src/lib.rs` | The harness: `Scenario`, `JudgedResult`, `JudgeConfig`, `run_scenario`, `parse_verdict`, `default_scenarios`. |
-| `rust/evals/tests/llm_judge.rs` | Gated live-gateway integration test that runs the whole suite and asserts on the aggregate. |
+| `rust/evals/src/lib.rs` | The harness: `Scenario`, `Competency`, `Scorecard`, `JudgedResult`, `JudgeConfig`, `run_scenario`, `parse_verdict`, `default_scenarios`, `extended_scenarios`. |
+| `rust/evals/src/corpus.rs` | The frozen retrieval corpus + labeled query set (Part 1). |
+| `rust/evals/src/retrieval.rs` | The deterministic retrieval runner + recall@k / MRR (Part 1). |
+| `rust/evals/tests/retrieval_quality.rs` | **Ungated** search-quality gate + the four degradation proofs. |
+| `rust/evals/tests/llm_judge.rs` | Gated live-gateway test over `default_scenarios`, asserting the aggregate mean. |
+| `rust/evals/tests/extended_judge.rs` | Gated live-gateway test over the harder `extended_scenarios`, lenient floor. |
+| `rust/evals/tests/regression.rs` | Gated **regression layer**: all 15 scenarios, per-competency floors, writes the scorecard JSON. |
 
 ## How a scenario is judged
 
@@ -69,6 +186,66 @@ delivery date — the same gap documented in
 substring check would not have. The aggregate stayed ≥ 4.0 (4.20), so the suite
 passes while loudly logging the miss for follow-up.
 
+## The regression layer: competencies, floors, and the scorecard
+
+`llm_judge` asserts one aggregate mean and `extended_judge` asserts a lenient
+floor. Neither answers the question a regression layer exists to answer: *which
+competency moved?* A drop in grounding and a rise in tone average out to the same
+number, and averaging them is how a real regression hides.
+
+Every `Scenario` therefore declares a typed `Competency`. It is a **required
+field**, not a name→competency lookup table — a table restating the scenario list
+drifts the moment someone adds a scenario, and drifts silently; a required field
+will not compile without a declaration.
+
+| Competency | Floor | Scenarios |
+| --- | --- | --- |
+| `anti_hallucination` | 4.0 | `honest_no_knowledge`, `dev_honest_unknown_config`, `user_asserts_false_policy` |
+| `safety` | 4.0 | `prompt_injection_in_kb`, `out_of_scope_refusal` |
+| `grounding` | 3.5 | `grounded_answer`, `contradictory_kb`, `dev_grounded_api_usage`, `dev_debugging_grounded` |
+| `tool_use` | 3.5 | `tool_use_supported_answer`, `distraction_needle` |
+| `tone` | 3.5 | `tone_helpfulness` |
+| `multi_turn_reasoning` | 2.5 | `multi_turn_coherence`, `multi_turn_planted_fabrication`, `numeric_month_boundary` |
+
+The floors are deliberately uneven. Inventing a fact loses a customer's trust
+irrecoverably, so anti-hallucination and safety are held highest. Cross-turn
+memory is a *known engine gap* (`KnowledgeChatRuntime` builds a fresh `Agent` per
+turn), so the multi-turn floor catches collapse rather than pretending the gap is
+closed — lowering a floor to hide a known gap and lowering it to describe one are
+different acts, and this is the second.
+
+`tests/regression.rs` runs all 15 scenarios, prints the scorecard, writes
+`rust/target/eval-scorecard.json`, and fails on any breached floor. The scorecard
+is written **before** the assertion, so a failing night still leaves the row that
+explains what it failed on.
+
+## Nightly CI + score history
+
+`.github/workflows/nightly-evals.yml` runs the judged regression suite across a
+matrix of agent models (`SMOOTH_AGENT_EVAL_MODEL`), judged by a stronger family
+(`SMOOTH_AGENT_JUDGE_MODEL`), and appends each night's scorecard to an
+Actions-cached `eval-history.jsonl` rendered as a trend table in the job summary.
+
+Two failure modes it refuses to have:
+
+1. **Silently not running.** The job sets `SMOOTH_AGENT_EVALS_REQUIRED=1`, which
+   makes the suite *fail* rather than skip without credentials, and a preflight
+   step fails first with an actionable message. A missing key cannot produce a
+   green night.
+2. **Being fooled by log output.** Nothing greps, tallies, or `^`-anchors a test
+   log — the gate is the `cargo test` exit code. `CARGO_TERM_COLOR: never` is set
+   regardless, so no ANSI escape can confuse anything downstream.
+
+> **Prerequisite:** the workflow reads the `SMOOAI_GATEWAY_KEY` repository secret
+> (the same smooai-org LLM virtual key `scripts/run-evals.sh` fetches from
+> `@smooai/config`). Until that secret exists the nightly job fails at the
+> preflight step — loudly, which is the intended behavior, but it does mean the
+> secret has to be added before the first useful night.
+
+The deterministic retrieval eval is **not** duplicated in the nightly job: it
+cannot drift between nights, only between commits, and `rust.yml` already runs it
+on every PR.
+
 ## Same-model-judging limitation & the judge-model knob
 
 By default the **agent and judge are the same model** (`claude-haiku-4-5`). A