From def10e78a55ba5ba2e8840d6f6dd4652b835ebad Mon Sep 17 00:00:00 2001 From: Brent Rager Date: Sat, 22 Aug 2026 21:32:57 -0400 Subject: [PATCH] Mark G5 (widget frontend e2e) closed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The widget's Playwright specs existed but no CI job ever invoked them, so the suite was standing coverage that had never executed — and 2 of 8 were red when first run. Records what closed it, plus the two findings worth generalising: a "mock" spec that silently depended on production, and a streaming assertion that could not detect streaming being removed. Implementation: SmooAI/chat-widget#46 Co-Authored-By: Claude Opus 5 (1M context) --- docs/Planning/Feature Gaps.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/Planning/Feature Gaps.md b/docs/Planning/Feature Gaps.md index 8893540a..2ae8dec5 100644 --- a/docs/Planning/Feature Gaps.md +++ b/docs/Planning/Feature Gaps.md @@ -58,9 +58,13 @@ Mature knowledge platforms sync per-connector permissions and filters retrieval Mature knowledge platforms have a `regression/` layer + nightly LLM-provider-chat. We're adding LLM-judge evals — formalize it. - **TDD**: grow `rust/evals` into the regression layer — a fixed scenario set with rubric thresholds (grounding, **anti-hallucination/honest-don't-know**, tool-use appropriateness, multi-turn reasoning), plus a **retrieval-quality** eval (seed a corpus, assert recall@k / MRR on labeled queries — deterministic, no LLM). Add a `nightly` CI job that runs the judged evals across models. Track score history to catch regressions. -### G5. Frontend e2e (Playwright) for the chat widget -Mature platforms ship extensive web + Playwright suites. Our new `SmooAI/chat-widget` has none yet. +### G5. Frontend e2e (Playwright) for the chat widget — ✅ running in CI +Mature platforms ship extensive web + Playwright suites. - **TDD**: a Playwright spec first — load the widget against a locally-booted `smooth-operator-server`, send a message, assert streamed assistant tokens render + a grounded answer appears. Wire it into the widget repo's CI. +- ✅ **Done — and the interesting part was that the specs already existed but had never run.** `SmooAI/chat-widget` had 6 spec files; its `ci.yml` ran typecheck/unit/build and **never invoked `test:e2e`**, so the suite was standing coverage that had never executed. Run for the first time, **2 of 8 were red**: `repro-stream-mock.spec.ts` mocked `WebSocket` but not `fetch`, so on mount the widget POSTed the **real production** `/internal/resume-by-fingerprint`, got a 403, and the console error tripped its own page-error assertion — a "mock" spec that depended on prod being reachable. Fixed with a `page.route` stub. +- **Streaming had no real guard.** The existing spec asserted only the *final* assistant text, which a widget that ignores every `stream_token` frame and paints the `eventual_response` blob also satisfies — confirmed by deleting streaming from the mock and watching it stay green. `e2e/streaming.spec.ts` closes it: the mock **withholds `eventual_response`** and all assertions sample inside that window, so only stream tokens can have produced on-screen text; it asserts incremental, monotonic growth to the full reply. +- **CI split so a missing secret cannot look like a pass**: `ci.yml` → `E2E (credential-free)` runs the hermetic specs on every PR (9 tests, ~6s); `e2e-live.yml` (nightly) builds a lean in-memory `smooth-operator-server` (`--no-default-features`) with `SMOOTH_AGENT_SEED_KB=1` and asserts the grounded **17-day return window** answer, **failing loudly** when `SMOOAI_GATEWAY_KEY` is absent rather than skipping green. `SMOOTH_AGENT_SERVER_BIN` replaced a hardcoded developer-local `shared-target` path that had never existed on a runner. (SmooAI/chat-widget#46) +- **Lesson worth generalising**: a suite that has never run is not coverage. `grep -c` on a spec file counts *attempts*, not passes — the only proof is a CI job whose green depends on those assertions executing. ### G6. Deployment-integration tests in CI Mature knowledge platforms run compose + k8s + helm tests in CI; we only `helm lint`/`helm template`.