Skip to content

test(golden): graduate hard-query xfails to hard assertions - #218

Merged
silversurfer562 merged 1 commit into
mainfrom
test/graduate-hard-golden-xfails
Aug 18, 2026
Merged

test(golden): graduate hard-query xfails to hard assertions#218
silversurfer562 merged 1 commit into
mainfrom
test/graduate-hard-golden-xfails

Conversation

@silversurfer562

Copy link
Copy Markdown
Member

The hard-difficulty golden queries have carried xfail(strict=False) since the marks were added to document the keyword-retriever gap, gated on an embeddings retriever (task 2.5). Two facts make the marks obsolete:

  • All three hard queries pass on main today (they showed as XPASS) via alias expansion + path-keyed summaries — the marks only mask future regressions.
  • The embeddings retriever they were waiting on is permanently deferred (docs/specs/archive/alias-expansion-sweep/README.md), so the "until embeddings" condition can never arrive.

Scope: removes _entry_marks so all difficulties assert hard, and records the graduation in the module docstring. The paraphrased info-only xfail set and the aggregate R@3 watermark are deliberately untouched — their threshold-gating decision remains deferred per M13.3.

Verification: golden suite 41 passed / 2 xfailed (known paraphrased misses gqp-015b, gqp-031a) / 78 xpassed; full suite green locally on top of #217.

Supersedes the stale May branch fix/golden-drop-stale-xfail (deleted in today's housekeeping sweep).

🤖 Generated with Claude Code

The hard-difficulty golden queries carried xfail(strict=False) pending
an embeddings retriever (task 2.5 gate). All three now pass via alias
expansion + path-keyed summaries, and the embeddings retriever is
permanently deferred (docs/specs/archive/alias-expansion-sweep/) — the
marks only masked regressions. Paraphrased info-only xfails and the
aggregate watermark are untouched; their gating decision remains
deferred per M13.3.

Golden suite: 41 passed, 2 xfailed (paraphrased), 78 xpassed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
attune-rag Ready Ready Preview Aug 18, 2026 11:35pm

@github-actions

Copy link
Copy Markdown
Contributor

Perf delta — within baseline

Blocking on regression in: keyword_retriever_retrieve.cpu, rag_pipeline_run.cpu. Other metrics are advisory and don't block merge.

Metric Baseline mean (s) Current mean (s) Δ Threshold (s) Status
directory_corpus_load.cpu 0.000060 0.000052 -13.3% 0.000089 ok
directory_corpus_load.wall 0.000060 0.000052 -13.3% 0.000089 ok
keyword_retriever_retrieve.cpu 0.005573 0.003761 -32.5% 0.007221 ok
keyword_retriever_retrieve.wall 0.005573 0.003761 -32.5% 0.007219 ok
rag_pipeline_run.cpu 0.000639 0.000559 -12.5% 0.000919 ok
rag_pipeline_run.wall 0.000638 0.000559 -12.4% 0.000918 ok

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@silversurfer562
silversurfer562 merged commit b3af29b into main Aug 18, 2026
22 checks passed
@silversurfer562
silversurfer562 deleted the test/graduate-hard-golden-xfails branch August 18, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant