Skip to content

fix: 2026-08-10 audit follow-up — count/flag/link accuracy, semantic passages - #9

Merged
silversurfer562 merged 1 commit into
mainfrom
claude/determined-gates-380ecd
Aug 11, 2026
Merged

fix: 2026-08-10 audit follow-up — count/flag/link accuracy, semantic passages#9
silversurfer562 merged 1 commit into
mainfrom
claude/determined-gates-380ecd

Conversation

@silversurfer562

Copy link
Copy Markdown
Member

Summary

Accuracy fixes from the 2026-08-10 full library audit: three numeric-claim false-positive classes stopped, two silent false-negative classes closed, markdown link titles handled, and the semantic layer now judges against real source passages instead of the content itself. Every fix carries paired clean + hallucinated corpus cases.

Fixed

  • Comma-grouped numbers are one claim — "1,234 tests" no longer extracts the "234" fragment and errors against tests=1234; grouped year-valued counts ("2,026 widgets") stay checked.
  • Decimals and version components are not counts — "94.53" and the "10" in "Python 3.10" are no longer claims.
  • Short count-source labels are no longer silently dead — "api"/"eps" labels now match (exact word boundary), but only as a fallback when the label has no longer word, so stopwords in mixed labels ("number of tests") can't drag unrelated numbers into the source.
  • Flags in single-span commands and shell fences are checked`mytool --flag` and ```bash fences were entirely unchecked; only a flag backticked alone was seen, despite a code comment claiming otherwise. Corpus flag cases updated to the realistic single-span form.
  • Markdown link titles and <angle-bracket> targets no longer break path existence checks; dead paths with titles are still flagged.
  • Semantic layer judges against VerifyContext.passages (new field) instead of the content itself, which was vacuously faithful by construction; with no passages the judge is skipped with a warning. No-judge vs. protocol-mismatch warnings are now distinct messages.
  • Import resolution passes the module name via argv instead of f-string interpolation into python -c (defense in depth).

Test plan

  • Suite grew 83 → 99 tests; 12 new corpus cases guard each fix from both the precision and recall side (corpus gate ≥ 0.95 both).
  • All five original audit repros verified fixed by direct probe; adversarial probes for stopword labels, --flag=value, console-fence help output, and year/comma interplay behave correctly.
  • ruff + black clean; mutation gate runs in CI.

🤖 Generated with Claude Code

…passages

Accuracy fixes from the full library audit, each with paired clean +
hallucinated corpus cases:

- Comma-grouped numbers ("1,234") are one claim; decimals ("94.53") and
  version components ("Python 3.10") are no longer claims at all.
- Short count-source labels ("api", "eps") are no longer silently dead;
  short words match on exact word boundaries and only when the label has
  no longer word, so stopwords in mixed labels ("number of tests") can't
  drag unrelated numbers into the source.
- Flags inside single-span commands (`mytool --flag`) and bash/sh/shell/
  console/zsh fences are now checked — previously only a flag backticked
  alone was seen, despite a comment claiming otherwise.
- Markdown link titles ([text](path "Title")) and <angle-bracket> targets
  no longer break path existence checks.
- The semantic layer judges against caller-declared VerifyContext.passages
  instead of the content itself (vacuously faithful); with no passages the
  judge is skipped with a warning. Bad-judge and no-judge warnings are
  distinct messages.
- Import resolution passes the module name via argv (defense in depth).
- Corpus flag cases use the realistic single-span form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@silversurfer562
silversurfer562 merged commit 51389cb into main Aug 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant