fix: 2026-08-10 audit follow-up — count/flag/link accuracy, semantic passages

CHANGELOG.md

@@ -7,6 +7,63 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+Accuracy fixes from the 2026-08-10 library audit: three numeric-claim
+false-positive classes stopped, two silent false-negative classes closed
+(short count labels, single-span/fenced CLI flags), markdown link titles
+handled, and the semantic layer now judges against real source passages
+instead of the content itself. Every fix carries a regression corpus case.
+
+### Fixed
+
+- **Comma-grouped numbers are one claim.** "1,234 tests" previously
+  extracted the "234" fragment and flagged an error-severity count
+  mismatch against `tests=1234`; `1,234` is now extracted as the single
+  value 1234 (commas stripped). Grouped values also stay checked when
+  year-valued — "2,026 widgets" is a count, never a year.
+- **Decimals and version components are not counts.** "94.53" near a
+  source keyword extracted 94 and 53 as separate claims and flagged both;
+  the "10" in "Python 3.10" was flagged against a nearby source. Digit
+  runs adjacent to a decimal point are now skipped.
+- **Short count-source labels are no longer silently dead.** Label words
+  of length ≤ 3 ("api", "eps") were dropped by the keyword filter, so
+  those sources could never match any claim — a silent false negative.
+  Short words now require an exact word-boundary match on both sides
+  ("api" matches "api" but not "rapid"), and participate only when the
+  label has no longer word — in a mixed label ("number of tests") a short
+  word is usually a stopword, and letting "of" match ordinary prose would
+  drag unrelated numbers into the source. Longer words keep the
+  leading-boundary match so plural drift still hits.
+- **Flags inside real command spans and shell fences are now checked.**
+  The extractor only matched a flag backticked alone (`` `--flag` ``);
+  the common form `` `mytool --flag` `` — the whole command in one span —
+  matched nothing, despite a comment claiming it was handled, and flags
+  inside ```` ```bash ```` fences were unchecked. Both are now extracted
+  (fences: bash/sh/shell/console/zsh), with the command guessed from the
+  span or line itself before falling back to preceding prose.
+- **Markdown link titles no longer break path checks.**
+  `[text](docs/a.md "Read me")` treated the whole `docs/a.md "Read me"`
+  string as the path and errored even when the file exists. The optional
+  title is stripped and `<angle-bracket>` targets are unwrapped; dead
+  paths with titles are still flagged.
+- **The semantic layer no longer judges content against itself.**
+  `verify()` passed the generated content as its own source passages —
+  vacuously faithful by construction. `VerifyContext` gains a `passages`
+  field that is forwarded to the judge; when `semantic=True` with no
+  passages, verify degrades gracefully (warning, judge not called).
+- **The no-judge warning no longer misreports a bad judge.** A judge that
+  was provided but fails the `Judge` protocol check was reported as "no
+  judge was provided"; the message now names the failing object and the
+  protocol mismatch.
+
+### Changed
+
+- Import resolution passes the module name to the child interpreter via
+  `argv` instead of f-string interpolation into the `-c` program —
+  defense in depth (`ast` already guarantees identifier-safe names).
+- Corpus flag cases use the realistic single-span form
+  (`` `mytool --verbose` ``) instead of the unnatural split form; the
+  split form remains covered by `evasion_substring_flag`.
+
 ## [0.2.2] - 2026-07-17
 
 Accuracy + reliability patch from a full library review: five silent

src/attune_verify/_extract.py

@@ -42,7 +42,15 @@ class NumericClaim:
     re.MULTILINE | re.DOTALL,
 )
 _LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)")
-_NUM_RE = re.compile(r"\b(\d{2,})\b")  # 2+ digit numbers (skip single digits)
+# 2+ digit numbers (skip single digits). Comma-grouped values ("1,234") are one
+# claim — the first alternative captures the whole group before the bare \d{2,}
+# can grab a fragment. Digit runs touching a decimal point ("94.53", the "10"
+# in "Python 3.10") are decimal/version components, not counts, and are skipped
+# via the surrounding lookarounds.
+_NUM_RE = re.compile(r"(?<!\w)(?<!\d\.)(\d{1,3}(?:,\d{3})+|\d{2,})(?!\w)(?!\.\d)")
+# A markdown link target may carry a quoted/parenthesized title after the path
+# ('docs/a.md "Read me"') or wrap the path in <angle brackets>.
+_LINK_TITLE_RE = re.compile(r"""^(\S+)\s+("[^"]*"|'[^']*'|\([^)]*\))$""")
 
 
 def extract_code_fences(content: str) -> List[CodeFence]:
@@ -63,30 +71,50 @@ def extract_code_fences(content: str) -> List[CodeFence]:
 
 
 def extract_links(content: str) -> List[MarkdownLink]:
-    """Extract all markdown links from content."""
+    """Extract all markdown links from content.
+
+    Targets are normalized: an optional markdown title
+    (``docs/a.md "Read me"``) is stripped and ``<angle-bracket>`` wrapping is
+    removed, so checkers see only the path.
+    """
     links = []
     for match in _LINK_RE.finditer(content):
         line = content[: match.start()].count("\n") + 1
         links.append(
             MarkdownLink(
                 text=match.group(1),
-                target=match.group(2),
+                target=_clean_link_target(match.group(2)),
                 line=line,
             )
         )
     return links
 
 
+def _clean_link_target(raw: str) -> str:
+    """Strip an optional title and angle-bracket wrapping from a link target."""
+    target = raw.strip()
+    title_match = _LINK_TITLE_RE.match(target)
+    if title_match:
+        target = title_match.group(1)
+    if target.startswith("<") and target.endswith(">"):
+        target = target[1:-1]
+    return target
+
+
 def extract_numeric_claims(content: str) -> List[NumericClaim]:
-    """Extract numeric claims (2+ digit numbers) with surrounding context."""
+    """Extract numeric claims (2+ digit numbers) with surrounding context.
+
+    Comma-grouped numbers ("1,234") are one claim with the commas stripped;
+    decimal and version components ("94.53", "Python 3.10") are not claims.
+    """
     claims = []
     for match in _NUM_RE.finditer(content):
         line = content[: match.start()].count("\n") + 1
         start = max(0, match.start() - 40)
         end = min(len(content), match.end() + 40)
         claims.append(
             NumericClaim(
-                value=int(match.group(1)),
+                value=int(match.group(1).replace(",", "")),
                 context=content[start:end].replace("\n", " "),
                 line=line,
             )

src/attune_verify/_verify.py

@@ -25,7 +25,8 @@ def verify(content: str, context: VerifyContext) -> VerifyResult:
 
     Deterministic checkers always run and are independent — a failure in
     one does not abort the others. The semantic layer runs only when
-    context.semantic is True and a judge is available.
+    context.semantic is True and both a judge and source passages are
+    available.
 
     Args:
         content: LLM-generated content to verify.
@@ -108,14 +109,31 @@ def _run_semantic(result: VerifyResult, content: str, context: VerifyContext) ->
     """Run the semantic layer if a judge is available."""
     from attune_verify.semantic.protocol import Judge  # noqa: PLC0415
 
-    if context.judge is None or not isinstance(context.judge, Judge):
+    if context.judge is None:
+        detail = (
+            "Semantic layer requested (context.semantic=True) "
+            "but no judge was provided in VerifyContext.judge"
+        )
+    elif not isinstance(context.judge, Judge):
+        detail = (
+            "Semantic layer requested (context.semantic=True) but the "
+            f"provided judge ({type(context.judge).__name__}) does not "
+            "satisfy the Judge protocol (missing a compatible score())"
+        )
+    elif not context.passages:
+        # Without independent source passages the judge would score the
+        # content against itself — vacuously faithful, so skip instead.
+        detail = (
+            "Semantic layer requested (context.semantic=True) but no "
+            "source passages were provided in VerifyContext.passages"
+        )
+    else:
+        detail = None
+    if detail is not None:
         result.findings.append(
             Finding(
                 kind=FindingKind.SEMANTIC,
-                detail=(
-                    "Semantic layer requested (context.semantic=True) "
-                    "but no judge was provided in VerifyContext.judge"
-                ),
+                detail=detail,
                 evidence="",
                 severity="warning",
             )
@@ -126,7 +144,7 @@ def _run_semantic(result: VerifyResult, content: str, context: VerifyContext) ->
         verdict = context.judge.score(
             query="Verify this generated content for faithfulness",
             answer=content,
-            passages=content,
+            passages=context.passages,
         )
         result.semantic_ran = True
         if not verdict.faithful:

src/attune_verify/checkers/counts.py

@@ -83,12 +83,38 @@ def _find_close_label(
     """
     context_lower = context.lower()
     for label in sources:
-        words = label.lower().split()
-        if any(re.search(rf"\b{re.escape(w)}", context_lower) for w in words if len(w) > 3):
+        if any(_word_matches(w, context_lower) for w in _match_words(label)):
             return label
     return None
 
 
+def _match_words(label: str) -> list[str]:
+    """Return the label words eligible for keyword matching.
+
+    Short words (<= 3 chars) participate only when the label has no longer
+    word to match on. A label like "api" was silently dead when short words
+    were dropped outright; but in a mixed label like "number of tests" a
+    short word is usually a stopword — letting "of" match ordinary prose
+    would drag unrelated numbers into the source.
+    """
+    words = label.lower().split()
+    long_words = [w for w in words if len(w) > 3]
+    return long_words if long_words else words
+
+
+def _word_matches(word: str, text: str) -> bool:
+    """Match a label keyword in text, calibrated to the keyword's length.
+
+    Long words match on a leading boundary only, so plural drift still hits
+    ("test" ~ "tests"). Short words ("api", "eps") prefix-match far too
+    loosely, so they require an exact word-boundary match on both sides —
+    previously they were dropped entirely, silently killing their source.
+    """
+    if len(word) > 3:
+        return re.search(rf"\b{re.escape(word)}", text) is not None
+    return re.search(rf"\b{re.escape(word)}\b", text) is not None
+
+
 def _year_like(value: int) -> bool:
     return _YEAR_MIN <= value <= _YEAR_MAX
 
@@ -99,10 +125,11 @@ def _label_follows_number(claim: NumericClaim, label: str) -> bool:
     "2026 widgets" reads as a widget count; "2026 versions of the widgets"
     reads as a year that merely has the keyword nearby.
     """
-    match = re.search(rf"\b{claim.value}\b((?:\s+\S+){{1,2}})", claim.context.lower())
+    # The claim value has commas stripped, but the context still shows the
+    # written form — match either ("2,026 widgets" is a count, never a year).
+    grouped = re.escape(f"{claim.value:,}")
+    match = re.search(rf"\b(?:{claim.value}|{grouped})\b((?:\s+\S+){{1,2}})", claim.context.lower())
     if match is None:
         return False
     following = match.group(1)
-    return any(
-        re.search(rf"\b{re.escape(w)}", following) for w in label.lower().split() if len(w) > 3
-    )
+    return any(_word_matches(w, following) for w in _match_words(label))

src/attune_verify/checkers/flags.py

@@ -4,11 +4,18 @@
 
 import re
 import subprocess
-from typing import Dict, FrozenSet, List
+from typing import Dict, FrozenSet, List, Optional
 
+from attune_verify._extract import _FENCE_RE, extract_code_fences
 from attune_verify.result import Finding, FindingKind
 
-_FLAG_RE = re.compile(r"`(--[\w-]+)`")
+# One inline code span (`mytool --flag`); fences are handled separately.
+_INLINE_CODE_RE = re.compile(r"`([^`\n]+)`")
+# A flag token anywhere in code text. Stops before "=value"; the negative
+# lookbehind keeps it from matching the tail of a longer flag or a "---" rule.
+_FLAG_TOKEN_RE = re.compile(r"(?<![\w-])(--\w[\w-]*)")
+# Fence languages whose content is command lines worth flag-checking.
+_SHELL_LANGS = frozenset({"bash", "sh", "shell", "console", "zsh"})
 
 
 def check_flags(
@@ -30,36 +37,68 @@ def check_flags(
         List of findings for unverifiable or unknown flags.
     """
     findings: List[Finding] = []
-    # Find patterns like "`--flag`" or "`command --flag`"
-    for match in _FLAG_RE.finditer(content):
-        flag = match.group(1)
-        surrounding = content[max(0, match.start() - 30) : match.start()]
-        cmd = _guess_command(surrounding)
-        help_text = _get_help(cmd, help_commands, allowed_help_cmds)
-        if help_text is None:
-            findings.append(
-                Finding(
-                    kind=FindingKind.UNKNOWN_FLAG,
-                    detail=(
-                        f"Flag '{flag}' could not be verified "
-                        f"(no --help output available for command '{cmd}')"
-                    ),
-                    evidence=match.group(0),
-                    severity="warning",
-                )
+    # Inline spans: `--flag` alone or a whole command in one span
+    # (`mytool --flag`). Fence bodies are stripped first so they are never
+    # double-scanned as inline code.
+    prose = _FENCE_RE.sub("", content)
+    for match in _INLINE_CODE_RE.finditer(prose):
+        span = match.group(1)
+        for flag_match in _FLAG_TOKEN_RE.finditer(span):
+            cmd = _guess_command(span[: flag_match.start()])
+            if cmd == "unknown":
+                # Bare `--flag` span: the command is named in the prose
+                # before it ("Run mytool with `--flag`").
+                cmd = _guess_command(prose[max(0, match.start() - 30) : match.start()])
+            finding = _verify_flag(
+                flag_match.group(1), cmd, f"`{span}`", help_commands, allowed_help_cmds
             )
-        elif not _flag_in_help(flag, help_text):
-            findings.append(
-                Finding(
-                    kind=FindingKind.UNKNOWN_FLAG,
-                    detail=f"Flag '{flag}' not found in '{cmd} --help'",
-                    evidence=match.group(0),
-                    severity="error",
+            if finding is not None:
+                findings.append(finding)
+    # Shell fences: each line is a command whose flags are claims too.
+    for fence in extract_code_fences(content):
+        if fence.language not in _SHELL_LANGS:
+            continue
+        for line in fence.content.splitlines():
+            command_line = line.strip().lstrip("$").strip()
+            for flag_match in _FLAG_TOKEN_RE.finditer(command_line):
+                cmd = _guess_command(command_line[: flag_match.start()])
+                finding = _verify_flag(
+                    flag_match.group(1), cmd, command_line, help_commands, allowed_help_cmds
                 )
-            )
+                if finding is not None:
+                    findings.append(finding)
     return findings
 
 
+def _verify_flag(
+    flag: str,
+    cmd: str,
+    evidence: str,
+    help_commands: Dict[str, str],
+    allowed_help_cmds: FrozenSet[str],
+) -> Optional[Finding]:
+    """Check one flag against its command's help; None when it verifies."""
+    help_text = _get_help(cmd, help_commands, allowed_help_cmds)
+    if help_text is None:
+        return Finding(
+            kind=FindingKind.UNKNOWN_FLAG,
+            detail=(
+                f"Flag '{flag}' could not be verified "
+                f"(no --help output available for command '{cmd}')"
+            ),
+            evidence=evidence,
+            severity="warning",
+        )
+    if not _flag_in_help(flag, help_text):
+        return Finding(
+            kind=FindingKind.UNKNOWN_FLAG,
+            detail=f"Flag '{flag}' not found in '{cmd} --help'",
+            evidence=evidence,
+            severity="error",
+        )
+    return None
+
+
 def _flag_in_help(flag: str, help_text: str) -> bool:
     """Return True if flag appears in help as a whole token.
 

src/attune_verify/checkers/imports.py

@@ -105,12 +105,19 @@ def _modules_from_node(node: ast.AST) -> list[str]:
 
 
 def _resolves(module: str, env_python: str) -> bool:
-    """Return True if module is importable in env_python."""
+    """Return True if module is importable in env_python.
+
+    The module name travels via argv, not f-string interpolation into the
+    -c program — defense in depth (ast already guarantees identifier-safe
+    names, but the child program must not depend on that).
+    """
     result = subprocess.run(
         [
             env_python,
             "-c",
-            f"import importlib.util; " f"print(importlib.util.find_spec('{module}') is not None)",
+            "import importlib.util, sys; "
+            "print(importlib.util.find_spec(sys.argv[1]) is not None)",
+            module,
         ],
         capture_output=True,
         text=True,