Skip to content

Static patterns: \s+/\s* cross paragraph boundaries — 1,485 occurrences across all 15 pattern files (systemic #405) #446

Description

@yashrajp22

Summary

#405 was one instance of a systemic problem: Python's \s matches newlines, so every \s+/\s* in a static pattern can silently bridge a paragraph boundary (blank line) and stitch together two unrelated sentences. #417 fixed the one reported regex. The disease is everywhere else:

  • 1,485 occurrences of \s+/\s* across all 15 static_patterns_*.py files (counted on current main).
  • All of them run via re.finditer(pattern, content, re.IGNORECASE | re.MULTILINE) over the whole file content, so nothing stops a match from spanning paragraphs.

Reproducible example, live on main today

The pattern directly below the one #417 fixed, in the same EA1 list:

(r"(?:allow|grant|enable)\s+(?:access\s+to\s+)?(?:all|any|every)\s+tools?", 0.8)

against this benign markdown:

lists the features you enable

Any tools mentioned below require review.

matches enable\n\nAny tools → EA1/0.8 false positive bridging two unrelated paragraphs. Every prose-style pattern in all 15 files has the same exposure.

Why per-pattern fixes are the wrong tool

Replacing \s+ with [ \t]+ per pattern is wrong for prose: a sentence soft-wrapped mid-phrase (allow access to\nall tools) is one sentence and should match. The bug is only crossing blank lines (paragraph boundaries). Hand-tuning 1,485 quantifiers case-by-case is unmaintainable whack-a-mole — #405/#417 was one mole.

Proposed direction (one fix, all 15 files)

Fix it once in the shared runner instead of in every pattern. Options, in rough order of preference:

  1. Split content into paragraphs (re.split(r"\n[ \t]*\n", content)) in the shared matching loop and run patterns per paragraph, offsetting line numbers. No pattern changes at all. Single-line config patterns (like EA1's wildcard) are unaffected.
  2. Rewrite quantifiers at compile time: a small helper that transforms \s+/\s* into (?:[^\S\n]|\n(?![ \t]*\n))+ (whitespace, but a newline only when not followed by a blank line) before re.compile. Keeps whole-content matching for anything that legitimately needs it.
  3. Per-pattern audits — only as a last resort, for patterns that intentionally span paragraphs (none found so far).

Option 1 is the smallest diff and the easiest to reason about; it also makes matched text/line attribution cleaner (no more 'enable\n\nAny tools' spans in reports).

Acceptance criteria

  • The example above no longer fires.
  • Existing single-line and same-paragraph positive tests still pass unchanged.
  • A regression test asserts that no static pattern can match across a blank line.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions