Skip to content

Add verifying clobber suite: accuracy/leak/crash fuzzing on all cores - 1 - #110

Merged
Qubitium merged 1 commit into
mainfrom
test/verifying-clobber
Aug 24, 2026
Merged

Add verifying clobber suite: accuracy/leak/crash fuzzing on all cores - 1#110
Qubitium merged 1 commit into
mainfrom
test/verifying-clobber

Conversation

@Qubitium

Copy link
Copy Markdown
Contributor

Summary

Adds tests/test_clobber_verify.py — a verifying clobber suite. The existing test_clobber/test_clobber_thread hammer the API with mostly well-formed inputs and only require "no unexpected exception"; this suite generates random but valid regexes of all kinds and verifies the accuracy / leak / crash properties of every operation, on all cores − 1 workers (31 on the 32-core test box).

What it checks

Phase What is verified
Differential accuracy (cores−1 threads) Structured generator producing re-compatible patterns (literals, classes, greedy/lazy quantifiers, alternation, capturing/named groups, backrefs, conditionals, lookarounds, anchors, inline flags, str + bytes). Every op — search/match/fullmatch with random pos/endpos (incl. astral-plane subjects), findall, finditer, split, subn — compared span-for-span, group-for-group against re.
PCRE2-extended invariants (cores−1 threads) Second pool adds \p{..}, atomic groups, possessive quantifiers, \R, \Q..\E over subjects up to 2 KiB; checks structural invariants: spans in-bounds & ascending, m.group(i) == subject[start:end], findall/finditer/subn counts agree, search equals first finditer match.
Shared-finditer exactly-once One iterator drained concurrently by all workers; harvested union must equal the single-threaded reference — no lost, duplicated, or torn matches (directly exercises the FindIter lock from #108 for accuracy, not just absence of crashes).
Leak stability Steady-state rounds over a fixed case set; asserts bounded sys.getallocatedblocks() and RSS growth.
Crash/deadlock Shared-deadline join watchdog; a hang dumps all stacks and names the in-flight pattern of each wedged worker. On free-threaded builds the suite also asserts sys._is_gil_enabled() is False.

Generator soundness guards (each one prevents a class of false positives found while building this): group numbers tracked by opening-paren order so backrefs/conditionals only reference closed groups; nullable group bodies are never quantified (re/PCRE2 empty-repeat semantics differ); boundedness propagates into nested groups plus a per-pattern unbounded-quantifier budget, so the limit-less re oracle can't hit exponential/deep-polynomial backtracking.

Upstream PCRE2 10.46 findings (worth reporting to PCRE2Project)

The fuzzer immediately surfaced three engine-level wrong-result behaviors, all reproduced with raw libpcre2-8 via ctypes (no pypcre involved):

  1. JIT start-optimization loses matches0?a[b ]{0,2} on b' ab ': pcre2_jit_match → no match, pcre2_match(1, 4). Compiling with PCRE2_NO_START_OPTIMIZE makes JIT agree, implicating the JIT fast-forward scan with an optional first atom. Many variants found (e.g. (?s)(?i)[_ ]?b.{3,5}2+\w?, d?1([a-f]{0,2}(?=[^_]))-). Since pypcre enables JIT by default, affected patterns silently return wrong results through no fault of pypcre — you may want to raise this upstream and/or consider a mitigation. The suite checks JIT parity separately and reports divergences without failing (strict mode: PYPCRE_CLOBBER_STRICT_JIT=1).
  2. Auto-possessification changes results around explicit possessives(\S*?|.?2*)(b{2,2}|(c??[a-fg-k ]+|[\d\w]??_)){1,1}(\-{1,2})?+a on b'd_S2ea1-' fails to match unless compiled with PCRE2_NO_AUTO_POSSESS (re: (0, 6)). Excluded from the differential pool.
  3. {0,0} zero-repetition groups((z)x|(?=w)y){0,0} (reduces to the empty pattern) matches nothing; re matches empty at every position. Excluded from the differential pool.

Results

  • Free-threaded 3.14.7, GIL confirmed off at runtime, 31 workers, 5-minute run: 8,655,968 differential cases + 5,943,857 invariant cases — zero pypcre accuracy failures, leak growth 74 blocks / 0 B RSS, no crashes, no deadlocks.
  • GIL 3.14.7: 270k + 47k cases in the 60 s configuration — clean.
  • Full pytest suite green on both builds with the new module included.
  • Tunables: PYPCRE_CLOBBER_VERIFY_SECONDS (default 45 s differential phase), PYPCRE_CLOBBER_SEED for reproduction (every failure message embeds the seed and full case).

🤖 Generated with Claude Code

test_clobber and test_clobber_thread hammer the API but only require "no
unexpected exception".  tests/test_clobber_verify.py generates random but
VALID regexes from a structured builder and verifies every result:

- differential accuracy vs re (all cores - 1 workers): patterns stay
  inside the re-compatible dialect (literals, classes, greedy/lazy
  quantifiers, alternation, capturing/named groups, backrefs, conditionals,
  fixed-width lookbehind, lookahead, anchors, inline flags, str and bytes)
  and every op (search/match/fullmatch with random pos/endpos incl.
  astral-plane subjects, findall, finditer, split, subn) is compared
  span-for-span and group-for-group against re.  The generator tracks
  group numbers by opening-paren order so backrefs/conditionals only
  reference closed groups, never quantifies a nullable group body, and
  propagates boundedness into nested groups with a per-pattern unbounded-
  quantifier budget so the limit-less re oracle cannot blow up.
- PCRE2-extended invariants: a second pool adds \p{..}, atomic groups,
  possessive quantifiers, \R, \Q..\E on up to 2 KiB subjects and checks
  structural invariants (spans in bounds and ascending, group slices match
  the subject, findall/finditer/subn agree, search equals first finditer
  match).
- shared-finditer exactly-once: one iterator drained concurrently by all
  workers; the harvested union must equal the single-threaded reference
  with no lost or duplicated matches.
- leak stability: steady-state rounds over a fixed case set must not grow
  allocated blocks or RSS beyond noise.
- crash/deadlock: workers run under a shared-deadline join watchdog that
  dumps stacks and names the in-flight pattern of any wedged worker.

On a free-threaded build the suite asserts sys._is_gil_enabled() is False
(the extension must keep declaring Py_MOD_GIL_NOT_USED).

JIT parity is checked separately and reported without failing (strict mode
via PYPCRE_CLOBBER_STRICT_JIT=1): the fuzzer found that PCRE2 10.46's JIT
returns wrong results on patterns pypcre cannot work around, e.g.
0?a[b ]{0,2} fails to match " ab " under pcre2_jit_match while pcre2_match
returns (1, 4); PCRE2_NO_START_OPTIMIZE makes JIT agree, implicating the
JIT fast-forward start optimization.  Also excluded from the differential
pool as verified upstream/engine behavior (raw libpcre2 reproduces both
without pypcre): auto-possessification changing results around explicit
possessive quantifiers, and {0,0} zero-repetition groups over alternations
with nested capture + lookahead matching nothing where re matches empty.

Validated on CPython 3.14.7 free-threaded (GIL=0 at runtime, 31 workers,
5-minute run: 8.65M differential + 5.94M invariant cases, zero accuracy
failures, zero leak growth) and 3.14.7 GIL builds; duration tunable via
PYPCRE_CLOBBER_VERIFY_SECONDS, seed via PYPCRE_CLOBBER_SEED.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Qubitium
Qubitium merged commit 09ab546 into main Aug 24, 2026
19 checks passed
Qubitium added a commit that referenced this pull request Aug 24, 2026
…111)

The JIT fast-forward (start-of-match optimization) in PCRE2 10.46 and
10.47 skips valid start positions for patterns that begin with an
optional atom followed by a literal and a bounded class:
pcre2_jit_match finds nothing for 0?a[b ]{0,2} on " ab " while
pcre2_match returns (1, 4).  Verified against raw libpcre2 builds of the
10.46 and 10.47 releases (both affected) and current upstream main
(10.48-dev, fixed).  Since JIT is enabled by default, affected patterns
silently returned wrong results on every currently shipping PCRE2.

Following the jit_anchor_fixup_needed() precedent, the runtime is probed
once at module init (jit_start_optimize_broken): compile the probe
pattern, run pcre2_jit_match vs pcre2_match(PCRE2_NO_JIT) on the same
code, and compare.  On a broken runtime, Pattern_create injects
PCRE2_NO_START_OPTIMIZE into the compile options of JIT-bound patterns
— an option bit available on every PCRE2 version, so no new symbol
dependency — and masks it back out of the pattern's public flags (the
bit shares a value with Flag.SUBSTITUTE_MATCHED on the Python side; a
caller-requested NO_START_OPTIMIZE is preserved).  Fixed runtimes probe
clean and compile exactly as before.

Validation on the affected 10.46 system runtime: the verifying clobber
suite (PR #110) ran 3.55M differential cases with
PYPCRE_CLOBBER_STRICT_JIT=1 (JIT-vs-interpreter divergence = failure)
and found ZERO divergences, versus roughly ten per minute before the
workaround.  New tests/test_jit_start_optimize_workaround.py pins the
known repro family (must pass on both broken and fixed runtimes) and
asserts the injected bit does not leak into Pattern.flags.  Full pytest
suite green on CPython 3.14.7 free-threaded (GIL=0) and GIL builds.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant