Add verifying clobber suite: accuracy/leak/crash fuzzing on all cores - 1 - #110
Merged
Conversation
test_clobber and test_clobber_thread hammer the API but only require "no
unexpected exception". tests/test_clobber_verify.py generates random but
VALID regexes from a structured builder and verifies every result:
- differential accuracy vs re (all cores - 1 workers): patterns stay
inside the re-compatible dialect (literals, classes, greedy/lazy
quantifiers, alternation, capturing/named groups, backrefs, conditionals,
fixed-width lookbehind, lookahead, anchors, inline flags, str and bytes)
and every op (search/match/fullmatch with random pos/endpos incl.
astral-plane subjects, findall, finditer, split, subn) is compared
span-for-span and group-for-group against re. The generator tracks
group numbers by opening-paren order so backrefs/conditionals only
reference closed groups, never quantifies a nullable group body, and
propagates boundedness into nested groups with a per-pattern unbounded-
quantifier budget so the limit-less re oracle cannot blow up.
- PCRE2-extended invariants: a second pool adds \p{..}, atomic groups,
possessive quantifiers, \R, \Q..\E on up to 2 KiB subjects and checks
structural invariants (spans in bounds and ascending, group slices match
the subject, findall/finditer/subn agree, search equals first finditer
match).
- shared-finditer exactly-once: one iterator drained concurrently by all
workers; the harvested union must equal the single-threaded reference
with no lost or duplicated matches.
- leak stability: steady-state rounds over a fixed case set must not grow
allocated blocks or RSS beyond noise.
- crash/deadlock: workers run under a shared-deadline join watchdog that
dumps stacks and names the in-flight pattern of any wedged worker.
On a free-threaded build the suite asserts sys._is_gil_enabled() is False
(the extension must keep declaring Py_MOD_GIL_NOT_USED).
JIT parity is checked separately and reported without failing (strict mode
via PYPCRE_CLOBBER_STRICT_JIT=1): the fuzzer found that PCRE2 10.46's JIT
returns wrong results on patterns pypcre cannot work around, e.g.
0?a[b ]{0,2} fails to match " ab " under pcre2_jit_match while pcre2_match
returns (1, 4); PCRE2_NO_START_OPTIMIZE makes JIT agree, implicating the
JIT fast-forward start optimization. Also excluded from the differential
pool as verified upstream/engine behavior (raw libpcre2 reproduces both
without pypcre): auto-possessification changing results around explicit
possessive quantifiers, and {0,0} zero-repetition groups over alternations
with nested capture + lookahead matching nothing where re matches empty.
Validated on CPython 3.14.7 free-threaded (GIL=0 at runtime, 31 workers,
5-minute run: 8.65M differential + 5.94M invariant cases, zero accuracy
failures, zero leak growth) and 3.14.7 GIL builds; duration tunable via
PYPCRE_CLOBBER_VERIFY_SECONDS, seed via PYPCRE_CLOBBER_SEED.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Qubitium
added a commit
that referenced
this pull request
Aug 24, 2026
…111) The JIT fast-forward (start-of-match optimization) in PCRE2 10.46 and 10.47 skips valid start positions for patterns that begin with an optional atom followed by a literal and a bounded class: pcre2_jit_match finds nothing for 0?a[b ]{0,2} on " ab " while pcre2_match returns (1, 4). Verified against raw libpcre2 builds of the 10.46 and 10.47 releases (both affected) and current upstream main (10.48-dev, fixed). Since JIT is enabled by default, affected patterns silently returned wrong results on every currently shipping PCRE2. Following the jit_anchor_fixup_needed() precedent, the runtime is probed once at module init (jit_start_optimize_broken): compile the probe pattern, run pcre2_jit_match vs pcre2_match(PCRE2_NO_JIT) on the same code, and compare. On a broken runtime, Pattern_create injects PCRE2_NO_START_OPTIMIZE into the compile options of JIT-bound patterns — an option bit available on every PCRE2 version, so no new symbol dependency — and masks it back out of the pattern's public flags (the bit shares a value with Flag.SUBSTITUTE_MATCHED on the Python side; a caller-requested NO_START_OPTIMIZE is preserved). Fixed runtimes probe clean and compile exactly as before. Validation on the affected 10.46 system runtime: the verifying clobber suite (PR #110) ran 3.55M differential cases with PYPCRE_CLOBBER_STRICT_JIT=1 (JIT-vs-interpreter divergence = failure) and found ZERO divergences, versus roughly ten per minute before the workaround. New tests/test_jit_start_optimize_workaround.py pins the known repro family (must pass on both broken and fixed runtimes) and asserts the injected bit does not leak into Pattern.flags. Full pytest suite green on CPython 3.14.7 free-threaded (GIL=0) and GIL builds. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tests/test_clobber_verify.py— a verifying clobber suite. The existingtest_clobber/test_clobber_threadhammer the API with mostly well-formed inputs and only require "no unexpected exception"; this suite generates random but valid regexes of all kinds and verifies the accuracy / leak / crash properties of every operation, on all cores − 1 workers (31 on the 32-core test box).What it checks
re-compatible patterns (literals, classes, greedy/lazy quantifiers, alternation, capturing/named groups, backrefs, conditionals, lookarounds, anchors, inline flags, str + bytes). Every op —search/match/fullmatchwith randompos/endpos(incl. astral-plane subjects),findall,finditer,split,subn— compared span-for-span, group-for-group againstre.\p{..}, atomic groups, possessive quantifiers,\R,\Q..\Eover subjects up to 2 KiB; checks structural invariants: spans in-bounds & ascending,m.group(i) == subject[start:end],findall/finditer/subncounts agree,searchequals firstfinditermatch.sys.getallocatedblocks()and RSS growth.sys._is_gil_enabled()is False.Generator soundness guards (each one prevents a class of false positives found while building this): group numbers tracked by opening-paren order so backrefs/conditionals only reference closed groups; nullable group bodies are never quantified (re/PCRE2 empty-repeat semantics differ); boundedness propagates into nested groups plus a per-pattern unbounded-quantifier budget, so the limit-less
reoracle can't hit exponential/deep-polynomial backtracking.Upstream PCRE2 10.46 findings (worth reporting to PCRE2Project)
The fuzzer immediately surfaced three engine-level wrong-result behaviors, all reproduced with raw
libpcre2-8via ctypes (no pypcre involved):0?a[b ]{0,2}onb' ab ':pcre2_jit_match→ no match,pcre2_match→(1, 4). Compiling withPCRE2_NO_START_OPTIMIZEmakes JIT agree, implicating the JIT fast-forward scan with an optional first atom. Many variants found (e.g.(?s)(?i)[_ ]?b.{3,5}2+\w?,d?1([a-f]{0,2}(?=[^_]))-). Since pypcre enables JIT by default, affected patterns silently return wrong results through no fault of pypcre — you may want to raise this upstream and/or consider a mitigation. The suite checks JIT parity separately and reports divergences without failing (strict mode:PYPCRE_CLOBBER_STRICT_JIT=1).(\S*?|.?2*)(b{2,2}|(c??[a-fg-k ]+|[\d\w]??_)){1,1}(\-{1,2})?+aonb'd_S2ea1-'fails to match unless compiled withPCRE2_NO_AUTO_POSSESS(re:(0, 6)). Excluded from the differential pool.{0,0}zero-repetition groups —((z)x|(?=w)y){0,0}(reduces to the empty pattern) matches nothing;rematches empty at every position. Excluded from the differential pool.Results
PYPCRE_CLOBBER_VERIFY_SECONDS(default 45 s differential phase),PYPCRE_CLOBBER_SEEDfor reproduction (every failure message embeds the seed and full case).🤖 Generated with Claude Code