diff --git a/README.md b/README.md index 72e792a..e965cde 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,9 @@ # Contact: qubitium@modelcloud.ai, x.com/qubitium --> -# PyPcre (Python PCRE2 Binding) +# PyPcre (Python PCRE2 Binding) ๐Ÿงฌ -Modern `nogil` Python bindings for the PCRE2 library with `stdlib.re` API compatibility. +Fast, free-threaded Python bindings for `PCRE2` with a stable `stdlib.re`-compatible API. โšก

GitHub release @@ -19,52 +19,92 @@ Modern `nogil` Python bindings for the PCRE2 library with `stdlib.re` API compat -## Latest News -* 03/22/2026 [0.2.15](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.15): Python 3.15 `re` compatibility (`prefixmatch`, `NOFLAG`) -* 03/21/2026 [0.2.14](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.14): Python 3.14 compatibility -* 03/02/2026 [0.2.11](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.11): Auto-detect `Visual Studio` in Windows environments during install and compile. -* 02/24/2026 [0.2.10](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.10): Allow a `Visual Studio` (VS) compiler version check override via an environment variable. -* 12/15/2025 [0.2.8](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.8): Fixed multi-arch Linux OS compatibility when both x86_64 and i386 `pcre2` libraries are installed. -* 10/20/2025 [0.2.4](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.4): Removed the dependency on a system `python3-dev` package. `Python.h` will be downloaded optimistically from python.org when needed. +## Latest News ๐Ÿš€ +* 04/13/2026 [0.3.0](https://github.com/ModelCloud/PyPcre/releases/tag/v0.3.0): Lower-overhead public `Match` objects, faster hot-path `match()` / `search()` / `fullmatch()` / `findall()`, and tighter free-threaded execution. โšก +* 03/22/2026 [0.2.15](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.15): Python 3.15 `re` compatibility (`prefixmatch`, `NOFLAG`) โœ… +* 03/21/2026 [0.2.14](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.14): Python 3.14 compatibility ๐Ÿ +* 03/02/2026 [0.2.11](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.11): Auto-detect `Visual Studio` in Windows environments during install and compile. ๐ŸชŸ +* 02/24/2026 [0.2.10](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.10): Allow a `Visual Studio` (VS) compiler version check override via an environment variable. ๐Ÿงฐ +* 12/15/2025 [0.2.8](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.8): Fixed multi-arch Linux OS compatibility when both x86_64 and i386 `pcre2` libraries are installed. ๐Ÿง +* 10/20/2025 [0.2.4](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.4): Removed the dependency on a system `python3-dev` package. `Python.h` will be downloaded optimistically from python.org when needed. ๐Ÿ“ฆ * 10/12/2025 [0.2.3](https://github.com/ModelCloud/PyPcre/releases/tag/v0.2.3): ๐Ÿค— Full `GIL=0` compliance for Python >= 3.13T. Reduced cache thread contention. Improved performance across all APIs. Expanded CI test coverage. FreeBSD, Solaris, and Windows compatibility validated. * 10/09/2025 [0.1.0](https://github.com/ModelCloud/PyPcre/releases/tag/v0.1.0): ๐ŸŽ‰ First release. Thread-safe, with auto JIT, auto pattern caching, and optimistic linking to the system library for fast installs. -## Why PyPcre +## Why PyPcre โšก -PyPcre is a modern PCRE2 binding designed to be both fast and thread-safe in a `GIL=0` world. In the era of the global interpreter lock, Python had real threads but often only limited concurrency, aside from a handful of low-level APIs and packages. As Python moves toward a fuller `GIL=0` design, true multi-threaded concurrency becomes practical and brings Python closer to parity with other modern languages. +PyPcre pairs Python's familiar `re`-compatible API with the real `PCRE2` engine. You keep the ergonomics of the standard library while gaining a more capable regex engine, optional JIT, explicit threading support, and a binding designed and tested for free-threaded Python. ๐Ÿง โšก -Many Python regular expression packages either segfault under `GIL=0` or suffer suboptimal performance because they were not designed with threaded execution in mind. +### Big Wins ๐Ÿ† -PyPcre is fully CI-tested. Every API and PCRE2 flag is exercised in a continuous development environment backed by the ModelCloud.AI team. Fuzz (clobber) tests are also run to catch memory safety, accuracy, and memory leak regressions. +- ๐Ÿงฌ **Full power of PCRE2**: PyPcre uses the real `PCRE2` engine, so you get native compile options, semantics, JIT, and upstream tuning. +- ๐Ÿ”ฅ **More expressive regex syntax**: `PCRE2` supports constructs beyond stdlib `re`, including atomic groups `(?>...)`, possessive quantifiers `++`, branch-reset groups `(?|...)`, richer lookarounds, and backtracking control verbs like `(*SKIP)(*FAIL)`. +- ๐Ÿงต **Thread-safe into `nogil`**: PyPcre is built for `PYTHON_GIL=0`, with CI coverage, lock-aware caches, reusable match/JIT resources, and `parallel_map()` for multi-subject fan-out. +- โšก **Fast on real workloads**: `PCRE2` JIT plus cached compiled patterns lets PyPcre match or beat `re` and `regex` on many common scans, especially multiline searches, lookaround-heavy patterns, and free-threaded execution. +- ๐Ÿ›ก๏ธ **Safer operational story**: PyPcre prefers the system `libpcre2-8` shared library so normal OS package updates can bring security and bug-fix benefits without a bundled fork. +- โœ… **Validated thoroughly**: the project runs API tests, fuzz tests, memory-safety checks, local `valgrind` leak checks, and `massif` heap profiles. Recent local profiling found `0` definite leaks and `0` possible leaks in both the public API and raw binding paths. -For safety, PyPcre preferentially links against the OS-provided `libpcre2` package so it can benefit from upstream security patches. You can force a full source build with the `PYPCRE_BUILD_FROM_SOURCE=1` environment variable. +### Quick Comparison ๐ŸฅŠ -## Installation +| Area | PyPcre | `stdlib.re` | `regex` | +| --- | --- | --- | --- | +| Engine | Full `PCRE2` โœ… | CPython stdlib engine | Separate engine, not `PCRE2` | +| `PCRE2` syntax and flags | Full access โœ… | No | No | +| Syntax power | Very rich โœ… | More limited | Rich, but different from `PCRE2` | +| JIT execution | `PCRE2` JIT โœ… | No | No | +| `re`-compatible API surface | Stable and familiar โœ… | Native | Similar, but not the main goal | +| Free-threaded support | Built and tested for `PYTHON_GIL=0` โœ… | No explicit PyPcre-style layer | Not a project focus here | +| Built-in threaded subject fan-out | `parallel_map()` โœ… | No | No | +| System library updates | Uses system `libpcre2-8` by default โœ… | N/A | N/A | + +### Benchmark Highlights ๐Ÿ + +Measured on a `Python 3.14.3` free-threaded build on x86_64 Linux with compiled-pattern reuse. Times are best-of-5; lower is better. + +| Workload | Operation | PyPcre | `re` | `regex` | PyPcre edge | +| --- | --- | ---: | ---: | ---: | --- | +| First `ERROR` line in a multiline log buffer | `search` | `3.68 ms` | `51.72 ms` | `5.67 ms` | `14.0x` vs `re`, `1.54x` vs `regex` | +| Extract only `WARN` / `ERROR` lines | `findall` | `6.41 ms` | `91.84 ms` | `91.14 ms` | `14.3x` vs `re`, `14.2x` vs `regex` | +| Per-line full-name extraction | `findall` | `22.28 ms` | `172.38 ms` | `218.29 ms` | `7.74x` vs `re`, `9.80x` vs `regex` | +| Lookbehind + negative-lookahead extraction | `findall` | `50.23 ms` | `53.35 ms` | `57.03 ms` | `1.06x` vs `re`, `1.14x` vs `regex` | +| UUID extraction | `findall` | `77.49 ms` | `83.19 ms` | `134.87 ms` | `1.07x` vs `re`, `1.74x` vs `regex` | +| Boundary-aware token scan | `findall` | `127.76 ms` | `128.03 ms` | `146.37 ms` | effectively tied with `re`, `1.15x` vs `regex` | + +### Free-Threaded Benchmark Highlights ๐Ÿงต + +Measured in the same environment with `8` threads sharing one compiled pattern. Times are best-of-3; lower is better. + +| Workload | Threads | PyPcre | `re` | `regex` | PyPcre edge | +| --- | ---: | ---: | ---: | ---: | --- | +| First `ERROR` line in a multiline log buffer | `8` | `25.34 ms` | `38.83 ms` | `40.34 ms` | `1.53x` vs `re`, `1.59x` vs `regex` | +| Extract only `WARN` / `ERROR` lines | `8` | `28.58 ms` | `65.54 ms` | `73.55 ms` | `2.29x` vs `re`, `2.57x` vs `regex` | +| Per-line full-name extraction | `8` | `31.68 ms` | `123.44 ms` | `164.80 ms` | `3.90x` vs `re`, `5.20x` vs `regex` | + +PyPcre is the stronger all-around choice when you want more than the baseline: full `PCRE2` features, more expressive syntax, JIT, explicit free-threaded support, and a stable `re`-compatible API surface. It keeps Python ergonomics while giving you a substantially more capable engine. ๐Ÿš€ + +## Installation ๐Ÿ“ฆ ```bash pip install PyPcre ``` -The package prefers linking against the system `libpcre2-8` shared library for fast installs and to inherit security updates from the OS. See [Building](#building) for manual build details. +By default, the package links against the system `libpcre2-8` shared library for fast installs and to inherit OS security updates. See [Building](#building) for manual build details. -## Platform Support (Validated) +## Platform Support (Validated) โœ… `Linux`, `macOS`, `Windows`, `WSL`, `FreeBSD` -## Usage +## Usage ๐Ÿ› ๏ธ -If you already rely on the standard library `re`, migrating is as -simple as changing your import: +If you already use the standard library `re`, migration is often just an import swap: ```python import pcre as re ``` -The high-level API keeps the standard library shape, so most existing `re` -code can move over with little or no rewriting. +The high-level API stays close to the standard library, so most existing `re` code can move over with little or no rewriting. -### Quick start +### Quick start ๐Ÿš€ ```python from pcre import compile, findall, match, search, Flag @@ -76,7 +116,7 @@ pattern = compile(rb"\d+", flags=Flag.MULTILINE) numbers = pattern.findall(b"line 1\nline 22") ``` -### User-facing API +### API Overview ๐Ÿงญ - Module helpers: `prefixmatch`, `match`, `search`, `fullmatch`, `finditer`, `findall`, `split`, `sub`, `subn`, `compile`, `escape`, `purge`, and @@ -94,7 +134,7 @@ numbers = pattern.findall(b"line 1\nline 22") - Errors are raised as `pcre.PcreError`; `error` and `PatternError` are kept as compatibility aliases. -### Common examples +### Common examples ๐Ÿงช Compiled patterns: @@ -124,7 +164,7 @@ pattern = compile(br"\w+") print(pattern.findall(b"ab cd")) # [b'ab', b'cd'] ``` -### Stdlib `re` compatibility +### Stdlib `re` compatibility ๐Ÿ” - Module-level helpers and the `Pattern` class follow the same call shapes as the standard library `re` module, including `pos`, `endpos`, and `flags` @@ -148,7 +188,7 @@ print(pattern.findall(b"ab cd")) # [b'ab', b'cd'] patterns so escaping semantics remain identical. - String patterns enable Unicode behavior by default. Byte patterns do not. -### `regex` package compatibility +### `regex` package compatibility ๐Ÿ”„ The [`regex`](https://pypi.org/project/regex/) package interprets `\uXXXX` and `\UXXXXXXXX` escapes as UTF-8 code points, while PCRE2 expects @@ -166,7 +206,7 @@ Set the default behavior globally with `pcre.configure(compat_regex=True)` so that subsequent calls to `compile()` and the module-level helpers apply the conversion without repeating the flag. -### Common issues +### Common issues โš ๏ธ - Unsupported stdlib flags such as `re.DEBUG`, `re.LOCALE`, and `re.ASCII` raise `ValueError`. If you want ASCII-style behavior, use `pcre.ASCII` or @@ -178,7 +218,7 @@ the conversion without repeating the flag. - Most users do not need to tune caching, JIT, or threading. The defaults are intended to work well out of the box. -### Optional runtime controls +### Optional runtime controls ๐ŸŽ›๏ธ - `pcre.configure(jit=False)` disables JIT globally. `Flag.JIT` and `Flag.NO_JIT` let you override that per pattern. @@ -188,10 +228,9 @@ the conversion without repeating the flag. `shutdown_thread_pool()`, `Flag.THREADS`, and `Flag.NO_THREADS` are available if you want to opt into or restrict threaded execution. -## Building +## Building ๐Ÿ—๏ธ -The extension links against an existing PCRE2 installation (the `libpcre2-8` -variant). Install the development headers for your platform before building, +The extension links against an existing `libpcre2-8` installation. Install the development headers for your platform before building, for example `apt install libpcre2-dev` on Debian/Ubuntu, `dnf install pcre2-devel` on Fedora/RHEL derivatives, or `brew install pcre2` on macOS. diff --git a/pcre/pcre.py b/pcre/pcre.py index 3a98efe..84b4880 100644 --- a/pcre/pcre.py +++ b/pcre/pcre.py @@ -28,7 +28,7 @@ NO_UTF: int = int(Flag.NO_UTF) NO_UCP: int = int(Flag.NO_UCP) from .re_compat import ( - Match, + Match as _CompatMatch, TemplatePatternStub, coerce_group_value, coerce_subject_slice, @@ -53,6 +53,9 @@ _CPattern = _pcre2.Pattern PcreError = _pcre2.PcreError +Match = getattr(_pcre2, "Match", _CompatMatch) +_ATTACH_MATCH = getattr(_pcre2, "_attach_match", None) +_RAW_MATCH_TYPE = getattr(_pcre2, "Match", None) FlagInput = int | _std_re.RegexFlag | Iterable[int | _std_re.RegexFlag] @@ -65,6 +68,14 @@ _THREAD_MODE_AUTO = "auto" +def _can_attach_match(raw: Any) -> bool: + return ( + _ATTACH_MATCH is not None + and _RAW_MATCH_TYPE is not None + and isinstance(raw, _RAW_MATCH_TYPE) + ) + + def _resolve_jit_setting(jit: bool | None) -> bool: if jit is None: return _DEFAULT_JIT @@ -162,13 +173,6 @@ def _normalise_flags(flags: FlagInput) -> int: raise TypeError("flags must be an int, stdlib re flag, or an iterable of those") -def _call_with_optional_end(method, subject: Any, pos: int, endpos: int | None, options: int): - resolved_end = resolve_endpos(subject, endpos) - if endpos is None: - return method(subject, pos=pos, options=options), resolved_end - return method(subject, pos=pos, endpos=resolved_end, options=options), resolved_end - - class Pattern: """High-level wrapper around the C-backed :class:`pcre_ext_c.Pattern`.""" @@ -223,8 +227,10 @@ def enable_auto_threads(self) -> None: self._thread_mode = _THREAD_MODE_AUTO def _update_group_hint(self, match: Match) -> None: + if self._groups_hint is not None: + return groups_count = len(match.groups()) - if self._groups_hint is None or groups_count > self._groups_hint: + if groups_count > 0: self._groups_hint = groups_count def _wrap_match( @@ -236,7 +242,9 @@ def _wrap_match( ) -> Match | None: if raw is None: return None - wrapped = Match(self, raw, subject, pos, end_boundary) + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) + wrapped = _CompatMatch(self, raw, subject, pos, end_boundary) self._update_group_hint(wrapped) return wrapped @@ -248,8 +256,22 @@ def match( endpos: int | None = None, options: int = 0, ) -> Match | None: - subject = prepare_subject(subject) - raw, resolved_end = _call_with_optional_end(self._pattern.match, subject, pos, endpos, options) + if type(subject) is memoryview: + subject = subject.tobytes() + if endpos is None: + raw = self._pattern.match(subject, pos=pos, options=options) + if raw is None: + return None + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) + resolved_end = len(subject) + else: + resolved_end = resolve_endpos(subject, endpos) + raw = self._pattern.match(subject, pos=pos, endpos=resolved_end, options=options) + if raw is None: + return None + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) return self._wrap_match(raw, subject, pos, resolved_end) prefixmatch = match @@ -262,8 +284,22 @@ def search( endpos: int | None = None, options: int = 0, ) -> Match | None: - subject = prepare_subject(subject) - raw, resolved_end = _call_with_optional_end(self._pattern.search, subject, pos, endpos, options) + if type(subject) is memoryview: + subject = subject.tobytes() + if endpos is None: + raw = self._pattern.search(subject, pos=pos, options=options) + if raw is None: + return None + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) + resolved_end = len(subject) + else: + resolved_end = resolve_endpos(subject, endpos) + raw = self._pattern.search(subject, pos=pos, endpos=resolved_end, options=options) + if raw is None: + return None + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) return self._wrap_match(raw, subject, pos, resolved_end) def fullmatch( @@ -274,8 +310,22 @@ def fullmatch( endpos: int | None = None, options: int = 0, ) -> Match | None: - subject = prepare_subject(subject) - raw, resolved_end = _call_with_optional_end(self._pattern.fullmatch, subject, pos, endpos, options) + if type(subject) is memoryview: + subject = subject.tobytes() + if endpos is None: + raw = self._pattern.fullmatch(subject, pos=pos, options=options) + if raw is None: + return None + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) + resolved_end = len(subject) + else: + resolved_end = resolve_endpos(subject, endpos) + raw = self._pattern.fullmatch(subject, pos=pos, endpos=resolved_end, options=options) + if raw is None: + return None + if _can_attach_match(raw): + return _ATTACH_MATCH(raw, self) return self._wrap_match(raw, subject, pos, resolved_end) def finditer( @@ -286,7 +336,8 @@ def finditer( endpos: int | None = None, options: int = 0, ) -> Generator[Match, None, None]: - subject = prepare_subject(subject) + if type(subject) is memoryview: + subject = subject.tobytes() origin_pos = pos resolved_end = resolve_endpos(subject, endpos) backend_iter = getattr(self._pattern, "finditer", None) @@ -298,9 +349,7 @@ def finditer( raw_iter = None if raw_iter is not None: for raw in raw_iter: - match_obj = Match(self, raw, subject, origin_pos, resolved_end) - self._update_group_hint(match_obj) - yield match_obj + yield self._wrap_match(raw, subject, origin_pos, resolved_end) return search_end = resolved_end if endpos is not None else -1 @@ -312,8 +361,7 @@ def finditer( if raw is None: break - match_obj = Match(self, raw, subject, origin_pos, resolved_end) - self._update_group_hint(match_obj) + match_obj = self._wrap_match(raw, subject, origin_pos, resolved_end) yield match_obj start, end = match_obj.span() @@ -334,6 +382,25 @@ def findall( endpos: int | None = None, options: int = 0, ) -> List[Any]: + if type(subject) is memoryview: + subject = subject.tobytes() + backend_iter = getattr(self._pattern, "finditer", None) + if backend_iter is not None: + compiled_end = -1 if endpos is None else resolve_endpos(subject, endpos) + try: + raw_iter = backend_iter(subject, pos=pos, endpos=compiled_end, options=options) + except TypeError: + raw_iter = None + if raw_iter is not None: + results: List[Any] = [] + for raw in raw_iter: + groups = raw.groups() + if groups: + results.append(groups[0] if len(groups) == 1 else groups) + else: + results.append(raw.group(0)) + return results + results: List[Any] = [] for match_obj in self.finditer(subject, pos=pos, endpos=endpos, options=options): groups = match_obj.groups() diff --git a/pcre/re_compat.py b/pcre/re_compat.py index c36653b..2a7ece9 100644 --- a/pcre/re_compat.py +++ b/pcre/re_compat.py @@ -151,6 +151,24 @@ def render_template(parsed: Any, match: "Match", *, is_bytes: bool, empty: Any) return join_parts(pieces, is_bytes=is_bytes) +def expand_match_template(match: Any, template: Any) -> Any: + is_bytes = is_bytes_like(match.string) + empty = b"" if is_bytes else "" + if is_bytes: + if not is_bytes_like(template): + raise TypeError("template must be bytes-like for bytes matches") + template = bytes(template) + else: + if not isinstance(template, str): + raise TypeError("template must be str for text matches") + + parsed = _parser.parse_template( + template, + TemplatePatternStub(match.re.groups, match.re.groupindex), + ) + return render_template(parsed, match, is_bytes=is_bytes, empty=empty) + + def maybe_infer_group_count(pattern_source: Any) -> int | None: normalised = pattern_source if isinstance(normalised, memoryview): @@ -270,21 +288,7 @@ def span(self, group: Any = 0) -> tuple[int, int]: return self._match.span(group) def expand(self, template: Any) -> Any: - is_bytes = is_bytes_like(self._string) - empty = b"" if is_bytes else "" - if is_bytes: - if not is_bytes_like(template): - raise TypeError("template must be bytes-like for bytes matches") - template = bytes(template) - else: - if not isinstance(template, str): - raise TypeError("template must be str for text matches") - - parsed = _parser.parse_template( - template, - TemplatePatternStub(self.re.groups, self.re.groupindex), - ) - return render_template(parsed, self, is_bytes=is_bytes, empty=empty) + return expand_match_template(self, template) @property def re(self) -> Any: diff --git a/pcre_ext/pcre2.c b/pcre_ext/pcre2.c index bd89b76..a93aa84 100644 --- a/pcre_ext/pcre2.c +++ b/pcre_ext/pcre2.c @@ -27,9 +27,11 @@ resolve_pcre2_prerelease(void) return raw; } +/* Process-wide library metadata cached once during module initialization. */ static char pcre2_library_version[64] = "unknown"; static ATOMIC_VAR(int) pcre2_version_initialized = 0; #if defined(PCRE2_USE_OFFSET_LIMIT) +/* -1 unknown, 0 unsupported, 1 supported by the loaded PCRE2 runtime. */ static ATOMIC_VAR(int) offset_limit_support = ATOMIC_VAR_INIT(-1); #endif @@ -162,6 +164,7 @@ static void Match_dealloc(MatchObject *self) { Py_XDECREF(self->pattern); + Py_XDECREF(self->public_pattern); Py_XDECREF(self->subject); Py_XDECREF(self->utf8_owner); pcre_free(self->ovector); @@ -176,6 +179,64 @@ Match_repr(MatchObject *self) return PyUnicode_FromFormat("", start, end, self->pattern->pattern); } +static inline PyObject * +match_public_pattern(MatchObject *self) +{ + if (self->public_pattern != NULL) { + return self->public_pattern; + } + return (PyObject *)self->pattern; +} + +static int +match_resolve_span(MatchObject *self, + Py_ssize_t index, + Py_ssize_t *start_out, + Py_ssize_t *end_out, + int allow_missing) +{ + /* + * Convert the raw byte-oriented ovector entry into the user-visible span. + * For bytes subjects the PCRE2 offsets are already correct. For text + * subjects we translate byte offsets back to Python code-point indexes. + */ + if (index < 0 || (uint32_t)index >= self->ovec_count) { + PyErr_SetString(PyExc_IndexError, "group index out of range"); + return -1; + } + + Py_ssize_t start = self->ovector[index * 2]; + Py_ssize_t end = self->ovector[index * 2 + 1]; + if (start < 0 || end < 0) { + if (allow_missing) { + *start_out = -1; + *end_out = -1; + return 0; + } + return 1; + } + + if (self->subject_is_bytes) { + *start_out = start; + *end_out = end; + return 0; + } + + const char *data = self->utf8_data; + Py_ssize_t start_index = utf8_offset_to_index(data, start); + if (start_index < 0 && PyErr_Occurred()) { + return -1; + } + Py_ssize_t end_index = utf8_offset_to_index(data, end); + if (end_index < 0 && PyErr_Occurred()) { + return -1; + } + + *start_out = start_index; + *end_out = end_index; + return 0; +} + static int resolve_group_key(MatchObject *self, PyObject *key, Py_ssize_t *index) { @@ -309,28 +370,16 @@ Match_span(MatchObject *self, PyObject *args) if (resolve_group_key(self, key, &index) < 0) { return NULL; } - if (index < 0 || (uint32_t)index >= self->ovec_count) { - PyErr_SetString(PyExc_IndexError, "group index out of range"); + Py_ssize_t start = 0; + Py_ssize_t end = 0; + int rc = match_resolve_span(self, index, &start, &end, 0); + if (rc < 0) { return NULL; } - Py_ssize_t start = self->ovector[index * 2]; - Py_ssize_t end = self->ovector[index * 2 + 1]; - if (start < 0 || end < 0) { + if (rc > 0) { Py_RETURN_NONE; } - if (self->subject_is_bytes) { - return Py_BuildValue("(nn)", start, end); - } - const char *data = self->utf8_data; - Py_ssize_t start_index = utf8_offset_to_index(data, start); - if (start_index < 0 && PyErr_Occurred()) { - return NULL; - } - Py_ssize_t end_index = utf8_offset_to_index(data, end); - if (end_index < 0 && PyErr_Occurred()) { - return NULL; - } - return Py_BuildValue("(nn)", start_index, end_index); + return Py_BuildValue("(nn)", start, end); } static PyObject * @@ -415,6 +464,139 @@ Match_get_string(MatchObject *self, void *closure) return self->subject; } +static PyObject * +Match_get_re(MatchObject *self, void *closure) +{ + PyObject *pattern = match_public_pattern(self); + Py_INCREF(pattern); + return pattern; +} + +static PyObject * +Match_get_pos(MatchObject *self, void *closure) +{ + return PyLong_FromSsize_t(self->public_pos); +} + +static PyObject * +Match_get_endpos(MatchObject *self, void *closure) +{ + return PyLong_FromSsize_t(self->public_endpos); +} + +static PyObject * +Match_get_lastindex(MatchObject *self, void *closure) +{ + if (self->ovec_count <= 1) { + Py_RETURN_NONE; + } + + for (Py_ssize_t index = (Py_ssize_t)self->ovec_count - 1; index >= 1; --index) { + Py_ssize_t start = self->ovector[index * 2]; + Py_ssize_t end = self->ovector[index * 2 + 1]; + if (start >= 0 && end >= 0) { + return PyLong_FromSsize_t(index); + } + } + + Py_RETURN_NONE; +} + +static PyObject * +Match_get_lastgroup(MatchObject *self, void *closure) +{ + PyObject *lastindex_obj = Match_get_lastindex(self, closure); + if (lastindex_obj == NULL || lastindex_obj == Py_None) { + return lastindex_obj; + } + + PyObject *key = NULL; + PyObject *value = NULL; + Py_ssize_t pos = 0; + while (PyDict_Next(self->pattern->groupindex, &pos, &key, &value)) { + int matches = PyObject_RichCompareBool(value, lastindex_obj, Py_EQ); + if (matches < 0) { + Py_DECREF(lastindex_obj); + return NULL; + } + if (matches) { + Py_INCREF(key); + Py_DECREF(lastindex_obj); + return key; + } + } + + Py_DECREF(lastindex_obj); + Py_RETURN_NONE; +} + +static PyObject * +Match_get_regs(MatchObject *self, void *closure) +{ + PyObject *result = PyTuple_New(self->ovec_count); + if (result == NULL) { + return NULL; + } + + for (uint32_t index = 0; index < self->ovec_count; ++index) { + Py_ssize_t start = 0; + Py_ssize_t end = 0; + if (match_resolve_span(self, (Py_ssize_t)index, &start, &end, 1) < 0) { + Py_DECREF(result); + return NULL; + } + PyObject *span = Py_BuildValue("(nn)", start, end); + if (span == NULL) { + Py_DECREF(result); + return NULL; + } + PyTuple_SET_ITEM(result, index, span); + } + + return result; +} + +static PyObject * +Match_expand(MatchObject *self, PyObject *template_obj) +{ + /* Delegate template parsing to the Python compatibility helper. */ + PyObject *module = PyImport_ImportModule("pcre.re_compat"); + if (module == NULL) { + return NULL; + } + + PyObject *helper = PyObject_GetAttrString(module, "expand_match_template"); + Py_DECREF(module); + if (helper == NULL) { + return NULL; + } + + PyObject *result = PyObject_CallFunctionObjArgs( + helper, + (PyObject *)self, + template_obj, + NULL + ); + Py_DECREF(helper); + return result; +} + +static int +match_set_public_pattern(MatchObject *self, PyObject *public_pattern) +{ + /* The high-level wrapper reuses this C object and swaps in its owner here. */ + if (public_pattern == NULL) { + Py_XDECREF(self->public_pattern); + self->public_pattern = NULL; + return 0; + } + + Py_INCREF(public_pattern); + Py_XDECREF(self->public_pattern); + self->public_pattern = public_pattern; + return 0; +} + static PyMethodDef Match_methods[] = { {"group", (PyCFunction)Match_group, METH_VARARGS, PyDoc_STR("Return one or more capture groups.")}, {"groups", (PyCFunction)Match_groups, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Return all capture groups as a tuple." )}, @@ -422,11 +604,18 @@ static PyMethodDef Match_methods[] = { {"span", (PyCFunction)Match_span, METH_VARARGS, PyDoc_STR("Return the (start, end) span for a group." )}, {"start", (PyCFunction)Match_start, METH_VARARGS, PyDoc_STR("Return the start index for a group." )}, {"end", (PyCFunction)Match_end, METH_VARARGS, PyDoc_STR("Return the end index for a group." )}, + {"expand", (PyCFunction)Match_expand, METH_O, PyDoc_STR("Apply a replacement template to the match." )}, {NULL, NULL, 0, NULL}, }; static PyGetSetDef Match_getset[] = { + {"re", (getter)Match_get_re, NULL, PyDoc_STR("Pattern object used for the match."), NULL}, {"string", (getter)Match_get_string, NULL, PyDoc_STR("Original subject."), NULL}, + {"pos", (getter)Match_get_pos, NULL, PyDoc_STR("Original search start position."), NULL}, + {"endpos", (getter)Match_get_endpos, NULL, PyDoc_STR("Original search end position."), NULL}, + {"lastindex", (getter)Match_get_lastindex, NULL, PyDoc_STR("Index of the last matched capturing group."), NULL}, + {"lastgroup", (getter)Match_get_lastgroup, NULL, PyDoc_STR("Name of the last matched capturing group."), NULL}, + {"regs", (getter)Match_get_regs, NULL, PyDoc_STR("Tuple of span pairs for the whole match and each group."), NULL}, {NULL, NULL, NULL, NULL, NULL}, }; @@ -450,6 +639,7 @@ typedef struct { int subject_is_bytes; Py_ssize_t subject_length_bytes; Py_ssize_t logical_length; + Py_ssize_t origin_pos; Py_ssize_t current_pos; Py_ssize_t current_byte; Py_ssize_t resolved_end; @@ -468,11 +658,18 @@ typedef struct { int utf8_is_ascii; } FindIterObject; +/* + * Iteration over Unicode subjects frequently needs byte<->code-point + * conversions. These caches keep the common forward-only scan cheap rather than + * rescanning the full subject for every match. + */ static MatchObject *create_match_object(PatternObject *pattern, PyObject *subject_obj, PyObject *utf8_owner, const char *utf8_data, Py_ssize_t utf8_length, + Py_ssize_t pos, + Py_ssize_t endpos, uint32_t ovec_count, PCRE2_SIZE *ovector); @@ -480,6 +677,7 @@ static MatchObject *create_match_object(PatternObject *pattern, static inline Py_ssize_t utf8_index_to_offset_fast(const char *data, Py_ssize_t data_len, Py_ssize_t index) { + /* Walk UTF-8 once, collapsing ASCII runs so index->byte conversion stays cheap. */ if (index <= 0) { return 0; } @@ -524,6 +722,7 @@ utf8_index_to_offset_fast(const char *data, Py_ssize_t data_len, Py_ssize_t inde static Py_ssize_t finditer_byte_to_index(FindIterObject *self, Py_ssize_t target_byte) { + /* Convert a byte offset back to a code-point index using the forward cache. */ if (target_byte < 0) { self->byte_to_index_cached_index = 0; self->byte_to_index_cached_byte = 0; @@ -595,6 +794,7 @@ finditer_byte_to_index(FindIterObject *self, Py_ssize_t target_byte) static Py_ssize_t finditer_index_to_byte(FindIterObject *self, Py_ssize_t target_index) { + /* Convert a code-point index to a byte offset using the forward cache. */ if (target_index < 0) { self->index_to_byte_cached_index = 0; self->index_to_byte_cached_byte = 0; @@ -820,6 +1020,8 @@ FindIter_iternext(FindIterObject *self) self->utf8_owner, self->utf8_data, self->subject_length_bytes, + self->origin_pos, + self->resolved_end, (uint32_t)expected_pairs, ovector); if (match == NULL) { @@ -888,9 +1090,15 @@ create_match_object(PatternObject *pattern, PyObject *utf8_owner, const char *utf8_data, Py_ssize_t utf8_length, + Py_ssize_t pos, + Py_ssize_t endpos, uint32_t ovec_count, PCRE2_SIZE *ovector) { + /* + * Materialize a standalone match snapshot. The ovector is copied because + * PCRE2 reuses match-data buffers from caches across calls and threads. + */ MatchObject *match = PyObject_New(MatchObject, &MatchType); if (match == NULL) { return NULL; @@ -910,6 +1118,7 @@ create_match_object(PatternObject *pattern, Py_INCREF(pattern); match->pattern = pattern; + match->public_pattern = NULL; Py_INCREF(subject_obj); match->subject = subject_obj; @@ -918,6 +1127,8 @@ create_match_object(PatternObject *pattern, match->utf8_owner = utf8_owner; match->utf8_data = utf8_data; match->utf8_length = utf8_length; + match->public_pos = pos; + match->public_endpos = endpos; match->subject_is_bytes = PyBytes_Check(subject_obj); return match; @@ -943,6 +1154,7 @@ Pattern_create_finditer(PatternObject *pattern, iter->subject_is_bytes = 0; iter->subject_length_bytes = 0; iter->logical_length = 0; + iter->origin_pos = 0; iter->current_pos = 0; iter->current_byte = 0; iter->resolved_end = 0; @@ -1045,6 +1257,7 @@ Pattern_create_finditer(PatternObject *pattern, } iter->current_pos = pos; + iter->origin_pos = pos; iter->current_byte = current_byte; iter->resolved_end = resolved_end; iter->resolved_end_byte = resolved_end_byte; @@ -1569,6 +1782,8 @@ Pattern_execute(PatternObject *self, PyObject *subject_obj, Py_ssize_t pos, utf8_owner, buffer, subject_length_bytes, + pos, + adjusted_endpos >= 0 ? adjusted_endpos : logical_length, (uint32_t)expected_pairs, ovector); @@ -1968,6 +2183,33 @@ module_configure(PyObject *Py_UNUSED(module), PyObject *args, PyObject *kwargs) Py_RETURN_FALSE; } +static PyObject * +module_attach_match(PyObject *Py_UNUSED(module), PyObject *args) +{ + /* + * The public Python wrapper keeps using the same C MatchObject and only + * stamps in the high-level owner here. That avoids a second Python object + * allocation on every successful search/match call. + */ + PyObject *match_obj = NULL; + PyObject *pattern_obj = NULL; + if (!PyArg_ParseTuple(args, "OO", &match_obj, &pattern_obj)) { + return NULL; + } + + if (!PyObject_TypeCheck(match_obj, &MatchType)) { + PyErr_SetString(PyExc_TypeError, "expected pcre.Match instance"); + return NULL; + } + + if (match_set_public_pattern((MatchObject *)match_obj, pattern_obj) < 0) { + return NULL; + } + + Py_INCREF(match_obj); + return match_obj; +} + static PyObject *module_memory_allocator(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)); static PyObject *module_get_pcre2_version(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)); static void initialize_pcre2_version(void); @@ -1979,6 +2221,7 @@ static PyMethodDef module_methods[] = { {"search", (PyCFunction)module_search, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Search a string for a pattern." )}, {"fullmatch", (PyCFunction)module_fullmatch, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Match a pattern against the entire string." )}, {"configure", (PyCFunction)module_configure, METH_VARARGS | METH_KEYWORDS, PyDoc_STR("Get or set module-wide defaults (currently only 'jit')." )}, + {"_attach_match", (PyCFunction)module_attach_match, METH_VARARGS, PyDoc_STR("Attach a public pattern owner to a low-level match object." )}, {"get_match_data_cache_size", (PyCFunction)module_get_match_data_cache_size, METH_NOARGS, PyDoc_STR("Return the capacity of the reusable match-data cache." )}, {"set_match_data_cache_size", (PyCFunction)module_set_match_data_cache_size, METH_VARARGS, PyDoc_STR("Set the capacity of the reusable match-data cache." )}, {"clear_match_data_cache", (PyCFunction)module_clear_match_data_cache, METH_NOARGS, PyDoc_STR("Release all cached PCRE2 match-data buffers." )}, @@ -2151,7 +2394,7 @@ PyInit_pcre_ext_c(void) goto error_cache; } - if (PyModule_AddStringConstant(module, "__version__", "0.1.0") < 0) { + if (PyModule_AddStringConstant(module, "__version__", "0.3.0") < 0) { goto error_cache; } diff --git a/pcre_ext/pcre2_module.h b/pcre_ext/pcre2_module.h index ee5fb19..b4996a1 100644 --- a/pcre_ext/pcre2_module.h +++ b/pcre_ext/pcre2_module.h @@ -59,15 +59,25 @@ typedef struct { int first_literal_caseless; } PatternObject; +/* + * MatchObject is the single match carrier used by both the low-level C API and + * the higher-level Python wrapper. `pattern` always points at the compiled C + * pattern that produced the match, while `public_pattern` optionally points at + * the high-level `pcre.Pattern` wrapper so public `match.re` parity can be + * preserved without allocating a second Python-side wrapper object. + */ typedef struct { PyObject_HEAD PatternObject *pattern; + PyObject *public_pattern; PyObject *subject; PyObject *utf8_owner; const char *utf8_data; Py_ssize_t utf8_length; Py_ssize_t *ovector; uint32_t ovec_count; + Py_ssize_t public_pos; + Py_ssize_t public_endpos; int subject_is_bytes; } MatchObject; diff --git a/pyproject.toml b/pyproject.toml index c313d8a..fd8ef89 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,7 +9,7 @@ build-backend = "setuptools.build_meta" [project] name = "PyPcre" -version = "0.2.15" +version = "0.3.0" description = "Modern, GIL-friendly, Fast Python bindings for PCRE2 with auto caching and JIT of compiled patterns." readme = "README.md" requires-python = ">=3.9"