Add C++ parsergen implementation with tests and CNI layer

[mikecovlee]

No description provided.

[Copilot]

Pull request overview

This PR introduces a C++ implementation of ParserGen (native libparsergen + CovScript CNI extension parsergen_cxx.cse) and updates the CovScript side to a unified API surface so tests can run against either implementation.

Changes:

• Add a full C++ ParserGen library (lexer/parser/generator + unicode support) and a CNI layer exporting the unified API to CovScript.
• Update CovScript ParserGen and ECS grammar packaging to support unified constructors/helpers (make_grammar_from, make_lex_error, add_language, etc.) and implementation selection via PARSERGEN_IMPL.
• Add C++ tests/benchmarks and extend GitHub Actions CI to build/test the C++ implementation and run drop-in replacement tests.

Reviewed changes

Copilot reviewed 45 out of 46 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/run_parser.ecs	Switch ECS language registration to unified add_language.
tests/ecs_repl.ecs	Precompile lexical rules once and reuse in the REPL lexer.
tests/ecs_parser.csp	Make ParserGen implementation selectable and split lexical patterns vs compiled lexical rules.
parsergen.csp	Add unified API helpers/methods and state resets; add per-language encoding support.
parsergen_debug.csp	Mirror unified API helpers/methods and state resets in debug implementation.
CXX_API.md	Document the C++ native API of libparsergen.
csbuild/parsergen_cxx.json	Add csbuild extension manifest for parsergen_cxx.
csbuild/make.sh	Add Unix build script to produce parsergen_cxx.cse.
csbuild/make.bat	Add Windows build script to produce parsergen_cxx.cse.
csbuild/format.sh	Add Unix formatting script for C++ sources.
csbuild/format.bat	Add Windows formatting script for C++ sources.
cpp/test/test_v1_old.csc	Add legacy-style test (currently needs lex compilation fix).
cpp/test/test_typeid.csc	Add CNI typeid interoperability test.
cpp/test/test_partial_parser.csc	Add partial-parser EOF-hook test via CNI.
cpp/test/test_ecs_cpp.cpp	Add native C++ ECS parsing demo/test.
cpp/test/test_dropin_all.csc	Add “drop-in replace” compatibility test suite for CNI implementation.
cpp/test/test_cni_add_language.csc	Add CNI-focused unified add_language + lex_string tests.
cpp/test/test_add_language.csc	Add unified add_language usage example/test for CovScript implementation.
cpp/test/main.cpp	Add native C++ unit tests for core behaviors (JSON/TINY/empty/recovery).
cpp/test/ecs_parser.hpp	Add C++ ECS grammar definition used by native C++ tests/benchmarks.
cpp/test/bench_native.csc	Add benchmark script using unified add_language.
cpp/test/bench_lex_parse.csc	Add benchmark separating lex vs parse on CovScript implementation.
cpp/test/bench_lex_parse_cpp.cpp	Add native C++ lex/parse benchmark.
cpp/test/ast_dump_native.csc	Add AST dumper script using unified generator API.
cpp/src/unicode_lexer.cpp	Implement C++ unicode lexer using PCRE2-32.
cpp/src/parser.cpp	Implement C++ parser, partial parser, and recovering parser.
cpp/src/lexer.cpp	Implement C++ lexer and regex helpers.
cpp/src/generator.cpp	Implement C++ generator (file/string/stream parsing and error formatting).
cpp/include/parsergen/unicode.hpp	Add C++ codecvt charset implementations (ascii/utf8/gbk).
cpp/include/parsergen/token.hpp	Add C++ token and lex_error types.
cpp/include/parsergen/syntax.hpp	Add C++ syntax factory API.
cpp/include/parsergen/parsergen.hpp	Add umbrella header for the C++ API.
cpp/include/parsergen/parser.hpp	Add C++ parser class declarations.
cpp/include/parsergen/lexer.hpp	Add C++ lexer/unicode lexer declarations and grammar type.
cpp/include/parsergen/generator.hpp	Add C++ generator declarations.
cpp/include/parsergen/bootset.hpp	Add C++ bootset/prediction set type.
cpp/include/parsergen/ast.hpp	Add C++ AST structures and printer.
cpp/cni/parsergen_cni.cpp	Add CovScript CNI bindings for the unified API (parsergen_cxx).
cpp/CMakeLists.txt	Add CMake build for libparsergen, tests, and optional CNI build.
API.md	Add unified API reference across CovScript and C++ CNI implementations.
.gitmodules	Add submodules for covscript-regex and utfcpp.
.gitignore	Ignore C++ build artifacts and local test data.
.github/workflows/ci.yml	Extend CI to build/test C++ implementation and run drop-in tests across OSes.
.editorconfig	Add EditorConfig to standardize formatting across file types.

Suppressed comments (2)

parsergen.csp:249

• unicode_lexer_type.run() later assigns wpos = pos and passes pos into error(...), but pos is [col, line, cursor]. This makes token/error positions contain an extra cursor element instead of [col, line].
  parsergen_debug.csp:288
• unicode_lexer_type.run() later does wpos = pos / error(..., pos) even though pos is [col, line, cursor]. This causes token/error positions to contain an extra element instead of [col, line].

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated no new comments.

Suppressed comments (3)

API.md:79

• add_grammar(lang, gram) is documented as equivalent to add_language(lang, "ascii", gram), but in the CovScript implementation add_language recompiles gram.lex via regex.build_optimize(...) while add_grammar simply stores the grammar as-is. This matters because existing grammars (e.g. tests/run_parser.ecs) commonly store compiled regex objects in gram.lex, which would not be valid input for add_language as currently implemented.

| `add_grammar(lang, gram)` | `add_language(lang, "ascii", gram)` | 保留 |

parsergen.csp:1141

• generator.add_language assumes gram.lex contains string patterns and unconditionally calls regex.build_optimize(it.second). However, other in-repo grammars store compiled regex objects in gram.lex (see tests/run_parser.ecs), and API.md currently implies add_grammar can be replaced by add_language. Consider accepting either precompiled regex objects or pattern strings here to avoid runtime type errors and make the unified API less surprising.
  parsergen_debug.csp:1374
• Same as parsergen.csp: generator.add_language always recompiles gram.lex via regex.build_optimize(...), which will fail if callers pass a grammar whose lex entries are already compiled regex objects (common in existing scripts). Supporting both compiled-regex and string-pattern inputs would reduce breakage and better match the unified API docs.