C++: a 16-character raw-string delimiter (the standard maximum) fails to parse — capacity guard is checked before the '(' terminator

[inth3shadows]

Summary

A raw-string delimiter of exactly 16 characters — the maximum [lex.string] allows — fails to parse. The whole translation unit after it collapses into ERROR nodes. 15 characters is fine, so the one length that is legal-but-rejected is exactly the spec's limit.

The cause is a check ordering in scan_raw_string_delimiter, and the fix is a one-line reorder. I have built and tested it; details and a differential sweep below.

Repro

// compiles cleanly with `g++ -std=c++17 -fsyntax-only`
const char* kTemplate = R"FFFFFFFFFFFFFFFF(
struct Ignored { int v; };
)FFFFFFFFFFFFFFFF";

int after_the_raw_string(int x) {
  return x + 1;
}

FFFFFFFFFFFFFFFF is 16 chars.

Parsed with the C API against this repo at HEAD (src/parser.c + src/scanner.c, linked with the tree-sitter runtime — no editor or downstream tool involved):

delim=14  hasError=false  function_definition=1  ERROR=0
delim=15  hasError=false  function_definition=1  ERROR=0
delim=16  hasError=true   function_definition=0  ERROR=3     <-- legal, rejected
delim=17  hasError=true   function_definition=0  ERROR=3
delim=18  hasError=true   function_definition=0  ERROR=3

after_the_raw_string disappears entirely at 16.

Cause

src/scanner.c, opening-delimiter loop:

for (;;) {
    if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
        iswspace(lexer->lookahead)) {
        return false;                              // (1) capacity guard
    }
    if (lexer->lookahead == '(') {
        return scanner->delimiter_length > 0;      // (2) legal terminator
    }
    scanner->delimiter[scanner->delimiter_length++] = lexer->lookahead;
    advance(lexer);
}

With a 16-char delimiter, the 16 d-chars are consumed and delimiter_length == 16. On the next iteration lookahead is ( — the legal terminator, check (2) — but check (1) runs first, sees the buffer is full, and returns false. The delimiter is never accepted even though nothing is wrong with it: the buffer is full, not overflowing.

15 works only because the loop still has a free slot when it reaches (.

Fix

Test the terminator before the capacity guard. A full buffer followed by ( is a complete, legal delimiter; a full buffer followed by another d-char is the genuine overflow.

     for (;;) {
-        if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
-            iswspace(lexer->lookahead)) {
-            return false;
-        }
         if (lexer->lookahead == '(') {
             // Rather than create a token for an empty delimiter, we fail and
             // let the grammar fall back to a delimiter-less rule.
             return scanner->delimiter_length > 0;
         }
+        if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
+            iswspace(lexer->lookahead)) {
+            return false;
+        }
         scanner->delimiter[scanner->delimiter_length++] = lexer->lookahead;
         advance(lexer);
     }

No bounds change: delimiter[] is still only written when delimiter_length < MAX_DELIMITER_LENGTH, so the array cannot overflow.

Verification

Built stock and patched scanners from the same sources and swept delimiter lengths 0-18, comparing function_definition counts:

length	stock	patched
0-15	1	1	identical
16	0	1	fixed
17, 18	0	0	identical — still rejected

The patch changes behaviour at exactly one length, the one that is legal today and rejected. 17+ stays rejected, so the spec limit is preserved, and length 0 (the empty-delimiter fallback the comment describes) is unchanged.

What I did not do: I could not run the corpus suite — npx tree-sitter-cli did not finish downloading in my environment — so this is verified by differential parsing rather than by tree-sitter test. Worth running before merging.

Happy to open a PR with the patch and a corpus case if useful.

Why it matters downstream

Files most likely to carry long descriptive delimiters — code generators, template/scaffold files, embedded shader or SQL blobs — are exactly where FILE_TEMPLATE_V1, SHADER_SOURCE_V2 or CMAKE_TEMPLATE_1 read as natural, and all are 16. When it happens the file yields no symbols at all, with no diagnostic.

Distinct from #245 (error-recovery synthesising a raw_string_delimiter, fixed in 3d8d510) — nothing here is in error recovery; the input is well-formed C++.

Environment: this repo at HEAD, tree-sitter runtime from tree-sitter/tree-sitter HEAD, gcc 13.3.0, Ubuntu 24.04.

[inth3shadows]

Closing the gap I flagged above: the CLI did eventually install, so I ran the corpus suite on both scanners from the same checkout.

stock    Total parses: 198; successful parses: 198; failed parses: 0; success 100.00%
patched  Total parses: 198; successful parses: 198; failed parses: 0; success 100.00%

Identical — no regression from the reorder.

Both runs also emit Failed to read query file node_modules/tree-sitter-c/queries/highlights.scm during the syntax-highlighting step. That is a missing dev dependency in my shallow clone, not a test failure, and it appears on the stock scanner too, so it is unrelated to this change.

So the patch now has: a native differential sweep (lengths 0-18, behaviour differs at exactly one length — 16), and tree-sitter test parity against stock. Still happy to send it as a PR with a corpus case for the 16-char delimiter if you would like one.