gen1-to-gen2: v1 block spans, and the rest of the v1 spellings - #3855
Merged
Conversation
aleksisch
force-pushed
the
achurkin/gen1-to-gen2-das
branch
6 times, most recently
from
August 25, 2026 12:08
0decdfc to
a9470df
Compare
aleksisch
force-pushed
the
achurkin/gen1-to-gen2-das
branch
3 times, most recently
from
August 25, 2026 16:11
0bd5204 to
4157bf5
Compare
It carries a fork of the v1 bison grammar and re-parses the file after every rewrite, which is why that fork has to accept mixed gen1+gen2 input. A converter that reads the tree daslang itself parses needs neither, and lands two commits later under the same name. Everything the binary reached goes with it: the vendored grammar, the `gen1_to_gen2` target and its install, `daslang --run-fmt` (its only in-binary caller), the CodeQL exclusion for the generated parser, the packaging rows that shipped the exe, and the release-phase checks that ran it. The formatter proper is `utils/das-fmt/dasfmt.das` and is untouched - the two have never been the same tool. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`parse_file` parses a file and everything it requires and stops there - no infer, no optimization, no simulate. The tree still mirrors the source one-to-one, which is what a source-rewriting tool needs: infer lowers make-syntax, resolves generics, folds constants and inserts generated nodes, none of which map back to a source span, and it rejects a file that does not type-check - which a converter still has to be able to read. Two v1 span fixes, both about where an indent block ends. It ran one column past the end of every line it closed on, because the dedent took the newline token's END. And a block closed by end of file in a file whose last line carries no newline ended at the line BEFORE its last statement, because `last_token_end` is maintained by the newline rule, which never runs for that line. The blocks and the `with` a class-method body is wrapped in are not in the source; they are flagged generated so a span consumer skips them rather than reporting the method name as a block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Source is immutable bytes. The file is parsed once with infer off, every rewrite
is recorded as an edit in ORIGINAL byte offsets, and the edits are applied in a
single forward walk. Nesting needs no special handling: a rule only ever edits
the delimiters of its own construct, which lie between its children, so a
parent's edits and its children's are disjoint ranges and one pass composes them.
That is what removes the previous tool's re-parse loop, and with it the reason
its grammar fork had to accept mixed gen1+gen2 input.
Comment preservation is structural rather than per-rule discipline: untouched
bytes are copied verbatim, a deletion whose range holds a comment or a string is
refused, gap-collapse rules require a blank gap, a line-edge insertion steps past
a block comment, and every conversion is gated on the input's ordered comment
sequence equalling the output's.
Converted families: make-syntax in every v1 shape, the `[{ }]` / `{{ }}`
wrappers and both comprehension spellings, indentation into braces, conditions
and for-heads into parentheses, declaration bodies, piped literals, argument
annotations, `typeinfo(trait x)`, tuple destructuring, typed make-arrays,
enumeration and bitfield values, call-style casts, type-function calls, keyword
call macros, multi-value return, `static_if` / `static_elif`, global `let`/`var`
blocks, and a continued initializer - which gen2 would otherwise read as a
separate unary statement and delete silently.
Measured by migrating the historical v1 corpus in place (tests/ at 027cbc2^,
`.inc` fragments included) and compiling the result: 71 of the 72 files that
compile as v1 today convert to gen2 that compiles, none loses a comment. The
exception is the spoof-template file, which is skipped by design - its template
body is text the parser never sees as code, so no rule can place its braces.
`utils/internal/lineinfo-audit` gains `--parse-only` and `--gen1`, which is how
the converter's anchors are validated: that tree comes entirely from the parser,
so a span violation in it is a real parser bug. A borrowed position in the
compiled tree is what lowering does, and is not something to flag away - marking
such nodes `generated` quiets the audit at the cost of every other consumer of
that flag, lint included.
`run_utils_tests` collects the directory, so CI runs the converter's 75 dastest
tests; the MCP `convert_to_gen2` tool runs the script through the daslang it was
launched with, which is why the `.das` files are installed alongside the other
utils.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aleksisch
force-pushed
the
achurkin/gen1-to-gen2-das
branch
from
August 25, 2026 17:52
4157bf5 to
180a047
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A v1→gen2 converter written in daslang, replacing the C++ one. Two commits: the converter,
then the removal of the tool it replaces.
Measured by migrating the historical v1 corpus in place (the 104
.dasfiles undertests/at027cbc2d8^,.incfragments included) and compiling the result as gen2. Only72 of those 104 compile as v1 with today's compiler — the rest are deliberately-failing
failed_*fixtures and files the current compiler rejects for reasons that predate anyconversion.
71 of those 72 convert to gen2 that compiles, and none loses a comment. The exception is
the spoof-template file, which is skipped by design: its template body is text the parser
never sees as code, so no rule can place its braces.
Per-file conversion cannot be the measurement — a file that
requires a sibling orincludes a fragment only compiles once the whole tree has moved.gen1-to-gen2: a v1 -> gen2 converter in daslangSource is immutable bytes. The file is parsed ONCE with infer off (
parse_file, added here),every rewrite is recorded as an edit in original byte offsets, and the edits are applied in
a single forward walk. Nesting needs no special handling: a rule only ever edits the delimiters
of its own construct, which lie between its children, so a parent's edits and its children's
are disjoint ranges and one pass composes them. That is what removes the C++ tool's re-parse
loop — and with it the reason its grammar fork had to accept mixed gen1+gen2 input.
Comment preservation is structural rather than per-rule discipline: untouched bytes are copied
verbatim, a deletion whose range holds a comment or a string is refused, gap-collapse rules
require a blank gap, a line-edge insertion steps past a block comment, and every conversion is
gated on the input's ordered comment sequence equalling the output's.
Two parser fixes it depends on, both about where a v1 block ends: spans that ran one column
past the end of every line they closed on, and a block closed by EOF in a file whose last line
carries no newline — which ended at the line before its last statement.
utils/lineinfo-auditgains--parse-onlyand--gen1, which is how the converter's anchorsare validated: that tree comes entirely from the parser, so a span violation in it is a real
parser bug. A borrowed position in the compiled tree is what lowering does, and is not
something to flag away — marking such nodes
generatedquiets the audit at the cost of everyother consumer of that flag, lint included.
dasFormatter: the daslang converter replaces the C++ oneThe sources, the vendored v1 bison grammar, the
gen1_to_gen2target and thedaslang -run-fmtentry point all go, and the daslang converter takes the directory.run_utils_testscollects the directory rather than the C++ binary's own
--tests, so what CI runs is theconverter's 67 dastest tests. The MCP
convert_to_gen2tool runs the script through thedaslang it was launched with, which is why the
.dasfiles are now installed alongside theother utils.
daslang --run-fmtwas the only in-binary caller of the C++ formatter. The formatter proper isutils/das-fmt/dasfmt.das, unaffected.Tests
Every family and every bug fix has a fixture case and an assertion, and each was checked with a
negative control — disable the rule, watch that assertion (and usually the fixture's
compiles_as_gen2check) fail, restore. Shapes the formatter rewrites (for/*i*/i,take_t<int>(1, 2)) are written to temporary files rather than kept in a checked-in fixture.Local:
utils/dasFormatter67/67 ·tests/ast19/19 · lint and formatter verify clean onevery changed
.dasfile.Before review
origin/masterand needs a rebase. I did notrebase it unattended — the working tree is shared and the generated parser files would
conflict.
--no-verify.tests/run was 9573/9681 with every failure traced to unbuilt modules(
dasHV, strudel,dasPUGIXML,json) — the branch base is missing master'smodules/dasPUGIXML/.das_moduleregistration oflinq_fold_xml, which breaks AOT stubgeneration locally. The rebase should clear that.
🤖 Generated with Claude Code