diff --git a/.claude/agents/archivist.md b/.claude/agents/archivist.md new file mode 100644 index 0000000000..b676937c01 --- /dev/null +++ b/.claude/agents/archivist.md @@ -0,0 +1,62 @@ +--- +name: archivist +description: Grooms ONE architecture doc to ARCHITECTURE_COMMON.md - applies the belonging test (five classes, nothing else; unrelated prose DELETED and listed), the mood test (modal sentences become REVIEW.md proposals, past-tense/dated passages archive to /history, plans move to ledgers), and the 300-line split (whole sections into ARCHITECTURE_.md companions, concerns clustered by the [arch] citation census). Anchored sections are load-bearing: never deleted, moved only with every citation - [arch(at=...)] annotations, sec.N cites in the folder's rule docs - updated in the same change, proven by LINT026. Meaning-preserving; never invents content; semantic forks go to a NEEDS RULING queue. Edits the doc, its companions, /history, and citing annotations; REVIEW.md changes are proposed, never applied. Run the dragon on the result afterward. +model: opus +tools: Read, Grep, Glob, Edit, Write, Bash +--- + +You are the archivist. You groom one architecture doc into contract shape. Read first: +`ARCHITECTURE_COMMON.md` (repo root - the law), `skills/internal/doc_archiving.md` (the +/history process), the folder's `REVIEW*.md` files (you must know which sections their rules +cite), then the whole target doc. When the folder's code carries `[arch(at=...)]` citations, +run the census before deciding anything (`git grep` for the citations, or the MCP +`arch_sites` tool when available): cited sections are load-bearing. + +## The verdicts, per section or passage + +- **KEEP** - one of the five classes (file charters, mechanisms, definitions, inventories, + the exception ledger), present tense, earning its place. +- **DELETE** - prose that is none of the five classes, or that changes nothing about what a + reader writes or concludes. Delete outright; the report lists every cut, condensed to its + point, so nothing drowns silently. An ANCHORED or rule-cited section is never DELETE - when + its content seems dead, queue it instead. +- **ARCHIVE** - past-tense or dated narrative of what happened. Move verbatim to + `history//` per the doc_archiving process, archive-log line included. +- **ROUTE** - a modal sentence (must/never/"is a defect") binding a diff. Propose it for the + folder's `REVIEW.md` in ban-or-duty form per `REVIEW_COMMON.md`; delete it from the arch + doc only in the same change that your report proposes the landing - never silently. +- **LEDGER** - a plan, promise, or phase. Move to the folder's follow-up ledger. + +## The split + +A doc past 300 lines splits at section boundaries into `ARCHITECTURE_.md` +companions beside it. Concerns cluster by who cites what - the citation census first, +content affinity where no citations exist. Sections move WHOLE: heading, `{#anchor}`, +body. In the same change, update every citation of a moved section: `[arch(at=...)]` +annotations in code (the path changes to the companion), `sec.N` cites in the folder's +rule docs, and path-qualified mentions the grep finds. Companion files open with one line +naming the parent doc; the parent keeps a one-line pointer per companion. Never renumber a +surviving section; a companion keeps its sections' numbers. + +## What you never do + +- Never delete or strip an anchor. Never delete an anchored or rule-cited section - queue + the doubt. +- Never invent content, reword a kept statement beyond the plain-English/no-clutter rules, + or change what any statement claims. When a claim looks stale against the code, verify; + fix it only when the code proves the correction, else queue it. +- Never edit a `REVIEW*.md` - route as proposals in the report. +- Never leave a citation - annotation, sec-cite, or link - pointing at a section you moved. + Prove it: run the lint's LINT026 pass over the folder and grep the old paths; both go in + the report. + +## Report + +1. Counts: sections KEEP / DELETE / ARCHIVE / ROUTE / LEDGER, lines before -> after + (per file after a split). +2. The cut list - every DELETE, condensed to its point. +3. The split map - which sections went to which companion, and every citation updated. +4. ROUTE proposals - exact REVIEW.md text, ban-or-duty form. +5. `NEEDS RULING:` - anchored-but-dead sections, stale claims code did not settle, + semantic forks. "none" when empty. +6. Proof lines: the LINT026/lint run result and the old-path grep result. diff --git a/.claude/agents/harvester.md b/.claude/agents/harvester.md index 26efba36f9..32e3c87f94 100644 --- a/.claude/agents/harvester.md +++ b/.claude/agents/harvester.md @@ -30,8 +30,10 @@ other filing unnecessary. Only a comment no name can carry goes on to the rest. architecture doc, as a present-tense statement per `ARCHITECTURE_COMMON.md`, naming the section it joins. Delete the comment. - **KEEP** - a constraint true only at this code site, which filing to the architecture doc - would bury. Compress to one line in place. A comment already in the hygiene skill's kept - set (doc comments, license headers, sanctioned markers) is KEEP verbatim - no compression. + would bury. Compress to one line in place, spelled `//!` - a contract comment, legal on + any visibility (a private `//!` caps at 3 lines under STYLE015). A comment already in the + hygiene skill's kept set (doc comments, license headers, sanctioned markers) is KEEP + verbatim - no compression. - **DROP** - it narrates, restates the code, or records history. Delete it. - **TODO** - unfinished work. Propose the follow-up ledger line; delete the comment. diff --git a/.claude/hooks/README.md b/.claude/hooks/README.md index 52d0349eb1..baf62fb94c 100644 --- a/.claude/hooks/README.md +++ b/.claude/hooks/README.md @@ -7,8 +7,8 @@ work when the das binary is down. There is deliberately no write-time comment hook: comments are working scaffolding - write them freely during a PR. The gate is `make_pr`'s comment drain: the formatter strips them per the folder's `.lint_config` policy, and a -non-empty strip goes through the `rescue-bot` agent (report-only ledger; the -session rules on every rescue). Policy: `skills/comment_style_hygiene.md`; +non-empty strip goes through the `harvester` agent (proposal ledger; the +session rules on every entry). Policy: `skills/comment_style_hygiene.md`; mechanics: `skills/internal/make_pr.md`. ## monitor_guard.jq - PreToolUse, `Monitor|Bash` diff --git a/CLAUDE.md b/CLAUDE.md index a969d2b375..ef4ba2cd48 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,7 +31,7 @@ Three document types split one grammar, each with a repo-root contract: - **`/history`** - past tense: what happened. Nothing current cites it. An implementation folder - `daslib/`, a module's `daslib/` or engine folder - holds no `.md` -besides `REVIEW*.md`, `ARCHITECTURE.md`, and `LAWS.md` (lint-enforced: `.lint_config` +besides `REVIEW*.md`, `ARCHITECTURE*.md`, and `LAWS.md` (lint-enforced: `.lint_config` `rule_docs_only`). Ledgers and plans live at the module root or in `plans/` (repo root). None of these documents installs - the SDK bundle gate bans them; `REVIEW_COMMON.md` and `ARCHITECTURE_COMMON.md` at the install root are the two vendoring exceptions. @@ -39,6 +39,21 @@ of these documents installs - the SDK bundle gate bans them; `REVIEW_COMMON.md` The mood test routes misplaced text: must/never binding a diff -> REVIEW.md; present-tense fact -> ARCHITECTURE.md; dated or past-tense -> /history. +**Before editing a function annotated `[arch(at=".md#")]`, read the section it +cites.** MCP `arch_of` returns each of a file's citations with its section text; `arch_sites` +returns a document's anchors with the code citing them, dead anchors and dangling citations +included. Citation spelling, path resolution, and failure reasons: `skills/mcp_tools.md`. + +**An architecture-doc heading that code cites carries the `{#anchor}` its citation names.** +LINT026 fails a citation naming no such file or anchor, and in a folder whose `.lint_config` +sets `[docs] enforce_arch = true`, an anchor no `[arch]` cites. + +**A `REVIEW*.md` or `ARCHITECTURE*.md` past 300 lines, in a folder whose `.lint_config` sets +`[docs] rule_docs_only = true` or `[docs] enforce_arch = true`, splits into companions** - +`ARCHITECTURE_.md` for an architecture doc, a `REVIEW_.md` the parent +checklist's opening routes to for a checklist. LINT027 is the gate; the `archivist` agent +does the architecture-doc splits. + ## GitHub Operations Use GitHub MCP tools (`mcp__github__*`) for all GitHub operations (PRs, issues, reads) - they avoid shell escaping entirely. Fallback when unavailable: `gh` CLI with `--body-file` for any text containing backticks (shell escape characters in every supported shell). @@ -69,7 +84,7 @@ Task-specific instructions are split into skill files under `skills/`. You MUST |---|---| | `skills/project_overview.md` | First significant task - design philosophy, three execution tiers, macros-as-design-lens | | `skills/internal/build_and_debug.md` | Build flags, AOT build commands, exit-code/crash diagnosis, `options log_infer_passes` | -| `skills/mcp_tools.md` | Full MCP tool table + live-API reference | +| `skills/mcp_tools.md` | Full MCP tool table + live-API reference; writing or following an `[arch(at=...)]` citation | | `skills/das_formatting.md` | Creating or modifying any `.das` file | | `skills/comment_style_hygiene.md` | Writing or reviewing comments, names, or local code shape in ANY language | | `skills/writing_tests.md` | Writing or editing any dastest test | @@ -271,7 +286,7 @@ For path/filename ops use `fio` helpers (`base_name`/`dir_name`/`path_join`/...) **Complexity/length lint (STYLE037/STYLE038): new code meets both limits from the start.** On a hit in existing code, the suppress-vs-split resolution policy is `skills/style_lint.md` - never force a split on an honest shape. -**Comment hygiene, all languages: `skills/comment_style_hygiene.md` is the rulebook.** For `.das` in a folder whose `.lint_config` arms `force_clean_comments` the headline is that no comment outside the skill's kept set survives to a commit - scaffolding is welcome while you work, and the PR gate drains it (`skills/internal/make_pr.md`, the Comment drain row); elsewhere the kept set still holds and every other `.das` comment answers to the skill's deletion test. For C/C++ it is no NEW comments. The kept sets, teaching-code boundary, naming and code-shape rules all live in the skill; the `style-hygiene-auditor` agent applies them to every PR's new code (mandatory run in `skills/internal/make_pr.md`, findings persuade rather than block). +**Comment hygiene, all languages: `skills/comment_style_hygiene.md` is the rulebook.** No folder arms formatter stripping any more: scaffolding is welcome while you work, and the PR gate HARVESTS the diff's added comments (`skills/internal/make_pr.md`, the Comment harvest row) - renames first, rules and facts into the folder's documents, site-local keepers as `//!` contract one-liners; every surviving `.das` comment answers to the skill's deletion test. For C/C++ it is no NEW comments. The kept sets, teaching-code boundary, naming and code-shape rules all live in the skill; the `style-hygiene-auditor` agent applies them to every PR's new code (mandatory run in `skills/internal/make_pr.md`, findings persuade rather than block). ## Key Directories diff --git a/CMakeLists.txt b/CMakeLists.txt index f0162b6180..3aee1914bf 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2072,8 +2072,10 @@ install(DIRECTORY ${PROJECT_SOURCE_DIR}/utils/find-dupe/tests/ FILES_MATCHING PATTERN "*.das" ) -# Install utils/common (git-aware staleness signature used by -# utils/mcp/tools/cpp_common for source-search staleness tracking). +# Install utils/common. The GLOB covers every shared module, which is what keeps the shipped MCP +# server startable: it runs interpreted with no -project, and its require graph reaches +# utils/common/arch_citations.das (via utils/mcp/tools/arch_common) and +# utils/common/git_signature.das (via utils/mcp/tools/cpp_common). file(GLOB DAS_UTILS_COMMON_FILES ${PROJECT_SOURCE_DIR}/utils/common/*.das) install(FILES ${DAS_UTILS_COMMON_FILES} DESTINATION utils/common) install(DIRECTORY ${PROJECT_SOURCE_DIR}/utils/common/tests/ diff --git a/LAWS.md b/LAWS.md index f64a16c122..1af31188ef 100644 --- a/LAWS.md +++ b/LAWS.md @@ -21,3 +21,8 @@ compacted, or cited as rules. | 2026-08-27 | .claude/agents/harvester.md (RENAME verdict) | "what the previous resque-boat agent did for comments - the most interesting part - is suggest renames. like 'this comment is rename in disguise'. i don't want to loose that at all. its important" - RENAME added to the harvester's verdicts; comment stays until the rename lands | | 2026-08-27 | .claude/agents/harvester.md (RENAME precedence) | "rename should be first option, before any rescue attempts. it is the strongest." - RENAME tested before RULE/FACT/KEEP/DROP/TODO; only a comment no name can carry goes on to filing | | 2026-08-27 | CLAUDE.md (document system), CMakeLists.txt (install gate), ci/smoke_test_bundle.sh, daslib/.lint_config + utils/lint | "no random .md next to shipped code ... REVIEW.md and REVIEW.das go there, if they exist. so does ARCHITECTURE.md ... but there should be no other .md there whatsoever. lint should catch mechanically ... and related CMAKE install thing is to not install .md unless explicitly listed" + "LAWS.md can stay in the folder just like REVIEW.md, but should never ever be installed as well" - rule_docs_only lint tag; install manifest gate + bundle smoke test ban ARCHITECTURE.md/LAWS.md/ledgers; ARCHITECTURE_COMMON.md ships beside REVIEW_COMMON.md | +| 2026-08-27 | src/builtin/module_builtin_runtime.cpp, utils/lint (LINT026), utils/common/arch_citations.das, utils/mcp arch tools | the [arch] system: "annotation in the builtin, on C++ side, which does nothing. but lint does all the work" + reverse direction "if enabled - it should sweep .md and if there are anchors which can't be found in the code - same lint error" + on forward-always/reverse-opt-in as shipped: "i like whats shipped." | +| 2026-08-27 | daslib/style_lint.das (STYLE015), skills/comment_style_hygiene.md, .claude/agents/harvester.md | "//! - lint will flag on non-public functions. do we need 3rd type of comments?" -> recommendation reuse //!; ruling: "lets keep //! on non-public" - private //! is a contract comment, 3-line cap like public docs; harvester KEEP emits //! | +| 2026-08-27 | .claude/agents/archivist.md (new), utils/lint (LINT027 line gate) | ARCHITECTURE.md cleanup: "1. size - needs split by sections past 300 lines (u pick the limit...) 2. some are bunch of unrelated prose. that stuff must just go. so its an extra agent for cleanup" + "we do both this PR. i'd like to not drop anchors" - archivist agent (belonging test deletes unrelated prose, census-driven splits, anchors never dropped); 300-line lint gate on rule docs in tagged folders; both arch docs groomed in this PR | +| 2026-08-27 | skills/internal/make_pr.md | "should we update make_pr skill?" - the comment-drain row becomes the diff-scoped harvest row (the ruled harvest regime, trial passed on dasllama_image.das); rescue-bot references retired from the flow; the format row states no folder strips comments; the tree-frozen-during-preflight rule lands from this session's lib/-hiding incident | +| 2026-08-27 | .claude/agents/ (rescue-bot, rescue-sweep-bot -> history/agents/), strip-advisory strings, .claude/hooks/README.md | "yes, lets retire both. they go to /history/agents i guess. this is evolution" - both rescue bots archived; the dormant strip advisory names the harvester as successor | diff --git a/ci/make_cpp_mcp_bundle.sh b/ci/make_cpp_mcp_bundle.sh index bbf5848739..13004993ca 100755 --- a/ci/make_cpp_mcp_bundle.sh +++ b/ci/make_cpp_mcp_bundle.sh @@ -32,7 +32,9 @@ mkdir -p "$OUT/bin" "$OUT/utils/mcp/tools" "$OUT/utils/common" cp "$BIN" "$OUT/bin/" cp utils/mcp/*.das "$OUT/utils/mcp/" cp utils/mcp/tools/*.das "$OUT/utils/mcp/tools/" -cp utils/common/git_signature.das "$OUT/utils/common/" +# every shared module, not just the ones cpp_main reaches: the whole tools/ directory is copied +# above, and a tool whose require target is missing from the bundle is a trap for the next reader +cp utils/common/*.das "$OUT/utils/common/" cp -r daslib "$OUT/daslib" # ast-grep rule files: the outline / cpp_* tools load these at runtime via # get_das_root() (tools/outline.das, tools/cpp_common.das), so the bundle needs diff --git a/daslib/ARCHITECTURE.md b/daslib/ARCHITECTURE.md index c4fb2cc6c8..22ee443240 100644 --- a/daslib/ARCHITECTURE.md +++ b/daslib/ARCHITECTURE.md @@ -3,348 +3,11 @@ Design rationale a maintainer cannot recover from the code alone. One numbered section per module; entries are anchored to symbols. -## 1. perf_lint - -- **Two parallel loop spaces.** `loop_depth`/`in_closure` (+ `VarStackEntry.depth`) is the - contract for every loop rule: any closure body is deferred, hence not "in" the loop. - PERF031 needs the opposite for inline block arguments (`peek_data(s) $(d) { ... }` runs - in place), so `inline_aware_loop_depth`/`in_deferred` (+ `inline_aware_depth`) count - loops inside inline block-argument bodies as real loop levels and defer only lambdas, - local functions, generators, and generated blocks. `inline_block_entry_depth` records - the loop depth at each inline-block entry because a `return` there unwinds only to that - depth: loops at or below it keep re-invoking the block, so only barrier 0 proves the - body runs once. -- **Closure-guard placement is load-bearing, per rule.** Four idioms coexist - - `in_closure > 0` early return, `in_closure == 0` gate, `in_deferred == 0` gate, no guard - - and the guard's POSITION inside a visitor method encodes which rules are meaningful - inside a lambda. PERF001 deliberately has no guard: `loop_depth` never advances inside - closures, so its loop check is already scoped; syntactic rules (PERF003, PERF021) fire - anywhere because the pattern is wrong regardless of where it runs. Moving a guard by two - statements changes which rules fire. -- **`perf024_skip_iptr` is a one-slot handshake** between `preVisitExprCallArgument` - (classifies an annotated-arg use as safe, stashes the variable's identity) and the next - `preVisitExprVar` (consumes the stash instead of disqualifying). It is sound because the - classifier accepts only a bare `ExprVar` after one `ExprRef2Value` peel, so the very - next variable the walker reaches is that argument's own. Widening the peel set breaks - the adjacency and mis-applies the stash. -- **`add_ptr_ref` is the qmacro-splice marker.** PERF023 rests on `apply_qrules` wrapping - every `$e(...)` tag in `add_ptr_ref(...)`; matching the name suffices because the - templates_boost wrapper is its only producer post-expansion. Its depth tracking fires - before the `in_closure` gate - the rules block is itself a closure. -- **Hot-path walk (PERF026-028).** `HotCache` keeps per-function scan results sliced into - two flat arrays; `sinks_span_by_fn` doubles as the presence key and the two tables are - strictly co-populated (one insert site each, adjacent). The visited set keys on - (callee hash, `nearest_root_file_site` pointer): reaching one callee from two root-file - call sites must report twice, so a function is re-walked once per distinct anchor - - a cost taken for per-call-site diagnostics. Generated/generic frames re-anchor to the - caller side because their own `at` points into the template. A call to any function - with a `@scratch` parameter prunes that callee's whole subtree from the walk - the prune - is deliberately TRANSITIVE, removing the callee's whole subtree and not just the sink, - because narrowing the prune to the sink buries scratch-heavy callers in deep-resize - findings on buffers the helper family sizes correctly. -- **PERF030 release credits are consume-once**: a `delete x` licenses exactly one - subsequent `x <- ...`, then the warning re-arms; the lowered delete family is matched - by name in its four spellings (`_::finalize`, `finalize`, ``builtin`finalize``, - ``builtin`finalize_dim``). -- **`"$"` in module allowlists is the builtin module's name** - `Module_BuiltIn` registers - as `Module("$")` (`src/builtin/module_builtin.cpp`), so macro-time reflection reports - core builtins under `"$"` beside `"builtin"` and `"strings"`. -- **`Function.tempStringResult` marks a FRESH per-call string allocation** - never a view - or a passthrough (`include/daScript/ast/ast.h`). A temp-string builtin (`to_char`, - `repeat`) on a hot path IS heap traffic; the PERF026 string-returning catch-all - deliberately has no temporary exemption. PERF020's `_type.flags.temporary` check is a - different axis: a `#`-typed VALUE being cast, where `string(x#)` is a load-bearing - temporary-to-permanent clone. -- **PERF021's tail-argument comparison passes `require_pure = false`** - workhorse-cast - tails are flags and literals after the injected Fake* skips, so purity has nothing to - protect there; the default protects rules that collapse duplicated subexpressions from - changing evaluation counts. -- **`report_key` bit layout**: line in the low 20 bits, column shifted above it, XOR'd with - the FNV-64-prime-mixed `fileInfo` pointer and the rule-code hash. Lines past 2^20 alias - - accepted; a collision silently drops one warning. -- **`parse_range_leg` operator polarity**: `upper_op` is the var-on-left operator marking - the upper-bound leg (`<=` closed-in-range, `>` open-out-of-range), `lower_op` the lower - (`>=` / `<`). Yoda (const-on-left) forms flip, so the const-on-left arm tests against - `lower_op` - the `bounds_above` assignments differ between the two arms by design. -- **PERF022 is a three-function protocol over one stack.** `perf022_push_sentinel` pushes a - frame in `preVisitExprFor` unconditionally, so the stack stays lock-step with for-loop - nesting; `preVisitExprForBody` may flip the frame's `matched`; `perf022_pop_and_check` - fires when a frame survives matched-and-undisqualified - at the for-loop's `at`, the - actionable line a reader rewrites, not the push site. -- **A generic-body exemption exists exactly where the remedy is not instantiation-safe.** - PERF020's fix (delete the cast) is wrong for the sibling instantiations of the same - source line, so it bails on `fromGeneric`; PERF019/PERF021 rewrites (fuse under one - cast, hoist over the ternary) stay valid for every instantiation, so they fire - everywhere. - -## 2. lint_config - -- **`.lint_config` is parsed once per process.** `[lint_macro]` apply() fires once per - module in the require chain, so an uncached read re-parses the repo file N times per - compilation; the file is repo policy and cannot change mid-compilation, so - `ensure_cache_loaded` splits it once into off/on/path-exclude caches. -- **`LINT019` in a line's nolint list exempts the WHOLE line from stale-checking** - - chosen tradeoff: the in-tree escape hatch is `nolint:PERF006,LINT019` on lines whose - code fires only in downstream compiles; per-code vouching would need new syntax. - -## 3. lint - -- **`lint_error` reports only nodes whose lineinfo belongs to the current function's own - file.** An inliner splice or macro-manufactured node from another file gets its - diagnostics from that file's own compile; reporting it here would fail a module on - idioms it cannot edit. -- **LINT022 collects references from three sources because none alone sees them all**: - the ordinary walk (resolved pointers); a latched second walk over generic bodies - (`visit(prog)` skips generics, and uninferred bodies carry only spelled alias names); - and declarations plus string literals (signatures and name-based lookups leave only a - literal). Self-reference is pre-inserted so it cannot keep a private type alive. - Functions are checked only when inlining is off - inlining splices small private - callees away and leaves a body whose only callers were inlined away, indistinguishable - from dead code. Public symbols, lifecycle/macro hooks, virtual methods, and generic - bodies are skipped: their callers are invisible to lint. -- **Argument rules skip what the author does not own, and converge leaf-first.** - LINT012/013/014/023 stay silent on class methods, `[extern]` stubs, `finalize` - overloads, and address-taken functions; LINT014 also skips a parameter whose mutability - a callee demands - the leaf is flagged first and each fix re-exposes the next caller. - A returned `var` argument keeps its `var` (a non-copyable result cannot move from - const); a used `_name` parameter is never LINT004'd (interface and keyword-clash names). -- **Closure bodies are per-rule, not global.** LINT010 counts a closure body as a branch - (it may run later or never - writes inside must not kill outside stores, reads inside - must not keep an outside init live); LINT021 counts the same body as an escape - a - captured 64-bit local is never a narrow. -- **`expr_equal` compares `describe(a) == describe(b)`, never pointers** - every - Expression has exactly one parent, so sibling pointer equality is effectively never - true. LINT007 excludes `ExprConst*` operands: identical constants are macro-splice - residue, not copy-paste typos. -- **LINT015 fires on `+` and `-` only** - the only operators both binary and unary, so a - split continuation line re-parses as a valid discarded unary statement; every other - operator produces a loud parse error instead. -- **LINT023's store set extends the compiler's.** `under_clone` marks only bare-variable - `=`/`:=` targets; `<-`, swizzle LHS, and statement-level `++`/`--` are marked by the - visitor. A pointer parameter stays a candidate: a write through the pointee reaches the - caller and counts as a use, not a store. -- **The 64-bit family fires only where the overflow hazard survives.** LINT021 treats - only unbounded-top math as transparent (`min`/`clamp`/`sign` cap the value - deliberate - saturation); LINT024 requires a call among the product's leaves - that is the unbounded - factor, a product of plain locals is tile geometry. -- **`canVisitWithAliasSubexpression` returns false on purpose** - `assume` keeps the - template block at the definition site and clones it per use; the block is never invoked - there, so descending would report its arguments unused. -- **C++ mirror pairs**: `lint022_optimized` / `lint022_calls_may_be_inlined` <-> - `Program::getOptimize` / `Program::patchInline`; `stale_scan_line` <-> - `rtti_is_nolint_suppressed`; `is_inline_temp_name` <-> `INLINE_TEMP_PREFIX`. Nothing - fails when one side moves alone. - -## 4. style_lint - -- **Lint runs under policies that skip const-folding and leave some types unset.** Take a - variable's type from `Variable._type`, not an `ExprVar`'s; expect `T('c')` to stay an - `ExprCall`; expect `-1` as an `ExprOp1` over a constant; when a type is missing, bias - conservative ("needs unsafe"). -- **Rules whose forms are identical in the AST read the source span instead** - `defer <|`, - generators, STYLE003's `$()`, STYLE028's three invoke spellings. STYLE005 uses LineInfo - identity: the parser gives braceless and postfix forms a synthetic block sharing the - terminator's LineInfo, so `blk.at != inner.at` <=> the user wrote braces. Both of these - tests - the source-span read and the LineInfo identity check - are the rule's only - evidence for the distinction, and both break silently when lowering or LineInfo - assignment changes. -- **STYLE024/025 mirror the compiler's unsafe rules with a deliberate asymmetry: in doubt, - mark "needs unsafe".** A false "needs" costs a missed note; a false "redundant" tells - the user to delete a wrap the compiler requires. `let v & = unsafe(expr)` is the one - recorded exception - the wrap is required by the reference binding, not the inner - expression. STYLE025 narrows only when the single statement has an expression form. -- **A require counts as used when its symbols cannot be traced, not only when seen.** - Generic instances attribute to `fromGeneric._module`; bare names in never-instantiated - generic bodies keep their providers alive; re-export chains are followed; a module - providing any compile-time macro or lifecycle hook is used - it leaves no AST reference. - STYLE029 fires only when exactly one re-exported module is used: two or more is an - aggregation facade. -- **Metric rules attribute code to the module that can fix it.** STYLE037/038 walk the - `fromGeneric` chain to its root (`[template]` generics instantiate INTO the caller); - generated carriers stay unmetered; closures are not length-checked separately - the - host trips first. -- **STYLE041 is sound because of a reference count, not the structural scan.** The scan - classifies writes, plumb reads, and the single consuming `if`, arming the candidate with - that count; `preVisitExprVar` counts every reference in the function, and the warning - fires only when the counts match. Anything the scan does not descend into therefore - rejects the candidate instead of being assumed harmless. -- **STYLE036 keys on a flag the compiler clears.** Infer strips a substitution contract - when a generic binds; one still set on a resolved cast target was never consumed - but - an `auto`/unresolved-alias target has simply not been substituted yet, hence the - `isAutoOrAlias` bail. -- **STYLE039 requires non-ASCII in both the value and its source span** - escape-spelled - bytes in ASCII source are log-safe, and span-slicing keeps a neighbouring comment's - typography from firing. -- **The unsafe-frame walk**: `preVisitExpression` pushes one frame per expression node, - `visitExpression` pops it - the popped frame is the subtree's summary, stored in - `unsafeExprs` for `visitExprUnsafe` (STYLE024 vs STYLE025), propagated to the parent - slot, and fired as STYLE024 when an `unsafe(...)` target's subtree had count 0. The - balance panic in each entry point is the guard against a `canVisit*` override skipping a - subtree the frame walk still counts. -- **Two compiler regimes run this visitor.** The `[lint_macro]` path runs inside a normal - compile (const-folding ON - `T('c')` folds to a typed const); the standalone runner and - MCP path run under lint policies (folding OFF - `T('c')` stays an `ExprCall`, `-1` stays - an `ExprOp1`). Shape-sensitive rules carry both arms (`numeric_char_cast_target`'s - ExprCall branch + source-span re-parse); deleting the "redundant" second arm kills the - rule in the other regime. -- **`expr.canShadow` on a for-variable exempts it from LINT002/003/004** - the flag marks - macro-generated loops, and lint does not judge names the user never spelled. -- **The `expect` probe scans the whole file while `lint-skip-file` is header-capped** - - an `expect` directive anywhere makes the file a compile-error fixture (its position is - semantic to dastest), whereas `lint-skip-file` deeper than the header would let quoted - prose unlint a file. -- **The `blk.list` index rescans are O(block^2) by choice** - the per-statement visitor - callback carries no index, and a shared index stack would have to push/pop with nested - blocks; block statement lists are short. -- **Threshold provenance**: STYLE038's 80 ~ p97 of this tree's function lengths; STYLE041 - caps report-arg recursion at 8 and fails closed (deeper is unknown shape); - `LINT_SKIP_HEADER_LINES = 16` exists so prose quoting the directive cannot unlint a - file; STYLE037's 20 and `note_type`'s depth 24 have no recorded derivation. -- **C++ mirror pairs**: the STYLE024/025 unsafe map <-> infer's `unsafe_*` rules; - `style036_inert_contract` <-> infer's contract clearing. Nothing fails when one side - moves alone. - -## 5. aot_cpp - -- **C++ identifier mangling**: `aotSuffixNameEx` prepends `_S`/`_E`/`_V`/`_f_` when a das - name is a C++ keyword, holds a non-alnum char, or is `DELETE` (winnt.h). Structs and - enums share ONE C++ namespace while daslang keeps separate tables, so `struct X` + - `enum X` in one module is legal das and a C++ redefinition - `buildStructEnumCollisions` - finds the pairs per module and forces both suffixes. -- **The emitter targets the MSVC/GCC/Clang intersection; the redundant-looking spellings - are the portable ones** - `INT64_C`, the explicit `0x8000000000000000` enum arm, - `to_cpp_double`'s named non-finite tokens (mirrors `src/simulate/runtime_string.cpp` and - moves with it), elaborated `struct X`, double parens around `das_iterator` sources - (most vexing parse), non-const value loop variables (`first()`/`next()` write into it). -- **Sequenced binary operands**: any op2 whose operands are not both side-effect-free is - emitted through `das_ordered2{ L, R }` plus an immediately-invoked lambda. The struct and - its member names live in `include/daScript/simulate/aot.h` and move with the emitter's - `__lr.left`/`__lr.right` spellings. -- **Stack-frame `new`/ascend**: per-block storage declared once, the USE site - re-initializes per evaluation (memset for `new`, whole-value overwrite for ascend) - - dropping the reinit reuses the previous iteration's value. -- **fp16 and the 8/16-bit lattice go through the vec4f policy ABI**: `SimPolicy_HalfVec` is - vec4f-shaped even at width 1, so scalar fp16 always casts across the policy boundary, - and a lattice-vector policy result must come back typed for `das_equ_val` deduction. - Lattice swizzles use element-typed helpers - generic `das_swizzle` caps at 4 fields, - `v_extract` is 32-bit-lane only. -- **A struct needs a COMPLETE C++ type more often than field access suggests** - globals - need `sizeof` even untouched, and by-value field types emit depth-first before their - owner, or the type degrades to a forward declaration. -- **`aotModuleName` has a C++ twin that moves in lockstep** - - `src/ast/ast_typedecl.cpp`'s `aotModuleName`/`describeCppTypeEx` spell the same type - names into the same TU through `DebugInfoHelper`'s cached cpp names - (`debug_helper_find_*_cppname`, the cross-platform `TypeSize<...>` sinks). The - main-module namespace override is therefore one knob with two ends: - `set_aot_main_module_name` writes the daslib global AND forwards to - `set_aot_main_module_name_cpp`; a spelling rule changed on one side only produces a TU - where the definition and its debug-info references disagree. - -## 6. aot_standalone - -- **The generated constructor IS the init protocol** - a standalone context never calls - `Context::runInitScript`, so the ctor reproduces its observable semantics inline: - `memset(context.globals, 0, context.getGlobalSize())` mirrors runInitScript's globals - memset (`src/runtime/context.cpp`), then the same-TU `__init_script(&context, true)`, - then each `[init]` function directly. `aot_cpp.das`'s `preVisitGlobalLet` writes the - `__init_script` signature - a daslib-to-daslib pairing nothing checks. -- **`__init_shared` is hardcoded `true`** - a fresh standalone context always owns its - shared globals. -- **The ctor's base `Context(N)` folds the init headroom in, because there is no separate - init stack** - init locals are C++ locals in AOT. A script that sets `options stack` to a - positive value gets `N = options stack + globalInitStackSize`, honored exactly, the - interpreter's floor included: the floor protects INTERPRETED init, while standalone init - is AOT, so an embedded target's stack budget wins. Zero and negative fall through to - `N = max(policies.stack, 16384) + globalInitStackSize`; `Context(0)` owns no stack, so any - prologue would assert. The fall-through arm mirrors the interpreter's init-stack formula - in `src/ast/ast_simulate.cpp` - a pair, deliberately divergent on the explicit-option arm. - `globalInitStackSize` reaches the emitter through the rtti `Program` binding in - `src/builtin/module_builtin_rtti.cpp`, so an `invoke` during init still has das stack for - its prologue. -- **There is no `!stopFlags` guard between `[init]` calls** - a panic propagates out of the - ctor instead of soft-stopping the sequence. -- **Global init order is fixed by `var->index` assignment**: `StandaloneContextGen`'s - `preVisitGlobalLet` emits required modules' globals via ordered `for_each_module` - before the adapter walks the entry module's own - correct only because - `ast_allocate_stack.cpp` assigns `var->index` through the same dependency-first - module order with the entry module last, and `runInitScript` executes ascending - index. `[init]` function order is not re-derived at all: the emitter reads the - simulated context's list through rtti `for_each_init_function`, so the C++ late-init - sort stays the single source of truth. -- **Cross-module limits fail loud at emit time**: only main-module, AOT-emitted `[init]` - functions can be called from the ctor (required-module and `[no_aot]` ones are collected - emit errors with the reason), because the standalone TU only emits the entry module's - function bodies. -- **Every used function must have an AOT body** - a standalone context has no - interpreter, so a used `noAot` function (the `[no_aot]` annotation, or `NoAotMarker` - finding a type AOT cannot express) is a collected emit error, never a - `fnByMangledName` call that would crash at runtime. `prepareProgramForEmission` runs - `NoAotMarker` first (the regular AOT paths run it too; standalone must match) and - then `checkAllUsedFunctionsCanAot` walks used, non-builtin functions. -- **Type definitions live in the header, once** - struct/enum definitions (the - dependency dump plus the entry module's own `declarations` capture) are emitted into - the `.das.h`, which the `.das.cpp` includes; the source never redefines them. They sit - in module-named namespaces directly under `das`, NOT under the per-program - `_anon_` namespace - that hash changes with the program, so it can never be a - public name. The entry module is promoted with an empty name, which would put its - types in an anonymous namespace (a distinct type per TU - an ODR/link break for any - struct in an exported signature); `aot_cpp`'s `set_aot_main_module_name` names it - after the context for the duration of standalone emission, so class signatures, the - source TU, and the embedder all resolve the same `das::::T`. The header carries - `#pragma once` and the required modules' `aotRequire` includes so it stands alone in - an embedder TU; two DIFFERENT contexts' headers sharing a das dependency still cannot - be included in one TU (the shared types have no per-type guards). - -## 7. flatten - -- **Predicated lowering carries one live-mask per exit flavor** - `__flat_live` for - return, a per-loop break mask (persists across unrolled copies) and continue mask - (re-minted per copy). A write's predicate ANDs every active mask plus the structural - predicate; a narrow term excludes its own mask so it self-cancels. An inlined callee - gets a fresh live mask and lowers with `ctx.loopMasks` moved OUT, so its break/continue - can never reach the caller's loops. -- **`flatten_preshade_cse` is a joint fixpoint, not a pipeline** - extraction, regroup, - CSE and alias elimination mutually enable each other; the `_preshader_`/`_cse_` counters - are owned by that loop and re-seeded from surviving suffixes (per-call numbering - re-mints a live name). -- **A CSE/regroup tally counts exactly the regions its rewrite can change** - a duplicate - counted where the rewrite cannot reach never drops below 2 and runs the fixpoint to its - iteration cap. -- **`__flat_ret` carries `safeWhenUninitialized` only while every write is a - self-referential select** - a lowering change that makes the bare-decl read observable - turns the flag into a real uninitialized read. -- **CSE is local value numbering over one converged basic block, and it is complete** - - pure subtrees keyed by `describe()`; value-stability = reads no reassigned name; a store - through index/field/swizzle destabilizes its base; an unrecognized node fails closed as - mutable-reading. Uniform duplicates route to the preshader. -- **The copy-prop/CSE walks stay O(size)** - one name-to-statement index, one structural - walk. A `string` materialized per `ExprVar` in a visitor callback breaks that: each - `describe()` allocates a string that lives to the end of the pass, so the walk goes - quadratic in heap bytes, not only in time. -- **`MutCollect` is what CSE trusts to say whether a name is stable, so it counts every - store spelling, not the one the lowering emits.** CSE treats a name outside its set as - constant for the whole block; a missed store is a shared subexpression across a mutation. - Copies are only the visible half - `<-` also zeroes its SOURCE, `:=` lowers to a - `builtin`clone`(dst, src)` CALL rather than an `ExprClone`, `++`/`+=` are their own - nodes, and a by-reference - argument writes with no assignment node anywhere. Hence the argument arm keys on the - callee's parameter type (non-const and `ref` or a ref type), not on a node kind. -- **`delete` on a container of `ExpressionPtr` frees the BUFFER, never the nodes** - - `delete array` frees the pointees only for das-heap `T`, and `Expression` is a - handled C++ type whose instances are not heap chunks at all (the - measurement: an `array` of das structs returns its pointees to `heap_bytes_allocated`, - an `array` returns only the buffer and the nodes surface in the exit GC - report). That is why `make_float_ctor`'s const-fold early return may leave its lanes - un-consumed while the ctor path `emplace`s them away, why every `unsafe { delete args }` - after it is sound over borrowed tree nodes, and why a struct field holding a borrowed - node needs no `@do_not_delete`. Node lifetime belongs to the AST GC: a lane the const - fold drops is unreachable and collected at the enclosing `ast_gc_guard`. -- **The whitelist admits value-returning primitives only.** `lower_stmt`'s fall-through arm - lowers an unrecognized statement for its lifted sub-lets and drops the statement itself, - which is correct exactly while every surviving call is pure - so `lift_expr` refuses a - whitelisted call that writes through a by-reference argument (`sincos`) rather than let - the drop delete the store. Predicating such a write would need per-out-param temps the - lowering does not own. +Three companions carry a concern each; a section number is unique across all four files. + +- `ARCHITECTURE_LINT.md` - sec. 1-4: perf_lint, lint_config, lint, style_lint. +- `ARCHITECTURE_EMIT.md` - sec. 5-7, 28-29: aot_cpp, aot_standalone, flatten, the shader rails. +- `ARCHITECTURE_LINQ.md` - sec. 11-17, 33, 37: the linq family, sql_linq, sql_migrate. ## 8. ast_verify @@ -412,163 +75,6 @@ module; entries are anchored to symbols. `carry_tag_safe_flags` copies the parser's unsafe-wrap flags across `$c` substitution or the safety is lost where the result lands. -## 11. linq - -- **`top_n_by_with_cmp` exists for the fold splice and has no daslib call site.** When an - order-by key body is pure and inlineable, `linq_fold_common` emits it with that body - spliced into BOTH sides of the comparator - killing the per-comparison comparator dispatch - and the per-side `key(v)` dispatch the key-lambda overloads pay. It is not a redundant - twin of `top_n_by`. - -## 12. linq_boost - -- **`BucketLambdaStamper` exists because a bucket-surface lambda cannot infer on its own.** - On the `group_by_lazy` element shape `tuple>`, an untyped lambda in - `._1 |> select/sum/...()` has nothing to bind against in the fully generic - tier-2 parameters (error 30303); the chain type knows `E`, so the macro stamps the - lambda's parameter before the rewrite - the same move as the outer-parameter injection in - `visit()`. -- **`result_param1_type` is the join result-lambda auto-typing hook.** param0 is always TA; - param1 is whatever the override returns - `_join` -> TB, `_group_join` -> a bare `array` - with const/ref cleared to match the bucket type. `_left_join` / `_right_join` / - `_full_outer_join` deliberately do not override: their result parameter is `$Option` on - one or both sides, so those result lambdas stay user-spelled. - -## 13. linq_das - -- **The substituting scanners share one token model.** `substitute_idents`, - `mentions_ident` and `rewrite_group_var` all read it: plain `"..."` content is verbatim, a - `{...}` interpolation body is code that is scanned and substituted, one level of nested - string literal inside an interpolation is verbatim again, and identifier position - classifies the same way everywhere - after-`.` field access and before-single-`=` labels - are not references, with `is_label_position` the shared judge. -- **`JOIN_TI` is a spellable identifier on purpose.** `__`-prefixed names are reserved by - the compiler, so the transparent-identifier carrier cannot be hidden outside the user's - namespace and must stay a single plain token - hence the explicit rejection of a range - variable named `linq_join_ti` instead of an unspellable name. -- **`build_src`'s two arms are a fusion decision, not a formatting one.** A fused source - (chain head, join) is built as `from_in(...)` so `_fold` can route it; an UNFUSED operator - argument - the uncorrelated multiple-`from`, whose `_cross_join` runs at tier-3 - passthrough - is passed as the bare parenthesized expression, so `each`'s - `[unsafe_outside_of_for]` does not trip and the arrayxarray overload is selected. -- **The correlated-flatten collection selector borrows, it never copies.** `unsafe(each(tail))` - is chosen over `to_sequence` (whose const-array overload clones per row) and over - `where_`'s array overload (which materializes per row). Borrowing a TEMPORARY tail - (`from x in [c.id]`, a function result) is safe because daslang heap-allocates arrays and - does not finalize them at scope exit - the backing storage outlives the borrow and is - reclaimed by GC. -- **A `FromInMacro` reject returns null so infer can stabilize.** Errors clear on every pass, - so the not-yet-inferred-source arm's error survives only when the source never infers at - all - that is what makes that arm a deferral rather than a final verdict. -- **`from_in` identifies typed sources by matching module + type NAME as strings**, not by - requiring the provider modules: `sqlite_boost::SqlRunner` -> `select_from`, pugixml - `xml_node` -> `from_xml_node`, `json::JsonValue?` -> `from_json` - linq_das stays - dependency-free on sqlite/pugixml/json. Both node builders are `[unsafe_outside_of_for]`, - so the emitted call carries the `unsafe` wrap a `_fold` chain requires. A table source is - rejected with a fix message before those arms: it already carries its row shape, and the - typed builders would all mis-fire. - -## 14. linq_fold_common - -- **group_by's per-key update is a dummy plus an address compare, not a `key_exists` - probe.** `entry &= tab?[uk] ?? dummy` costs one hash op per element on hits; - `addr(entry) == addr(dummy)` identifies the miss, which then fills the dummy, inserts it, - and re-defaults it - the second hash op is paid on misses only. Average slots hold a - `(sum, count)` accumulator and divide with no zero guard: sound because a key enters the - table only on its first element. -- **`hoist_prelude` hand-builds a flat block on purpose.** The materializer prelude - the - hoisted default-row declaration on the field-prune path - must be a SIBLING of the walk; a - `$b(prelude)` splice nests it in its own scope, where the per-element reads cannot see it. -- **One `at` can reach two materializers.** A `group_by` chain closed by - `first`/`order`/`distinct`/`take` emits through both the array-walk materializer and the - handle materializer at the same source location, so `qn` - which keys on (prefix, at) - - keeps the two sites' binds apart only while each site carries its own prefix. -- **An `ExprFor` the typer has not re-inferred carries an EMPTY `iteratorVariables`.** A - decs bridge feeding a no-chain terminator (`from_decs_template(...).count()`) has no chain - op to force the second inference pass, so the iterator names are recovered from the push - tuple's values - each references its iter var by name under an `ExprRef2Value` wrap - and - match what the loop binds once a later pass infers it. - -## 15. linq_fold_sql - -- **`extract_sql_source` walks the call's own `arguments[0]` spine, not linq_fold's - flattened `top`** - the SQL DSL ops are not `linqCalls`, so `flatten_linq` stops at the - first one and never reaches the `select_from` source. `sql_linq_loaded(prog)` then gates - on `_sql` being in THIS program: a consumer can get `select_from` from a provider boost - without `daslib/sql_linq`, and that chain must fall through to the in-memory path. - -## 16. linq_fold_decs - -- **Column pruning is a four-step pipeline over the `decs_tup` bind, and every step exists - to reach bind elision.** Scan the body for `decs_tup.` uses; if the bind is - referenced as a WHOLE var, rewrite each whole-var use into a synthesized - `(userName1 = iter1, ...)` literal - semantically identical - and re-scan; then flatten the - surviving field reads to the iter vars and drop the bind entirely. Unused `get_ro` slots - disappear along with one tuple-make plus N field reads per iteration. The unpruned bind - stays only for whole-var refs that survive the rewrite. -- **The decs random-index fast paths rest on these source facts.** A plain `[decs_template]` - field compiles to `get_ro` (an indexable array) while a default-init field compiles to - `get_default_ro` - an iterator, not indexable - so `decs_can_random_index` bails and the - walk stands. `get_ro` is `[unsafe_outside_of_for]`, so each cloned source AND the index - read are marked `alwaysSafe` (mirroring `decs_boost`'s `append_index_lookup`). - `for_each_archetype` visits archetypes in order and skips empty ones, so a no-predicate - `last()` overwriting a result per archetype ends holding the global last in - O(num_archetypes), and `reverse |> take(N)` collects only the tail with the boundary - archetype's head indexed past rather than walked. -- **Trivial-select elision removes the copy a `_select(_.userName)` would emit.** The - pruned inner-for already keeps that component's iter var, so the synthetic bind would - flatten to a pure copy; renaming the chain's `finalBind` to the iter var skips the binding - entirely. Detecting the shape needs TWO `ExprRef2Value` peels: the typer wraps both the - projection root and the `ExprVar` inside the `ExprField`. -- **The decs join hook keeps hash-collect and probe inline rather than routing through - `wrap_source_loop`** - that is what preserves the count-no-`where` fast path, which - answers from bucket lengths instead of walking pairs. Its key lambdas are synthesized - upstream by the LinqJoin macro as 1-arg single-return blocks, so the BLOCK's return type - is the key type; a change to that synthesized shape breaks the decs join silently. - -## 17. sql_linq - -- **Equal phase numbers are deliberate.** `PHASE_ORDER_BY == PHASE_SELECT` - ORDER BY - commutes with the projection in SQL (it may reference projected aliases or source - columns) - and `PHASE_GROUP_BY == PHASE_DISTINCT == PHASE_SET_OP`. Giving either member - its own number spuriously diverts the canonical `_group_by |> _order_by |> _select` into - a nested SELECT. -- **Emitted SQL is a template over two in-band markers.** `?` is a bind, `\x01` a runtime - identifier (a dynamic `_order_by` column name); `next_placeholder` scans outside - single-quoted literals and always resumes just past a placeholder, so the scan starts - with no quote open. `sql_to_frags_ex` pairs each marker with `orderedBinds` / - `orderedInlineIds` in occurrence order and preserves a marker as text once its list is - exhausted. Nothing reaches the text unquoted: a compile-time constant folds in with SQL - doubling, a non-constant emits a runtime `sql_quote_id` / `sql_quote_lit` call, and every - other runtime value stays a `?`. -- **Key contexts render with constants inlined because their fragment is re-used.** A - computed `_group_by` / `_order_by` key fragment is emitted at several SQL positions - (SELECT, GROUP BY, ORDER BY) while a placeholder can be bound only once, so - `render_inlined_key_sql` sets `q.inlineConstants`, admits only `ExprConst*` (strings - through `sql_quote_lit`), and rejects any key that pushed a bind. -- **`_distinct_by(K)` is dialect-routed on `caps.distinct_on`.** Without it, SQLite's - bare-aggregate form `SELECT *, MIN/MAX(pk) FROM t GROUP BY K` keeps the `*` columns of - the min-pk (or max-pk) row per K - SQLite-only semantics; strict engines reject bare - columns beside GROUP BY. With it, `SELECT DISTINCT ON (K) * ... ORDER BY K, pk [DESC]` - a - PostgreSQL extension DuckDB also implements. MIN/ASC is linq's "first row per K", MAX/DESC - is `reverse() |> _distinct_by(K)` "last row per K" - both only while pk is monotonic with - insertion order. -- **A join's `into` projection registry is snapshotted because the outer projection - clobbers the live one.** `process_join_call` copies the live projection into - `joinProjRecordNames` + the `joinSelectCol*` arrays, which survive - `analyze_grouped_projection`'s clear-and-repopulate. Post-join `_.` resolves - through that snapshot in every post-join consumer - `pred_to_sql`'s column-ref arm, - `collect_one_order_key`, `push_group_key`, `try_translate_group_aggregate`, - `try_translate_global_aggregate` - all through `find_projection_alias` + - `render_projection_alias_sql`; a new post-join alias consumer goes through that pair too. - A registry miss is rejected loudly everywhere alike: falling back to base-table resolution - would leak the unqualified base-table namespace into post-join predicates. -- **`normalize_single_source_arg_names` exists for the linq_das front end.** `_sql` - resolves a single source against the placeholder `_`, but LINQ-syntax lowering splices - the user's range variable verbatim (`$(c) => c.field`), so a single-parameter chain-op - lambda's bound variable is renamed to `_`. Gated on a pure single-source chain - join and - set-op key lambdas bind distinct sources and are left alone. - ## 18. dupe_detect - **A run is unextractable when its meaning depends on the scope around it** - the gate @@ -635,21 +141,7 @@ module; entries are anchored to symbols. - **The writer is hand-rolled because the builtins do not round-trip** (`\v`, raw 0x7f, float exponent thresholds); a value scan that runs into a bare-key char rewinds and - re-lexes as a key - a new numeric form needs the same rewind. - -## 28. shader_block_layout - -- **Two rails, deliberately separate** - the LAYOUT rail admits int64/uint64 as block - members (`compute_block_layout` special-cases them) while the ARITHMETIC rail rejects - 64-bit INT (`arith_width_ok` allows width 64 only for floats); `cpu_only_lattice_width` - keys both emitters' fail-closed diagnostic. - -## 29. shader_lingua_franca - -- **Every symbol is either an exact CPU mirror of its GPU semantics or a `[sideeffects]` - dummy every rail lowers by name** - the dummies return zero on the host, so a CPU replay - reproduces GPU semantics only for the real-bodied set. Unsigned overloads never fold into - signed twins (glslang picks the unsigned opcode). + re-lexes as a key. ## 30. archive @@ -676,13 +168,6 @@ module; entries are anchored to symbols. into strings, so `parse_response` / `parse_response_batch` free their own tree before returning. -## 33. sql_migrate - -- **The audit table is provider-neutral by construction** (client-side epoch seconds, - BIGINT); duplicate versions are caught in two layers because neither sees everything; the - `struct_convert_field` overload set is a specificity ladder - deleting a "duplicate" - silently re-routes conversions. - ## 34. fio - **The glob matcher follows POSIX fnmatch on degenerate patterns** (unterminated `[`, `]` @@ -698,82 +183,3 @@ module; entries are anchored to symbols. - **`_table_index_and_init` exists for infer's `default_init_containers` rewrite of non-store `tab[key]`** - it has no daslib call site and is not dead. - -## 37. linq_fold module layout and the SourceAdapter contract - -- **Charters.** `linq_fold` is dispatch only: source recognition (`try_splice_patterns`), - adapter construction, the tier-2 `fold_linq_default` cascade, the `LinqFold` call macro, - and the single registrar `register_all_linq_fold_rows`. `linq_fold_common` holds what no - source can influence - the pattern-table types, the walker, the predicate library, the - chain pre-passes, the generic emit lanes, the `splice_patterns` registry, and the abstract - `SourceAdapter`. Each remaining file owns one source: `linq_fold_array` (array, zip, - array-join), `linq_fold_decs`, `linq_fold_json`, `linq_fold_table`, and pugixml's - `linq_fold_xml`. `linq_fold_sql` is the exception - it recognizes a `[sql_table]` source - and hands the whole chain to the `_sql` macro, so it has no adapter and no emit. A source - file requires `linq_fold_common` and never a sibling source. -- **`try_splice_patterns` tries recognizers in a fixed order** - sql, decs, xml (behind the - pugixml `static_if`), json, table - and the array arm runs last with no recognizer of its - own: it is what claims a chain nobody else claimed. -- **The adapter contract is four abstract methods; everything else on the base has a - default.** `bind_name`, `element_type`, `wrap_source_loop` and `wrap_invoke` are abstract, - so every source answers them. The per-operation hooks (`emit_loop_or_count`, - `emit_reverse_skip_into_tail`, `emit_reverse_last_backward`, `emit_distinct_take_loop`, - `build_group_by_adapter`, `emit_join_hook`) return null on the base, and a null sends the - caller down its generic path. The capability methods (`can_group_by`, `can_join`, - `can_reserve_by_length`, `has_own_loop_or_count_lane`, `supports_direct_return`, - `effective_dispatch`, `defers_materialization`, `count_shortcut`, `invoke_param_type`, - `name_prefix`) are the source answering questions about itself. daslang classes have no - `is`/`as` downcast, so an emit fn holding a `SourceAdapter?` reaches source-specific - behavior only through these methods - there is no place to write a per-source branch, and - adding a source needs no edit to a central enum or switch. -- **The capability defaults are fail-closed, and `supports_direct_return` is the one that - matters.** A wrong `true` emits a mid-loop `return` that escapes a nested-callback walk - (decs's `for_each_archetype`); a wrong `false` only costs the slower state-var + - find-stop + tail path. -- **`loop_source_expr` / `loop_source_name` is what lights up the array-shaped lanes.** The - shared lanes spell their loop and their `length(...)` reads with the name, and read the - expr for compile-time facts - the `type_has_length` reserve gate and the join's srcA - element-type check. Array and table override both, xml and json override the name only, - decs overrides neither. -- **`ProjectedSourceAdapter` absorbs a leading source projection.** When a row captures the - optional `srcsel` slot (`source |> _select(f) |> order/distinct/take`), the dispatcher - wraps the real adapter in this decorator: it binds `projName = f(rawElem)` on top of the - per-element body and delegates loop and invoke to the inner adapter, so every emit sees the - projected element unchanged. It leaves `loop_source_expr` / `loop_source_name` / - `can_reserve_by_length` at the base defaults, which disables the source-direct fast paths - - those would bypass the projection. -- **Rows are per operator family, not per source.** The seven `build_*_rows` fns - (order_family, loop_or_count, reverse, distinct, group_by, join, zip) emplace 17 rows into - the one `splice_patterns` table; only zip's builder lives outside `linq_fold_common`. - Source gating happens inside a row - through a `RequiresPredicate` that asks the adapter - (`can_group_by_source`, `can_join_source`, `array_source`, `non_array_source`) or through - a hook that returns null. Registration order is match priority: the dispatcher walks the - table top to bottom and the first row whose chain matches and whose emit returns non-null - wins, so more specific chains are declared first. A row whose slot chain is a strict prefix - of a later row's shadows that row (`chain_prefix_of`). -- **The walker consumes the whole call list or fails.** `match_pattern` walks the flattened - calls left to right against the row's slots: `c_one` must match and advances both cursors, - `c_opt` advances both on a match and skips the slot otherwise, `c_chain` matches greedily - while the call is in the slot's name set and always succeeds - including with zero matches, - which still creates the (empty) `captures.many` entry, so an emit fn can rely on the key - existing. Any call left unconsumed after the last slot is a no-match. The `requires` - predicates run last, against the populated captures. -- **`Captures.single_name` exists because a pre-pass renames the op without touching the - AST.** `normalize_order_reverse` folds a trailing `reverse` into the order op by swapping - the tuple's `LinqCall` record, leaving the `ExprCall` alone; deriving the name back from - the `ExprCall` (`call_norm_name`) would silently undo the swap. The walker therefore - records the `LinqCall.name` at match time, and any emit fn that cares about the - post-normalize name reads `single_name`. -- **`alias_table` names the op-name groups a `m_alias` slot admits, and a missing name fails - silently.** A terminator the lanes handle but `loop_terminator_family` omits never matches - the terminator slot, so the chain leaves the splice and lands on the tier-2 cascade with no - diagnostic - the symptom is a slow chain, not an error. -- **Predicates and emit fns are named module-level functions, referenced as - `@@ name` / `@@ name`.** An anonymous `@@(...)` lambda compiles - to a `_localfunction_*` symbol the LLVM JIT pass cannot resolve; a named function has a - stable address. -- **The table lane's key and value binds are asymmetric on purpose.** `keys(tab)` yields - non-const elements (writable temp copies), so the key lanes rebind the loop var through a - `let` before the body sees it; `values(tab)` over the const table param already yields - `V& const` and binds directly. Keys are workhorse types, so the copy is free, and - downstream `==const` composition (a `push_clone` of a bare projected key) needs the const. diff --git a/daslib/ARCHITECTURE_EMIT.md b/daslib/ARCHITECTURE_EMIT.md new file mode 100644 index 0000000000..b4b49df7a5 --- /dev/null +++ b/daslib/ARCHITECTURE_EMIT.md @@ -0,0 +1,159 @@ +# daslib architecture notes - emission: AOT C++, standalone contexts, shaders + +Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across the family. + +## 5. aot_cpp + +- **C++ identifier mangling**: `aotSuffixNameEx` prepends `_S`/`_E`/`_V`/`_f_` when a das + name is a C++ keyword, holds a non-alnum char, or is `DELETE` (winnt.h). Structs and + enums share ONE C++ namespace while daslang keeps separate tables, so `struct X` + + `enum X` in one module is legal das and a C++ redefinition - `buildStructEnumCollisions` + finds the pairs per module and forces both suffixes. +- **The emitter targets the MSVC/GCC/Clang intersection; the redundant-looking spellings + are the portable ones** - `INT64_C`, the explicit `0x8000000000000000` enum arm, + `to_cpp_double`'s named non-finite tokens (mirrors `src/simulate/runtime_string.cpp` and + moves with it), elaborated `struct X`, double parens around `das_iterator` sources + (most vexing parse), non-const value loop variables (`first()`/`next()` write into it). +- **Sequenced binary operands**: any op2 whose operands are not both side-effect-free is + emitted through `das_ordered2{ L, R }` plus an immediately-invoked lambda. The struct and + its member names live in `include/daScript/simulate/aot.h` and move with the emitter's + `__lr.left`/`__lr.right` spellings. +- **Stack-frame `new`/ascend**: per-block storage declared once, the USE site + re-initializes per evaluation (memset for `new`, whole-value overwrite for ascend) - + dropping the reinit reuses the previous iteration's value. +- **fp16 and the 8/16-bit lattice go through the vec4f policy ABI**: `SimPolicy_HalfVec` is + vec4f-shaped even at width 1, so scalar fp16 always casts across the policy boundary, + and a lattice-vector policy result must come back typed for `das_equ_val` deduction. + Lattice swizzles use element-typed helpers - generic `das_swizzle` caps at 4 fields, + `v_extract` is 32-bit-lane only. +- **A struct needs a COMPLETE C++ type more often than field access suggests** - globals + need `sizeof` even untouched, and by-value field types emit depth-first before their + owner, or the type degrades to a forward declaration. +- **`aotModuleName` has a C++ twin that moves in lockstep** - + `src/ast/ast_typedecl.cpp`'s `aotModuleName`/`describeCppTypeEx` spell the same type + names into the same TU through `DebugInfoHelper`'s cached cpp names + (`debug_helper_find_*_cppname`, the cross-platform `TypeSize<...>` sinks). The + main-module namespace override is therefore one knob with two ends: + `set_aot_main_module_name` writes the daslib global AND forwards to + `set_aot_main_module_name_cpp`; a spelling rule changed on one side only produces a TU + where the definition and its debug-info references disagree. + +## 6. aot_standalone + +- **The generated constructor IS the init protocol** - a standalone context never calls + `Context::runInitScript`, so the ctor reproduces its observable semantics inline: + `memset(context.globals, 0, context.getGlobalSize())` mirrors runInitScript's globals + memset (`src/runtime/context.cpp`), then the same-TU `__init_script(&context, true)`, + then each `[init]` function directly. `aot_cpp.das`'s `preVisitGlobalLet` writes the + `__init_script` signature - a daslib-to-daslib pairing nothing checks. +- **`__init_shared` is hardcoded `true`** - a fresh standalone context always owns its + shared globals. +- **The ctor's base `Context(N)` folds the init headroom in, because there is no separate + init stack** - init locals are C++ locals in AOT. A script that sets `options stack` to a + positive value gets `N = options stack + globalInitStackSize`, honored exactly, the + interpreter's floor included: the floor protects INTERPRETED init, while standalone init + is AOT, so an embedded target's stack budget wins. Zero and negative fall through to + `N = max(policies.stack, 16384) + globalInitStackSize`; `Context(0)` owns no stack, so any + prologue would assert. The fall-through arm mirrors the interpreter's init-stack formula + in `src/ast/ast_simulate.cpp` - a pair, deliberately divergent on the explicit-option arm. + `globalInitStackSize` reaches the emitter through the rtti `Program` binding in + `src/builtin/module_builtin_rtti.cpp`, so an `invoke` during init still has das stack for + its prologue. +- **There is no `!stopFlags` guard between `[init]` calls** - a panic propagates out of the + ctor instead of soft-stopping the sequence. +- **Global init order is fixed by `var->index` assignment**: `StandaloneContextGen`'s + `preVisitGlobalLet` emits required modules' globals via ordered `for_each_module` + before the adapter walks the entry module's own - correct only because + `ast_allocate_stack.cpp` assigns `var->index` through the same dependency-first + module order with the entry module last, and `runInitScript` executes ascending + index. `[init]` function order is not re-derived at all: the emitter reads the + simulated context's list through rtti `for_each_init_function`, so the C++ late-init + sort stays the single source of truth. +- **Cross-module limits fail loud at emit time**: only main-module, AOT-emitted `[init]` + functions can be called from the ctor (required-module and `[no_aot]` ones are collected + emit errors with the reason), because the standalone TU only emits the entry module's + function bodies. +- **Every used function must have an AOT body** - a standalone context has no + interpreter, so a used `noAot` function (the `[no_aot]` annotation, or `NoAotMarker` + finding a type AOT cannot express) is a collected emit error, never a + `fnByMangledName` call that would crash at runtime. `prepareProgramForEmission` runs + `NoAotMarker` first (the regular AOT paths run it too; standalone must match) and + then `checkAllUsedFunctionsCanAot` walks used, non-builtin functions. +- **Type definitions live in the header, once** - struct/enum definitions (the + dependency dump plus the entry module's own `declarations` capture) are emitted into + the `.das.h`, which the `.das.cpp` includes; the source never redefines them. They sit + in module-named namespaces directly under `das`, NOT under the per-program + `_anon_` namespace - that hash changes with the program, so it can never be a + public name. The entry module is promoted with an empty name, which would put its + types in an anonymous namespace (a distinct type per TU - an ODR/link break for any + struct in an exported signature); `aot_cpp`'s `set_aot_main_module_name` names it + after the context for the duration of standalone emission, so class signatures, the + source TU, and the embedder all resolve the same `das::::T`. The header carries + `#pragma once` and the required modules' `aotRequire` includes so it stands alone in + an embedder TU; two DIFFERENT contexts' headers sharing a das dependency still cannot + be included in one TU (the shared types have no per-type guards). + +## 7. flatten + +- **Predicated lowering carries one live-mask per exit flavor** - `__flat_live` for + return, a per-loop break mask (persists across unrolled copies) and continue mask + (re-minted per copy). A write's predicate ANDs every active mask plus the structural + predicate; a narrow term excludes its own mask so it self-cancels. An inlined callee + gets a fresh live mask and lowers with `ctx.loopMasks` moved OUT, so its break/continue + can never reach the caller's loops. +- **`flatten_preshade_cse` is a joint fixpoint, not a pipeline** - extraction, regroup, + CSE and alias elimination mutually enable each other; the `_preshader_`/`_cse_` counters + are owned by that loop and re-seeded from surviving suffixes (per-call numbering + re-mints a live name). +- **A CSE/regroup tally counts exactly the regions its rewrite can change** - a duplicate + counted where the rewrite cannot reach never drops below 2 and runs the fixpoint to its + iteration cap. +- **`__flat_ret` carries `safeWhenUninitialized` only while every write is a + self-referential select** - a lowering change that makes the bare-decl read observable + turns the flag into a real uninitialized read. +- **CSE is local value numbering over one converged basic block, and it is complete** - + pure subtrees keyed by `describe()`; value-stability = reads no reassigned name; a store + through index/field/swizzle destabilizes its base; an unrecognized node fails closed as + mutable-reading. Uniform duplicates route to the preshader. +- **The copy-prop/CSE walks stay O(size)** - one name-to-statement index, one structural + walk. A `string` materialized per `ExprVar` in a visitor callback breaks that: each + `describe()` allocates a string that lives to the end of the pass, so the walk goes + quadratic in heap bytes, not only in time. +- **`MutCollect` is what CSE trusts to say whether a name is stable, so it counts every + store spelling, not the one the lowering emits.** CSE treats a name outside its set as + constant for the whole block; a missed store is a shared subexpression across a mutation. + Copies are only the visible half - `<-` also zeroes its SOURCE, `:=` lowers to a + `builtin`clone`(dst, src)` CALL rather than an `ExprClone`, `++`/`+=` are their own + nodes, and a by-reference + argument writes with no assignment node anywhere. Hence the argument arm keys on the + callee's parameter type (non-const and `ref` or a ref type), not on a node kind. +- **`delete` on a container of `ExpressionPtr` frees the BUFFER, never the nodes** - + `delete array` frees the pointees only for das-heap `T`, and `Expression` is a + handled C++ type whose instances are not heap chunks at all (the + measurement: an `array` of das structs returns its pointees to `heap_bytes_allocated`, + an `array` returns only the buffer and the nodes surface in the exit GC + report). That is why `make_float_ctor`'s const-fold early return may leave its lanes + un-consumed while the ctor path `emplace`s them away, why every `unsafe { delete args }` + after it is sound over borrowed tree nodes, and why a struct field holding a borrowed + node needs no `@do_not_delete`. Node lifetime belongs to the AST GC: a lane the const + fold drops is unreachable and collected at the enclosing `ast_gc_guard`. +- **The whitelist admits value-returning primitives only.** `lower_stmt`'s fall-through arm + lowers an unrecognized statement for its lifted sub-lets and drops the statement itself, + which is correct exactly while every surviving call is pure - so `lift_expr` refuses a + whitelisted call that writes through a by-reference argument (`sincos`) rather than let + the drop delete the store. Predicating such a write would need per-out-param temps the + lowering does not own. + +## 28. shader_block_layout + +- **Two rails, deliberately separate** - the LAYOUT rail admits int64/uint64 as block + members (`compute_block_layout` special-cases them) while the ARITHMETIC rail rejects + 64-bit INT (`arith_width_ok` allows width 64 only for floats); `cpu_only_lattice_width` + keys both emitters' fail-closed diagnostic. + +## 29. shader_lingua_franca + +- **Every symbol is either an exact CPU mirror of its GPU semantics or a `[sideeffects]` + dummy every rail lowers by name** - the dummies return zero on the host, so a CPU replay + reproduces GPU semantics only for the real-bodied set. Unsigned overloads never fold into + signed twins (glslang picks the unsigned opcode). diff --git a/daslib/ARCHITECTURE_LINQ.md b/daslib/ARCHITECTURE_LINQ.md new file mode 100644 index 0000000000..63a24a98a9 --- /dev/null +++ b/daslib/ARCHITECTURE_LINQ.md @@ -0,0 +1,246 @@ +# daslib architecture notes - the linq and sql families + +Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across the family. + +## 11. linq + +- **`top_n_by_with_cmp` exists for the fold splice and has no daslib call site.** When an + order-by key body is pure and inlineable, `linq_fold_common` emits it with that body + spliced into BOTH sides of the comparator - killing the per-comparison comparator dispatch + and the per-side `key(v)` dispatch the key-lambda overloads pay. It is not a redundant + twin of `top_n_by`. + +## 12. linq_boost + +- **`BucketLambdaStamper` exists because a bucket-surface lambda cannot infer on its own.** + On the `group_by_lazy` element shape `tuple>`, an untyped lambda in + `._1 |> select/sum/...()` has nothing to bind against in the fully generic + tier-2 parameters (error 30303); the chain type knows `E`, so the macro stamps the + lambda's parameter before the rewrite - the same move as the outer-parameter injection in + `visit()`. +- **`result_param1_type` is the join result-lambda auto-typing hook.** param0 is always TA; + param1 is whatever the override returns - `_join` -> TB, `_group_join` -> a bare `array` + with const/ref cleared to match the bucket type. `_left_join` / `_right_join` / + `_full_outer_join` deliberately do not override: their result parameter is `$Option` on + one or both sides, so those result lambdas stay user-spelled. + +## 13. linq_das + +- **The substituting scanners share one token model.** `substitute_idents`, + `mentions_ident` and `rewrite_group_var` all read it: plain `"..."` content is verbatim, a + `{...}` interpolation body is code that is scanned and substituted, one level of nested + string literal inside an interpolation is verbatim again, and identifier position + classifies the same way everywhere - after-`.` field access and before-single-`=` labels + are not references, with `is_label_position` the shared judge. +- **`JOIN_TI` is a spellable identifier on purpose.** `__`-prefixed names are reserved by + the compiler, so the transparent-identifier carrier cannot be hidden outside the user's + namespace and must stay a single plain token - hence the explicit rejection of a range + variable named `linq_join_ti` instead of an unspellable name. +- **`build_src`'s two arms are a fusion decision, not a formatting one.** A fused source + (chain head, join) is built as `from_in(...)` so `_fold` can route it; an UNFUSED operator + argument - the uncorrelated multiple-`from`, whose `_cross_join` runs at tier-3 + passthrough - is passed as the bare parenthesized expression, so `each`'s + `[unsafe_outside_of_for]` does not trip and the arrayxarray overload is selected. +- **The correlated-flatten collection selector borrows, it never copies.** `unsafe(each(tail))` + is chosen over `to_sequence` (whose const-array overload clones per row) and over + `where_`'s array overload (which materializes per row). Borrowing a TEMPORARY tail + (`from x in [c.id]`, a function result) is safe because daslang heap-allocates arrays and + does not finalize them at scope exit - the backing storage outlives the borrow and is + reclaimed by GC. +- **A `FromInMacro` reject returns null so infer can stabilize.** Errors clear on every pass, + so the not-yet-inferred-source arm's error survives only when the source never infers at + all - that is what makes that arm a deferral rather than a final verdict. +- **`from_in` identifies typed sources by matching module + type NAME as strings**, not by + requiring the provider modules: `sqlite_boost::SqlRunner` -> `select_from`, pugixml + `xml_node` -> `from_xml_node`, `json::JsonValue?` -> `from_json` - linq_das stays + dependency-free on sqlite/pugixml/json. Both node builders are `[unsafe_outside_of_for]`, + so the emitted call carries the `unsafe` wrap a `_fold` chain requires. A table source is + rejected with a fix message before those arms: it already carries its row shape, and the + typed builders would all mis-fire. + +## 14. linq_fold_common + +- **group_by's per-key update is a dummy plus an address compare, not a `key_exists` + probe.** `entry &= tab?[uk] ?? dummy` costs one hash op per element on hits; + `addr(entry) == addr(dummy)` identifies the miss, which then fills the dummy, inserts it, + and re-defaults it - the second hash op is paid on misses only. Average slots hold a + `(sum, count)` accumulator and divide with no zero guard: sound because a key enters the + table only on its first element. +- **`hoist_prelude` hand-builds a flat block on purpose.** The materializer prelude - the + hoisted default-row declaration on the field-prune path - must be a SIBLING of the walk; a + `$b(prelude)` splice nests it in its own scope, where the per-element reads cannot see it. +- **One `at` can reach two materializers.** A `group_by` chain closed by + `first`/`order`/`distinct`/`take` emits through both the array-walk materializer and the + handle materializer at the same source location, so `qn` - which keys on (prefix, at) - + keeps the two sites' binds apart only while each site carries its own prefix. +- **An `ExprFor` the typer has not re-inferred carries an EMPTY `iteratorVariables`.** A + decs bridge feeding a no-chain terminator (`from_decs_template(...).count()`) has no chain + op to force the second inference pass, so the iterator names are recovered from the push + tuple's values - each references its iter var by name under an `ExprRef2Value` wrap - and + match what the loop binds once a later pass infers it. + +## 15. linq_fold_sql + +- **`extract_sql_source` walks the call's own `arguments[0]` spine, not linq_fold's + flattened `top`** - the SQL DSL ops are not `linqCalls`, so `flatten_linq` stops at the + first one and never reaches the `select_from` source. `sql_linq_loaded(prog)` then gates + on `_sql` being in THIS program: a consumer can get `select_from` from a provider boost + without `daslib/sql_linq`, and that chain must fall through to the in-memory path. + +## 16. linq_fold_decs + +- **Column pruning is a four-step pipeline over the `decs_tup` bind, and every step exists + to reach bind elision.** Scan the body for `decs_tup.` uses; if the bind is + referenced as a WHOLE var, rewrite each whole-var use into a synthesized + `(userName1 = iter1, ...)` literal - semantically identical - and re-scan; then flatten the + surviving field reads to the iter vars and drop the bind entirely. Unused `get_ro` slots + disappear along with one tuple-make plus N field reads per iteration. The unpruned bind + stays only for whole-var refs that survive the rewrite. +- **The decs random-index fast paths rest on these source facts.** A plain `[decs_template]` + field compiles to `get_ro` (an indexable array) while a default-init field compiles to + `get_default_ro` - an iterator, not indexable - so `decs_can_random_index` bails and the + walk stands. `get_ro` is `[unsafe_outside_of_for]`, so each cloned source AND the index + read are marked `alwaysSafe` (mirroring `decs_boost`'s `append_index_lookup`). + `for_each_archetype` visits archetypes in order and skips empty ones, so a no-predicate + `last()` overwriting a result per archetype ends holding the global last in + O(num_archetypes), and `reverse |> take(N)` collects only the tail with the boundary + archetype's head indexed past rather than walked. +- **Trivial-select elision removes the copy a `_select(_.userName)` would emit.** The + pruned inner-for already keeps that component's iter var, so the synthetic bind would + flatten to a pure copy; renaming the chain's `finalBind` to the iter var skips the binding + entirely. Detecting the shape needs TWO `ExprRef2Value` peels: the typer wraps both the + projection root and the `ExprVar` inside the `ExprField`. +- **The decs join hook keeps hash-collect and probe inline rather than routing through + `wrap_source_loop`** - that is what preserves the count-no-`where` fast path, which + answers from bucket lengths instead of walking pairs. Its key lambdas are synthesized + upstream by the LinqJoin macro as 1-arg single-return blocks, so the BLOCK's return type + is the key type; a change to that synthesized shape breaks the decs join silently. + +## 17. sql_linq + +- **Equal phase numbers are deliberate.** `PHASE_ORDER_BY == PHASE_SELECT` - ORDER BY + commutes with the projection in SQL (it may reference projected aliases or source + columns) - and `PHASE_GROUP_BY == PHASE_DISTINCT == PHASE_SET_OP`. Giving either member + its own number spuriously diverts the canonical `_group_by |> _order_by |> _select` into + a nested SELECT. +- **Emitted SQL is a template over two in-band markers.** `?` is a bind, `\x01` a runtime + identifier (a dynamic `_order_by` column name); `next_placeholder` scans outside + single-quoted literals and always resumes just past a placeholder, so the scan starts + with no quote open. `sql_to_frags_ex` pairs each marker with `orderedBinds` / + `orderedInlineIds` in occurrence order and preserves a marker as text once its list is + exhausted. Nothing reaches the text unquoted: a compile-time constant folds in with SQL + doubling, a non-constant emits a runtime `sql_quote_id` / `sql_quote_lit` call, and every + other runtime value stays a `?`. +- **Key contexts render with constants inlined because their fragment is re-used.** A + computed `_group_by` / `_order_by` key fragment is emitted at several SQL positions + (SELECT, GROUP BY, ORDER BY) while a placeholder can be bound only once, so + `render_inlined_key_sql` sets `q.inlineConstants`, admits only `ExprConst*` (strings + through `sql_quote_lit`), and rejects any key that pushed a bind. +- **`_distinct_by(K)` is dialect-routed on `caps.distinct_on`.** Without it, SQLite's + bare-aggregate form `SELECT *, MIN/MAX(pk) FROM t GROUP BY K` keeps the `*` columns of + the min-pk (or max-pk) row per K - SQLite-only semantics; strict engines reject bare + columns beside GROUP BY. With it, `SELECT DISTINCT ON (K) * ... ORDER BY K, pk [DESC]` - a + PostgreSQL extension DuckDB also implements. MIN/ASC is linq's "first row per K", MAX/DESC + is `reverse() |> _distinct_by(K)` "last row per K" - both only while pk is monotonic with + insertion order. +- **A join's `into` projection registry is snapshotted because the outer projection + clobbers the live one.** `process_join_call` copies the live projection into + `joinProjRecordNames` + the `joinSelectCol*` arrays, which survive + `analyze_grouped_projection`'s clear-and-repopulate. Post-join `_.` resolves + through that snapshot in every post-join consumer - `pred_to_sql`'s column-ref arm, + `collect_one_order_key`, `push_group_key`, `try_translate_group_aggregate`, + `try_translate_global_aggregate` - all through `find_projection_alias` + + `render_projection_alias_sql`. A registry miss is rejected loudly everywhere alike: falling + back to base-table resolution would leak the unqualified base-table namespace into post-join + predicates. +- **`normalize_single_source_arg_names` exists for the linq_das front end.** `_sql` + resolves a single source against the placeholder `_`, but LINQ-syntax lowering splices + the user's range variable verbatim (`$(c) => c.field`), so a single-parameter chain-op + lambda's bound variable is renamed to `_`. Gated on a pure single-source chain - join and + set-op key lambdas bind distinct sources and are left alone. + +## 33. sql_migrate + +- **The audit table is provider-neutral by construction** (client-side epoch seconds, + BIGINT); duplicate versions are caught in two layers because neither sees everything; the + `struct_convert_field` overload set is a specificity ladder - deleting a "duplicate" + silently re-routes conversions. + +## 37. linq_fold module layout and the SourceAdapter contract + +- **Charters.** `linq_fold` is dispatch only: source recognition (`try_splice_patterns`), + adapter construction, the tier-2 `fold_linq_default` cascade, the `LinqFold` call macro, + and the single registrar `register_all_linq_fold_rows`. `linq_fold_common` holds what no + source can influence - the pattern-table types, the walker, the predicate library, the + chain pre-passes, the generic emit lanes, the `splice_patterns` registry, and the abstract + `SourceAdapter`. Each remaining file owns one source: `linq_fold_array` (array, zip, + array-join), `linq_fold_decs`, `linq_fold_json`, `linq_fold_table`, and pugixml's + `linq_fold_xml`. `linq_fold_sql` is the exception - it recognizes a `[sql_table]` source + and hands the whole chain to the `_sql` macro, so it has no adapter and no emit. A source + file requires `linq_fold_common` and never a sibling source. +- **`try_splice_patterns` tries recognizers in a fixed order** - sql, decs, xml (behind the + pugixml `static_if`), json, table - and the array arm runs last with no recognizer of its + own: it is what claims a chain nobody else claimed. +- **The adapter contract is four abstract methods; everything else on the base has a + default.** `bind_name`, `element_type`, `wrap_source_loop` and `wrap_invoke` are abstract, + so every source answers them. The per-operation hooks (`emit_loop_or_count`, + `emit_reverse_skip_into_tail`, `emit_reverse_last_backward`, `emit_distinct_take_loop`, + `build_group_by_adapter`, `emit_join_hook`) return null on the base, and a null sends the + caller down its generic path. The capability methods (`can_group_by`, `can_join`, + `can_reserve_by_length`, `has_own_loop_or_count_lane`, `supports_direct_return`, + `effective_dispatch`, `defers_materialization`, `count_shortcut`, `invoke_param_type`, + `name_prefix`) are the source answering questions about itself. daslang classes have no + `is`/`as` downcast, so an emit fn holding a `SourceAdapter?` reaches source-specific + behavior only through these methods - there is no place to write a per-source branch, and + adding a source needs no edit to a central enum or switch. +- **The capability defaults are fail-closed, and `supports_direct_return` is the one that + matters.** A wrong `true` emits a mid-loop `return` that escapes a nested-callback walk + (decs's `for_each_archetype`); a wrong `false` only costs the slower state-var + + find-stop + tail path. +- **`loop_source_expr` / `loop_source_name` is what lights up the array-shaped lanes.** The + shared lanes spell their loop and their `length(...)` reads with the name, and read the + expr for compile-time facts - the `type_has_length` reserve gate and the join's srcA + element-type check. Array and table override both, xml and json override the name only, + decs overrides neither. +- **`ProjectedSourceAdapter` absorbs a leading source projection.** When a row captures the + optional `srcsel` slot (`source |> _select(f) |> order/distinct/take`), the dispatcher + wraps the real adapter in this decorator: it binds `projName = f(rawElem)` on top of the + per-element body and delegates loop and invoke to the inner adapter, so every emit sees the + projected element unchanged. It leaves `loop_source_expr` / `loop_source_name` / + `can_reserve_by_length` at the base defaults, which disables the source-direct fast paths - + those would bypass the projection. +- **Rows are per operator family, not per source.** The seven `build_*_rows` fns + (order_family, loop_or_count, reverse, distinct, group_by, join, zip) emplace 17 rows into + the one `splice_patterns` table; only zip's builder lives outside `linq_fold_common`. + Source gating happens inside a row - through a `RequiresPredicate` that asks the adapter + (`can_group_by_source`, `can_join_source`, `array_source`, `non_array_source`) or through + a hook that returns null. Registration order is match priority: the dispatcher walks the + table top to bottom and the first row whose chain matches and whose emit returns non-null + wins, so more specific chains are declared first. A row whose slot chain is a strict prefix + of a later row's shadows that row (`chain_prefix_of`). +- **The walker consumes the whole call list or fails.** `match_pattern` walks the flattened + calls left to right against the row's slots: `c_one` must match and advances both cursors, + `c_opt` advances both on a match and skips the slot otherwise, `c_chain` matches greedily + while the call is in the slot's name set and always succeeds - including with zero matches, + which still creates the (empty) `captures.many` entry, so an emit fn can rely on the key + existing. Any call left unconsumed after the last slot is a no-match. The `requires` + predicates run last, against the populated captures. +- **`Captures.single_name` exists because a pre-pass renames the op without touching the + AST.** `normalize_order_reverse` folds a trailing `reverse` into the order op by swapping + the tuple's `LinqCall` record, leaving the `ExprCall` alone; deriving the name back from + the `ExprCall` (`call_norm_name`) would silently undo the swap. The walker therefore + records the `LinqCall.name` at match time, and any emit fn that cares about the + post-normalize name reads `single_name`. +- **`alias_table` names the op-name groups a `m_alias` slot admits, and a missing name fails + silently.** A terminator the lanes handle but `loop_terminator_family` omits never matches + the terminator slot, so the chain leaves the splice and lands on the tier-2 cascade with no + diagnostic - the symptom is a slow chain, not an error. +- **Predicates and emit fns are named module-level functions, referenced as + `@@ name` / `@@ name`.** An anonymous `@@(...)` lambda compiles + to a `_localfunction_*` symbol the LLVM JIT pass cannot resolve; a named function has a + stable address. +- **The table lane's key and value binds are asymmetric on purpose.** `keys(tab)` yields + non-const elements (writable temp copies), so the key lanes rebind the loop var through a + `let` before the body sees it; `values(tab)` over the const table param already yields + `V& const` and binds directly. Keys are workhorse types, so the copy is free, and + downstream `==const` composition (a `push_clone` of a bare projected key) needs the const. diff --git a/daslib/ARCHITECTURE_LINT.md b/daslib/ARCHITECTURE_LINT.md new file mode 100644 index 0000000000..a126497025 --- /dev/null +++ b/daslib/ARCHITECTURE_LINT.md @@ -0,0 +1,204 @@ +# daslib architecture notes - the lint family + +Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across the family. + +## 1. perf_lint + +- **Two parallel loop spaces.** `loop_depth`/`in_closure` (+ `VarStackEntry.depth`) is the + contract for every loop rule: any closure body is deferred, hence not "in" the loop. + PERF031 needs the opposite for inline block arguments (`peek_data(s) $(d) { ... }` runs + in place), so `inline_aware_loop_depth`/`in_deferred` (+ `inline_aware_depth`) count + loops inside inline block-argument bodies as real loop levels and defer only lambdas, + local functions, generators, and generated blocks. `inline_block_entry_depth` records + the loop depth at each inline-block entry because a `return` there unwinds only to that + depth: loops at or below it keep re-invoking the block, so only barrier 0 proves the + body runs once. +- **Closure-guard placement is load-bearing, per rule.** Four idioms coexist - + `in_closure > 0` early return, `in_closure == 0` gate, `in_deferred == 0` gate, no guard - + and the guard's POSITION inside a visitor method encodes which rules are meaningful + inside a lambda. PERF001 deliberately has no guard: `loop_depth` never advances inside + closures, so its loop check is already scoped; syntactic rules (PERF003, PERF021) fire + anywhere because the pattern is wrong regardless of where it runs. Moving a guard by two + statements changes which rules fire. +- **`perf024_skip_iptr` is a one-slot handshake** between `preVisitExprCallArgument` + (classifies an annotated-arg use as safe, stashes the variable's identity) and the next + `preVisitExprVar` (consumes the stash instead of disqualifying). It is sound because the + classifier accepts only a bare `ExprVar` after one `ExprRef2Value` peel, so the very + next variable the walker reaches is that argument's own. Widening the peel set breaks + the adjacency and mis-applies the stash. +- **`add_ptr_ref` is the qmacro-splice marker.** PERF023 rests on `apply_qrules` wrapping + every `$e(...)` tag in `add_ptr_ref(...)`; matching the name suffices because the + templates_boost wrapper is its only producer post-expansion. Its depth tracking fires + before the `in_closure` gate - the rules block is itself a closure. +- **Hot-path walk (PERF026-028).** `HotCache` keeps per-function scan results sliced into + two flat arrays; `sinks_span_by_fn` doubles as the presence key and the two tables are + strictly co-populated (one insert site each, adjacent). The visited set keys on + (callee hash, `nearest_root_file_site` pointer): reaching one callee from two root-file + call sites must report twice, so a function is re-walked once per distinct anchor - + a cost taken for per-call-site diagnostics. Generated/generic frames re-anchor to the + caller side because their own `at` points into the template. A call to any function + with a `@scratch` parameter prunes that callee's whole subtree from the walk - the prune + is deliberately TRANSITIVE, removing the callee's whole subtree and not just the sink, + because narrowing the prune to the sink buries scratch-heavy callers in deep-resize + findings on buffers the helper family sizes correctly. +- **PERF030 release credits are consume-once**: a `delete x` licenses exactly one + subsequent `x <- ...`, then the warning re-arms; the lowered delete family is matched + by name in its four spellings (`_::finalize`, `finalize`, ``builtin`finalize``, + ``builtin`finalize_dim``). +- **`"$"` in module allowlists is the builtin module's name** - `Module_BuiltIn` registers + as `Module("$")` (`src/builtin/module_builtin.cpp`), so macro-time reflection reports + core builtins under `"$"` beside `"builtin"` and `"strings"`. +- **`Function.tempStringResult` marks a FRESH per-call string allocation** - never a view + or a passthrough (`include/daScript/ast/ast.h`). A temp-string builtin (`to_char`, + `repeat`) on a hot path IS heap traffic; the PERF026 string-returning catch-all + deliberately has no temporary exemption. PERF020's `_type.flags.temporary` check is a + different axis: a `#`-typed VALUE being cast, where `string(x#)` is a load-bearing + temporary-to-permanent clone. +- **PERF021's tail-argument comparison passes `require_pure = false`** - workhorse-cast + tails are flags and literals after the injected Fake* skips, so purity has nothing to + protect there; the default protects rules that collapse duplicated subexpressions from + changing evaluation counts. +- **`report_key` bit layout**: line in the low 20 bits, column shifted above it, XOR'd with + the FNV-64-prime-mixed `fileInfo` pointer and the rule-code hash. Lines past 2^20 alias - + accepted; a collision silently drops one warning. +- **`parse_range_leg` operator polarity**: `upper_op` is the var-on-left operator marking + the upper-bound leg (`<=` closed-in-range, `>` open-out-of-range), `lower_op` the lower + (`>=` / `<`). Yoda (const-on-left) forms flip, so the const-on-left arm tests against + `lower_op` - the `bounds_above` assignments differ between the two arms by design. +- **PERF022 is a three-function protocol over one stack.** `perf022_push_sentinel` pushes a + frame in `preVisitExprFor` unconditionally, so the stack stays lock-step with for-loop + nesting; `preVisitExprForBody` may flip the frame's `matched`; `perf022_pop_and_check` + fires when a frame survives matched-and-undisqualified - at the for-loop's `at`, the + actionable line a reader rewrites, not the push site. +- **A generic-body exemption exists exactly where the remedy is not instantiation-safe.** + PERF020's fix (delete the cast) is wrong for the sibling instantiations of the same + source line, so it bails on `fromGeneric`; PERF019/PERF021 rewrites (fuse under one + cast, hoist over the ternary) stay valid for every instantiation, so they fire + everywhere. + +## 2. lint_config + +- **`.lint_config` is parsed once per process.** `[lint_macro]` apply() fires once per + module in the require chain, so an uncached read re-parses the repo file N times per + compilation; the file is repo policy and cannot change mid-compilation, so + `ensure_cache_loaded` splits it once into off/on/path-exclude caches. +- **`LINT019` in a line's nolint list exempts the WHOLE line from stale-checking** - + chosen tradeoff: the in-tree escape hatch is `nolint:PERF006,LINT019` on lines whose + code fires only in downstream compiles; per-code vouching would need new syntax. + +## 3. lint + +- **`lint_error` reports only nodes whose lineinfo belongs to the current function's own + file.** An inliner splice or macro-manufactured node from another file gets its + diagnostics from that file's own compile; reporting it here would fail a module on + idioms it cannot edit. +- **LINT022 collects references from three sources because none alone sees them all**: + the ordinary walk (resolved pointers); a latched second walk over generic bodies + (`visit(prog)` skips generics, and uninferred bodies carry only spelled alias names); + and declarations plus string literals (signatures and name-based lookups leave only a + literal). Self-reference is pre-inserted so it cannot keep a private type alive. + Functions are checked only when inlining is off - inlining splices small private + callees away and leaves a body whose only callers were inlined away, indistinguishable + from dead code. Public symbols, lifecycle/macro hooks, virtual methods, and generic + bodies are skipped: their callers are invisible to lint. +- **Argument rules skip what the author does not own, and converge leaf-first.** + LINT012/013/014/023 stay silent on class methods, `[extern]` stubs, `finalize` + overloads, and address-taken functions; LINT014 also skips a parameter whose mutability + a callee demands - the leaf is flagged first and each fix re-exposes the next caller. + A returned `var` argument keeps its `var` (a non-copyable result cannot move from + const); a used `_name` parameter is never LINT004'd (interface and keyword-clash names). +- **Closure bodies are per-rule, not global.** LINT010 counts a closure body as a branch + (it may run later or never - writes inside must not kill outside stores, reads inside + must not keep an outside init live); LINT021 counts the same body as an escape - a + captured 64-bit local is never a narrow. +- **`expr_equal` compares `describe(a) == describe(b)`, never pointers** - every + Expression has exactly one parent, so sibling pointer equality is effectively never + true. LINT007 excludes `ExprConst*` operands: identical constants are macro-splice + residue, not copy-paste typos. +- **LINT015 fires on `+` and `-` only** - the only operators both binary and unary, so a + split continuation line re-parses as a valid discarded unary statement; every other + operator produces a loud parse error instead. +- **LINT023's store set extends the compiler's.** `under_clone` marks only bare-variable + `=`/`:=` targets; `<-`, swizzle LHS, and statement-level `++`/`--` are marked by the + visitor. A pointer parameter stays a candidate: a write through the pointee reaches the + caller and counts as a use, not a store. +- **The 64-bit family fires only where the overflow hazard survives.** LINT021 treats + only unbounded-top math as transparent (`min`/`clamp`/`sign` cap the value - deliberate + saturation); LINT024 requires a call among the product's leaves - that is the unbounded + factor, a product of plain locals is tile geometry. +- **`canVisitWithAliasSubexpression` returns false on purpose** - `assume` keeps the + template block at the definition site and clones it per use; the block is never invoked + there, so descending would report its arguments unused. +- **C++ mirror pairs**: `lint022_optimized` / `lint022_calls_may_be_inlined` <-> + `Program::getOptimize` / `Program::patchInline`; `stale_scan_line` <-> + `rtti_is_nolint_suppressed`; `is_inline_temp_name` <-> `INLINE_TEMP_PREFIX`. Nothing + fails when one side moves alone. + +## 4. style_lint + +- **Lint runs under policies that skip const-folding and leave some types unset.** Take a + variable's type from `Variable._type`, not an `ExprVar`'s; expect `T('c')` to stay an + `ExprCall`; expect `-1` as an `ExprOp1` over a constant; when a type is missing, bias + conservative ("needs unsafe"). +- **Rules whose forms are identical in the AST read the source span instead** - `defer <|`, + generators, STYLE003's `$()`, STYLE028's three invoke spellings. STYLE005 uses LineInfo + identity: the parser gives braceless and postfix forms a synthetic block sharing the + terminator's LineInfo, so `blk.at != inner.at` <=> the user wrote braces. Both of these + tests - the source-span read and the LineInfo identity check - are the rule's only + evidence for the distinction, and both break silently when lowering or LineInfo + assignment changes. +- **STYLE024/025 mirror the compiler's unsafe rules with a deliberate asymmetry: in doubt, + mark "needs unsafe".** A false "needs" costs a missed note; a false "redundant" tells + the user to delete a wrap the compiler requires. `let v & = unsafe(expr)` is the one + recorded exception - the wrap is required by the reference binding, not the inner + expression. STYLE025 narrows only when the single statement has an expression form. +- **A require counts as used when its symbols cannot be traced, not only when seen.** + Generic instances attribute to `fromGeneric._module`; bare names in never-instantiated + generic bodies keep their providers alive; re-export chains are followed; a module + providing any compile-time macro or lifecycle hook is used - it leaves no AST reference. + STYLE029 fires only when exactly one re-exported module is used: two or more is an + aggregation facade. +- **Metric rules attribute code to the module that can fix it.** STYLE037/038 walk the + `fromGeneric` chain to its root (`[template]` generics instantiate INTO the caller); + generated carriers stay unmetered; closures are not length-checked separately - the + host trips first. +- **STYLE041 is sound because of a reference count, not the structural scan.** The scan + classifies writes, plumb reads, and the single consuming `if`, arming the candidate with + that count; `preVisitExprVar` counts every reference in the function, and the warning + fires only when the counts match. Anything the scan does not descend into therefore + rejects the candidate instead of being assumed harmless. +- **STYLE036 keys on a flag the compiler clears.** Infer strips a substitution contract + when a generic binds; one still set on a resolved cast target was never consumed - but + an `auto`/unresolved-alias target has simply not been substituted yet, hence the + `isAutoOrAlias` bail. +- **STYLE039 requires non-ASCII in both the value and its source span** - escape-spelled + bytes in ASCII source are log-safe, and span-slicing keeps a neighbouring comment's + typography from firing. +- **The unsafe-frame walk**: `preVisitExpression` pushes one frame per expression node, + `visitExpression` pops it - the popped frame is the subtree's summary, stored in + `unsafeExprs` for `visitExprUnsafe` (STYLE024 vs STYLE025), propagated to the parent + slot, and fired as STYLE024 when an `unsafe(...)` target's subtree had count 0. The + balance panic in each entry point is the guard against a `canVisit*` override skipping a + subtree the frame walk still counts. +- **Two compiler regimes run this visitor.** The `[lint_macro]` path runs inside a normal + compile (const-folding ON - `T('c')` folds to a typed const); the standalone runner and + MCP path run under lint policies (folding OFF - `T('c')` stays an `ExprCall`, `-1` stays + an `ExprOp1`). Shape-sensitive rules carry both arms (`numeric_char_cast_target`'s + ExprCall branch + source-span re-parse); deleting the "redundant" second arm kills the + rule in the other regime. +- **`expr.canShadow` on a for-variable exempts it from LINT002/003/004** - the flag marks + macro-generated loops, and lint does not judge names the user never spelled. +- **The `expect` probe scans the whole file while `lint-skip-file` is header-capped** - + an `expect` directive anywhere makes the file a compile-error fixture (its position is + semantic to dastest), whereas `lint-skip-file` deeper than the header would let quoted + prose unlint a file. +- **The `blk.list` index rescans are O(block^2) by choice** - the per-statement visitor + callback carries no index, and a shared index stack would have to push/pop with nested + blocks; block statement lists are short. +- **Threshold provenance**: STYLE038's 80 ~ p97 of this tree's function lengths; STYLE041 + caps report-arg recursion at 8 and fails closed (deeper is unknown shape); + `LINT_SKIP_HEADER_LINES = 16` exists so prose quoting the directive cannot unlint a + file; STYLE037's 20 and `note_type`'s depth 24 have no recorded derivation. +- **C++ mirror pairs**: the STYLE024/025 unsafe map <-> infer's `unsafe_*` rules; + `style036_inert_contract` <-> infer's contract clearing. Nothing fails when one side + moves alone. diff --git a/daslib/REVIEW.md b/daslib/REVIEW.md index 69f61521dc..f03a67a46c 100644 --- a/daslib/REVIEW.md +++ b/daslib/REVIEW.md @@ -1,7 +1,8 @@ # daslib Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture doc: -`daslib/ARCHITECTURE.md`. Planned work: `plans/daslib_comment_sweep.md` (repo root). +`daslib/ARCHITECTURE.md` and its companions `ARCHITECTURE_LINT.md`, `ARCHITECTURE_EMIT.md`, +`ARCHITECTURE_LINQ.md`. Planned work: `plans/daslib_comment_sweep.md` (repo root). A diff touching the linq family - `linq*.das`, `sql_*.das` - applies `REVIEW_LINQ.md` too. **A diff that adds or changes a lint report path (`perf_warning` / `style_warning` / @@ -17,7 +18,7 @@ the actionable line is an intermediate call site. **A diff that moves an `in_closure` / `in_deferred` guard to a different statement position inside a lint visitor method is a defect.** The guard's position decides which rules apply -inside a lambda; the per-rule policy is in `ARCHITECTURE.md` sec. 1. +inside a lambda; the per-rule policy is in `ARCHITECTURE_LINT.md` sec. 1. **Never move `add_ptr_ref`'s depth tracking behind the `in_closure` gate.** The rules block is itself a closure, so a gated tracker never sees the splice. @@ -34,14 +35,14 @@ construct's visit. survives into the sibling loop's exit path and unbalances its counter. **A diff that makes a daslib predicate or emitted identifier depend on a C++-side -definition records the pair in `ARCHITECTURE.md`, in its module's section, naming both +definition records the pair in the architecture doc, in its module's section, naming both sides.** Nothing catches it when one side later moves alone. **When a diff changes one side of a recorded daslib/C++ pair so the two no longer match, -it changes the other side and updates the pair's `ARCHITECTURE.md` entry in the same +it changes the other side and updates the pair's architecture-doc entry in the same diff.** -**A comment-sweep diff adds an `ARCHITECTURE.md` entry only where no name, shape, or test +**A comment-sweep diff adds an architecture-doc entry only where no name, shape, or test can carry the fact.** **Weakening `tests/lint/test_nolint_suppression.das` is a defect** - it pins that a string @@ -251,3 +252,7 @@ the first appearance moves it, and skipping the clone gives one node two parents every place `rst.das` computes it - the stub pass (`generate_module_stubs`) and the documenting pass (`documents`) - keeping them byte-for-byte equal.** When the two diverge, the page prints a bare signature and the symbol re-stubs. + +**A diff that adds a numeric value form to the toml lexer routes it through `rewind_to_bare` +on a bare-key character.** Without the rewind, a bare key that starts like a number lexes as +a value. diff --git a/daslib/REVIEW_LINQ.md b/daslib/REVIEW_LINQ.md index 3ccabd89f8..f44af7d7a9 100644 --- a/daslib/REVIEW_LINQ.md +++ b/daslib/REVIEW_LINQ.md @@ -2,7 +2,7 @@ **A diff touching the linq family - `linq*.das`, `sql_*.das` - applies this checklist together with `REVIEW.md`.** `REVIEW_COMMON.md` (repo root) binds this file too. -Architecture doc: `ARCHITECTURE.md`. +Architecture doc: `ARCHITECTURE_LINQ.md`. **A fused emit that binds a terminator's default or compare argument lazily, more than once, or below the top of the generated invoke, every `*_or_default` decs lane included, is a defect** - @@ -104,6 +104,12 @@ array - `selectCols`, `selectColAliases`, `selectColSqlFragments`, `selectColTyp cover only part of the set and leave the rest to their caller; a partial push desyncs the SELECT list from the row builder, with no error. +**A diff that adds a post-join `_.` consumer to sql_linq resolves the alias through +`find_projection_alias` + `render_projection_alias_sql`, and rejects a miss rather than +falling back to resolving the name against the base table.** After a join the alias lives in +the join projection snapshot, so base-table resolution answers with an unqualified +base-table column - a wrong column, with no error. + **Never pick a projection slot's SQL by `q.seenJoin` - pick it by which of that slot's own entries is non-empty: SQL fragment first, then aliased column, then unqualified column.** @@ -122,13 +128,13 @@ helper it returns, so a swapped pair attaches index DDL to the wrong overload. **Never narrow or widen the linq_das clause-keyword test - a clause keyword is a whole word at bracket nesting depth zero, not preceded by `.` and not preceded by a `>` that tails `|>`, `=>` -or `->`.** Narrowing -lets an in-body aggregate parse as a `select` clause; widening the exclusion to any `>` hides a -clause keyword that legitimately follows a generic bracket or a comparison. +or `->`.** Narrowing lets an in-body aggregate parse as a `select` clause; widening the +exclusion to any `>` hides a clause keyword that legitimately follows a generic bracket or a +comparison. **A diff that changes the token model in one substituting linq_das scanner - `substitute_idents`, `mentions_ident`, `rewrite_group_var` - changes it in the others, in the same change** (the -shared model: `ARCHITECTURE.md` sec. 13). A model change in one scanner desyncs +shared model: `ARCHITECTURE_LINQ.md` sec. 13). A model change in one scanner desyncs `mentions_ident` from the rewrite it gates, and the emitter then renames a parameter the spliced projection still references. diff --git a/daslib/linq_fold_array.das b/daslib/linq_fold_array.das index 4cc77550cb..71c830f021 100644 --- a/daslib/linq_fold_array.das +++ b/daslib/linq_fold_array.das @@ -8,7 +8,7 @@ module linq_fold_array shared public //! linq_fold array source adapters: ArrayAdapter / ZipAdapter / ArrayJoinAdapter plus the //! array-shaped emit lanes (counter / accumulator / early-exit / loop-or-count / zip) and -//! their row-builders. Requires linq_fold_common. See daslib/ARCHITECTURE.md sec. 37. +//! their row-builders. Requires linq_fold_common. See daslib/ARCHITECTURE_LINQ.md sec. 37. require daslib/linq public require daslib/ast_boost diff --git a/daslib/linq_fold_common.das b/daslib/linq_fold_common.das index e939ed46cf..1d600b5be6 100644 --- a/daslib/linq_fold_common.das +++ b/daslib/linq_fold_common.das @@ -8,7 +8,7 @@ module linq_fold_common shared public //! linq_fold source-independent kernel: pattern-table types, walker, predicate library, //! chain transforms, generic emit lanes, and the abstract ``SourceAdapter`` base. Carries -//! the shared ``splice_patterns`` registry. See daslib/ARCHITECTURE.md sec. 37. +//! the shared ``splice_patterns`` registry. See daslib/ARCHITECTURE_LINQ.md sec. 37. require daslib/linq public require daslib/ast_boost diff --git a/daslib/linq_fold_decs.das b/daslib/linq_fold_decs.das index 7815cdf7b9..f0cc7029a0 100644 --- a/daslib/linq_fold_decs.das +++ b/daslib/linq_fold_decs.das @@ -8,7 +8,7 @@ module linq_fold_decs shared public //! linq_fold decs (ECS) source adapters: DecsAdapter / DecsJoinAdapter, the decs-bridge tuple //! scanners/rewriters, and the decs-specific emit hooks (count_shortcut, reverse-skip-into-tail, //! join, min_by/max_by). Terminators ride the generic lanes in linq_fold_common via DecsAdapter. -//! Requires linq_fold_common. See daslib/ARCHITECTURE.md sec. 37. +//! Requires linq_fold_common. See daslib/ARCHITECTURE_LINQ.md sec. 37. require daslib/linq public require daslib/ast_boost diff --git a/daslib/linq_fold_json.das b/daslib/linq_fold_json.das index 6690c95847..da5c58905c 100644 --- a/daslib/linq_fold_json.das +++ b/daslib/linq_fold_json.das @@ -18,7 +18,7 @@ module linq_fold_json shared public //! ``for_each_archetype``), so this module pulls in no json dependency. The field-pruning //! row-usage scanners (``RowUsageScanner`` / ``collect_row_usage`` / ``RowFieldFlattener`` / //! ``flatten_row_to_locals`` / ``hoist_prelude``) are shared with the XML and table adapters and -//! live in ``daslib/linq_fold_common.das``. See daslib/ARCHITECTURE.md sec. 37. +//! live in ``daslib/linq_fold_common.das``. See daslib/ARCHITECTURE_LINQ.md sec. 37. require daslib/ast_boost require daslib/ast_match diff --git a/daslib/linq_fold_sql.das b/daslib/linq_fold_sql.das index 54425e9b4e..1608d86a12 100644 --- a/daslib/linq_fold_sql.das +++ b/daslib/linq_fold_sql.das @@ -10,7 +10,7 @@ module linq_fold_sql shared public //! materialized full-table ``array`` in memory. There is no emitting adapter — ``_sql`` is a //! complete parallel lowering; this module only recognizes the source. Companion to //! ``daslib/linq_fold``; provider-neutral (matches the ``[sql_table]`` annotation, not a runner -//! type), so it loads unconditionally. See daslib/ARCHITECTURE.md sec. 37. +//! type), so it loads unconditionally. See daslib/ARCHITECTURE_LINQ.md sec. 37. require daslib/linq_fold_common public require daslib/ast_boost diff --git a/daslib/linq_fold_table.das b/daslib/linq_fold_table.das index 371a1e600c..c9f45048cc 100644 --- a/daslib/linq_fold_table.das +++ b/daslib/linq_fold_table.das @@ -15,7 +15,7 @@ module linq_fold_table shared public //! field-pruning); bare ``count()`` folds to O(1) ``length(tab)``; a plain ``distinct`` over raw keys/kv //! elements is dropped (keys are unique by construction). In-tree companion to ``daslib/linq_fold`` //! (required unconditionally; the matcher returns null for non-table chains). Emits ``keys`` / ``values`` -//! / the kv zip BY NAME at the user's splice site. See daslib/ARCHITECTURE.md sec. 37. +//! / the kv zip BY NAME at the user's splice site. See daslib/ARCHITECTURE_LINQ.md sec. 37. require daslib/ast_boost require daslib/ast_match diff --git a/daslib/lint_config.das b/daslib/lint_config.das index 051fed19e3..f5d24cba31 100644 --- a/daslib/lint_config.das +++ b/daslib/lint_config.das @@ -36,6 +36,10 @@ module lint_config shared private //! ``rule_docs_only = true`` marks a folder that carries rule documents only; //! ``rule_docs_only_at`` reads that one file, and the LINT025 pass in //! ``utils/lint/main.das`` reports any other ``.md`` sitting beside the code. +//! ``enforce_arch = true`` marks a folder whose ``.md`` anchors are all owed an +//! ``[arch]`` citation; ``enforce_arch_at`` reads it, and the LINT026 pass in +//! the same runner reports every anchor no code cites. Either tag also arms that +//! runner's LINT027 line gate over the folder's rule documents. require daslib/fio require daslib/json @@ -284,21 +288,34 @@ def private with_config_table(path, table_name : string; blk : block<(var tbl : } } -//! True when the ``.lint_config`` at `config_path` sets ``[docs] rule_docs_only = true`` — the folder -//! carries rule documents only (``REVIEW*.md`` / ``ARCHITECTURE*.md`` / ``LAWS.md``). A folder property, -//! NOT the ``[format]`` cascade: only that one file is read, and anything missing reads as false. -def public rule_docs_only_at(config_path : string) : bool { +// One bool key of the [docs] table of exactly `config_path` - a folder property, so no cascade +// and no inheritance. A missing file, missing table, missing key or non-bool value reads as false. +def private docs_flag_at(config_path, key : string) : bool { var result = false with_config_table(config_path, "docs") $(var docs_jv) { var docs & = unsafe(docs_jv.value as _object) - return if (!key_exists(docs, "rule_docs_only")) - var val_jv : JsonValue? = docs["rule_docs_only"] + return if (!key_exists(docs, key)) + var val_jv : JsonValue? = docs[key] return if (val_jv == null || !(val_jv.value is _bool)) result = val_jv.value as _bool } return result } +//! True when the ``.lint_config`` at `config_path` sets ``[docs] rule_docs_only = true`` — the folder +//! carries rule documents only (``REVIEW*.md`` / ``ARCHITECTURE*.md`` / ``LAWS.md``). A folder property, +//! NOT the ``[format]`` cascade: only that one file is read, and anything missing reads as false. +def public rule_docs_only_at(config_path : string) : bool { + return docs_flag_at(config_path, "rule_docs_only") +} + +//! True when the ``.lint_config`` at `config_path` sets ``[docs] enforce_arch = true`` — every +//! ``{#anchor}`` in the folder's ``.md`` owes an ``[arch]`` citation from the ``.das`` beside it, +//! which LINT026 reports in reverse. A folder property like ``rule_docs_only_at``. +def public enforce_arch_at(config_path : string) : bool { + return docs_flag_at(config_path, "enforce_arch") +} + def private config_dir_of(file : string) : string { let dir = to_generic_path(dir_name(to_generic_path(file))) return empty(dir) ? "." : dir diff --git a/daslib/perf_lint.das b/daslib/perf_lint.das index fb6ad3ff41..f1d6f67cd8 100644 --- a/daslib/perf_lint.das +++ b/daslib/perf_lint.das @@ -1249,7 +1249,7 @@ class PerfLintVisitor : AstVisitor { } def perf022_push_sentinel(expr : ExprFor?) : void { - //! The warning fires at the for-loop's `at`, not the push site — the three-function protocol is in daslib/ARCHITECTURE.md. + //! The warning fires at the for-loop's `at`, not the push site — the three-function protocol is in daslib/ARCHITECTURE_LINT.md. var state : Perf022State perf022_stack |> push(state) unsafe { diff --git a/daslib/style_lint.das b/daslib/style_lint.das index 03f016606f..59bd9be9c5 100644 --- a/daslib/style_lint.das +++ b/daslib/style_lint.das @@ -21,7 +21,7 @@ module style_lint shared private //! STYLE012 — array var initialized via push/emplace; use an array literal //! STYLE013 — struct (or `new`-allocated struct pointer) var with default/empty init followed by a run of field assignments //! STYLE014 — comment block exceeds 3 lines at module/public scope (on by default under daslib/, opt-in elsewhere via ``options _comment_hygiene = true``; suppress with '//!@nolint' or '// nolint:STYLE014') -//! STYLE015 — comment block exceeds 1 line inside a 'def private' (on by default under daslib/, opt-in elsewhere via ``options _comment_hygiene = true``; suppress with '// nolint:STYLE015') +//! STYLE015 — comment block inside a 'def private' over its cap: 3 lines for a '//!' contract doc, 1 line for a plain '//' block (on by default under daslib/, opt-in elsewhere via ``options _comment_hygiene = true``; suppress with '// nolint:STYLE015') //! STYLE016 — adjacent guards leading to identical early-exit can be combined with '||' //! STYLE017 — 'if (cond) return true; else return false' — use 'return cond' (or 'return !cond') //! STYLE018 — redundant boolean comparison ('b == true' / 'b != false') — use 'b' / '!b' @@ -2450,7 +2450,9 @@ class StyleLintVisitor : AstVisitor { } } let private_ctx = inside_function && enclosing_priv - let threshold = private_ctx ? 1 : 3 + // a '//!' block is a contract doc wherever it sits, so a private one gets the public cap + let contract_doc = starts_with(first_line_text, "//!") + let threshold = (private_ctx && !contract_doc) ? 1 : 3 if (line_count <= threshold) return var at = sample_at at.line = block_start @@ -2459,7 +2461,7 @@ class StyleLintVisitor : AstVisitor { at.last_column = 1u var msg = "" if (private_ctx) { - msg = "STYLE015: {line_count}-line comment inside 'def private' exceeds 1-line cap; private symbols don't get public docs - keep at most one WHY line, or suppress on the first line with '// nolint:STYLE015'" + msg = "STYLE015: {line_count}-line comment inside 'def private' exceeds {threshold}-line cap; a '//!' contract doc there caps at 3 lines, a plain '//' block at one WHY line - or suppress on the first line with '//!@nolint' (//! blocks) or '// nolint:STYLE015' (// blocks)" } else { msg = "STYLE014: comment block of {line_count} lines exceeds 3-line cap; trim multi-paragraph prose to a 1-line WHY (move design notes to .md), or suppress on the first line with '//!@nolint' (//! blocks) or '// nolint:STYLE014' (// blocks)" } @@ -2514,7 +2516,7 @@ class StyleLintVisitor : AstVisitor { } def override visitExpression(var expr : ExpressionPtr) : ExpressionPtr { - //! The frame is the subtree's unsafe-requirement summary; mechanics in daslib/ARCHITECTURE.md. + //! The frame is the subtree's unsafe-requirement summary; mechanics in daslib/ARCHITECTURE_LINT.md. let n = length(unsafe_stack) let frame = unsafe_stack[n - 1] unsafe_stack |> pop diff --git a/doc/reflections/das2rst.das b/doc/reflections/das2rst.das index 9f90b0089a..a2f73e917f 100644 --- a/doc/reflections/das2rst.das +++ b/doc/reflections/das2rst.das @@ -1364,7 +1364,7 @@ def document_module_lint(_root : string) { def document_module_lint_config(_root : string) { var mod = find_module("lint_config") var groups <- array( - group_by_regex("Configuration", mod, %regex~(load_lint_config|load_lint_config_from_path|load_env_disabled|seed_default_disabled|build_lint_macro_disabled|lint_config_forces_on|rule_docs_only_at)$%%), + group_by_regex("Configuration", mod, %regex~(load_lint_config|load_lint_config_from_path|load_env_disabled|seed_default_disabled|build_lint_macro_disabled|lint_config_forces_on|rule_docs_only_at|enforce_arch_at)$%%), group_by_regex("Path excludes", mod, %regex~(load_path_excludes_from_path|matches_path_excludes|is_lint_path_excluded)$%%), group_by_regex("Path-based rule defaults", mod, %regex~(is_daslib_source|is_shipped_library_source|is_core_library_source)$%%), group_by_regex("Lint-surface predicates", mod, %regex~(is_user_authored_body|is_lint_fixture_name|lint_file_skip_reason)$%%), diff --git a/doc/source/reference/language/lint.rst b/doc/source/reference/language/lint.rst index 0708d90167..3107df50a1 100644 --- a/doc/source/reference/language/lint.rst +++ b/doc/source/reference/language/lint.rst @@ -775,6 +775,79 @@ never fires in an untagged folder. [docs] rule_docs_only = true +LINT026 — ``[arch]`` citation does not resolve +============================================== + +``[arch(at=".md#")]`` binds a function, structure or class to a +section of an architecture document. The annotation itself is an inert marker +in the builtin module, so citing a document needs no ``require`` and costs a +build nothing; the compiler checks only the citation's *shape* — exactly one +``at=`` string, a document path and an anchor, both non-empty, separated by +``#``. Repeat the annotation to cite more than one section. + +Whether a citation *resolves* is this rule's question, and the runner answers +it with its own tree walk over the folders being linted: + +.. code-block:: das + + [arch(at="ARCHITECTURE.md#residency")] + def upload_weights(var ctx : Ctx) { + // ... + } + +The path resolves **relative to the citing file's folder**, the document must +exist, and the anchor must appear in it exactly once as a heading suffix — +``## Residency ramp {#residency}``, at any heading level. An anchor declared +twice is reported like a missing one: the citation reads as precise while +pointing at whichever section the reader reaches first. + +The rule also runs in reverse, for folders that ask for it: + +.. code-block:: toml + + # .lint_config beside the code the document describes + [docs] + enforce_arch = true + +In an armed folder every ``{#anchor}`` in every ``.md`` beneath it must be +cited by at least one ``[arch]`` in the ``.das`` beneath it — an anchor is a +promise that some code answers for the section, so an uncited one is either +code that forgot to say so or a section that was never anyone's contract. +Strip the anchor to demote such a section to narrative. Like ``[docs] +rule_docs_only``, ``enforce_arch`` is a folder property with no cascade: each +directory answers for itself. + +Both directions read source text rather than the AST, so a citation is checked +even in a file the linting environment cannot compile — and so a structure's +citation, which the compiler accepts unvalidated, is shape-checked here. A +match with ``//`` earlier on its line is prose and is skipped; one inside a +string literal is not. + +LINT027 — rule document exceeds 300 lines +========================================== + +The third folder rule: in a folder whose ``.lint_config`` carries ``[docs] +rule_docs_only = true`` or ``[docs] enforce_arch = true``, every +``REVIEW*.md`` and ``ARCHITECTURE*.md`` sitting there is capped at 300 lines. +Past that a checklist stops being read end to end, and a rule nobody reaches +is not a rule. + +.. code-block:: toml + + # either tag arms the line gate + [docs] + rule_docs_only = true + +The fix is a split, never a trim — every criterion survives, moved into a +companion the reader can hold: ``ARCHITECTURE_.md`` for an +architecture document, a routed ``REVIEW_.md`` for a checklist. +``LAWS.md`` is append-only ruling provenance rather than a document anyone +reads end to end, and carries no cap. + +Like the other two folder rules, the pass runs once per invocation over the +directories being linted and the folders of the files being linted, and never +fires in an untagged folder — the tags are folder properties with no cascade. + LINT018 — narrowed size argument of a call with a 64-bit overload ================================================================== @@ -2300,30 +2373,33 @@ to suppress the lint — the rest of the block stays visible. For a ``//`` block (no doc-comment), put ``// nolint:STYLE014`` on the first line; those blocks never reach the doc generator. -STYLE015 — comment block exceeds 1 line inside a ``def private`` -================================================================ +STYLE015 — comment block inside a ``def private`` over its cap +============================================================== .. note:: This check is **opt-in**. Enable it by adding ``options _comment_hygiene = true`` at the top of your file, or pass ``--comment-hygiene`` to the standalone utility. -Private symbols don't surface in any doc generator, so multi-line -comment prose inside a ``def private`` body is dead weight. Trim to one -line, or suppress with ``// nolint:STYLE015`` on the first line of the -block. +A ``//!`` block inside a ``def private`` is contract documentation — +callers inside the module still need the contract — so it gets the same +three-line cap STYLE014 gives a public doc. A plain ``//`` block there is +narration and caps at one WHY line. Whichever cap the block hits, +``// nolint:STYLE015`` on its first line suppresses it. .. das-doc: alt .. code-block:: das def private bad() { // First line — explanation // STYLE015 - // Second line — fires (>1 line in private) + // Second line — fires (>1 line for a // block) ... } def private good() { - // single WHY line — silent + //! What the caller gets back. // silent — //! caps at 3 + //! What it must pass in. + //! When it panics. ... } diff --git a/doc/source/stdlib/handmade/function_annotation-builtin-arch.rst b/doc/source/stdlib/handmade/function_annotation-builtin-arch.rst new file mode 100644 index 0000000000..69297728be --- /dev/null +++ b/doc/source/stdlib/handmade/function_annotation-builtin-arch.rst @@ -0,0 +1 @@ +Binds the function to a section of an architecture document: ``[arch(at=".md#")]``, where the path resolves relative to the citing file's folder. Repeat the annotation to cite more than one section. The compiler checks only the shape - exactly one ``at=`` string, a document path and an anchor, both non-empty, separated by ``#``; whether the citation resolves to a real section is checked by the LINT026 rule in the lint runner, which the annotated code does not need to require - the marker itself is metadata only and costs nothing. A structure or class takes the same spelling, unvalidated by the compiler, since a module holds one annotation per name. diff --git a/history/README.md b/history/README.md index 5d614a74c2..d4571ff785 100644 --- a/history/README.md +++ b/history/README.md @@ -24,10 +24,12 @@ Process and criteria: `skills/internal/doc_archiving.md`. `modules/dasSQLITE/PROVIDER_CONTRACT.md` - `dasSpirv/` - the completed PHASE6-9 working plans (content captured in `modules/dasSpirv/ARCHITECTURE.md` and `history/dasSpirv/MASTERPLAN_LOG.md`) +- `agents/` - retired agent charters, superseded by their successors (the harvester + absorbed both rescue bots) - `dasVulkan/` - the original boost-layer design plan, carried over when the module moved in-tree; the living docs are `modules/dasVulkan/CLAUDE.md` and its `ROADMAP.md` - `linq_fold/` - the linq_fold arc's plans, audits and the archived masterplan; the living - reference is `daslib/ARCHITECTURE.md` sec. 37, results stay at `benchmarks/sql/results.md` + reference is `daslib/ARCHITECTURE_LINQ.md` sec. 37, results stay at `benchmarks/sql/results.md` - `examples/` - plans, findings, and research notes behind shipped examples - `tests/` - one-off test-migration audits @@ -142,4 +144,7 @@ listed in the log below - search it first when hunting for a doc. - 2026-08-24 `modules/dasLLAMA/quant_lane_audit_plan.md` -> `history/dasLLAMA/quant_lane_audit_plan.md` - the format x kernel-lane x platform coverage audit (holes plugged; the kq-native and fused-arm work it drove shipped) - 2026-08-27 `modules/dasSpirv/MASTERPLAN.md` (plan sections + implementation log) -> `history/dasSpirv/MASTERPLAN_LOG.md` - the Phase 0-11 SPIR-V emitter build-out, its phase plans, and the tutorial-prerequisite backlog (the doc's present-tense half stayed as `modules/dasSpirv/ARCHITECTURE.md`) - 2026-08-27 `modules/dasMetal/MASTERPLAN.md` (plan sections + implementation log) -> `history/dasMetal/MASTERPLAN_LOG.md` - the Phase 0-8 MSL emitter build-out and the dasLLAMA GPU-resident prefill/decode chase (the doc's present-tense half, including the shared cross-backend ledger, stayed as `modules/dasMetal/ARCHITECTURE.md`) -- 2026-08-27 `daslib/linq_fold.md` -> `history/linq_fold/linq_fold.md` - the PR A-G splice-pattern refactor masterplan, its phase logs and decision log (the module layout, `SourceAdapter` contract and walker contract were distilled into `daslib/ARCHITECTURE.md` sec. 37, which the `linq_fold_*` headers cite) +- 2026-08-27 `daslib/linq_fold.md` -> `history/linq_fold/linq_fold.md` - the PR A-G splice-pattern refactor masterplan, its phase logs and decision log (the module layout, `SourceAdapter` contract and walker contract were distilled into `daslib/ARCHITECTURE_LINQ.md` sec. 37, which the `linq_fold_*` headers cite) +- 2026-08-27 `modules/dasLLAMA/ARCHITECTURE.md` (retired-note passages) -> `history/dasLLAMA/architecture_retired_notes.md` - the deleted whisper-decoder attention kernel and the three "superseded/corrected" narrations of the archived design docs, archived when ARCHITECTURE.md was groomed and split into its seven companions +- 2026-08-27 `.claude/agents/rescue-bot.md` -> `history/agents/rescue-bot.md` - fresh-scaffolding rescue at the PR gate, superseded by the harvester (make_pr row 0a0) +- 2026-08-27 `.claude/agents/rescue-sweep-bot.md` -> `history/agents/rescue-sweep-bot.md` - legacy comment-sweep rescue, superseded by harvest-on-first-touch diff --git a/.claude/agents/rescue-bot.md b/history/agents/rescue-bot.md similarity index 100% rename from .claude/agents/rescue-bot.md rename to history/agents/rescue-bot.md diff --git a/.claude/agents/rescue-sweep-bot.md b/history/agents/rescue-sweep-bot.md similarity index 100% rename from .claude/agents/rescue-sweep-bot.md rename to history/agents/rescue-sweep-bot.md diff --git a/history/dasLLAMA/architecture_retired_notes.md b/history/dasLLAMA/architecture_retired_notes.md new file mode 100644 index 0000000000..fbfee201f1 --- /dev/null +++ b/history/dasLLAMA/architecture_retired_notes.md @@ -0,0 +1,27 @@ +# dasLLAMA architecture - retired notes + +Past-tense passages archived out of `modules/dasLLAMA/ARCHITECTURE.md` when it was groomed to +the `ARCHITECTURE_COMMON.md` contract and split into companions. Each passage is verbatim, with +the section it sat in. The present-tense facts they carried stayed in the live documents. + +## From sec.1.5 GPU backends + +- **Retired: the single-pass whisper-decoder attention (`MetalWdecAttn`/`enc_wdec_attn`)** - + deleted in the metal-asr review round (2026-08-17, the `bbatkin/metal-asr` PR; the kernel is + in git history). The chunked part/comb pair replaced it during bring-up (1470->709 ms on the + turbo decode) and its `float[1504]` tgmem bound was the only reason for the driver's old + ta/n_text_ctx<=1504 shape cap, which left with it. + +## From sec.3 Inherited invariants, the `x64_arch.md` block + +(The x64-era "fallback rail" framing - AOT/interp as correct-but-untuned tiers - is retired; +intrinsic fallback bodies exist for off-ARCH correctness *inside* the JIT, not as runnable +tiers.) + +## From sec.3 Inherited invariants, the `kv_cache_compression.md` block + +**Correction:** that doc's "BNNS not pursued" tail is superseded by the AMX arc - + +## From sec.3 Inherited invariants, the `API_REWORK.md` block + +**CORRECTED:** the hybrid Gated-DeltaNet lane that doc lists as deferred Tier-3 **SHIPPED** - diff --git a/history/linq_fold/linq_fold.md b/history/linq_fold/linq_fold.md index d539c32887..1015db2b34 100644 --- a/history/linq_fold/linq_fold.md +++ b/history/linq_fold/linq_fold.md @@ -2,7 +2,7 @@ Archived 2026-08-27 from `daslib/linq_fold.md`. The operative half - the module layout, the `SourceAdapter` contract, and the pattern-table walker contract - was distilled into -`daslib/ARCHITECTURE.md` sec. 37, which the `linq_fold_*` module headers now cite. +`daslib/ARCHITECTURE_LINQ.md` sec. 37, which the `linq_fold_*` module headers now cite. Everything below is that document's plan material, phase logs, PR sketches and decision log, verbatim; the projected alias/predicate tables it carries were snapshots of a planned state and the shipped code has since diverged from them. diff --git a/modules/dasLLAMA/.lint_config b/modules/dasLLAMA/.lint_config new file mode 100644 index 0000000000..766cbe0035 --- /dev/null +++ b/modules/dasLLAMA/.lint_config @@ -0,0 +1,3 @@ +# Every {#anchor} in this module's .md files is owed an [arch] citation from module code (LINT026). +[docs] +enforce_arch = true diff --git a/modules/dasLLAMA/ARCHITECTURE.md b/modules/dasLLAMA/ARCHITECTURE.md index b40bef5e50..b4d33a415a 100644 --- a/modules/dasLLAMA/ARCHITECTURE.md +++ b/modules/dasLLAMA/ARCHITECTURE.md @@ -1,10 +1,5 @@ # dasLLAMA Architecture -**Who reads this: me.** It is the memory that survives a compaction, a new session, or a -hand-off - the thing to read when the answer to "where does this code go, and why is it shaped -like this" is no longer in context. It must therefore be readable cold, with no session state: -durable facts only, no history, no PR numbers, no "recently". - The three documents divide as follows, and the division is load-bearing - the same fact written in two of them will drift: @@ -15,17 +10,12 @@ in two of them will drift: | `REVIEW.md` | `/code-review`, and us while writing | criteria checkable against a diff | Shipped-arc plan docs live in `history/dasLLAMA/` (the archive log is `history/README.md`); -a `*_plan.md` in this folder is an ACTIVE arc's working plan. - -**References flow one way: REVIEW cites ARCHITECTURE, never the reverse.** Sections here are -numbered so they can be cited (`ARCHITECTURE sec.2.2`). Nothing here may cite a REVIEW rule - -those are unnumbered review criteria by design, and a citation to one is a dangling pointer the -moment the checklist is reordered. +a `*_plan.md` in this folder is an ACTIVE arc's working plan. Passages retired from this +document are archived in `history/dasLLAMA/architecture_retired_notes.md`. -**Charters below are boundaries, not descriptions.** README already says what every file is; -repeating that here is the failure mode this split exists to prevent. An entry earns its place by -answering a question a description cannot: what does NOT go here, and where does the adjacent -thing live instead. Files with no contested edge get one short line. +Seven companions carry this document's sections, each keeping its sections' numbers. The +routing block under each numbered heading below is the index: it names every companion that +section runs to, and the sections that companion holds. --- @@ -33,968 +23,26 @@ thing live instead. Files with no contested edge get one short line. When placement charters disagree on one function, the sec.1 charter line decides. -Every file under `dasllama/` appears here. Two carry generated CONTENT rather than being -generated themselves: `dasllama_env.das` holds the `[EnvConfig]` knob declarations (`ENVIRONMENT.md` is -generated FROM it by `harness/gen_env_doc.das`), and `dasllama_unicode.das` is hand-written -around transcoded Unicode data tables (the RANGES/WS tables - regenerate by re-transcoding -`$LCPP/src/unicode-data.cpp`, never hand-edit the tables). - -### 1.1 Engine core - -- **`dasllama.das`** - the public API facade and nothing else: `load_model` -> `create_session` -> - generate, re-exported names, the doc surface. No engine logic; a function that does work belongs - in the module that owns the concern, and the facade re-exports it. -- **`dasllama_version.das`** - `DASLLAMA_VERSION`, the module's own release counter, and nothing - else. Decoupled from the daslang version and from `LLVM_JIT_CODEGEN_VERSION`; ANY kernel work - bumps it (REVIEW.md), so equal versions mean an equal kernel roster - the sidecar exchange - keys validity on (version, box). A zero-require leaf so the tuner's noise half, the benches, - and the facade chain (`dasllama_common` re-exports it) all reach it without weight. -- **`dasllama_common.das`** - the engine: `Model`/`Session`/`Config`, the forward loops, the - override registries, the runtime knobs. **Not** the load walk (sec.1.3) and **not** GPU residency - (sec.1.5) - both left, and the seam each left behind is a registered hook, so neither comes back. - It remains the module's debt sink; what sits here that is family-specific or platform-specific is - debt, **not precedent**. Nothing platform-specific may be added; new shared concerns get their own - file rather than another thousand lines here. Ledgered exceptions the sidecar-apply seam forces - (the knob rail lives here, and the backends require this module, so their sidecar knobs cannot): - the metal tensor crowns, and the three metal prefill raced-constant knobs beside them - both are - applied by `apply_box_profile_runtime` and CONSUMED by the Metal backends at their own init. - `REVIEW.das`'s `check_fastmath_default` licensed set is empty by design: no engine kernel opts - out of fastmath, and an entrant ships its failing-under-fastmath test with the kernel. -- **`dasllama_transformer.das`** - the block-composition seam, and the require umbrella: the - `[init]`-only side-effect requires (arch registrations, GPU tiers, and any module that - requires the engine back, which the umbrella therefore hosts to break the cycle). -- **`dasllama_blocks.das`** - the std/dense/MoE transformer block kernels, decode and prefill, - plus the two default block sets the arch files bind. `forward()` never names a kernel here - it - dispatches through the `ArchBlocks` function pointers, which is why the family lives outside the - hub with no hook. -- **`dasllama_moe.das`** - MoE expert routing and dispatch: the top-k router, per-expert FFN - accumulation, the shared expert, and the decode instruments. The block kernels reach it only - through `moe_ffn_core`. -- **`dasllama_attn_prefill.das`** - prefill attention, threaded over heads. `prefill_attention` - is the only entry; the classic/blocked/flash head kernels and their KV-codec ladders are its - implementation. -- **`dasllama_batch.das`** - the batched decode step: one pass of the weights over B sessions, - GEMVs widened to B-row GEMMs, attention still per-(row, head) against each session's own cache. -- **`dasllama_sampling.das`** - token sampling and the generation drivers. A leaf on top of - `forward`/`eval_batch`; the engine never calls back in. -- **`dasllama_ple.das`** - gemma-4 E-series per-layer embeddings and the gemma4 MoE FFN. The - forward sequence reaches it only through the hooks it registers at init. -- **`dasllama_config.das`** - `DlimConfiguration`: every input that changes `.dlim` image BYTES, - in one struct, plus its identity formatter. A knob that does not change image bytes does not - belong here; a knob that does and is missing is an image-aliasing bug. -- **`dasllama_tools.das`** - the per-ToolMode tool wire codecs (definition serializers, - replay/result text builders, reply parsers; Harmony namespace/channels, gemma-4 DSL, mistral - control tokens, llama JSON). Every byte of tool wire text is produced here - pure string+JSON - functions, model-free testable; the chat layer assembles the output into ChatParts and the - server parses through the parsers. Wire-format provenance: `history/dasLLAMA/tool_formats_plan.md`. -- **`dasllama_chat.das`** - conversation turns and chat-template application. Per-arch template - *content* is registered by the arch file (sec.1.6), not written here. -- **`dasllama_par.das`** - `maybe_parallel_for` and nothing else. Threading policy (job counts, - thresholds) belongs to the caller that knows the shape. - -### 1.2 Formats and data movement - -These six own the module's data-shape vocabulary. The boundaries between them are the ones most -often gotten wrong, so each says explicitly where the neighbouring half goes. - -- **`dasllama_plane.das`** - the borrowed-plane vocabulary: a plane is a pointer into a prepared - image plus its element count; the image owns the bytes, a carrier owns nothing but its backing. - Requires nothing in dasllama - the image rail binds planes, every carrier holds them. -- **`dasllama_kqformat.das`** - format IDENTITY: the `KqFmt` enum, the per-format descriptor table - (plane strides, block geometry, stream codes), format predicates. It requires nothing else in - dasllama, because it is the taxonomy everything keys off. ONE id space - the enum; integer ids - exist only at the IR/kernel-param boundary. `kq_sb` is the superblock-lattice predicate: a - `fmt != q8` test does not imply the lattice, so branch on the predicate. -- **`dasllama_convert.das`** - every tensor format CONVERSION: quantize/dequantize/transcode/encode, - codec byte readers, numeric widen/narrow - regardless of platform or caller; plus the Q8_0/Q4_0 - format identity itself (block geometry, scale layout, the quantization-quality detector). - Metadata-coupled drivers (GGUF lookup, threading, guards) stay with their containers and - dispatch in. ONE carve-out: a conversion that IS a KV-cache format's store/read half lives with - its codec family (sec. below). -- **`dasllama_repack.das`** - every disk-order -> compute-order kernel-LAYOUT transform (grp - interleaves, disk-order extractors, panel unpacks), any format, any platform. Number sources - (tune stamps, bake overrides) stay with their owners and pass plain parameters in. -- **`dasllama_kv_codec.das`** - the KV-cache runtime codec, one family per cache format, kept - WHOLE: store (quantize), read (dequant), the attention score dot, and the V-accumulate axpy all - share the format's block byte geometry, so a layout change can never span modules. These are - per-token and per-tokenxposition `[tuned]` hot kernels, not load-time converters. The tq4 FWHT - rotation and sign vector live with their family. Codec DISPATCH (`KVDtype`) stays at common's - `kv_store_row`/`kv_load_row`/`kv_dot`/`kv_axpy` seam; the f16 row converts are the generic pair - in `dasllama_convert` because they are dual-use beyond the cache. -- **`dasllama_rope.das`** - RoPE angle and TABLE GENERATION: the theta schedule, `rope_freqs`, - fscale/mscale, every materialized layout - pure functions over plain parameters. Model-facing - wrappers stay in common. APPLICATION kernels stay with their backends: the CPU `rope_scaled_*` - leaves and the GPU fused rope-store kernels are shape-specialized deliberately, and - single-sourcing them is generator-rail work, never a hand-merge. Float multiply order in the - builders is contractual (parity-pinned) - never "unify" it. -- **`dasllama_gguf.das`** - the GGUF container: KV/tensor descriptors, the byte-level reader, the - mapping. Codecs live in `dasllama_convert`; this file finds bytes, it does not decode them. -- **`dasllama_layout.das`** - disk-format -> compute-layout transforms at LOAD scope: the blob - transform, the CPU repack walkers, the GPU tier gathers, the per-region q8->Metal-34B - un-repack (`q8_region_to_metal_blob`, the ASR-decoder driver's upload form), and the refusal - half (`can this model take the blob form`) split out so the image writer can commit without - loading. -- **`dasllama_tokenizer.das`** - the tokenizer facade: backend selection off the GGUF metadata and - the one encode/decode/piece surface models and the chat layer call. Re-exports both backends, so - a consumer requires this file and never picks a backend by hand. -- **`dasllama_spm.das`** - the SentencePiece backend (Llama-2 family, Phi-3, Gemma): score-greedy - merges over vocab pieces, `<0xXX>` byte fallback. -- **`dasllama_bpe.das`** - the byte-level BPE backend (Llama-3 / tiktoken family): vocab load, the - GPT-2 byte alphabet, ranked merges, encode/decode. Split from SPM because the two algorithms - share no state; a third merge algorithm gets a fourth file. Two sanctioned family-name tests - live here rather than in `dasllama_pretok`: the `pre`-name selector inside `bpe_encode`, and the - gemma-4 newline-run split in `bpe_encode_spm_space`; `load_bpe_tokenizer_gguf`'s per-family - metadata defaults are the third and last. -- **`dasllama_pretok.das`** - the pre-tokenizer: one hand-compiled split function per family - (llama3/qwen2/qwen35, gpt-2, gpt-4o, tekken), selected by the BPE `pre` name. Regex-port growth - lands here, never in the merge engine - the two change for different reasons (new model family - vs. algorithm work). Every arm with an on-disk upstream corpus vocab is gated by its case in - `test_tokenizer.das` (llama3, qwen2, qwen35, gpt-2); tekken has no corpus case, and gpt-4o is - pinned by frozen ids in `test_parity.das` only. - -### 1.3 The load and image rail - -**`dasllama_load.das`** is the GGUF load walk: metadata to `Config`, the plane layout, disk-format -detection, the eager and streamed conversion ladders, and the load entry points. It owns nothing the -forward path touches at run time - a loaded `Model` is the whole handoff - and it requires -`dasllama_common` back for `Model`/`Session`, so the transformer umbrella requires it `public` and -breaks the cycle. That re-export is what keeps every consumer on the facade. - -- **`dasllama_image.das`** - the prepared-model `.dlim` rail, and it is ONE rail (sec.2.1). Nothing - outside this file may read weights into a live carrier; image backings are released only - through the single release path in `dasllama_common` that the carrier finalizers call - never - an ad-hoc unmap. - -### 1.4 CPU kernel tiers - -- **`dasllama_math.das`** - the numeric ABSTRACTION: typedefs, active backend pointers, public - wrappers, dispatch shaping. Kernels themselves live in a tier file; a kernel body here is a - placement defect. -- **`dasllama_math_default.das`** - the portable backend, always registered, always correct, - out-ranked by any platform tier. -- **`dasllama_math_aarch64_neon.das`** - the arm64 SDOT/tbl tier. Its `[init]` never fires - off-arch (sec.3, three-layer safety model), so an intrinsic here needs a correct scalar fallback - body, not a guard at the call site. -- **`dasllama_math_accelerate.das`** - the Accelerate/BNNS float tier (AMX on M1-M3, SME on M4+), - for genuinely-float planes only. BLAS-for-quant is ruled out structurally (sec.3). -- **`dasllama_math_gen.das`** / **`dasllama_gemm_gen.das`** / **`dasllama_gemm_schema.das`** / - **`dasllama_gemm_register.das`** - the generated GEMM tier: the runtime registration, the tile - generator, the layout/perm schema shared by generator and runtime, and the `[tune]` family - registration. A hand-written tile that the generator could emit belongs in the generator. -- **`dasllama_tune.das`** - the per-box loop-hint tuner (`[tuned]` / `[dasllama_grid]`). Tuning - POLICY lives here; tuned VALUES live in the box's sidecar, never in source. - -### 1.5 GPU backends - -A GPU backend is a FAMILY of role files - matching things in matching files across backends, so -that a question answered for one backend has an obvious address in the other. The roles: - -| role | holds | must not hold | -|---|---|---| -| the kernel home
`dasllama_metal_kernels`, `dasllama_vulkan_classes` | kernel source, the derived-access/PSO census | device state, engine types | -| `dasllama__common`
`dasllama_metal_common`, `dasllama_vulkan_common` | device state, buffer/command plumbing, hazard + capture rail, profiler | driver policy | -| `dasllama__decode`
`dasllama_metal_decode`, `dasllama_vulkan_decode` | the resident token-step driver + decode-time arms | kernel bodies | -| `dasllama__prefill`
`dasllama_metal_prefill`, `dasllama_vulkan_prefill` | the batched prefill driver + batch arms | kernel bodies | -| `dasllama__shapes`
`dasllama_metal_shapes` | PORTABLE servability gates - no GPU C++ require, so any box can bake | device calls | -| the tower driver
`dasllama_metal_tower` | one-shot embedder/encoder encodes (gemma4uv chain, the gemma4v ViT, gemma3v SigLIP and qwen3v block loops - qwen3v adds the vision NEOX rope, the fused-qkv weight-offset GEMMs, and the inline deepstack tap + tail merger chains - the whisper-class block loop, the conv frontends + the qwen3a padded-weight slab) - no session, no KV, no mirror; registers the gemma4uv, gemma4v, gemma3v, qwen3v, encoder_blocks, tower-conv and qwen3a-conv hooks | kernel bodies, decoder state | -| the ASR-decoder driver
`dasllama_metal_asr_dec` | the whisper decoder on Metal: the 34B weight blob, the f16 resident cross/self K/V, window-granular cross-KV + decode-step serves; registers the whisper cross-KV and decode hooks (family registries in `dasllama_whisper`) | kernel bodies, LLM session state | -| the kernel-access lens
`dasllama_metal_lens` (Metal), `dasllama_vulkan_dispatch` (Vulkan - the `[vk_dispatch]` macro derives access per class) | the kernel-access macro | anything else | - -- **Vulkan additionally has an ENTRY, `dasllama_math_vulkan.das`** - capability probe/arm, `.dlim` - identity source, cross-arm routers, the `[init]` installs. It re-exports the family `public`, - and its NAME is the transformer umbrella's `?vulkan` require contract (deliberately LAST in the - umbrella: the vulkan drivers are the hot-edit modules, and require order is the jit obj-cache - layout): **never rename it.** -- **Vulkan additionally has `dasllama_vulkan_seams.das`** - the thin whole-op call seams the tier - and suites dispatch through (`vk_add_rms`, `vk_rope_kv_store`, `vk_decode_attn`). It exists - because of a require direction: common cannot require the classes module (classes requires - common back), so any seam that encodes a class kernel must sit above both. A seam here wraps - ensure/set/enc - kernel bodies and driver policy stay out. -- **Metal has NO `math_` entry** - the family enters via the transformer's `?das_metal` requires - plus unconditional shapes. Its below-common piece is **`dasllama_metal_gemm.das`** (the batch - GEMM donor that common requires `?das_metal`), which owns its device by necessity: - metal_common -> dasllama_common -> metal_gemm would cycle. -- **Backend-only capabilities live in their matching ROLE file, not in new grab-bags** - vulkan's - weight arena, streamed mirrors, heat cache, host-import, coopmat; metal's blob transform and MTP. -- **The tower driver owns NO PSOs.** Its kernels (LN, f32 mul_mm, the two gelu flavors, - posadd, the gemma4v clamp / rope2d / GEGLU-quick, the head restride - gemma3v's and, offset-bound, the qwen3v/prefill slicers) live in the kernel home, so `metal_decode_init` compiles and `metal_kernels_release` - releases them like every other registry PSO; the borrowed prefill builders (`pf_enc_bf16_mm`, - `enc_add_bias_rows`, `enc_rope` - the qwen3v vision NEOX apply - and the attention trio - `enc_qk_mm`/`enc_rowstat`/`enc_av_mm`; this list is the closed borrowed set REVIEW_GPU.md's - tower rules key on) come up through - `metal_prefill_pso_init`, prefill's public bring-up seat, and `plane_buffer` in common is - public for the same wrap-a-plane reason. The tower's own objects (the ones buffer, its - scratch pool) release through `metal_tower_shutdown`. -- **The tower driver is a Metal-only role** - Vulkan has no tower twin; audio/vision encodes - on the Vulkan tier stay CPU (the gemma4v ViT, gemma3v SigLIP and qwen3v block loops - included: on Vulkan and on plain CPU boxes those towers serve their q8 lanes). Likewise the non-causal media span: Metal serves it through - `AttnArgs.uend` - including the FUSED image turn (head + media rows + tail as ONE eval, the - per-query mask through `AttnArgs.ulo`); the Vulkan resident prefill declines span evals - (`followup_general.md` #23's remaining half) and registers the split-span capability - (`register_prefill_override_split_span`), so `eval_embd_span_` keeps the three-eval splice - while vulkan is the active override. The qwen mrope quantum rides the same shape through a - second capability seat (`register_prefill_override_mrope_tables`): Metal's `enc_rope` reads - the per-token table rows `prefill_rope_tables` builds from the grid map, so it serves mrope - unchanged and registers the seat; an override without it (vulkan builds angles from a scalar - position) declines the quantum to the CPU loop by name. The deepstack quantum is the third - seat (`register_prefill_override_ds_adds`): Metal uploads the caller's wide quantum WHOLE - (the `Session.wide_src` borrow), slices x and the slice-major ds planes on-device through - the offset-bound head restride, and encodes one `enc_add` at the slice offset after each - tapped layer's residual - no new kernel (the CPU-side split, `ds_split_quantum`, survives - as the CPU-loop fallback and the warm/MTP edge); an override without the seat declines - deepstack quanta by name, so Metal serves them and Vulkan does not. -- **Per-layer FFN widths (MatFormer E-series, at most two - `ffn_second_hidden`) serve on Metal - only**: the decode and prefill drivers bind the width per layer (dense trunks, no MTP; batch - keeps the layer-0 hoist behind its uniformity decline). The Vulkan tier has no PLE arm, so - E-series never reaches its width question. -- **Family-shared kernel classes live in `dasllama_metal_kernels`.** The `[metal_dispatch]` lens - generates `enc_*` builders and MSL globals into the module the class COMPILES in, so co-location - follows the class - "the builder needs the driver module" is never a placement reason. Prefill's - prefill-only classes are convergence debt, not precedent. -- **Retired: the single-pass whisper-decoder attention (`MetalWdecAttn`/`enc_wdec_attn`)** - - deleted in the metal-asr review round (2026-08-17, the `bbatkin/metal-asr` PR; the kernel is - in git history). The chunked part/comb pair replaced it during bring-up (1470->709 ms on the - turbo decode) and its `float[1504]` tgmem bound was the only reason for the driver's old - ta/n_text_ctx<=1504 shape cap, which left with it. -- **Ledgered kernel-binding asymmetries** - a REVIEW rule firing on one of these is expected, and - this entry is the sanction: the moe mul_mm TENSOR twins (`MetalMoeMulMmQ8T` / - `MetalMoeMulMmMx4T`) keep the pre-family compact kargs slots while their base classes bind the - family numbers, so no shared bind path may span the two layouts; and the in-engine moe mul_mm - A/B race harnesses (`dasllama_metal_prefill.das`) encode through `kn_moe_mm_family_tail` - rather than a per-class `enc_*` builder. -- **`dasllama_gpu_tier.das`** - the device-cooperation SPI: hook types, install/unset slots, - route/mark/want/status state, engine-facing forwarders. Vulkan implements it (per-op offload plus - resident plumbing); Metal deliberately does not, because UMA makes residency moot there and Metal - integrates as a whole-forward driver through common's override registries (the ASR-decoder - driver is the one exception: whisper is not a `Model`, so its hooks are family registries in - `dasllama_whisper`, same decline contract). -- **`dasllama_gpu_resident.das`** - the WHOLE-MODEL residency rail: bake the device layout offline - into the flavor image, upload a model's stacks to the tier, and drive decode/prefill entirely on - device. It is device-AGNOSTIC - it holds no device call and requires no GPU module, reaching the - hardware only through the `dasllama_gpu_tier` SPI and entering the engine only through common's - override registries. `"vulkan"` is the tier string it registers under, not a dependency, which is - why it compiles on every box. It requires common back for `Model`/`Session`, so like the Metal - drivers it is required from the transformer umbrella, never from common. -- **`dasllama_kernel_access.das`** - the shared body-walk read/write classifier both GPU lenses run - on, plus the dispatch-lens micro-grammar (the grid/tg/params spec tokenizers and the shared - AST-emission core: `is_digit_tok`, `role_ok`, `derived_role`, `mk_uint_cast`, `mk_call1`, - `mk_grid_dim`, `param_type`). One owner by design - the two lenses' grammars drifted apart - when each carried a private copy (metal folded only the literal "1" where vulkan folded any - integer). Backend-specific lowering stays in that backend's lens. - -**PSO lifecycle - the family shares ONE device and queue** (`metal_common_init`; the second-device -question was surveyed and closed against). The decode PSO set lives as `g_pso_*` in -`dasllama_metal_common`, is compiled by `metal_decode_init` in `dasllama_metal_kernels` and -released by `metal_kernels_release` there - the kernels module owns its set's lifecycle even -though the vars live with the device state. Prefill's `g_pf_pso_*` set is prefill-private end to -end: `metal_prefill_init` compiles, `metal_prefill_shutdown` releases. - -**Race and tune code for a kernel family lives beside the family.** The shared scaffolding -(`race_buf`, `race_envelope_ok`, `race_pair_ms`, `MetalTensorRaceResult`) is `_common`'s; -each module races its OWN families (`metal_tensor_race_decode` in kernels, `metal_tensor_race` -in prefill) and the tuner calls those public entries. - -**Decline REASONS are enum values in the shapes module** (`MetalDecodeDecline`, -`MetalPrefillDecline`); decline COUNTING lives in `_common` beside `require_or_panic`, for -both paths. - -**The allowed asymmetries between the backends - this list is closed; a new one lands with its -entry here:** - -- **Metal sits ABOVE `dasllama_common`** (typed `Model`/`Session` access, shapes unconditional); - **Vulkan sits BELOW it** (untyped pointer/array seams - the family never requires common). - Both tiers ENTER from the transformer umbrella (`?das_metal` requires; the single `?vulkan` - require of the `dasllama_math_vulkan` facade); the inversion that remains is the - kernels<->common require DIRECTION, and it is why their seam shapes differ. -- **UMA vs discrete VRAM**: Metal never grows Vulkan's VRAM machinery - arenas, upload - economics, mirrors and hydration are Vulkan's alone. Metal's residency artifacts are the - `MTLResidencySet` pin in `_common` (`DASLLAMA_METAL_RESIDENCY`) plus its keep-alive - heartbeat (`DASLLAMA_METAL_HEARTBEAT_S`, a dasMetal background re-request that stops the OS - collecting the set over a CPU-only window, sec.2.12) - a driver-cost shield, not a - placement mechanism; memory is still memory. -- **The weights-epoch drop is Metal-only.** `bump_weights_epoch`'s listener seat - (`register_weights_epoch_listener`) has one subscriber: `_common`'s `metal_weights_drop`, - which runs the registered reload preps (`register_reload_prep`; the decode driver registers - `discard_pre`), quiesces, and releases the address-keyed region caches. Vulkan's reload - story is the unmap notify (`set_moe_gpu_unmap_notify`) - a different seam for a different - ownership model. -- **Lens depth**: both lenses generate `enc_*` builders from kernel classes - Metal via - `[metal_dispatch]`, Vulkan via `[vk_dispatch]` (per-class set layouts + push constants; the - class-kernel arc retired the hand-built 6-slot set ladders outright) - and both speak the - multi-kernel form (`kernel=` names the method, one macro instance per kernel, declared roles - must cover every kernel). -- **`family=` is Vulkan-only.** A vulkan family shares the per-class surface - the `VkdClass` - global, the `set_*` builder, the pipe slots - across classes with one binding layout. Metal's - `enc_*` builder is the entire generated surface, so there is nothing for a family to share; - cross-class PSO/source sharing on Metal is a PSO-lifecycle question, not a lens one. -- **`@default` is Metal-only.** A `[metal_dispatch]` field may name a fallback global - (`@default = g_one`) that the generated builder binds when the caller passes null; - `[vk_dispatch]` has no counterpart - vulkan callers pass a real buffer at every slot. If - vulkan grows an optional-bind shape, it lands as this same annotation, not a new spelling. -- **The workgroup-footprint gate is Vulkan-only.** `[vk_dispatch]` sums a class's `@workgroup` - members and its generated `ensure_*` declines by name (`vkd_wg_fits`) before the pipeline - build, because MoltenVK's over-cap failure is an opaque `INITIALIZATION_FAILED` - and the - resident driver declines residency with it (`vk_rdec_prepare`). `[metal_dispatch]` has no - footprint gate: Metal's own pipeline compile fails loudly with the footprint in the error. -- **Vulkan has no shapes module yet** - `resident_upload` declines ad hoc by feature name; the - gap is `followup_vulkan.md` item 1, not a precedent to copy. - -Vulkan is the deliberately-designed model of this shape; Metal converges as it is touched. - -### 1.6 Architecture registrations - -Thirteen files registering eighteen names: -`dasllama_arch_llama.das` * `dasllama_arch_phi3.das` * `dasllama_arch_qwen2.das` * `dasllama_arch_qwen2moe.das` * `dasllama_arch_qwen3.das` * `dasllama_arch_qwen3moe.das` * `dasllama_arch_qwen35.das` * `dasllama_arch_gemma2.das` * `dasllama_arch_gemma3.das` * `dasllama_arch_gemma4.das` * `dasllama_arch_glm4moe.das` * `dasllama_arch_gptoss.das` * `dasllama_arch_mistral3.das`. They are DECLARATIVE: an arch -file builds an `ArchDesc` (name * `configure` * the `ArchBlocks` fn-ptr quad * `ChatTemplate` * -`LlmCaps`) and calls `register_arch` at `[init]`. Adding an arch touches no forward loop. +- `ARCHITECTURE_ENGINE.md` - sec.1.1-1.4, 1.6, 1.8, 1.9: the engine, format, load, CPU-tier, + arch-registration, support and serving charters. +- `ARCHITECTURE_GPU.md` - sec.1.5: the GPU backend role table and the backend asymmetries. +- `ARCHITECTURE_MEDIA.md` - sec.1.7, 1.7b: the encoder-tower, audio, ASR and vision charters. -Family behavior is distributed by `Config` flag, not dispatched by name - an `if (arch == "...")` -on a shared path is the anti-pattern. Only a genuinely new dataflow earns its own block pointer. -`register_arch` MOVES the descriptor, so an alias must clone the template first (sec.3). - -### 1.7 Encoder towers, audio, and ASR - -- **`dasllama_asr_types.das`** - the ASR floor: the capability/segment/timestamp types every - family file and the facade share (`AsrCaps`/`AsrTimestamps`/`TranscribeSegment`, plus - `asr_ctx_guard`). Family modules require this instead of each other - a shape needed by two - families moves up here, never sideways. -- **`dasllama_tower.das`** - the family-neutral encoder-tower home, composed by every audio AND - vision tower: the oracle-exact activations (erf / tanh-LUT / quick-LUT gelu, swiglu, geglu), - the LayerNorm/RMS row forms, bias and residual row adds, the - `mm_blob_b`/`mm_bf16_b`/`mm_plane_b` GEMM wrappers, `Clamp`/`read_clamp`, - `im2col_rgb_patches`, `rope_neox_2d_rows`, `rope_neox_tab_rows`, `avg_pool2d_rows`, - `interpolate_grid_bilinear_aa`, `tower_read_conv_pair_folded`, the padded stage readers - (`tower_read_gemm_q8`, `tower_stage_q8_zero_rows`/`tower_stage_q8_pad_cols` and their f32 - twins `tower_stage_f32_zero_rows`/`tower_stage_f32_pad_cols` - the load-scope padded - stagers for FFN widths that are not 32-aligned - and `tower_zero_span`), blocked `attention_bidir` and - its per-window form `attention_bidir_windows`, and the encode-stage prof rail. The one home: - a family file that re-implements one of these is a defect, and nothing here names a family - type. -- **`dasllama_audio.das`** - the audio encoder tower: the mel front-ends (mtmd and whisper.cpp - flavors), `AudioTower` with its staging, q8-quantize, and image rails, `EncoderState`, and the - whisper-class encode + block loop with its GPU hooks. Composes `dasllama_tower.das`. -- **`dasllama_audio_io.das`** - decode-any-format -> 16 kHz mono f32 PCM. The only file that talks to - miniaudio. -- **`dasllama_asr.das`** - the ASR facade: capability declaration, timestamp granularity, the - backend-neutral entry points. -- **`dasllama_whisper.das`** / **`dasllama_parakeet.das`** / **`dasllama_canary.das`** / - **`dasllama_qwen3a.das`** / **`dasllama_gemma4a.das`** - one file per model family, each owning its - weights, its decode loop, and its quirks. Shared tower pieces go up into `dasllama_tower`, not - sideways between families. -- **`dasllama_vad.das`** - Silero-VAD weights and per-stream state. - -### 1.7b Vision - -- **`dasllama_vision.das`** - the image preprocessing rail: dynamic-resolution geometry, the - letterbox resize (aspect-preserving bilinear onto a centered black canvas), u8->f32 normalize, - and the `DASLLAMA_VISION_DUMP` PPM writer. The only preprocessing home. -- **`dasllama_vision_io.das`** - image decode to RGB8, from a file or a byte blob. The only file - that talks to stbimage; re-exported by the facade (like `dasllama_audio_io`) so consumers - decode through one seam - the engine itself takes decoded pixels. -- **`dasllama_gemma4uv.das`** - the gemma4uv embedder (gemma-4 dense): mmproj load and the - im2col -> LayerNorm -> GEMM -> position-table -> projection forward. One file per vision - projector family, following the audio tower pattern; shared pieces move up into - `dasllama_tower.das` (the encoder-tower home), never sideways. - A shipped mmproj mixes element types per tensor - gemma-4's "BF16" file stores the patch - embedder as F32 and only the projection as BF16 - which is why a weight plane's element type - follows its source tensor, per tensor, never a per-file verdict. -- **`dasllama_gemma4v.das`** - the gemma4v ViT tower (gemma-4 E-series): mmproj load (bf16 GEMM - planes as the file has them, the clamp sidecars as a blob table) and the 16-block pre-norm RMS - forward - clamped GEMMs, per-head q/k RMS, two-axis NEOX rope, weightless V RMS, unscaled - bidirectional attention, GEGLU-quick - then the 3x3 pool, RMS and projection. Composes the - `dasllama_tower.das` tower pieces; owns only its layout and the block loop. -- **`dasllama_gemma3v.das`** - the gemma3 SigLIP tower (size-invariant across the gemma-3 line; - 4B and 12B are the tested pairs): mmproj load (the file's f16 planes widen exactly to f32; - the ffn pair serves at the layout's padded 4352 width, so every q8-lane GEMM quantizes) and the 27-block pre-norm - LayerNorm forward - biased GEMMs, learned position add, scaled bidirectional attention, - GELU-tanh - then post-LN, the 4x4 pool, weighted RMS and projection. The canvas is FIXED at - 896^2 (the learned table covers exactly that grid), and the family's image_mean/std (0.5) is - PREPROCESSING, not graph - `encode_image_` scales, `gemma3v_encode` takes planes raw like the - mtmd fixtures do. Composes `dasllama_tower.das`; owns only its layout and the block loop. - Two sanctioned exceptions to the family-quirk placement rule live outside this file: the - image span markers sit on the SHARED `gemma_chat` template in `dasllama_common.das` (template - detection cannot tell gemma3 from gemma2 - both spell `` - and the chat - layer's `image_vocab_ok` gate refuses a text-only vocab); and the carrier deliberately serves - ONE f32 plane (F32/F16/BF16 widen exactly; anything else refuses by name at stage) instead of - gemma4v's per-tensor plane split - the Metal leg runs the f32 mulmm, so there is no bf16 - plane to preserve. -- **`dasllama_qwen3v.das`** - the qwen3vl vision tower (Omni AND dense Qwen3-VL, - `qwen3vl_merger`): mmproj load (per-tensor bf16/f32 planes; the temporal patch-conv PAIR - folds to one f32 GEMM at stage - a still evaluates both convs on one frame, so conv_0+conv_1 - = (W_0+W_1)*img) and the pre-LN forward - fused-qkv biased GEMMs, vision-mrope (h/w ladder - tables from `build_rope_tabs_vision`, full-head NEOX apply), scaled bidirectional - attention, GELU-tanh - after a spatial-merge REORDER of the patch stream with the 48x48 - learned position table resized to the grid by an antialiased bilinear resample; then post-LN and - the 2x2 merger MLP (the x4 reshape is free - merge partners are already adjacent). Dense - carriers add DEEPSTACK taps (collected from the TENSOR list, never the metadata flags): - after each tap block a LayerNorm->fc1->GELU->fc2 merger over the x4-merged rows emits one more - proj_dim slice, and the output rows widen to (1+n_deepstack)*proj_dim - slice 0 = the main - merger, the decoder adds slice l+1 after layer l. The family token budget [8, 4096] is - mtmd's, not the gemma-scoped DASLLAMA_VISION_* knobs; image_mean/std (0.5) is - PREPROCESSING like gemma3v. CPU serving default: the block GEMMs serve as Q8_0 planes - (read-time transcode, the gemma3v recipe); the exact lane stages EVERYTHING f32-in-blob at - ff_pad both dims - blocks, mergers, and taps - because the Metal tower (`register_qwen3v_gpu`, - blocks + rope + taps + tail on-device, tap/tail proj outputs land in `ds_stash`/`ds_slice`) - and the float-batch tier both read f32 planes, and the same lane is the parity rail (the - f32 sites are this charter's sanction, not a fallback). Pin knobs - `set_qwen3v_q8`/`reset_qwen3v_q8`, two image tags; the lane policy prefers f32 when the - GPU tower or accel serves. Composes `dasllama_tower.das`; owns - only its layout, the reorder walk, and the block loop. SANCTIONED over the 1 GiB staged-mint line (this family - and qwen25v): the Omni (2.1 GB) and Qwen2.5-Omni (2.6 GB) mmprojs stage source+image at - once with no cap - the same shape their audio halves already stage - until - `followup_general.md` 24's streaming mint covers towers. -- **`dasllama_qwen25v.das`** - the qwen2.5 window-attention ViT (Qwen2.5-Omni/VL, projector - `qwen2.5o`): 32 RMS-normed blocks with separate biased q/k/v GEMMs and a gated-silu FFN - whose hidden width comes from the TENSOR dims (the metadata's feed_forward_length lies); - rope-only positions, per-channel CLIP normalize (`vision_normalize3`). Rows sort by 112 px - windows of 2x2-merged groups, non-full blocks attend per contiguous window range - (`attention_bidir_windows` in the tower - no mask tensor exists), every `n_wa_pattern`-th - block attends in full, and the merger un-sorts back to group-row-major. Its decoder is the - `qwen2vl` arch whose plain MROPE reads the rope sections as contiguous ranges - (`Config.mrope_interleaved` false -> `build_rope_tabs_mrope`; the qwen3vl family sets the - flag and keeps the interleaved walk). This tower serves the file's planes ONLY - no q8 - lane: its residual rows carry the Qwen2-VL-lineage outlier channels that per-32-block - activation requant cannot represent (a q8q8 lane measures 2.0 x rms vs the oracle, where a - DELETED layer measures less - the gate cannot discriminate; the weights themselves - quantize fine at 0.007 x rms, and a float-activation q8 GEMM wins nothing on these - compute-bound shapes since the CPU speedup IS the int8xint8 dot). -- **`dasllama_vision_embedder.das`** - the vision carrier: `VisionEmbedder` / `VisionState`, the - `AsrModel` shape for vision - one union through every seam, the family sniffed from the mmproj - (`clip.vision.projector_type`, or a `.dlim`'s baked tag) at load, one-line arms. Outside a - family's own file, a family type is named only here, in `dasllama_metal_tower.das`'s family - hooks, and in files under `tests/` - the set `REVIEW.das`'s seam check enforces. -- **`dasllama_audio_embedder.das`** - the audio carrier: `AudioEmbedder` / `AudioState`, the - vision carrier's audio twin - one union through every seam (server media worker, facade - `encode_audio`, tutorials), the family probed from the mmproj's audio tensor (or a `.dlim`'s - baked tag) at load, one-line arms. Outside a family's own file, an audio family type is named - only here, in `dasllama_asr.das`'s union field and one-line arms (the ASR rail's own carrier), - in the metal family hooks (`dasllama_metal_tower.das`, `dasllama_metal_asr_dec.das`), in - `benchmarks/asr/asr_bench.das` (the ASR bench pins a family serving knob for its correctness - rail), in `utils/dasllama-convert/main.das` (the per-family `.dlim` bake tool dispatches on family by - design), and in files under `tests/` - the set `REVIEW.das`'s seam check enforces. - -Vision oracle provenance (the convention `REVIEW.md`'s fixture rule points at): real image -fixtures and mmproj files live in the models dir with `.sha` pins, fetched never generated -(their `performance/fetch_models.das` entries are the checkable pins); the mtmd reference dumps -live beside them in `gemma4-vision-oracle/`, `gemma3-vision-oracle/` and -`qwen3vl-vision-oracle/`, whose `mint.sh` (gemma4uv, qwen3v Omni), `mint_4b.sh` (qwen3v -deepstack), `mint_25o.sh` (qwen25v), `mint_e2b.sh` / `mint_e4b.sh` -(gemma4v) and `mint_gemma3.sh` (gemma3v) record the -exact `llama-mtmd-debug` / `llama-mtmd-cli` invocation that minted each -dump, so regeneration is a command, not archaeology. An encode oracle dump is minted on the -CPU, `-fa off`, from the f32-widened mmproj twin - the only true-f32 reference arm (the -reference's Metal "f32" GEMM stages half operands, its flash-attention path casts K/V to f16, -and the shipped bf16 mmproj rounds activations to bf16; its own four arms spread -<= 6.5e-3 on the gemma4v tokens). - -### 1.8 Instrumentation and support - -- **`dasllama_parity.das`** - CPU-reference caches for the parity instruments. Test-facing, but - library-side because the caches outlive a single suite. -- **`dasllama_prefix.das`** - the prefix/page cache for evaluated token history. -- **`dasllama_lint.das`** - the facade boundary as a compile-time lint (DASLLAMA001): every - engine module carries it, so a consumer requiring anything under `modules/dasLLAMA/` but the - entry modules (facade, scheduler, exchange pair) fails to compile. Escape: - `options _dasllama_internal = true` - engine files, this module's own tests/harnesses/ - benchmarks/rigs, and the ruled consumers: `utils/dasllama-convert` (the bake tool reads - the mint rail), `utils/dasllama-server/model_catalog.das` (the env rail + the model-set - serving view; it re-exports `model_specs` so the server's catalog tests can gate the - serve rows), `utils/dasllama-server/test_openai_server*.das` (env-registry test knobs), - `modules/dasLLVM/daslib/llvm_user_modules.das` + `modules/dasLLVM/tests/test_{grid,tune,tuned}.das` - (the tune-generator contract). -- **`performance/model_specs.das`** - the ONE model-set table: per carrier, the profiled- - catalog fields, the official-board flag, the serving-catalog fields (`serve_*`, incl. - `serve_vision` naming a tower companion by name), the provenance pin (exact HF repo + - revision + sha256, or the on-box conversion recipe; companions hang off their owning entry, - and a companion shared by several rows is referenced from the others by name), and the - frozen parity evidence as data (pinned ids + arms). `llm_catalog` / `official_catalog` / - `models_provenance` are views over it; `serve_vision_tower` resolves a tower name to its - pinned companion, and `serve_asr_tower` is the one standalone pinned artifact the file - carries - the serving ASR tower (parakeet v3), consumed by `asr_catalog`'s v3 row in - `profile_common.das` (the pin's single source lives here, not there). -- **`performance/box_ident.das`** - the hardware-keyed box tag: the fleet table maps every - recorded box's CPU brand to its store tag (the M3 row is an EXACT match so bigger M3s slug), - the `DASLLAMA_BOX` override wins, and unknown hardware slugs from the brand - never another - box's store. `profile_common.das`'s `box_name()` and `tests/run.das`'s manifest pick both - resolve through it; the old per-file `"m1"` literals are gone. -- **`performance/fetch_models.das`** - the fetch/verify driver over the provenance view. - Verify by default, `--fetch` downloads; it never converts on `--fetch`, never benches, and - it touches no tune state (`tune_policy(missing="fallback")`). BRINGUP.md sec.2 is the runbook. -- **`dasllama/dasllama_exchange_schema.das`** - engine-free validation for exchange submissions - (record stores + tune sidecars); the dasllama.io ladder service builds on it. -- **`dasllama/dasllama_exchange.das`** - the sidecar-exchange client (a sanctioned public entry point): the boot-time - lookup/apply (llvm_tune's scope resolver - a verified per-box match downloads instead of a - ~20-minute tune), the privacy-stripped submit rails, and the control-page surface - dasllama-server serves at `/exchange`. -- **`benchmarks/asr/mem_census.sh`** - the peak-memory census (`/usr/bin/time -l` around one - asr_bench process per cell; macOS only) - the interim footprint instrument until a footprint - leg lands in `gen_bench_records`; its numbers live in `PERF_LEDGER.md`, never the stores. - ---- - -### 1.9 Serving - -- **`dasllama_scheduler.das`** - the continuous-batching scheduler, the serving layer over the - facade (its one engine require is `dasllama/dasllama`). One synchronous thread: each - `scheduler_step` admits queued requests, runs one `eval_batch` decode step over every - decoding stream, then at most one bounded prefill chunk FCFS; paged serving donates finished - streams' KV pages to the prefix cache. Results flow out as `SchedEvent`s - no HTTP here. - `utils/dasllama-server` owns the writers; `tutorials/dasLLAMA/13_serving.das` is the - teaching consumer; `tests/test_scheduler.das` gates it against `generate()` references. +Every file under `dasllama/` appears in a companion. Two carry generated CONTENT rather +than being generated themselves: `dasllama_env.das` holds the `[EnvConfig]` knob declarations +(`ENVIRONMENT.md` is generated FROM it by `harness/gen_env_doc.das`), and `dasllama_unicode.das` +is hand-written around transcoded Unicode data tables (the RANGES/WS tables - regenerate by +re-transcoding `$LCPP/src/unicode-data.cpp`). ## 2. Mechanisms -The "why" behind criteria that REVIEW states in one line each. - -### 2.1 There is ONE way to load a model - -A weight carrier becomes a live struct through exactly two functions here, and nothing else may -read weights into one: - -- **`build_image`** walks a carrier's planes into a sink - a `.dlim` file, or a page-aligned memory - chunk when there is nowhere to write. -- **`parse_image`** turns `(base, bytes)` back into borrowed-plane fields, and does not know which - sink produced them. - -Cold and warm therefore yield the SAME struct. A cold load reaches it by building the image and -handing off *through the file* - write, drop the model, map - so the model and its image are never -both resident. That handoff costs a close and a re-map of a multi-GB file and is the *slower* cold -start on purpose, under the tiebreak in sec.3. `cache_via_image` is that handoff for every weight -carrier; the streaming forms transcode planes from the gguf mapping straight into the image so they -never materialize at all. - -**Suites load each carrier through its own loader - decoders via `load_model_`, towers and -embedders via their family or carrier loaders - never the image rail.** `load_model` / -`load_model_cached` mint identity-stamped `.dlim` flavors and GC-purge siblings. A suite -child's pinned identity (backend pin, wscale, tune manifest) differs from the serving rig's, -so a suite on the rail both re-mints multi-GB images the rig cannot use and purges the flavors -the rig depends on. Image-rail coverage belongs to the image suites alone. - -**An image is judged only where its identity can be recomputed - the LANE rule.** An identity's -(quant, tag) pair is its lane, and lanes coexist by design: a bake proves dead only its OWN lane's -siblings, plus BROKEN and version-stale images anywhere. Everything else is `FOREIGN` - kept, and -counted in the log. Two contexts produce lanes this process cannot reproduce: a GPU flavor (a blob -bake selects its own backend and winners, so the string differs even where the tag matches: -`portable|q8 mr4` against a planar `arm64-gen|q8 mr8`), and a family tag whose module is not loaded -(the tag registry is filled by each family's `[init]`, so a process that never required whisper has -no way to know a whisper image is current). A process that cannot recompute an identity has no -standing to call it dead - the orchestrator of a sweep least of all, since its cells measure through -an exe carrying its own baked winners. The one owner carve-out is the batch lifecycle: the board -rigs own the model dirs for a whole run, so `dlim_wipe` (verdict-blind, `dasllama_image.das`) -clears them behind the exe gate at batch start and after each model's last cell, with every image -re-baked from its gguf on demand. Judging stays forbidden; owning the directory for the batch is -what licenses deletion without judgment. - -### 2.2 Kernel SHAPE is compile-time; only DATA is runtime - -The test is one question: *for a given compiled kernel, can this value change between dispatches?* - -- **Yes -> DATA.** Context depth, row counts, buffer offsets, `kv_dim`, scales, head counts. It - belongs in a uniform or a kargs struct. -- **No -> SHAPE.** A codec's block stride, a scale-plane stride, a lane width, an unroll factor, a - format selector. It must NOT reach the kernel as a uniform, a kargs field, or a helper parameter. - -Shape belongs to the specialization: a separate kernel class and PSO, a per-codec overload, a -monomorphized generic, or a `static_if` on a compile-time witness. Handing a shape constant over as -a value and trusting the shader compiler to fold it back is an assumption, not a guarantee, and it -is worth nothing in the kernels that matter. The same rule bans indirection in a kernel body - no -function pointers, no vtables. - -**Verify against the EMITTED shader, never the das source.** Read the `*_msl` global or the SPIR-V -dump and confirm the constant is literal there: `blk * 34u`, not `blk * bstr`. A helper that looks -specialized in das can still lower to a runtime multiply. - -**kargs structure.** Twins of a family bind the SAME kargs type at the SAME binding, even where one -twin ignores a field. A twin that carries an extra scalar must not shift the others to different -slots, because that asymmetry propagates into the encoder as a per-form branch. Two tells that a -fold is overdue: - -1. *A value reaches the encoder twice* - a pooled scalar uniform BUFFER passed alongside the - identical value as a parameter (`bd` next to `d`). The buffer is uploaded and released per step - to carry a number the encoder already holds. -2. *A field is a function of the fields beside it.* An expert plane's block stride is - `kdim * ndim / blocksize`; a reciprocal scale is `1/sqrt(dim)`. Derive it in the builder - each - one passed separately is a second place to get it wrong. Likewise, when a kernel's grid IS the - geometry it reads, take the grid off the kargs rather than re-passing the numbers. - -**Nothing dispatches a kernel except its `enc_*` builder.** A hand-rolled bind list elsewhere - a -tune-race harness, a benchmark, a probe - duplicates the builder and desyncs the moment the -family's args change, silently: the slots still exist, the types still compile, and the kernel -reads a struct out of a 4-byte buffer. The dispatch census only catches a builder that binds kargs -on some paths and not others; a duplicate that binds NO kargs is invisible to it. - -### 2.2b Tensor-GEMM shapes that measured out (M5, interleaved-race evidence) - -The forms below were built, raced against the crowned tg-staged q8u GEMM at real model -shapes, and LOST; the probe that holds the GEMM numbers is -`benchmarks/matmul/bench_metal_nax_probe.das`, and the fused-attention numbers live in -`benchmarks/attn/bench_metal_pf_fused_attn.das`. Re-attempting one without new structure is -re-buying a measured loss. - -The probe also RETAINS the decided-and-shipped arms (the half-A stream, the dev-W all-device -form, the tall M-tile twin, the bk staging depths, the no-zero-init form) as hand-written MSL -reference implementations beside the refuted ones - they are the arc's bisect ledger, and the -sync duty is REVIEW_GPU.md's - the rule that a diff changing what a hand-binding arm mirrors -fixes or deletes that arm in the same change. - -- **Per-simdgroup register-fragment matmul2d (16x32x16, device -> `vec` fragments):** - 1.7-2.0x slower for weight GEMMs, vectorized loads and deep n-blocking included. The - fragment architecture pays for attention's streaming operand reuse (one resident Q against - a K/V walk), not for a GEMM's operand traffic - cooperative tg staging wins there. -- **Mixed-integer matmul2d operands (float x int8, and the i8 x i8 per-block-fold form):** - 3.5x slower - the mixed-int combinations exist in MPP's type lists but lower off the NAX - fast path. W8A8 claims from other stacks do not transfer through MPP. - -**Sanctioned float-A stamps** (the licensed set of the rule that a `matmul2d` operand reaches -the op as `float` only in a class stamped `[metal_kernel(float_a_ok=true)]`, each carrying -that stamp): every tensor template's `XT = float` stamp - the live fallback wherever the half -panel is absent (below the convert row floor, panel does not fit, half-X pinned off) - and the -batch-decode/classifier `MetalQ8GemmTensorT` family, whose half-X extension is an open ledger -item. A float A operand anywhere else is the defect the rule names. -- **Fused single-kernel attention (scores in threadgroup, online softmax):** loses 10-80% to - the pipelined three-pass at real shapes (`benchmarks/attn/bench_metal_pf_fused_attn.das`) - - Metal's cross-kernel pipelining plus full-width softmax beat tg-scope fusion. - -The positive laws these races established - half operands, stage-only-to-transform, -consecutive staging runs, relaxed_precision always - are `REVIEW_GPU.md` rules and the -`modules/dasMetal/REVIEW.das` descriptor gate; this section keeps only the refuted shapes -and why they lose. - -### 2.3 GPU-resident cache identity - -An address-keyed entry carries its SPAN, and a hit must cover the request - a shorter first upload -must never serve a wider later one. Different upload FORMS (plain span vs concat) live in separate -tables so they can never alias; the metal `RegionEntry` rail is the model. Buffers grown out of an -entry retire to a list released only at quiesce boundaries, because unretained command buffers may -still bind them. - -### 2.4 Complexity and length lint - -STYLE037 (cyclomatic) and STYLE038 (line count) are prompts to look, not orders to refactor. This -module has shapes that are irreducible by design and they take `// nolint:STYLE03x` with a one-line -reason: flat one-call-per-item runs (a registration or release list with one line per kernel), and -GPU kernel bodies whose phases are coupled by barriers, cooperative-matrix ops or register -residency and so cannot cross a function boundary without changing the shader. - -Split only where a real seam exists - genuine duplication, a distinct phase, a self-contained arm - -and only when the extracted helper stands on its own. Two corollaries this module keeps tripping -over: **a kargs fold that grows an already-over-cap kernel body is not a reason to abandon the -fold** (unpacking N fields adds N lines; take the growth and ledger the real seam), and **never -suppress a function you have just argued is reducible** - if it is on the follow-up ledger wanting a -dedup, it keeps its warning until the dedup lands. - -### 2.5 There is ONE benchmark rig, and the records are the baseline - -`benchmarks/lcpp_bench.das` is the only thing that measures performance. It is a *mirror* of -the upstream `llama-bench` - the same test shapes, rep counts and timing boundaries, applied to -our engine - so `pp` is one batched prefill of `-p` tokens from an empty cache per rep and `tg` -is `-n` single-token forwards with no logit read, each row one untimed warmup plus `-r` timed -reps. The real `llama-bench` runs only when `--ref ` is passed; that is how the upstream -columns were produced, and they are pinned, not re-measured. - -`performance/gen_bench_records.das` sweeps a board by spawning that rig once per cell, and -writes `performance/records/.json`. `gen_site_records.das` merges those into the file the -site renders. A stored row carries its own command, sha, version, tune stamp and exec format, so -a number is self-describing rather than a bare figure in a table. - -**Regression checking inverts the same rig:** `gen_bench_records.das --oracle --legs metal` -takes the store's das rows as the work list, re-measures each once, and gates one-sided against -its stored mean (fail past 5%, warn past 3%, gains flagged as suspicious). Upstream never runs, -the store is never written, and a text cell's timed child runs `--frozen` - a prepare pass bakes -and warms its image first (the batch starts wiped), so the timed cell never converts; ASR and -image-chat cells bake what they need mid-cell, like their publishing legs. A second -harness would produce numbers that cannot be compared to any of this, which is -why writing one is a review defect. - -**The tune stamp gates the comparison.** A manifest older than the binary fails every cell, and -an untuned invocation re-execs into a full retune rather than measuring - so re-mint the box -manifest and check its winners against the stored rows' `tune` stamps before trusting a delta. - -**A measured number proves its kernel provenance through `tune_gate()` -(`performance/profile_common.das`), one arm per world it can run in.** Three worlds, because -`tune_status()` populates in exactly one of them: a standalone exe checks the sidecar the -release shipped beside it; a `DAS_TUNE_MANIFEST` run checks that file; a plain script checks -that every `[tune]` row stamps a manifest winner. An invocation no arm covers refuses - or -worse, measures on fallback kernels - which is why every measuring entry point calls the gate -before its first timed rep. Two rig shapes fall outside "measuring entry point" by the -property itself, ledgered here: a kernel A/B lab dispatches its variants through its own arms -(never the `[tune]` selection), and `lcpp_bench.das`'s `--tok` cell dispatches no kernels at -all - neither can measure a fallback silently. - -**The retune re-exec bites scaffolding, and the pin for it is checked in.** Any bare `daslang` -run that requires the engine - a probe, a one-off script, a REPL experiment - re-execs into a -full retune when no manifest is armed. `performance/last_known_good_sidecar.json` exists for -exactly that: a frozen copy of a complete, noise-gated mint, tracked in git (the `*.tune.json` -ignore rule deliberately does not match it). Point `DAS_TUNE_MANIFEST` at it and the framework -never retunes; on a different box the identity mismatch just serves fallbacks, and a copy minted -before the current `DASLLAMA_VERSION` serves fallbacks on any box - the compile says which with -one `WARNING DAS_TUNE_MANIFEST` line per scope. That is the whole -contract - it suppresses the re-exec, it does not tune the box, and a number measured under it -is not a benchmark. Benches and the rig keep minting their own; refresh the copy when a -re-mint moves the crowns or `DASLLAMA_VERSION` bumps. - -### 2.6 Capability questions and readiness questions are different questions - -A predicate that mixes them cannot be reused. `prefill_decline` answers "can metal serve this -model" (capability) *and* "is this window staged" (readiness - are the rope tables built). A caller -that runs before the window is staged must ask the capability half only, or it gets "not yet" -forever and its feature silently never runs. Split such predicates rather than reordering the -caller; an optimistic capability answer is safe when the late path has a fallback, and here it does. - -### 2.7 A quantized activation carries its scale lattice (Vulkan) - -Two activation quant forms ride the vulkan rail, and they differ in the SCALE LATTICE, not the -int8 payload: the Q8_0 form scales per 32 values, the superblock form per 256 (with per-32 -sub-scales inside). A compiled kernel indexes ONE lattice - the q8 GEMV/GEMM rail reads per-32 -scales; the k-quant (k4/k5/k6/q40) kernels index the per-256 lattice. `kq_sb(fmt)` is the -predicate (sec.1.2), and it answers for the WEIGHT plane the dispatch consumes. - -Three consequences the code is shaped around: - -- **A quant/act encoder is picked by the CONSUMING plane's format, never by a rail-wide - default.** Every site that encodes activations for a GEMV/GEMM keys its encoder (and its - grid: superblock counts are `n/256`) on the consumer's `kq_sb`. The failure mode is silent - per-dispatch: the wrong lattice indexes garbage scales, outputs stay finite, and nothing - panics - only end-to-end token parity (`harness/parity.das`) catches it, so a resident - change is witnessed only by parity runs over both a q8 and a k-quant model. -- **The fused add-rms+requant twin exists only for the per-32 form.** The rail gates it on - "every consumer of this buffer is Q8_0-scaled" (`rd_x_quants_b32`), and the profiler stamp - shape must ride the SAME gate, or profiles desync from what actually dispatched. -- **A GEMV group sharing one activation buffer must be lattice-homogeneous.** q/k/v share one - quantized x; gate/up share another. Resident arming classifies each member's consumer form - and DECLINES a mixed group rather than serving one member wrong scales. - -### 2.8 Every program root declares its stack budget and its prefill intent - -`options stack` is main-module-only: it does not unify up from required modules, so no library in -the forward chain can declare the depth it needs. Every program that drives the engine - each test, -harness, benchmark, and tool - must therefore declare it, and dasLLAMA's frames are deep enough -(by-value `Session`s, the forward/prefill chain, the generated kernel tier) that the default is -never enough. - -The budget is **one number in every root**, currently 524288. Per-root numbers do not survive: a -frame that grows past the smallest declared budget breaks only the program that declared least, so -the limit is discovered by crashing - and the program that crashes is whichever one is run -rarest. A measurement rig sized below a test suite is the worst case of this, because the suite -stays green while the rig dies. The cost of the uniform number is reserved address space per -context; the cost of per-root numbers is a runtime crash found by the least-covered program. - -The second declaration is prefill intent. A model-loading root declares `allow_cpu_prefill()` -on the arms that hit the CPU-prefill guard - `set_metal_mode` with a runtime value declares -nothing, since `MetalMode.off` leaves the guard armed - and logs, once, which configuration it -ended on. The guard panics, and a panic takes every live stream down, so an undeclared root is -a serving outage waiting on its first long prompt. Both halves of root discipline are enforced -by `tests/test_program_roots.das`; weakening that test is a review defect -(`tests/REVIEW.md`). - -### 2.9 Environment knobs - -A knob is an `[EnvConfig]` field in `dasllama_env.das`, read as `g_env_*.`; the field is -also what generates its `ENVIRONMENT.md` row, so a knob declared anywhere else is invisible to -the documentation and to the registry test. The sanctioned forms beyond a plain typed field: - -- **Tri-state knobs** - presence matters, or the effective default is computed at runtime - are - `Option` fields. -- **Dynamic names** - a variable named by data, not by code - go through `env_is_set` / - `env_value_of`; there is no field to declare because the name is not known at compile time. -- **The config loads once at context init**, so `set_env_variable` mid-process is invisible to - the running config: arm a child process's environment instead. -- **A write of a foreign library's knob** (`set_env_variable` with a literal name) is allowed - only before that library first reads it, and the name must be a declared `[EnvConfig]` knob - - the registry test scans writes too, so a re-spelled name fails it. - -`tests/test_env_registry.das` enforces the lot in both directions (declared <-> documented, -read <-> registered, writes included). - -**Override announces.** A knob that is a gate escape, a policy override, or a threshold -recalibration - one whose presence makes a run measure, mint, or emit something its defaults -would not (`DASLLAMA_ALLOW_UNTUNED`, the `DAS_TUNE_NOISE_*` pair and `DAS_TUNE_POLICY` read -through llvm_tune's accessors) - announces at the point it changes the outcome: at least one -printed or logged line naming the knob by its env spelling. Set-but-inert stays silent; -per-site repeats are correct (the same knob may legitimately announce at its `[init]` wire, at -a gate it bypasses, and in a tool banner). Ordinary behavior knobs (thread counts, rail -selection, formats) are not overrides under this contract - their state belongs in the tools' -existing config/status lines, not in per-knob announces. - -### 2.10 Sanctioned instrumentation rails - -Engine timing goes through the rails that aggregate and tag it: the `jobque_profile` markers -(`profile_tag` / `profile_marker` and the `trace_*` wrappers in `dasllama_math.das`), the -`prof_add` / `forward_profile_*` decode buckets (`dasllama_common.das`), and the `asr_prof_add` -encode buckets (`dasllama_audio.das`). A new clock read paired with a print of the elapsed -interval bypasses the aggregation - it measures one call site once, is not filterable, and rots -where a rail entry would keep serving. Where a timed line IS the deliverable - `benchmarks/`, -`performance/`, `harness/`, and cold one-shot load/mint progress logs (image bake/map, load -stages, tokenizer build) - the rails do not apply. A clock whose value feeds logic is control -flow, not instrumentation; it is marked `// clock: control` so the sweep and any future lint -leave it alone. - -### 2.11 The [hot_path] coverage model - -`[hot_path]` sits at the REGION ENTRY - the `*_encode` / `*_decode` / step drivers - and its -`[no_alloc]` / `[no_env]` / `[no_io]` contracts arm transitively down the call graph, so -interior kernels stay bare. A new function needs the annotation itself only when no annotated -entry reaches it: a new entry point carries it, and a new backend entry (kernel-backend -override, batch donor) carries it too, because backends are also reached from un-annotated -harness paths. Reused buffers take `@scratch`; debug and profiling legs take `[cold_path]`. -The tokenizer encode/decode path is sanctioned UNCOVERED by the region contracts - its perf -gate is the `--tok` scaling rows, whose instrument (the size-ladder ratio) catches what the -contracts cannot. - -### 2.12 The post-CPU-burn GPU ramp - the residency heartbeat holds it; do not build a warm-up - -Figures in this section: M1 Max, 2026-08-23 - probes are quiet `-jit` runs with the rig's -tune manifest; cells are the released `lcpp_bench` exe (`--image --image-think -r 3 -t 8 ---ngl 99`); full tables in `history/dasLLAMA/qwen3vl_plan.md` slices M/J. - -After a CPU-only phase, the first Metal submission runs degraded - one time per window, -entirely in the kernel-side driver window (kernelStart->kernelEnd; queue hand-off and GPU -execution stay flat - split probe-verified 2026-08-23, `harness/residency_ramp_probe.das` + -`das_metal_boost`'s `metal_submit_trace`). The long-window mechanism is the OS collecting a -committed+requested `MTLResidencySet` during inactivity; the fix (SHIPPED) is the -residency HEARTBEAT - a dasMetal background thread re-requesting -residency every 5 ms for `DASLLAMA_METAL_HEARTBEAT_S` (default 180 s, 0 = A/B rail) after -the last served step, kicked from `residency_flush`. Measured on the qwen3v tower encode: -3000 ms burn drv 17.7 -> 3.0 ms. The `MTLResidencySet` pin itself -(`DASLLAMA_METAL_RESIDENCY`, on by default) holds the short-window case (-15 ms/encode on -the tower's first submission after its CPU stem); the heartbeat holds the long ones. -REFUTED by measurement (`PERF_LEDGER.md`, the heartbeat entry): every pre-payment - empty -command buffers and driver round-trips (0.107 ms - never asleep), single-dispatch kernels, -per-page touch kernels, light pulse-trains through the burn - a warm-up always pays its own -cost ON TOP of the slack it was meant to hide; do not re-attempt one. Also refuted: a -whole-map no-copy anchor buffer (page pre-wiring moves nothing) and the once-proposed -per-buffer tracking audit (resource count is not the lever) - prefill pools simply stay -tracked (untracked reds parity). The ~1-3 ms window-scaled residual is the driver/GPU -idle-state wake class, no user-space lever found. Related, same ledger: merely arming Metal -makes a CPU q8 tower encode ~1.7x slower - mechanism unnamed, deleted by a GPU-served tower. - ---- +- `ARCHITECTURE_IMAGE.md` - sec.2.1-2.1g: the prepared-image rail. +- `ARCHITECTURE_GPU.md` - sec.2.2b: the tensor-GEMM and fused-attention shapes that measured out. +- `ARCHITECTURE_RUNTIME.md` - sec.2.2-2.4, 2.6-2.9, 2.11, 2.12: kernel shape, caches, lint + policy, knobs, coverage, the GPU ramp. +- `ARCHITECTURE_MEASUREMENT.md` - sec.2.5, 2.10: the benchmark rig, the tune gate, and the + sanctioned instrumentation rails. ## 3. Inherited invariants -Durable "why it is built this way" facts harvested from the design docs archived under -`history/dasLLAMA/`. Violating one of these is a bug, not a style choice. - -**From `x64_arch.md`:** - -- **The LIBRARY never runs interpreted or AOT - the scaffolding may.** Inference - anything that - runs a model (eval / decode / prefill / generate) - is `-jit`-only, hard stop: an interpreted or - AOT model run is not "slow but correct", it is out of contract, and the library panics on it. - Loop hints and intrinsic lowering exist only in the JIT, and the JIT tier is the only one the - oracles gate. **Scaffolding is exempt:** conversion utilities (`.dlim` bake, image processing), - debug scripts, and batch/driver tools that spawn the real runner as a child may run interpreted. - Enforcement is `guard_interp_gguf_load` (`dasllama_image.das` - big-load panic + - `DASLLAMA_ALLOW_INTERP_LOAD` escape for the conversion path) plus `guard_interp_inference` - (`dasllama_math.das`), which panics at the library seams: `make_run_state`, ASR session - creation, and `vad_chunk_prob`. (The - x64-era "fallback rail" framing - AOT/interp as correct-but-untuned tiers - is retired; - intrinsic fallback bodies exist for off-ARCH correctness *inside* the JIT, not as runnable - tiers.) -- **Correctness before speed, token-for-token.** The engine is validated against external oracles - (llama2.c + upstream `simple_ids`) plus per-arch parity fixtures. A new kernel passes the suite - *and* the oracles with the new backend active before any perf claim. -- **Peak memory before cold-start latency - a DELIBERATE trade, and the standing tiebreak.** When a - load-time choice pits footprint against wall-clock, dasLLAMA takes the smaller footprint. The - asymmetry is not close: overshooting RAM on a big model is fatal (the OOM killer, or swap that - makes the whole box unusable), while a slower cold start costs seconds *once per process* and - costs the warm path - the common one - nothing at all. The prepared-image rail is where this bites - hardest and where the shape is set: a cold load writes its `.dlim`, drops the model, and maps the - file back rather than serving the copy it already has in RAM. That is a real close/reopen and a - real re-fault of a multi-GB file, and on an image larger than the page cache it is a re-read from - disk. Measured on Llama-3.1-8B-Q8_0 (9.6 GB image, M1 Max, `phys_footprint_peak`): planar cold - **3.7 s -> 4.5 s**, peak **12.3 GB -> 5.4 GB**. We bought a 2.3x footprint cut with ~20% of one cold - start. Apply the same tiebreak everywhere - bake, convert, KV growth, GPU staging - and when a - change goes the other way, it needs the measured pair (peak AND wall) and an explicit call, not an - assumption that faster is better. -- **Cold builds STREAM; nothing materializes a model to write one.** The rail that makes the trade - above cheap: a cold build transcodes each plane from the gguf mapping straight into the image, so - the model never exists in RAM at all and the peak is a few per-tensor temps over the small - RAM-resident scale planes. This covers the planar flavor and the metal blob flavor alike - the - blob's gguf-native 34B blocks assemble per tensor on the way out, and because a blob forbids CPU - repack, it does not even need the resident `qscales` plane the planar stream keeps. Metal cold on - the 8B: **19.5 GB -> 4.8 GB, 4.1 s -> 3.1 s** (faster AND smaller - the eager rail had been - splitting gguf's interleaved blocks apart only to reassemble them). The vulkan flavor rides it - too: its bake collects from the GPU walk of a model that is **already serving**, so it needs a - mapping rather than a load - which is what the warm path always did, and now the cold path does - the same instead of loading eagerly (**15.4 GB -> 6.6 GB**). A new flavor joins the streamed rail - or explains why it cannot. -- **Token-exact oracle tests pin the bit-exact path** (classic attention, scalar activation); - approximate/fast paths get separate tolerance tests. Rerouting an oracle test through a - non-bit-exact default makes it pass on the machine it was frozen on and flip elsewhere. -- **The three-layer safety model that makes a new ISA free:** (1) *registration gate* - an ISA's - `[init]` never fires off-arch, so `portable` stays active and every run is correct; (2) *scalar - fallbacks* - an intrinsic called off-target computes the right answer from its fallback body, only - the JIT emitter lowers it to hardware; (3) *tests drive public wrappers*, asserting only that *a* - row-major default is active, never an arch-specific name. A new backend is a file plus an `[init]`. -- **The u8xs8 sign trick, and which operand takes the `|a|`.** ARM `sdot` is signedxsigned; x64 - `vpdpbusd` is *unsignedxsigned*. The exact integer identity is - `dot(a,b) == dot(|a| as u8, sign(a) applied to b)` - and `|a|` goes on the **unsigned** operand, - because `|-128| = 128` fits u8 but not s8. Products fit s16, four-product sums fit s32: bit-exact, - no saturation. - -**From `kv_cache_compression.md`:** - -- **The KV layout is ragged per-layer, not flat.** `kv_row_total` *sums* per-layer `layer_kv_dim` - (layers differ under SWA/global splits), and a layer's base offset is a **prefix sum** - (`kv_row_prefix[l] * seq_len`). A codec keys strides and offsets off the prefix array - never off - one flat `kv_dim`. -- **Cross-layer KV sharing stores nothing.** When `kv_src[l] != l` both store sites are skipped and - the layer's offset **aliases the source layer's region**. Any codec must preserve that aliasing. -- **Convert the query, not the cache.** The query is `O(head_dim)` per step; the cache is - `O(context x head_dim)`. Dequantizing the cache on the decode read costs *read compressed + write - f32 + read f32* ~ 2x the f32 baseline - strictly worse than storing uncompressed. -- **Prefill and decode are asymmetric on purpose.** Prefill is compute-bound and tolerates external, - one-layer-at-a-time decompression (a fresh prompt reads the f32 `k_b`/`v_b` scratch and - decompresses nothing); the decode read is memory-bound and **must fuse** dequant in-loop. - Attention consumes one layer at a time - never materialize all layers at once. -- **BLAS is ruled out structurally, not incidentally.** `cblas_sgemm` is f32-only (no hgemm, no - int8), so any quantized operand must expand 4x into f32 scratch - and the weight matmul is bound on - bytes streamed, so quantized-native `sdot`/VNNI wins regardless of FLOPs. **Correction:** that - doc's "BNNS not pursued" tail is superseded by the AMX arc - `dasllama_math_accelerate.das` - ships an Accelerate float-batch override plus a BNNS-f16 lane for bf16 planes (`DASLLAMA_ACCEL` / - `DASLLAMA_ACCEL_F16`). BLAS-for-quant stays ruled out; Accelerate/BNNS on genuinely-float planes is - live and measured. - -**From `turboquant_research.md`:** - -- **Stage-2 QJL is skipped deliberately** in the tq4 KV codec - its unbiasedness buys variance that - softmax amplifies, and the damage grows with context. Five independent groups plus vLLM upstream - converged on Stage-1 only (rotation + Lloyd-Max Beta codebooks + per-block norm). Do not "restore" - QJL as a completeness fix. -- **Benchmark tq4 against `q4_0` + Hadamard, not f16.** f16 is the comparison the method wins by - construction; `q4_0`+Hadamard is the one it has repeatedly *lost* on small/dense models and long - contexts. Quoting an f16 ratio is the mistake this doc exists to prevent. -- **Two corrections that travel with the codec:** the estimator is **asymmetric** - the query is never - quantized, so a QJL dot is sign-XOR over fp lanes, not popcount (popcount needs the query binarized - too, an off-paper accuracy trade); and "3.5-bit" is **mixed per-channel bit allocation** over - head_dim, not fractional coding. - -**From `API_REWORK.md`:** - -- **The Tier-1/2/3 scope model.** Tier 1 = scalar `Config` knobs (RoPE variant/scaling, norm kind and - placement, activation, QKV bias, softcaps, sliding window, embed scale). Tier 2 = block swaps that - change dataflow (MoE, QK-norm, per-layer attention patterns, shared KV, PLE, sinks, parallel - attn+FFN). Tier 3 = deep forks needing a pluggable KV/attention core. **CORRECTED:** the hybrid - Gated-DeltaNet lane that doc lists as deferred Tier-3 **SHIPPED** - `dasllama_arch_qwen35.das` - registers `qwen35`/`qwen35moe`/`qwen3next`, gated by `tests/test_deltanet.das`. The arch registry - (18 names across 13 `dasllama/dasllama_arch_*.das` files) is the truth for what exists. -- **Arch files are declarative registrations.** An arch file builds an `ArchDesc` - (name * `configure` * the `ArchBlocks` fn-ptr quad * `ChatTemplate` * `LlmCaps`) and calls - `register_arch` at `[init]`. Adding an arch touches no forward loop. -- **Family behavior is distributed by `Config` flag, not dispatched by name.** GeGLU-vs-SwiGLU is an - `ffn_act` flag *inside* the shared dense FFN block; only a genuinely new dataflow (MoE) earns its - own `ffn_*` block pointer. An `if (arch == "...")` on a shared path is the anti-pattern. -- **`register_arch` MOVES the `ArchDesc`** (`g_arch_registry[name] <- desc`). Clone the template - **before** registering an alias, or the second call registers a gutted descriptor - the live - pattern in the qwen2 / qwen3 / qwen3moe / qwen35 files, each with the comment saying so. -- **What rides where:** `ArchBlocks` (copyable fn-ptrs) is copied onto the `Model` at load; the full - `ArchDesc` (non-copyable - it holds a `ChatTemplate`) stays in the registry. Image loads re-bind - blocks via `rebind_arch_blocks` and must **not** re-run `configure` - the serialized config already - carries configure's flags *plus* the GGUF's overrides. -- **The kernel/ISA seam is symmetric with the arch seam, plus one extra rule.** `dasllama_math.das` - owns the abstraction (typedefs, active pointers, public wrappers); ISA modules self-register at - `[init]`. Two selection tiers: `register_kernel_backend` auto-activates only the best **no-repack** - backend, so direct callers (tests, benches, row-major weights) always get a kernel that works on - unrepacked data; `select_matmul_backend_for_load()` is the **only** path that activates a - `needs_repack` backend - eagerly activating one runs an interleaved kernel over row-major data, - i.e. silent garbage. +`ARCHITECTURE_INVARIANTS.md` carries this section. diff --git a/modules/dasLLAMA/ARCHITECTURE_ENGINE.md b/modules/dasLLAMA/ARCHITECTURE_ENGINE.md new file mode 100644 index 0000000000..d5ef22fd07 --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_ENGINE.md @@ -0,0 +1,208 @@ +# dasLLAMA Architecture - engine file charters + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +### 1.1 Engine core + +- **`dasllama.das`** - the public API facade and nothing else: `load_model` -> `create_session` -> + generate, re-exported names, the doc surface. No engine logic; a function that does work belongs + in the module that owns the concern, and the facade re-exports it. +- **`dasllama_version.das`** - `DASLLAMA_VERSION`, the module's own release counter, and nothing + else. Decoupled from the daslang version and from `LLVM_JIT_CODEGEN_VERSION`; ANY kernel work + bumps it (REVIEW.md), so equal versions mean an equal kernel roster - the sidecar exchange + keys validity on (version, box). A zero-require leaf so the tuner's noise half, the benches, + and the facade chain (`dasllama_common` re-exports it) all reach it without weight. +- **`dasllama_common.das`** - the engine: `Model`/`Session`/`Config`, the forward loops, the + override registries, the runtime knobs. **Not** the load walk (sec.1.3) and **not** GPU residency + (`ARCHITECTURE_GPU.md` sec.1.5) - both left, and the seam each left behind is a registered hook, + so neither comes back. + It remains the module's debt sink; what sits here that is family-specific or platform-specific is + debt, **not precedent**. Ledgered exceptions the sidecar-apply seam forces + (the knob rail lives here, and the backends require this module, so their sidecar knobs cannot): + the metal tensor crowns, and the three metal prefill raced-constant knobs beside them - both are + applied by `apply_box_profile_runtime` and CONSUMED by the Metal backends at their own init. + `REVIEW.das`'s `check_fastmath_default` licensed set is empty by design: no engine kernel opts + out of fastmath, and an entrant ships its failing-under-fastmath test with the kernel. +- **`dasllama_transformer.das`** - the block-composition seam, and the require umbrella: the + `[init]`-only side-effect requires (arch registrations, GPU tiers, and any module that + requires the engine back, which the umbrella therefore hosts to break the cycle). +- **`dasllama_blocks.das`** - the std/dense/MoE transformer block kernels, decode and prefill, + plus the two default block sets the arch files bind. `forward()` never names a kernel here - it + dispatches through the `ArchBlocks` function pointers, which is why the family lives outside the + hub with no hook. +- **`dasllama_moe.das`** - MoE expert routing and dispatch: the top-k router, per-expert FFN + accumulation, the shared expert, and the decode instruments. The block kernels reach it only + through `moe_ffn_core`. +- **`dasllama_attn_prefill.das`** - prefill attention, threaded over heads. `prefill_attention` + is the only entry; the classic/blocked/flash head kernels and their KV-codec ladders are its + implementation. +- **`dasllama_batch.das`** - the batched decode step: one pass of the weights over B sessions, + GEMVs widened to B-row GEMMs, attention still per-(row, head) against each session's own cache. +- **`dasllama_sampling.das`** - token sampling and the generation drivers. A leaf on top of + `forward`/`eval_batch`; the engine never calls back in. +- **`dasllama_ple.das`** - gemma-4 E-series per-layer embeddings and the gemma4 MoE FFN. The + forward sequence reaches it only through the hooks it registers at init. +- **`dasllama_config.das`** - `DlimConfiguration`: every input that changes `.dlim` image BYTES, + in one struct, plus its identity formatter. A knob that does not change image bytes does not + belong here; a knob that does and is missing is an image-aliasing bug. +- **`dasllama_tools.das`** - the per-ToolMode tool wire codecs (definition serializers, + replay/result text builders, reply parsers; Harmony namespace/channels, gemma-4 DSL, mistral + control tokens, llama JSON). Every byte of tool wire text is produced here - pure string+JSON + functions, model-free testable; the chat layer assembles the output into ChatParts and the + server parses through the parsers. Wire-format provenance: `history/dasLLAMA/tool_formats_plan.md`. +- **`dasllama_chat.das`** - conversation turns and chat-template application. Per-arch template + *content* is registered by the arch file (sec.1.6), not written here. +- **`dasllama_par.das`** - `maybe_parallel_for` and nothing else. Threading policy (job counts, + thresholds) belongs to the caller that knows the shape. + +### 1.2 Formats and data movement + +- **`dasllama_plane.das`** - the borrowed-plane vocabulary: a plane is a pointer into a prepared + image plus its element count; the image owns the bytes, a carrier owns nothing but its backing. + Requires nothing in dasllama - the image rail binds planes, every carrier holds them. +- **`dasllama_kqformat.das`** - format IDENTITY: the `KqFmt` enum, the per-format descriptor table + (plane strides, block geometry, stream codes), format predicates. It requires nothing else in + dasllama, because it is the taxonomy everything keys off. ONE id space - the enum; integer ids + exist only at the IR/kernel-param boundary. `kq_sb` is the superblock-lattice predicate: a + `fmt != q8` test does not imply the lattice, so branch on the predicate. +- **`dasllama_convert.das`** - every tensor format CONVERSION: quantize/dequantize/transcode/encode, + codec byte readers, numeric widen/narrow - regardless of platform or caller; plus the Q8_0/Q4_0 + format identity itself (block geometry, scale layout, the quantization-quality detector). + Metadata-coupled drivers (GGUF lookup, threading, guards) stay with their containers and + dispatch in. ONE carve-out: a conversion that IS a KV-cache format's store/read half lives with + its codec family (sec. below). +- **`dasllama_repack.das`** - every disk-order -> compute-order kernel-LAYOUT transform (grp + interleaves, disk-order extractors, panel unpacks), any format, any platform. Number sources + (tune stamps, bake overrides) stay with their owners and pass plain parameters in. +- **`dasllama_kv_codec.das`** - the KV-cache runtime codec, one family per cache format, kept + WHOLE: store (quantize), read (dequant), the attention score dot, and the V-accumulate axpy all + share the format's block byte geometry, so a layout change can never span modules. These are + per-token and per-tokenxposition `[tuned]` hot kernels, not load-time converters. The tq4 FWHT + rotation and sign vector live with their family. Codec DISPATCH (`KVDtype`) stays at common's + `kv_store_row`/`kv_load_row`/`kv_dot`/`kv_axpy` seam; the f16 row converts are the generic pair + in `dasllama_convert` because they are dual-use beyond the cache. +- **`dasllama_rope.das`** - RoPE angle and TABLE GENERATION: the theta schedule, `rope_freqs`, + fscale/mscale, every materialized layout - pure functions over plain parameters. Model-facing + wrappers stay in common. APPLICATION kernels stay with their backends: the CPU `rope_scaled_*` + leaves and the GPU fused rope-store kernels are shape-specialized deliberately, and + single-sourcing them is generator-rail work, never a hand-merge. Float multiply order in the + builders is contractual (parity-pinned). +- **`dasllama_gguf.das`** - the GGUF container: KV/tensor descriptors, the byte-level reader, the + mapping. Codecs live in `dasllama_convert`; this file finds bytes, it does not decode them. +- **`dasllama_layout.das`** - disk-format -> compute-layout transforms at LOAD scope: the blob + transform, the CPU repack walkers, the GPU tier gathers, the per-region q8->Metal-34B + un-repack (`q8_region_to_metal_blob`, the ASR-decoder driver's upload form), and the refusal + half (`can this model take the blob form`) split out so the image writer can commit without + loading. +- **`dasllama_tokenizer.das`** - the tokenizer facade: backend selection off the GGUF metadata and + the one encode/decode/piece surface models and the chat layer call. Re-exports both backends, so + a consumer requires this file and never picks a backend by hand. +- **`dasllama_spm.das`** - the SentencePiece backend (Llama-2 family, Phi-3, Gemma): score-greedy + merges over vocab pieces, `<0xXX>` byte fallback. +- **`dasllama_bpe.das`** - the byte-level BPE backend (Llama-3 / tiktoken family): vocab load, the + GPT-2 byte alphabet, ranked merges, encode/decode. Split from SPM because the two algorithms + share no state; a third merge algorithm gets a fourth file. Two sanctioned family-name tests + live here rather than in `dasllama_pretok`: the `pre`-name selector inside `bpe_encode`, and the + gemma-4 newline-run split in `bpe_encode_spm_space`; `load_bpe_tokenizer_gguf`'s per-family + metadata defaults are the third and last. +- **`dasllama_pretok.das`** - the pre-tokenizer: one hand-compiled split function per family + (llama3/qwen2/qwen35, gpt-2, gpt-4o, tekken), selected by the BPE `pre` name. Regex-port growth + lands here, never in the merge engine - the two change for different reasons (new model family + vs. algorithm work). Every arm with an on-disk upstream corpus vocab is gated by its case in + `test_tokenizer.das` (llama3, qwen2, qwen35, gpt-2); tekken has no corpus case, and gpt-4o is + pinned by frozen ids in `test_parity.das` only. + +### 1.3 The load and image rail + +**`dasllama_load.das`** is the GGUF load walk: metadata to `Config`, the plane layout, disk-format +detection, the eager and streamed conversion ladders, and the load entry points. It owns nothing the +forward path touches at run time - a loaded `Model` is the whole handoff - and it requires +`dasllama_common` back for `Model`/`Session`, so the transformer umbrella requires it `public` and +breaks the cycle. That re-export is what keeps every consumer on the facade. + +- **`dasllama_image.das`** - the prepared-model `.dlim` rail, and it is ONE rail + (`ARCHITECTURE_IMAGE.md` sec.2.1). Nothing + outside this file may read weights into a live carrier; image backings are released only + through the single release path in `dasllama_common` that the carrier finalizers call - never + an ad-hoc unmap. + +### 1.4 CPU kernel tiers + +- **`dasllama_math.das`** - the numeric ABSTRACTION: typedefs, active backend pointers, public + wrappers, dispatch shaping. Kernels themselves live in a tier file; a kernel body here is a + placement defect. +- **`dasllama_math_default.das`** - the portable backend, always registered, always correct, + out-ranked by any platform tier. +- **`dasllama_math_aarch64_neon.das`** - the arm64 SDOT/tbl tier. Its `[init]` never fires + off-arch (`ARCHITECTURE_INVARIANTS.md` sec.3, three-layer safety model), so an intrinsic here + needs a correct scalar fallback body, not a guard at the call site. +- **`dasllama_math_accelerate.das`** - the Accelerate/BNNS float tier (AMX on M1-M3, SME on M4+), + for genuinely-float planes only. BLAS-for-quant is ruled out structurally + (`ARCHITECTURE_INVARIANTS.md` sec.3). +- **`dasllama_math_gen.das`** / **`dasllama_gemm_gen.das`** / **`dasllama_gemm_schema.das`** / + **`dasllama_gemm_register.das`** - the generated GEMM tier: the runtime registration, the tile + generator, the layout/perm schema shared by generator and runtime, and the `[tune]` family + registration. A hand-written tile that the generator could emit belongs in the generator. +- **`dasllama_tune.das`** - the per-box loop-hint tuner (`[tuned]` / `[dasllama_grid]`). Tuning + POLICY lives here; tuned VALUES live in the box's sidecar, never in source. + +### 1.6 Architecture registrations + +Thirteen files registering eighteen names: +`dasllama_arch_llama.das` * `dasllama_arch_phi3.das` * `dasllama_arch_qwen2.das` * `dasllama_arch_qwen2moe.das` * `dasllama_arch_qwen3.das` * `dasllama_arch_qwen3moe.das` * `dasllama_arch_qwen35.das` * `dasllama_arch_gemma2.das` * `dasllama_arch_gemma3.das` * `dasllama_arch_gemma4.das` * `dasllama_arch_glm4moe.das` * `dasllama_arch_gptoss.das` * `dasllama_arch_mistral3.das`. They are DECLARATIVE: an arch +file builds an `ArchDesc` (name * `configure` * the `ArchBlocks` fn-ptr quad * `ChatTemplate` * +`LlmCaps`) and calls `register_arch` at `[init]`. Adding an arch touches no forward loop. + +### 1.8 Instrumentation and support + +- **`dasllama_parity.das`** - CPU-reference caches for the parity instruments. Test-facing, but + library-side because the caches outlive a single suite. +- **`dasllama_prefix.das`** - the prefix/page cache for evaluated token history. +- **`dasllama_lint.das`** - the facade boundary as a compile-time lint (DASLLAMA001): every + engine module carries it, so a consumer requiring anything under `modules/dasLLAMA/` but the + entry modules (facade, scheduler, exchange pair) fails to compile. Escape: + `options _dasllama_internal = true` - engine files, this module's own tests/harnesses/ + benchmarks/rigs, and the ruled consumers: `utils/dasllama-convert` (the bake tool reads + the mint rail), `utils/dasllama-server/model_catalog.das` (the env rail + the model-set + serving view; it re-exports `model_specs` so the server's catalog tests can gate the + serve rows), `utils/dasllama-server/test_openai_server*.das` (env-registry test knobs), + `modules/dasLLVM/daslib/llvm_user_modules.das` + `modules/dasLLVM/tests/test_{grid,tune,tuned}.das` + (the tune-generator contract). +- **`performance/model_specs.das`** - the ONE model-set table: per carrier, the profiled- + catalog fields, the official-board flag, the serving-catalog fields (`serve_*`, incl. + `serve_vision` naming a tower companion by name), the provenance pin (exact HF repo + + revision + sha256, or the on-box conversion recipe; companions hang off their owning entry, + and a companion shared by several rows is referenced from the others by name), and the + frozen parity evidence as data (pinned ids + arms). `llm_catalog` / `official_catalog` / + `models_provenance` are views over it; `serve_vision_tower` resolves a tower name to its + pinned companion, and `serve_asr_tower` is the one standalone pinned artifact the file + carries - the serving ASR tower (parakeet v3), consumed by `asr_catalog`'s v3 row in + `profile_common.das` (the pin's single source lives here, not there). +- **`performance/box_ident.das`** - the hardware-keyed box tag: the fleet table maps every + recorded box's CPU brand to its store tag (the M3 row is an EXACT match so bigger M3s slug), + the `DASLLAMA_BOX` override wins, and unknown hardware slugs from the brand - never another + box's store. `profile_common.das`'s `box_name()` and `tests/run.das`'s manifest pick both + resolve through it; the old per-file `"m1"` literals are gone. +- **`performance/fetch_models.das`** - the fetch/verify driver over the provenance view. + Verify by default, `--fetch` downloads; it never converts on `--fetch`, never benches, and + it touches no tune state (`tune_policy(missing="fallback")`). BRINGUP.md sec.2 is the runbook. +- **`dasllama/dasllama_exchange_schema.das`** - engine-free validation for exchange submissions + (record stores + tune sidecars); the dasllama.io ladder service builds on it. +- **`dasllama/dasllama_exchange.das`** - the sidecar-exchange client (a sanctioned public entry point): the boot-time + lookup/apply (llvm_tune's scope resolver - a verified per-box match downloads instead of a + ~20-minute tune), the privacy-stripped submit rails, and the control-page surface + dasllama-server serves at `/exchange`. +- **`benchmarks/asr/mem_census.sh`** - the peak-memory census (`/usr/bin/time -l` around one + asr_bench process per cell; macOS only) - the interim footprint instrument until a footprint + leg lands in `gen_bench_records`; its numbers live in `PERF_LEDGER.md`, never the stores. + +### 1.9 Serving + +- **`dasllama_scheduler.das`** - the continuous-batching scheduler, the serving layer over the + facade (its one engine require is `dasllama/dasllama`). One synchronous thread: each + `scheduler_step` admits queued requests, runs one `eval_batch` decode step over every + decoding stream, then at most one bounded prefill chunk FCFS; paged serving donates finished + streams' KV pages to the prefix cache. Results flow out as `SchedEvent`s - no HTTP here. + `utils/dasllama-server` owns the writers; `tutorials/dasLLAMA/13_serving.das` is the + teaching consumer; `tests/test_scheduler.das` gates it against `generate()` references. + diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU.md b/modules/dasLLAMA/ARCHITECTURE_GPU.md new file mode 100644 index 0000000000..ca8c58cd42 --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_GPU.md @@ -0,0 +1,192 @@ +# dasLLAMA Architecture - GPU backends and refuted kernel shapes + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +### 1.5 GPU backends + +A GPU backend is a FAMILY of role files - matching things in matching files across backends, so +that a question answered for one backend has an obvious address in the other. The roles: + +| role | holds | must not hold | +|---|---|---| +| the kernel home
`dasllama_metal_kernels`, `dasllama_vulkan_classes` | kernel source, the derived-access/PSO census | device state, engine types | +| `dasllama__common`
`dasllama_metal_common`, `dasllama_vulkan_common` | device state, buffer/command plumbing, hazard + capture rail, profiler | driver policy | +| `dasllama__decode`
`dasllama_metal_decode`, `dasllama_vulkan_decode` | the resident token-step driver + decode-time arms | kernel bodies | +| `dasllama__prefill`
`dasllama_metal_prefill`, `dasllama_vulkan_prefill` | the batched prefill driver + batch arms | kernel bodies | +| `dasllama__shapes`
`dasllama_metal_shapes` | PORTABLE servability gates - no GPU C++ require, so any box can bake | device calls | +| the tower driver
`dasllama_metal_tower` | one-shot embedder/encoder encodes (gemma4uv chain, the gemma4v ViT, gemma3v SigLIP and qwen3v block loops - qwen3v adds the vision NEOX rope, the fused-qkv weight-offset GEMMs, and the inline deepstack tap + tail merger chains - the whisper-class block loop, the conv frontends + the qwen3a padded-weight slab) - no session, no KV, no mirror; registers the gemma4uv, gemma4v, gemma3v, qwen3v, encoder_blocks, tower-conv and qwen3a-conv hooks | kernel bodies, decoder state | +| the ASR-decoder driver
`dasllama_metal_asr_dec` | the whisper decoder on Metal: the 34B weight blob, the f16 resident cross/self K/V, window-granular cross-KV + decode-step serves; registers the whisper cross-KV and decode hooks (family registries in `dasllama_whisper`) | kernel bodies, LLM session state | +| the kernel-access lens
`dasllama_metal_lens` (Metal), `dasllama_vulkan_dispatch` (Vulkan - the `[vk_dispatch]` macro derives access per class) | the kernel-access macro | anything else | + +- **Vulkan additionally has an ENTRY, `dasllama_math_vulkan.das`** - capability probe/arm, `.dlim` + identity source, cross-arm routers, the `[init]` installs. It re-exports the family `public`, + and its NAME is the transformer umbrella's `?vulkan` require contract (deliberately LAST in the + umbrella: the vulkan drivers are the hot-edit modules, and require order is the jit obj-cache + layout): **never rename it.** +- **Vulkan additionally has `dasllama_vulkan_seams.das`** - the thin whole-op call seams the tier + and suites dispatch through (`vk_add_rms`, `vk_rope_kv_store`, `vk_decode_attn`). It exists + because of a require direction: common cannot require the classes module (classes requires + common back), so any seam that encodes a class kernel must sit above both. A seam here wraps + ensure/set/enc - kernel bodies and driver policy stay out. +- **Metal has NO `math_` entry** - the family enters via the transformer's `?das_metal` requires + plus unconditional shapes. Its below-common piece is **`dasllama_metal_gemm.das`** (the batch + GEMM donor that common requires `?das_metal`), which owns its device by necessity: + metal_common -> dasllama_common -> metal_gemm would cycle. +- **Backend-only capabilities live in their matching ROLE file, not in new grab-bags** - vulkan's + weight arena, streamed mirrors, heat cache, host-import, coopmat; metal's blob transform and MTP. +- **The tower driver owns NO PSOs.** Its kernels (LN, f32 mul_mm, the two gelu flavors, + posadd, the gemma4v clamp / rope2d / GEGLU-quick, the head restride - gemma3v's and, offset-bound, the qwen3v/prefill slicers) live in the kernel home, so `metal_decode_init` compiles and `metal_kernels_release` + releases them like every other registry PSO; the borrowed prefill builders (`pf_enc_bf16_mm`, + `enc_add_bias_rows`, `enc_rope` - the qwen3v vision NEOX apply - and the attention trio + `enc_qk_mm`/`enc_rowstat`/`enc_av_mm`; this list is the closed borrowed set REVIEW_GPU.md's + tower rules key on) come up through + `metal_prefill_pso_init`, prefill's public bring-up seat, and `plane_buffer` in common is + public for the same wrap-a-plane reason. The tower's own objects (the ones buffer, its + scratch pool) release through `metal_tower_shutdown`. +- **The tower driver is a Metal-only role** - Vulkan has no tower twin; audio/vision encodes + on the Vulkan tier stay CPU (the gemma4v ViT, gemma3v SigLIP and qwen3v block loops + included: on Vulkan and on plain CPU boxes those towers serve their q8 lanes). Likewise the non-causal media span: Metal serves it through + `AttnArgs.uend` - including the FUSED image turn (head + media rows + tail as ONE eval, the + per-query mask through `AttnArgs.ulo`); the Vulkan resident prefill declines span evals + (`followup_general.md` #23's remaining half) and registers the split-span capability + (`register_prefill_override_split_span`), so `eval_embd_span_` keeps the three-eval splice + while vulkan is the active override. The qwen mrope quantum rides the same shape through a + second capability seat (`register_prefill_override_mrope_tables`): Metal's `enc_rope` reads + the per-token table rows `prefill_rope_tables` builds from the grid map, so it serves mrope + unchanged and registers the seat; an override without it (vulkan builds angles from a scalar + position) declines the quantum to the CPU loop by name. The deepstack quantum is the third + seat (`register_prefill_override_ds_adds`): Metal uploads the caller's wide quantum WHOLE + (the `Session.wide_src` borrow), slices x and the slice-major ds planes on-device through + the offset-bound head restride, and encodes one `enc_add` at the slice offset after each + tapped layer's residual - no new kernel (the CPU-side split, `ds_split_quantum`, survives + as the CPU-loop fallback and the warm/MTP edge); an override without the seat declines + deepstack quanta by name, so Metal serves them and Vulkan does not. +- **Per-layer FFN widths (MatFormer E-series, at most two - `ffn_second_hidden`) serve on Metal + only**: the decode and prefill drivers bind the width per layer (dense trunks, no MTP; batch + keeps the layer-0 hoist behind its uniformity decline). The Vulkan tier has no PLE arm, so + E-series never reaches its width question. +- **Family-shared kernel classes live in `dasllama_metal_kernels`.** The `[metal_dispatch]` lens + generates `enc_*` builders and MSL globals into the module the class COMPILES in, so co-location + follows the class - "the builder needs the driver module" is never a placement reason. Prefill's + prefill-only classes are convergence debt, not precedent. +- **Ledgered kernel-binding asymmetries** - a REVIEW rule firing on one of these is expected, and + this entry is the sanction: the moe mul_mm TENSOR twins (`MetalMoeMulMmQ8T` / + `MetalMoeMulMmMx4T`) keep the pre-family compact kargs slots while their base classes bind the + family numbers, so no shared bind path may span the two layouts; and the in-engine moe mul_mm + A/B race harnesses (`dasllama_metal_prefill.das`) encode through `kn_moe_mm_family_tail` + rather than a per-class `enc_*` builder. +- **`dasllama_gpu_tier.das`** - the device-cooperation SPI: hook types, install/unset slots, + route/mark/want/status state, engine-facing forwarders. Vulkan implements it (per-op offload plus + resident plumbing); Metal deliberately does not, because UMA makes residency moot there and Metal + integrates as a whole-forward driver through common's override registries (the ASR-decoder + driver is the one exception: whisper is not a `Model`, so its hooks are family registries in + `dasllama_whisper`, same decline contract). +- **`dasllama_gpu_resident.das`** - the WHOLE-MODEL residency rail: bake the device layout offline + into the flavor image, upload a model's stacks to the tier, and drive decode/prefill entirely on + device. It is device-AGNOSTIC - it holds no device call and requires no GPU module, reaching the + hardware only through the `dasllama_gpu_tier` SPI and entering the engine only through common's + override registries. `"vulkan"` is the tier string it registers under, not a dependency, which is + why it compiles on every box. It requires common back for `Model`/`Session`, so like the Metal + drivers it is required from the transformer umbrella, never from common. +- **`dasllama_kernel_access.das`** - the shared body-walk read/write classifier both GPU lenses run + on, plus the dispatch-lens micro-grammar (the grid/tg/params spec tokenizers and the shared + AST-emission core: `is_digit_tok`, `role_ok`, `derived_role`, `mk_uint_cast`, `mk_call1`, + `mk_grid_dim`, `param_type`). One owner by design - the two lenses' grammars drifted apart + when each carried a private copy (metal folded only the literal "1" where vulkan folded any + integer). Backend-specific lowering stays in that backend's lens. + +**PSO lifecycle - the family shares ONE device and queue** (`metal_common_init`; the second-device +question was surveyed and closed against). The decode PSO set lives as `g_pso_*` in +`dasllama_metal_common`, is compiled by `metal_decode_init` in `dasllama_metal_kernels` and +released by `metal_kernels_release` there - the kernels module owns its set's lifecycle even +though the vars live with the device state. Prefill's `g_pf_pso_*` set is prefill-private end to +end: `metal_prefill_init` compiles, `metal_prefill_shutdown` releases. + +**Race and tune code for a kernel family lives beside the family.** The shared scaffolding +(`race_buf`, `race_envelope_ok`, `race_pair_ms`, `MetalTensorRaceResult`) is `_common`'s; +each module races its OWN families (`metal_tensor_race_decode` in kernels, `metal_tensor_race` +in prefill) and the tuner calls those public entries. + +**Decline REASONS are enum values in the shapes module** (`MetalDecodeDecline`, +`MetalPrefillDecline`); decline COUNTING lives in `_common` beside `require_or_panic`, for +both paths. + +**The allowed asymmetries between the backends - this list is closed; a new one lands with its +entry here:** + +- **Metal sits ABOVE `dasllama_common`** (typed `Model`/`Session` access, shapes unconditional); + **Vulkan sits BELOW it** (untyped pointer/array seams - the family never requires common). + Both tiers ENTER from the transformer umbrella (`?das_metal` requires; the single `?vulkan` + require of the `dasllama_math_vulkan` facade); the inversion that remains is the + kernels<->common require DIRECTION, and it is why their seam shapes differ. +- **UMA vs discrete VRAM**: Metal never grows Vulkan's VRAM machinery - arenas, upload + economics, mirrors and hydration are Vulkan's alone. Metal's residency artifacts are the + `MTLResidencySet` pin in `_common` (`DASLLAMA_METAL_RESIDENCY`) plus its keep-alive + heartbeat (`DASLLAMA_METAL_HEARTBEAT_S`, a dasMetal background re-request that stops the OS + collecting the set over a CPU-only window, `ARCHITECTURE_RUNTIME.md` sec.2.12) - a + driver-cost shield, not a placement mechanism; memory is still memory. +- **The weights-epoch drop is Metal-only.** `bump_weights_epoch`'s listener seat + (`register_weights_epoch_listener`) has one subscriber: `_common`'s `metal_weights_drop`, + which runs the registered reload preps (`register_reload_prep`; the decode driver registers + `discard_pre`), quiesces, and releases the address-keyed region caches. Vulkan's reload + story is the unmap notify (`set_moe_gpu_unmap_notify`) - a different seam for a different + ownership model. +- **Lens depth**: both lenses generate `enc_*` builders from kernel classes - Metal via + `[metal_dispatch]`, Vulkan via `[vk_dispatch]` (per-class set layouts + push constants; the + class-kernel arc retired the hand-built 6-slot set ladders outright) - and both speak the + multi-kernel form (`kernel=` names the method, one macro instance per kernel, declared roles + must cover every kernel). +- **`family=` is Vulkan-only.** A vulkan family shares the per-class surface - the `VkdClass` + global, the `set_*` builder, the pipe slots - across classes with one binding layout. Metal's + `enc_*` builder is the entire generated surface, so there is nothing for a family to share; + cross-class PSO/source sharing on Metal is a PSO-lifecycle question, not a lens one. +- **`@default` is Metal-only.** A `[metal_dispatch]` field may name a fallback global + (`@default = g_one`) that the generated builder binds when the caller passes null; + `[vk_dispatch]` has no counterpart - vulkan callers pass a real buffer at every slot. If + vulkan grows an optional-bind shape, it lands as this same annotation, not a new spelling. +- **The workgroup-footprint gate is Vulkan-only.** `[vk_dispatch]` sums a class's `@workgroup` + members and its generated `ensure_*` declines by name (`vkd_wg_fits`) before the pipeline + build, because MoltenVK's over-cap failure is an opaque `INITIALIZATION_FAILED` - and the + resident driver declines residency with it (`vk_rdec_prepare`). `[metal_dispatch]` has no + footprint gate: Metal's own pipeline compile fails loudly with the footprint in the error. +- **Vulkan has no shapes module yet** - `resident_upload` declines ad hoc by feature name; the + gap is `followup_vulkan.md` item 1, not a precedent to copy. + +Vulkan is the deliberately-designed model of this shape; Metal converges as it is touched. + +### 2.2b Tensor-GEMM shapes that measured out (M5, interleaved-race evidence) + +The forms below were built, raced against the crowned tg-staged q8u GEMM at real model +shapes, and LOST; the probe that holds the GEMM numbers is +`benchmarks/matmul/bench_metal_nax_probe.das`, and the fused-attention numbers live in +`benchmarks/attn/bench_metal_pf_fused_attn.das`. Re-attempting one without new structure is +re-buying a measured loss. + +The probe also RETAINS the decided-and-shipped arms (the half-A stream, the dev-W all-device +form, the tall M-tile twin, the bk staging depths, the no-zero-init form) as hand-written MSL +reference implementations beside the refuted ones - they are the arc's bisect ledger, and the +sync duty is `REVIEW_GPU.md`'s. + +- **Per-simdgroup register-fragment matmul2d (16x32x16, device -> `vec` fragments):** + 1.7-2.0x slower for weight GEMMs, vectorized loads and deep n-blocking included. The + fragment architecture pays for attention's streaming operand reuse (one resident Q against + a K/V walk), not for a GEMM's operand traffic - cooperative tg staging wins there. +- **Mixed-integer matmul2d operands (float x int8, and the i8 x i8 per-block-fold form):** + 3.5x slower - the mixed-int combinations exist in MPP's type lists but lower off the NAX + fast path. W8A8 claims from other stacks do not transfer through MPP. + +**Sanctioned float-A stamps** (the licensed set of the rule that a `matmul2d` operand reaches +the op as `float` only in a class stamped `[metal_kernel(float_a_ok=true)]`, each carrying +that stamp): every tensor template's `XT = float` stamp - the live fallback wherever the half +panel is absent (below the convert row floor, panel does not fit, half-X pinned off) - and the +batch-decode/classifier `MetalQ8GemmTensorT` family, whose half-X extension is an open ledger +item. A float A operand anywhere else is the defect the rule names. +- **Fused single-kernel attention (scores in threadgroup, online softmax):** loses 10-80% to + the pipelined three-pass at real shapes (`benchmarks/attn/bench_metal_pf_fused_attn.das`) - + Metal's cross-kernel pipelining plus full-width softmax beat tg-scope fusion. + +The positive laws these races established - half operands, stage-only-to-transform, +consecutive staging runs, relaxed_precision always - are `REVIEW_GPU.md` rules and the +`modules/dasMetal/REVIEW.das` descriptor gate; this section keeps only the refuted shapes +and why they lose. + diff --git a/modules/dasLLAMA/ARCHITECTURE_IMAGE.md b/modules/dasLLAMA/ARCHITECTURE_IMAGE.md new file mode 100644 index 0000000000..b554a1167e --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_IMAGE.md @@ -0,0 +1,92 @@ +# dasLLAMA Architecture - the prepared-image rail + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +### 2.1 There is ONE way to load a model + +A weight carrier becomes a live struct through exactly two functions here, and nothing else may +read weights into one: + +- **`build_image`** walks a carrier's planes into a sink - a `.dlim` file, or a page-aligned memory + chunk when there is nowhere to write. +- **`parse_image`** turns `(base, bytes)` back into borrowed-plane fields, and does not know which + sink produced them. + +Cold and warm therefore yield the SAME struct. A cold load reaches it by building the image and +handing off *through the file* - write, drop the model, map - so the model and its image are never +both resident. That handoff costs a close and a re-map of a multi-GB file and is the *slower* cold +start on purpose, under the tiebreak in `ARCHITECTURE_INVARIANTS.md` sec.3. `cache_via_image` is +that handoff for every weight carrier; the streaming forms transcode planes from the gguf +mapping straight into the image so they never materialize at all. + +**An image is judged only where its identity can be recomputed - the LANE rule.** An identity's +(quant, tag) pair is its lane, and lanes coexist by design: a bake proves dead only its OWN lane's +siblings, plus BROKEN and version-stale images anywhere. Everything else is `FOREIGN` - kept, and +counted in the log. Two contexts produce lanes this process cannot reproduce: a GPU flavor (a blob +bake selects its own backend and winners, so the string differs even where the tag matches: +`portable|q8 mr4` against a planar `arm64-gen|q8 mr8`), and a family tag whose module is not loaded +(the tag registry is filled by each family's `[init]`, so a process that never required whisper has +no way to know a whisper image is current). A process that cannot recompute an identity has no +standing to call it dead - the orchestrator of a sweep least of all, since its cells measure through +an exe carrying its own baked winners. The one owner carve-out is the batch lifecycle: the board +rigs own the model dirs for a whole run, so `dlim_wipe` (verdict-blind, `dasllama_image.das`) +clears them behind the exe gate at batch start and after each model's last cell, with every image +re-baked from its gguf on demand. Judging stays forbidden; owning the directory for the batch is +what licenses deletion without judgment. + +### 2.1a Page alignment is the no-copy contract {#image-page-alignment} + +Every plane section starts on a 16 KiB boundary (`IMAGE_PAGE`, the Apple-Silicon page) and the +image's total length is a page multiple. The alignment is what lets a mapped plane be wrapped for +the GPU with no copy - Metal's `bytesNoCopy` requires it - and what lets a load borrow a plane in +place instead of reading it. + +### 2.1b The meta blob leads with two strings {#dlim-meta-head} + +The meta blob sits at the image tail and leads with two strings - the identity the image was baked +for and its config JSON - then the section table and the walk's scalar stream. The strings lead so +a peek, or a load that is about to decline, can print what the image was baked for without parsing +anything else. + +### 2.1c Array payloads reach the archive in bulk {#image-bulk-serialize} + +Array payloads reach the archive in bulk - one stream call per array (`serialize_pod_array`), +string arrays as a length vector plus one byte blob (`serialize_strings`). `daslib/archive`'s +per-element generic dispatch costs on the order of 340 us per element, which puts a 128k-entry +vocabulary near 238 seconds; the bulk forms make the same work milliseconds. + +### 2.1d An interpreted gguf load pays; an image load does not {#image-interp-load} + +A gguf load's O(model) transform loops run about ten times slower interpreted - a tinyllama load +takes 53 s against 5.5 s jitted, and a 69 GB hybrid extrapolates to an hour (the repack itself is +native tune kernels and costs the same either way). A prepared image costs nothing interpreted, +because mapping and borrowing planes runs no such loop. That asymmetry is why the guard fires on +the gguf path and never on the image path. + +### 2.1e Publishing an image {#image-publish} + +An image is published by writing a temp file beside its destination and renaming over it. POSIX +makes that replace atomic. Windows has no rename-over, so the publish removes the destination +first and a concurrent reader can see a brief absence - which costs that reader a regenerate, +never a corrupt map. + +### 2.1f The image's size is known before the first byte {#image-sizing-exactness} + +The image's final size is known before a byte goes out: the meta blob is serialized first, so its +length is in hand, and every plane contributes its own bytes (`image_total_bytes`). The sink +preallocates from that number, so the file lands as one contiguous run and the memory chunk is +exact. The walk advances the writer by exactly the bytes it accepted, so the section table can +never drift from the file. An append past the chunk is therefore a disagreement between the sizing +pass and the walk - a bug, not a disk condition - and the chunk rail panics rather than limping on. +The one decline it survives is failing to get the chunk at all: that happens before the walk +starts, so the carrier is still whole and its caller keeps serving it. + +### 2.1g Identity names the backend, so the backend is selected first {#image-identity-backend-order} + +An image's identity names the active matmul backend, so the backend is selected before any +identity is computed or compared. `image_identity` is a pure formatter over `DlimConfiguration`; +the backend select happens inside the config's CPU source, so a caller needs no ordering ritual of +its own. A load pins the box profile first because that pin can change the backend, and the parse +runs the same load select the gguf loader runs, before any kernel touches planes packed for that +backend. + diff --git a/modules/dasLLAMA/ARCHITECTURE_INVARIANTS.md b/modules/dasLLAMA/ARCHITECTURE_INVARIANTS.md new file mode 100644 index 0000000000..9fb572e231 --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_INVARIANTS.md @@ -0,0 +1,126 @@ +# dasLLAMA Architecture - inherited invariants + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +## 3. Inherited invariants + +Durable "why it is built this way" facts harvested from the design docs archived under +`history/dasLLAMA/`. Violating one of these is a bug, not a style choice. + +**From `x64_arch.md`:** + +- **The LIBRARY never runs interpreted or AOT - the scaffolding may.** Inference - anything that + runs a model (eval / decode / prefill / generate) - is `-jit`-only, hard stop: an interpreted or + AOT model run is not "slow but correct", it is out of contract, and the library panics on it. + Loop hints and intrinsic lowering exist only in the JIT, and the JIT tier is the only one the + oracles gate. **Scaffolding is exempt:** conversion utilities (`.dlim` bake, image processing), + debug scripts, and batch/driver tools that spawn the real runner as a child may run interpreted. + Enforcement is `guard_interp_gguf_load` (`dasllama_image.das` - big-load panic + + `DASLLAMA_ALLOW_INTERP_LOAD` escape for the conversion path) plus `guard_interp_inference` + (`dasllama_math.das`), which panics at the library seams: `make_run_state`, ASR session + creation, and `vad_chunk_prob`. Intrinsic fallback bodies exist for off-ARCH correctness + *inside* the JIT, not as runnable tiers. +- **Correctness before speed, token-for-token.** The engine is validated against external oracles + (llama2.c + upstream `simple_ids`) plus per-arch parity fixtures. A new kernel passes the suite + *and* the oracles with the new backend active before any perf claim. +- **Peak memory before cold-start latency - a DELIBERATE trade, and the standing tiebreak.** When a + load-time choice pits footprint against wall-clock, dasLLAMA takes the smaller footprint. The + asymmetry is not close: overshooting RAM on a big model is fatal (the OOM killer, or swap that + makes the whole box unusable), while a slower cold start costs seconds *once per process* and + costs the warm path - the common one - nothing at all. The prepared-image rail is where this bites + hardest and where the shape is set: a cold load writes its `.dlim`, drops the model, and maps the + file back rather than serving the copy it already has in RAM. That is a real close/reopen and a + real re-fault of a multi-GB file, and on an image larger than the page cache it is a re-read from + disk. Measured on Llama-3.1-8B-Q8_0 (9.6 GB image, M1 Max, `phys_footprint_peak`): planar cold + **3.7 s -> 4.5 s**, peak **12.3 GB -> 5.4 GB**. We bought a 2.3x footprint cut with ~20% of one cold + start. Apply the same tiebreak everywhere - bake, convert, KV growth, GPU staging - and when a + change goes the other way, it needs the measured pair (peak AND wall) and an explicit call, not an + assumption that faster is better. +- **Cold builds STREAM; nothing materializes a model to write one.** The rail that makes the trade + above cheap: a cold build transcodes each plane from the gguf mapping straight into the image, so + the model never exists in RAM at all and the peak is a few per-tensor temps over the small + RAM-resident scale planes. This covers the planar flavor and the metal blob flavor alike - the + blob's gguf-native 34B blocks assemble per tensor on the way out, and because a blob forbids CPU + repack, it does not even need the resident `qscales` plane the planar stream keeps. Metal cold on + the 8B: **19.5 GB -> 4.8 GB, 4.1 s -> 3.1 s** (faster AND smaller - the eager rail had been + splitting gguf's interleaved blocks apart only to reassemble them). The vulkan flavor rides it + too: its bake collects from the GPU walk of a model that is **already serving**, so it needs a + mapping rather than a load - which is what the warm path always did, and now the cold path does + the same instead of loading eagerly (**15.4 GB -> 6.6 GB**). A new flavor joins the streamed rail + or explains why it cannot. +- **Token-exact oracle tests pin the bit-exact path** (classic attention, scalar activation); + approximate/fast paths get separate tolerance tests. Rerouting an oracle test through a + non-bit-exact default makes it pass on the machine it was frozen on and flip elsewhere. +- **The three-layer safety model that makes a new ISA free:** (1) *registration gate* - an ISA's + `[init]` never fires off-arch, so `portable` stays active and every run is correct; (2) *scalar + fallbacks* - an intrinsic called off-target computes the right answer from its fallback body, only + the JIT emitter lowers it to hardware; (3) *tests drive public wrappers*, asserting only that *a* + row-major default is active, never an arch-specific name. A new backend is a file plus an `[init]`. +- **The u8xs8 sign trick, and which operand takes the `|a|`.** ARM `sdot` is signedxsigned; x64 + `vpdpbusd` is *unsignedxsigned*. The exact integer identity is + `dot(a,b) == dot(|a| as u8, sign(a) applied to b)` - and `|a|` goes on the **unsigned** operand, + because `|-128| = 128` fits u8 but not s8. Products fit s16, four-product sums fit s32: bit-exact, + no saturation. + +**From `kv_cache_compression.md`:** + +- **The KV layout is ragged per-layer, not flat.** `kv_row_total` *sums* per-layer `layer_kv_dim` + (layers differ under SWA/global splits), and a layer's base offset is a **prefix sum** + (`kv_row_prefix[l] * seq_len`). A codec keys strides and offsets off the prefix array - never off + one flat `kv_dim`. +- **Cross-layer KV sharing stores nothing.** When `kv_src[l] != l` both store sites are skipped and + the layer's offset **aliases the source layer's region**. Any codec must preserve that aliasing. +- **Convert the query, not the cache.** The query is `O(head_dim)` per step; the cache is + `O(context x head_dim)`. Dequantizing the cache on the decode read costs *read compressed + write + f32 + read f32* ~ 2x the f32 baseline - strictly worse than storing uncompressed. +- **Prefill and decode are asymmetric on purpose.** Prefill is compute-bound and tolerates external, + one-layer-at-a-time decompression (a fresh prompt reads the f32 `k_b`/`v_b` scratch and + decompresses nothing); the decode read is memory-bound and **must fuse** dequant in-loop. + Attention consumes one layer at a time - never materialize all layers at once. +- **BLAS is ruled out structurally, not incidentally.** `cblas_sgemm` is f32-only (no hgemm, no + int8), so any quantized operand must expand 4x into f32 scratch - and the weight matmul is bound on + bytes streamed, so quantized-native `sdot`/VNNI wins regardless of FLOPs. + `dasllama_math_accelerate.das` ships an Accelerate float-batch override plus a BNNS-f16 lane + for bf16 planes (`DASLLAMA_ACCEL` / `DASLLAMA_ACCEL_F16`). BLAS-for-quant stays ruled out; + Accelerate/BNNS on genuinely-float planes is live and measured. + +**From `turboquant_research.md`:** + +- **Stage-2 QJL is skipped deliberately** in the tq4 KV codec - its unbiasedness buys variance that + softmax amplifies, and the damage grows with context. Five independent groups plus vLLM upstream + converged on Stage-1 only (rotation + Lloyd-Max Beta codebooks + per-block norm). Do not "restore" + QJL as a completeness fix. +- **Benchmark tq4 against `q4_0` + Hadamard, not f16.** f16 is the comparison the method wins by + construction; `q4_0`+Hadamard is the one it has repeatedly *lost* on small/dense models and long + contexts. Quoting an f16 ratio is the mistake this doc exists to prevent. +- **Two corrections that travel with the codec:** the estimator is **asymmetric** - the query is never + quantized, so a QJL dot is sign-XOR over fp lanes, not popcount (popcount needs the query binarized + too, an off-paper accuracy trade); and "3.5-bit" is **mixed per-channel bit allocation** over + head_dim, not fractional coding. + +**From `API_REWORK.md`:** + +- **The Tier-1/2/3 scope model.** Tier 1 = scalar `Config` knobs (RoPE variant/scaling, norm kind and + placement, activation, QKV bias, softcaps, sliding window, embed scale). Tier 2 = block swaps that + change dataflow (MoE, QK-norm, per-layer attention patterns, shared KV, PLE, sinks, parallel + attn+FFN). Tier 3 = deep forks needing a pluggable KV/attention core. The hybrid + Gated-DeltaNet lane is a Tier-3 fork that serves: `dasllama_arch_qwen35.das` + registers `qwen35`/`qwen35moe`/`qwen3next`, gated by `tests/test_deltanet.das`. The arch registry + (18 names across 13 `dasllama/dasllama_arch_*.das` files) is the truth for what exists. +- **Family behavior is distributed by `Config` flag, not dispatched by name.** GeGLU-vs-SwiGLU is an + `ffn_act` flag *inside* the shared dense FFN block; only a genuinely new dataflow (MoE) earns its + own `ffn_*` block pointer. An `if (arch == "...")` on a shared path is the anti-pattern. +- **`register_arch` MOVES the `ArchDesc`** (`g_arch_registry[name] <- desc`). Clone the template + **before** registering an alias, or the second call registers a gutted descriptor - the live + pattern in the qwen2 / qwen3 / qwen3moe / qwen35 files, each with the comment saying so. +- **What rides where:** `ArchBlocks` (copyable fn-ptrs) is copied onto the `Model` at load; the full + `ArchDesc` (non-copyable - it holds a `ChatTemplate`) stays in the registry. Image loads re-bind + blocks via `rebind_arch_blocks` and must **not** re-run `configure` - the serialized config already + carries configure's flags *plus* the GGUF's overrides. +- **The kernel/ISA seam is symmetric with the arch seam, plus one extra rule.** `dasllama_math.das` + owns the abstraction (typedefs, active pointers, public wrappers); ISA modules self-register at + `[init]`. Two selection tiers: `register_kernel_backend` auto-activates only the best **no-repack** + backend, so direct callers (tests, benches, row-major weights) always get a kernel that works on + unrepacked data; `select_matmul_backend_for_load()` is the **only** path that activates a + `needs_repack` backend - eagerly activating one runs an interleaved kernel over row-major data, + i.e. silent garbage. diff --git a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md new file mode 100644 index 0000000000..3eecddd6e4 --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md @@ -0,0 +1,66 @@ +# dasLLAMA Architecture - the benchmark rig and instrumentation rails + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +### 2.5 There is ONE benchmark rig, and the records are the baseline + +`benchmarks/lcpp_bench.das` is the only thing that measures performance. It is a *mirror* of +the upstream `llama-bench` - the same test shapes, rep counts and timing boundaries, applied to +our engine - so `pp` is one batched prefill of `-p` tokens from an empty cache per rep and `tg` +is `-n` single-token forwards with no logit read, each row one untimed warmup plus `-r` timed +reps. The real `llama-bench` runs only when `--ref ` is passed; that is how the upstream +columns were produced, and they are pinned, not re-measured. + +`performance/gen_bench_records.das` sweeps a board by spawning that rig once per cell, and +writes `performance/records/.json`. `gen_site_records.das` merges those into the file the +site renders. A stored row carries its own command, sha, version, tune stamp and exec format, so +a number is self-describing rather than a bare figure in a table. + +**Regression checking inverts the same rig:** `gen_bench_records.das --oracle --legs metal` +takes the store's das rows as the work list, re-measures each once, and gates one-sided against +its stored mean (fail past 5%, warn past 3%, gains flagged as suspicious). Upstream never runs, +the store is never written, and a text cell's timed child runs `--frozen` - a prepare pass bakes +and warms its image first (the batch starts wiped), so the timed cell never converts; ASR and +image-chat cells bake what they need mid-cell, like their publishing legs. A second +harness would produce numbers that cannot be compared to any of this. + +**The tune stamp gates the comparison.** A manifest older than the binary fails every cell, and +an untuned invocation re-execs into a full retune rather than measuring - so re-mint the box +manifest and check its winners against the stored rows' `tune` stamps before trusting a delta. + +**A measured number proves its kernel provenance through `tune_gate()` +(`performance/profile_common.das`), one arm per world it can run in.** Three worlds, because +`tune_status()` populates in exactly one of them: a standalone exe checks the sidecar the +release shipped beside it; a `DAS_TUNE_MANIFEST` run checks that file; a plain script checks +that every `[tune]` row stamps a manifest winner. An invocation no arm covers refuses - or +worse, measures on fallback kernels - which is why every measuring entry point calls the gate +before its first timed rep. Two rig shapes fall outside "measuring entry point" by the +property itself, ledgered here: a kernel A/B lab dispatches its variants through its own arms +(never the `[tune]` selection), and `lcpp_bench.das`'s `--tok` cell dispatches no kernels at +all - neither can measure a fallback silently. + +**The retune re-exec bites scaffolding, and the pin for it is checked in.** Any bare `daslang` +run that requires the engine - a probe, a one-off script, a REPL experiment - re-execs into a +full retune when no manifest is armed. `performance/last_known_good_sidecar.json` exists for +exactly that: a frozen copy of a complete, noise-gated mint, tracked in git (the `*.tune.json` +ignore rule deliberately does not match it). Point `DAS_TUNE_MANIFEST` at it and the framework +never retunes; on a different box the identity mismatch just serves fallbacks, and a copy minted +before the current `DASLLAMA_VERSION` serves fallbacks on any box - the compile says which with +one `WARNING DAS_TUNE_MANIFEST` line per scope. That is the whole +contract - it suppresses the re-exec, it does not tune the box, and a number measured under it +is not a benchmark. Benches and the rig keep minting their own; refresh the copy when a +re-mint moves the crowns or `DASLLAMA_VERSION` bumps. + +### 2.10 Sanctioned instrumentation rails + +Engine timing goes through the rails that aggregate and tag it: the `jobque_profile` markers +(`profile_tag` / `profile_marker` and the `trace_*` wrappers in `dasllama_math.das`), the +`prof_add` / `forward_profile_*` decode buckets (`dasllama_common.das`), and the `asr_prof_add` +encode buckets (`dasllama_audio.das`). A new clock read paired with a print of the elapsed +interval bypasses the aggregation - it measures one call site once, is not filterable, and rots +where a rail entry would keep serving. Where a timed line IS the deliverable - `benchmarks/`, +`performance/`, `harness/`, and cold one-shot load/mint progress logs (image bake/map, load +stages, tokenizer build) - the rails do not apply. A clock whose value feeds logic is control +flow, not instrumentation; it is marked `// clock: control` so the sweep and any future lint +leave it alone. + diff --git a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md new file mode 100644 index 0000000000..575faa9df1 --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md @@ -0,0 +1,139 @@ +# dasLLAMA Architecture - encoder tower, audio, and vision file charters + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +### 1.7 Encoder towers, audio, and ASR + +- **`dasllama_asr_types.das`** - the ASR floor: the capability/segment/timestamp types every + family file and the facade share (`AsrCaps`/`AsrTimestamps`/`TranscribeSegment`, plus + `asr_ctx_guard`). Family modules require this instead of each other - a shape needed by two + families moves up here, never sideways. +- **`dasllama_tower.das`** - the family-neutral encoder-tower home, composed by every audio AND + vision tower: the oracle-exact activations (erf / tanh-LUT / quick-LUT gelu, swiglu, geglu), + the LayerNorm/RMS row forms, bias and residual row adds, the + `mm_blob_b`/`mm_bf16_b`/`mm_plane_b` GEMM wrappers, `Clamp`/`read_clamp`, + `im2col_rgb_patches`, `rope_neox_2d_rows`, `rope_neox_tab_rows`, `avg_pool2d_rows`, + `interpolate_grid_bilinear_aa`, `tower_read_conv_pair_folded`, the padded stage readers + (`tower_read_gemm_q8`, `tower_stage_q8_zero_rows`/`tower_stage_q8_pad_cols` and their f32 + twins `tower_stage_f32_zero_rows`/`tower_stage_f32_pad_cols` - the load-scope padded + stagers for FFN widths that are not 32-aligned - and `tower_zero_span`), blocked `attention_bidir` and + its per-window form `attention_bidir_windows`, and the encode-stage prof rail. The one home: + a family file that re-implements one of these is a defect, and nothing here names a family + type. +- **`dasllama_audio.das`** - the audio encoder tower: the mel front-ends (mtmd and whisper.cpp + flavors), `AudioTower` with its staging, q8-quantize, and image rails, `EncoderState`, and the + whisper-class encode + block loop with its GPU hooks. Composes `dasllama_tower.das`. +- **`dasllama_audio_io.das`** - decode-any-format -> 16 kHz mono f32 PCM. The only file that talks to + miniaudio. +- **`dasllama_asr.das`** - the ASR facade: capability declaration, timestamp granularity, the + backend-neutral entry points. +- **`dasllama_whisper.das`** / **`dasllama_parakeet.das`** / **`dasllama_canary.das`** / + **`dasllama_qwen3a.das`** / **`dasllama_gemma4a.das`** - one file per model family, each owning its + weights, its decode loop, and its quirks. Shared tower pieces go up into `dasllama_tower`, not + sideways between families. +- **`dasllama_vad.das`** - Silero-VAD weights and per-stream state. + +### 1.7b Vision + +- **`dasllama_vision.das`** - the image preprocessing rail: dynamic-resolution geometry, the + letterbox resize (aspect-preserving bilinear onto a centered black canvas), u8->f32 normalize, + and the `DASLLAMA_VISION_DUMP` PPM writer. The only preprocessing home. +- **`dasllama_vision_io.das`** - image decode to RGB8, from a file or a byte blob. The only file + that talks to stbimage; re-exported by the facade (like `dasllama_audio_io`) so consumers + decode through one seam - the engine itself takes decoded pixels. +- **`dasllama_gemma4uv.das`** - the gemma4uv embedder (gemma-4 dense): mmproj load and the + im2col -> LayerNorm -> GEMM -> position-table -> projection forward. One file per vision + projector family, following the audio tower pattern; shared pieces move up into + `dasllama_tower.das` (the encoder-tower home), never sideways. + A shipped mmproj mixes element types per tensor - gemma-4's "BF16" file stores the patch + embedder as F32 and only the projection as BF16 - which is why a weight plane's element type + follows its source tensor, per tensor, never a per-file verdict. +- **`dasllama_gemma4v.das`** - the gemma4v ViT tower (gemma-4 E-series): mmproj load (bf16 GEMM + planes as the file has them, the clamp sidecars as a blob table) and the 16-block pre-norm RMS + forward - clamped GEMMs, per-head q/k RMS, two-axis NEOX rope, weightless V RMS, unscaled + bidirectional attention, GEGLU-quick - then the 3x3 pool, RMS and projection. Composes the + `dasllama_tower.das` tower pieces; owns only its layout and the block loop. +- **`dasllama_gemma3v.das`** - the gemma3 SigLIP tower (size-invariant across the gemma-3 line; + 4B and 12B are the tested pairs): mmproj load (the file's f16 planes widen exactly to f32; + the ffn pair serves at the layout's padded 4352 width, so every q8-lane GEMM quantizes) and the 27-block pre-norm + LayerNorm forward - biased GEMMs, learned position add, scaled bidirectional attention, + GELU-tanh - then post-LN, the 4x4 pool, weighted RMS and projection. The canvas is FIXED at + 896^2 (the learned table covers exactly that grid), and the family's image_mean/std (0.5) is + PREPROCESSING, not graph - `encode_image_` scales, `gemma3v_encode` takes planes raw like the + mtmd fixtures do. Composes `dasllama_tower.das`; owns only its layout and the block loop. + Two sanctioned exceptions to the family-quirk placement rule live outside this file: the + image span markers sit on the SHARED `gemma_chat` template in `dasllama_common.das` (template + detection cannot tell gemma3 from gemma2 - both spell `` - and the chat + layer's `image_vocab_ok` gate refuses a text-only vocab); and the carrier deliberately serves + ONE f32 plane (F32/F16/BF16 widen exactly; anything else refuses by name at stage) instead of + gemma4v's per-tensor plane split - the Metal leg runs the f32 mulmm, so there is no bf16 + plane to preserve. +- **`dasllama_qwen3v.das`** - the qwen3vl vision tower (Omni AND dense Qwen3-VL, + `qwen3vl_merger`): mmproj load (per-tensor bf16/f32 planes; the temporal patch-conv PAIR + folds to one f32 GEMM at stage - a still evaluates both convs on one frame, so conv_0+conv_1 + = (W_0+W_1)*img) and the pre-LN forward - fused-qkv biased GEMMs, vision-mrope (h/w ladder + tables from `build_rope_tabs_vision`, full-head NEOX apply), scaled bidirectional + attention, GELU-tanh - after a spatial-merge REORDER of the patch stream with the 48x48 + learned position table resized to the grid by an antialiased bilinear resample; then post-LN and + the 2x2 merger MLP (the x4 reshape is free - merge partners are already adjacent). Dense + carriers add DEEPSTACK taps (collected from the TENSOR list, never the metadata flags): + after each tap block a LayerNorm->fc1->GELU->fc2 merger over the x4-merged rows emits one more + proj_dim slice, and the output rows widen to (1+n_deepstack)*proj_dim - slice 0 = the main + merger, the decoder adds slice l+1 after layer l. The family token budget [8, 4096] is + mtmd's, not the gemma-scoped DASLLAMA_VISION_* knobs; image_mean/std (0.5) is + PREPROCESSING like gemma3v. CPU serving default: the block GEMMs serve as Q8_0 planes + (read-time transcode, the gemma3v recipe); the exact lane stages EVERYTHING f32-in-blob at + ff_pad both dims - blocks, mergers, and taps - because the Metal tower (`register_qwen3v_gpu`, + blocks + rope + taps + tail on-device, tap/tail proj outputs land in `ds_stash`/`ds_slice`) + and the float-batch tier both read f32 planes, and the same lane is the parity rail (the + f32 sites are this charter's sanction, not a fallback). Pin knobs + `set_qwen3v_q8`/`reset_qwen3v_q8`, two image tags; the lane policy prefers f32 when the + GPU tower or accel serves. Composes `dasllama_tower.das`; owns + only its layout, the reorder walk, and the block loop. SANCTIONED over the 1 GiB staged-mint line (this family + and qwen25v): the Omni (2.1 GB) and Qwen2.5-Omni (2.6 GB) mmprojs stage source+image at + once with no cap - the same shape their audio halves already stage - until + `followup_general.md` 24's streaming mint covers towers. +- **`dasllama_qwen25v.das`** - the qwen2.5 window-attention ViT (Qwen2.5-Omni/VL, projector + `qwen2.5o`): 32 RMS-normed blocks with separate biased q/k/v GEMMs and a gated-silu FFN + whose hidden width comes from the TENSOR dims (the metadata's feed_forward_length lies); + rope-only positions, per-channel CLIP normalize (`vision_normalize3`). Rows sort by 112 px + windows of 2x2-merged groups, non-full blocks attend per contiguous window range + (`attention_bidir_windows` in the tower - no mask tensor exists), every `n_wa_pattern`-th + block attends in full, and the merger un-sorts back to group-row-major. Its decoder is the + `qwen2vl` arch whose plain MROPE reads the rope sections as contiguous ranges + (`Config.mrope_interleaved` false -> `build_rope_tabs_mrope`; the qwen3vl family sets the + flag and keeps the interleaved walk). This tower serves the file's planes ONLY - no q8 + lane: its residual rows carry the Qwen2-VL-lineage outlier channels that per-32-block + activation requant cannot represent (a q8q8 lane measures 2.0 x rms vs the oracle, where a + DELETED layer measures less - the gate cannot discriminate; the weights themselves + quantize fine at 0.007 x rms, and a float-activation q8 GEMM wins nothing on these + compute-bound shapes since the CPU speedup IS the int8xint8 dot). +- **`dasllama_vision_embedder.das`** - the vision carrier: `VisionEmbedder` / `VisionState`, the + `AsrModel` shape for vision - one union through every seam, the family sniffed from the mmproj + (`clip.vision.projector_type`, or a `.dlim`'s baked tag) at load, one-line arms. Outside a + family's own file, a family type is named only here, in `dasllama_metal_tower.das`'s family + hooks, and in files under `tests/` - the set `REVIEW.das`'s seam check enforces. +- **`dasllama_audio_embedder.das`** - the audio carrier: `AudioEmbedder` / `AudioState`, the + vision carrier's audio twin - one union through every seam (server media worker, facade + `encode_audio`, tutorials), the family probed from the mmproj's audio tensor (or a `.dlim`'s + baked tag) at load, one-line arms. Outside a family's own file, an audio family type is named + only here, in `dasllama_asr.das`'s union field and one-line arms (the ASR rail's own carrier), + in the metal family hooks (`dasllama_metal_tower.das`, `dasllama_metal_asr_dec.das`), in + `benchmarks/asr/asr_bench.das` (the ASR bench pins a family serving knob for its correctness + rail), in `utils/dasllama-convert/main.das` (the per-family `.dlim` bake tool dispatches on family by + design), and in files under `tests/` - the set `REVIEW.das`'s seam check enforces. + +Vision oracle provenance (the convention `REVIEW.md`'s fixture rule points at): real image +fixtures and mmproj files live in the models dir with `.sha` pins, fetched never generated +(their `performance/fetch_models.das` entries are the checkable pins); the mtmd reference dumps +live beside them in `gemma4-vision-oracle/`, `gemma3-vision-oracle/` and +`qwen3vl-vision-oracle/`, whose `mint.sh` (gemma4uv, qwen3v Omni), `mint_4b.sh` (qwen3v +deepstack), `mint_25o.sh` (qwen25v), `mint_e2b.sh` / `mint_e4b.sh` +(gemma4v) and `mint_gemma3.sh` (gemma3v) record the +exact `llama-mtmd-debug` / `llama-mtmd-cli` invocation that minted each +dump, so regeneration is a command, not archaeology. An encode oracle dump is minted on the +CPU, `-fa off`, from the f32-widened mmproj twin - the only true-f32 reference arm (the +reference's Metal "f32" GEMM stages half operands, its flash-attention path casts K/V to f16, +and the shipped bf16 mmproj rounds activations to bf16; its own four arms spread +<= 6.5e-3 on the gemma4v tokens). + diff --git a/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md b/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md new file mode 100644 index 0000000000..ad05fb39fb --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md @@ -0,0 +1,191 @@ +# dasLLAMA Architecture - runtime mechanisms + +Companion to `ARCHITECTURE.md`; section numbers are that document's. + +### 2.2 Kernel SHAPE is compile-time; only DATA is runtime + +The test is one question: *for a given compiled kernel, can this value change between dispatches?* + +- **Yes -> DATA.** Context depth, row counts, buffer offsets, `kv_dim`, scales, head counts. It + belongs in a uniform or a kargs struct. +- **No -> SHAPE.** A codec's block stride, a scale-plane stride, a lane width, an unroll factor, a + format selector. It must NOT reach the kernel as a uniform, a kargs field, or a helper parameter. + +Shape belongs to the specialization: a separate kernel class and PSO, a per-codec overload, a +monomorphized generic, or a `static_if` on a compile-time witness. Handing a shape constant over as +a value and trusting the shader compiler to fold it back is an assumption, not a guarantee, and it +is worth nothing in the kernels that matter. + +**Verify against the EMITTED shader, never the das source.** Read the `*_msl` global or the SPIR-V +dump and confirm the constant is literal there: `blk * 34u`, not `blk * bstr`. A helper that looks +specialized in das can still lower to a runtime multiply. + +**kargs structure.** Twins of a family bind the SAME kargs type at the SAME binding, even where one +twin ignores a field. A twin that carries an extra scalar must not shift the others to different +slots, because that asymmetry propagates into the encoder as a per-form branch. Two tells that a +fold is overdue: + +1. *A value reaches the encoder twice* - a pooled scalar uniform BUFFER passed alongside the + identical value as a parameter (`bd` next to `d`). The buffer is uploaded and released per step + to carry a number the encoder already holds. +2. *A field is a function of the fields beside it.* An expert plane's block stride is + `kdim * ndim / blocksize`; a reciprocal scale is `1/sqrt(dim)`. Derive it in the builder - each + one passed separately is a second place to get it wrong. Likewise, when a kernel's grid IS the + geometry it reads, take the grid off the kargs rather than re-passing the numbers. + +**Nothing dispatches a kernel except its `enc_*` builder.** A hand-rolled bind list elsewhere - a +tune-race harness, a benchmark, a probe - duplicates the builder and desyncs the moment the +family's args change, silently: the slots still exist, the types still compile, and the kernel +reads a struct out of a 4-byte buffer. The dispatch census only catches a builder that binds kargs +on some paths and not others; a duplicate that binds NO kargs is invisible to it. + +### 2.3 GPU-resident cache identity + +An address-keyed entry carries its SPAN, and a hit must cover the request - a shorter first upload +must never serve a wider later one. Different upload FORMS (plain span vs concat) live in separate +tables so they can never alias; the metal `RegionEntry` rail is the model. Buffers grown out of an +entry retire to a list released only at quiesce boundaries, because unretained command buffers may +still bind them. + +### 2.3a Making weights live bumps the weights epoch {#weights-epoch-on-load} + +Every path that makes weights live bumps the weights epoch, the image rail included. A fresh +mapping or chunk can land on a deleted model's recycled addresses, and an address-keyed region +cache would otherwise serve the previous model's bytes out of an entry that still looks like a +hit. + +### 2.4 Complexity and length lint + +STYLE037 (cyclomatic) and STYLE038 (line count) are prompts to look, not orders to refactor. This +module has shapes that are irreducible by design and they take `// nolint:STYLE03x` with a one-line +reason: flat one-call-per-item runs (a registration or release list with one line per kernel), and +GPU kernel bodies whose phases are coupled by barriers, cooperative-matrix ops or register +residency and so cannot cross a function boundary without changing the shader. + +Split only where a real seam exists - genuine duplication, a distinct phase, a self-contained arm - +and only when the extracted helper stands on its own. Two corollaries this module keeps tripping +over: **a kargs fold that grows an already-over-cap kernel body is not a reason to abandon the +fold** - unpacking N fields adds N lines; take the growth and ledger the real seam. + +### 2.6 Capability questions and readiness questions are different questions + +A predicate that mixes them cannot be reused. `prefill_decline` answers "can metal serve this +model" (capability) *and* "is this window staged" (readiness - are the rope tables built). A caller +that runs before the window is staged must ask the capability half only, or it gets "not yet" +forever and its feature silently never runs. Split such predicates rather than reordering the +caller; an optimistic capability answer is safe when the late path has a fallback, and here it does. + +### 2.7 A quantized activation carries its scale lattice (Vulkan) + +Two activation quant forms ride the vulkan rail, and they differ in the SCALE LATTICE, not the +int8 payload: the Q8_0 form scales per 32 values, the superblock form per 256 (with per-32 +sub-scales inside). A compiled kernel indexes ONE lattice - the q8 GEMV/GEMM rail reads per-32 +scales; the k-quant (k4/k5/k6/q40) kernels index the per-256 lattice. `kq_sb(fmt)` is the +predicate (`ARCHITECTURE_ENGINE.md` sec.1.2), and it answers for the WEIGHT plane the dispatch +consumes. + +Three consequences the code is shaped around: + +- **A quant/act encoder is picked by the CONSUMING plane's format, never by a rail-wide + default.** Every site that encodes activations for a GEMV/GEMM keys its encoder (and its + grid: superblock counts are `n/256`) on the consumer's `kq_sb`. The failure mode is silent + per-dispatch: the wrong lattice indexes garbage scales, outputs stay finite, and nothing + panics - only end-to-end token parity (`harness/parity.das`) catches it, so a resident + change is witnessed only by parity runs over both a q8 and a k-quant model. +- **The fused add-rms+requant twin exists only for the per-32 form.** The rail gates it on + "every consumer of this buffer is Q8_0-scaled" (`rd_x_quants_b32`), and the profiler stamp + shape must ride the SAME gate, or profiles desync from what actually dispatched. +- **A GEMV group sharing one activation buffer must be lattice-homogeneous.** q/k/v share one + quantized x; gate/up share another. Resident arming classifies each member's consumer form + and DECLINES a mixed group rather than serving one member wrong scales. + +### 2.8 Every program root declares its stack budget and its prefill intent + +`options stack` is main-module-only: it does not unify up from required modules, so no library in +the forward chain can declare the depth it needs. Every program that drives the engine - each test, +harness, benchmark, and tool - must therefore declare it, and dasLLAMA's frames are deep enough +(by-value `Session`s, the forward/prefill chain, the generated kernel tier) that the default is +never enough. + +The budget is **one number in every root**, currently 524288. Per-root numbers do not survive: a +frame that grows past the smallest declared budget breaks only the program that declared least, so +the limit is discovered by crashing - and the program that crashes is whichever one is run +rarest. A measurement rig sized below a test suite is the worst case of this, because the suite +stays green while the rig dies. The cost of the uniform number is reserved address space per +context; the cost of per-root numbers is a runtime crash found by the least-covered program. + +The second declaration is prefill intent. A model-loading root declares `allow_cpu_prefill()` +on the arms that hit the CPU-prefill guard - `set_metal_mode` with a runtime value declares +nothing, since `MetalMode.off` leaves the guard armed - and logs, once, which configuration it +ended on. The guard panics, and a panic takes every live stream down, so an undeclared root is +a serving outage waiting on its first long prompt. Both halves of root discipline are enforced +by `tests/test_program_roots.das`. + +### 2.9 Environment knobs + +A knob is an `[EnvConfig]` field in `dasllama_env.das`, read as `g_env_*.`; the field is +also what generates its `ENVIRONMENT.md` row, so a knob declared anywhere else is invisible to +the documentation and to the registry test. The sanctioned forms beyond a plain typed field: + +- **Tri-state knobs** - presence matters, or the effective default is computed at runtime - are + `Option` fields. +- **Dynamic names** - a variable named by data, not by code - go through `env_is_set` / + `env_value_of`; there is no field to declare because the name is not known at compile time. +- **The config loads once at context init**, so `set_env_variable` mid-process is invisible to + the running config: arm a child process's environment instead. +- **A write of a foreign library's knob** (`set_env_variable` with a literal name) is allowed + only before that library first reads it, and the name must be a declared `[EnvConfig]` knob - + the registry test scans writes too, so a re-spelled name fails it. + +`tests/test_env_registry.das` enforces the lot in both directions (declared <-> documented, +read <-> registered, writes included). + +**Override announces.** A knob that is a gate escape, a policy override, or a threshold +recalibration - one whose presence makes a run measure, mint, or emit something its defaults +would not (`DASLLAMA_ALLOW_UNTUNED`, the `DAS_TUNE_NOISE_*` pair and `DAS_TUNE_POLICY` read +through llvm_tune's accessors) - announces at the point it changes the outcome: at least one +printed or logged line naming the knob by its env spelling. Set-but-inert stays silent; +per-site repeats are correct (the same knob may legitimately announce at its `[init]` wire, at +a gate it bypasses, and in a tool banner). Ordinary behavior knobs (thread counts, rail +selection, formats) are not overrides under this contract - their state belongs in the tools' +existing config/status lines, not in per-knob announces. + +### 2.11 The [hot_path] coverage model + +`[hot_path]` sits at the REGION ENTRY - the `*_encode` / `*_decode` / step drivers - and its +`[no_alloc]` / `[no_env]` / `[no_io]` contracts arm transitively down the call graph, so +interior kernels stay bare. A new function needs the annotation itself only when no annotated +entry reaches it: a new entry point carries it, and a new backend entry (kernel-backend +override, batch donor) carries it too, because backends are also reached from un-annotated +harness paths. Reused buffers take `@scratch`; debug and profiling legs take `[cold_path]`. +The tokenizer encode/decode path is sanctioned UNCOVERED by the region contracts - its perf +gate is the `--tok` scaling rows, whose instrument (the size-ladder ratio) catches what the +contracts cannot. + +### 2.12 The post-CPU-burn GPU ramp - the residency heartbeat holds it; do not build a warm-up + +Figures in this section: M1 Max, 2026-08-23 - probes are quiet `-jit` runs with the rig's +tune manifest; cells are the released `lcpp_bench` exe (`--image --image-think -r 3 -t 8 +--ngl 99`); full tables in `history/dasLLAMA/qwen3vl_plan.md` slices M/J. + +After a CPU-only phase, the first Metal submission runs degraded - one time per window, +entirely in the kernel-side driver window (kernelStart->kernelEnd; queue hand-off and GPU +execution stay flat - split probe-verified 2026-08-23, `harness/residency_ramp_probe.das` + +`das_metal_boost`'s `metal_submit_trace`). The long-window mechanism is the OS collecting a +committed+requested `MTLResidencySet` during inactivity; the fix is the +residency HEARTBEAT - a dasMetal background thread re-requesting +residency every 5 ms for `DASLLAMA_METAL_HEARTBEAT_S` (default 180 s, 0 = A/B rail) after +the last served step, kicked from `residency_flush`. Measured on the qwen3v tower encode: +3000 ms burn drv 17.7 -> 3.0 ms. The `MTLResidencySet` pin itself +(`DASLLAMA_METAL_RESIDENCY`, on by default) holds the short-window case (-15 ms/encode on +the tower's first submission after its CPU stem); the heartbeat holds the long ones. +REFUTED by measurement (`PERF_LEDGER.md`, the heartbeat entry): every pre-payment - empty +command buffers and driver round-trips (0.107 ms - never asleep), single-dispatch kernels, +per-page touch kernels, light pulse-trains through the burn - a warm-up always pays its own +cost ON TOP of the slack it was meant to hide; do not re-attempt one. Also refuted: a +whole-map no-copy anchor buffer (page pre-wiring moves nothing) and the once-proposed +per-buffer tracking audit (resource count is not the lever) - prefill pools simply stay +tracked (untracked reds parity). The ~1-3 ms window-scaled residual is the driver/GPU +idle-state wake class, no user-space lever found. Related, same ledger: merely arming Metal +makes a CPU q8 tower encode ~1.7x slower - mechanism unnamed, deleted by a GPU-served tower. + diff --git a/modules/dasLLAMA/ENVIRONMENT.md b/modules/dasLLAMA/ENVIRONMENT.md index b5d0fd9653..022757a0c7 100644 --- a/modules/dasLLAMA/ENVIRONMENT.md +++ b/modules/dasLLAMA/ENVIRONMENT.md @@ -85,8 +85,8 @@ Apple GPU backend. Absent on non-Apple builds, where setting them does nothing. | `DASLLAMA_METAL_GEMV_TG` | number | 4 | Rows per MoE router/expert GEMV threadgroup, clamped 1..32 (the dense GEMV geometry is fixed). | | `DASLLAMA_METAL_KQ_B8` | flag | on | Single-pass B8 twin for K-quant small-batch mv at B=5..8; 0 is the A/B rail. | | `DASLLAMA_METAL_KV_MIRROR_MB` | number | 4096 | Ceiling in MiB for the device-side KV mirror, clamped 64..4096. | -| `DASLLAMA_METAL_RESIDENCY` | flag | on | Pin the served working set (weight regions, planes, the recycled pool buffers) in an MTLResidencySet, macOS 15+: the per-commit residency pass stays a no-op, so the first submission after a CPU-only window stops repaying it (~15 ms/encode on the M1 Max released-rig image cell; ARCHITECTURE.md 2.12); 0 is the A/B rail. | -| `DASLLAMA_METAL_HEARTBEAT_S` | number | 180 | Seconds the residency-set heartbeat keeps re-requesting residency after the last served step (a requested set still un-wires over a long CPU-only window; ARCHITECTURE.md 2.12); 0 disables the heartbeat - the A/B rail. | +| `DASLLAMA_METAL_RESIDENCY` | flag | on | Pin the served working set (weight regions, planes, the recycled pool buffers) in an MTLResidencySet, macOS 15+: the per-commit residency pass stays a no-op, so the first submission after a CPU-only window stops repaying it (~15 ms/encode on the M1 Max released-rig image cell; ARCHITECTURE_RUNTIME.md 2.12); 0 is the A/B rail. | +| `DASLLAMA_METAL_HEARTBEAT_S` | number | 180 | Seconds the residency-set heartbeat keeps re-requesting residency after the last served step (a requested set still un-wires over a long CPU-only window; ARCHITECTURE_RUNTIME.md 2.12); 0 disables the heartbeat - the A/B rail. | | `DASLLAMA_METAL_HAZARD_PARANOID` | flag | off | Barrier at every dispatch instead of at tracker-detected hazards (correctness bisect). | | `DASLLAMA_METAL_HAZARD_STRICT` | flag | off | Treat every detected hazard as strict, widening barriers (correctness bisect). | | `DASLLAMA_METAL_PIPE_DEBUG` | flag | off | Per-step pipeline trace: GPU envelope and true inter-step handoff idle, first steps plus outliers. | diff --git a/modules/dasLLAMA/LAWS.md b/modules/dasLLAMA/LAWS.md index 8b0f35362c..fcb0686fef 100644 --- a/modules/dasLLAMA/LAWS.md +++ b/modules/dasLLAMA/LAWS.md @@ -30,3 +30,12 @@ threads of every row threadgroup cost 24% of the qwen3moe pp512 forward) - per-row validity/ownership in a bucketed dispatch is answered by a table the bucket builder stamps, never by a per-row search over the bucket directory. + +- **2026-08-27** (`REVIEW.md`, `ARCHITECTURE.md`): Boris ruled the harvester's ledger for + `dasllama/dasllama_image.das` lands as harvested - all six renames approved (`ImgWriter.h` + -> `dwrite_h`, `.mem` -> `chunk_base`, `.cap` -> `chunk_cap`, `IMAGE_WRITE_BAND` -> + `DWRITE_STAGING_BAND`, `vk_cfg` -> `vk_tag` with the alias dissolved), the four rules + appended to `REVIEW.md` in the harvester's own words, the eight facts landed as anchored + `ARCHITECTURE.md` subsections cited from the owning functions via `[arch(at=...)]`, and + the `cache_via_image` return contract promoted to a `//!` doc matching its sibling's + wording. The module arms `[docs] enforce_arch = true` so every minted anchor stays cited. diff --git a/modules/dasLLAMA/README.md b/modules/dasLLAMA/README.md index 4c635c9c92..37ad6a535e 100644 --- a/modules/dasLLAMA/README.md +++ b/modules/dasLLAMA/README.md @@ -97,6 +97,13 @@ modules/dasLLAMA/ .das_module # registers the dasllama/ namespace — both binaries (static rows derive from it) CMakeLists.txt # ADD_MODULE_DAS_FROM_DESCRIPTOR + install rules ARCHITECTURE.md # what-goes-where ledger — which module owns which concern + ARCHITECTURE_ENGINE.md # companion: the engine, format, load, CPU-tier, support and serving charters + ARCHITECTURE_GPU.md # companion: the GPU backend role table, the backend asymmetries, the refuted kernel shapes + ARCHITECTURE_MEDIA.md # companion: the encoder-tower, audio, ASR and vision charters + ARCHITECTURE_IMAGE.md # companion: the prepared-image (.dlim) rail + ARCHITECTURE_RUNTIME.md # companion: kernel shape, caches, knobs, coverage, the GPU ramp + ARCHITECTURE_MEASUREMENT.md # companion: the benchmark rig, the tune gate, the instrumentation rails + ARCHITECTURE_INVARIANTS.md # companion: the inherited invariants REVIEW.md # the per-change review checklist METHODOLOGY.md # how every published number is measured (the site's method page) PROFILE.md # the three profiling rigs — the routine in-process check, the oracle cell, the rig that publishes diff --git a/modules/dasLLAMA/REVIEW.das b/modules/dasLLAMA/REVIEW.das index 161f5fc1cc..0160b4d3e7 100644 --- a/modules/dasLLAMA/REVIEW.das +++ b/modules/dasLLAMA/REVIEW.das @@ -100,7 +100,8 @@ def private check_tutorial_floor(tut_texts, rst_texts : array) { let private AUDIO_CARRIER = "modules/dasLLAMA/dasllama/dasllama_audio_embedder.das" let private VISION_CARRIER = "modules/dasLLAMA/dasllama/dasllama_vision_embedder.das" -// The licensed family namers per modality - ARCHITECTURE.md §1 ledgers each with its reason. +// The licensed family namers per modality - the ARCHITECTURE_MEDIA.md carrier charters (§1.7, +// §1.7b) ledger each with its reason. var private AUDIO_SEAM_EXEMPT <- { "modules/dasLLAMA/dasllama/dasllama_asr.das", "modules/dasLLAMA/dasllama/dasllama_metal_tower.das", @@ -188,7 +189,7 @@ def private check_family_seams(carrier, kind_enum : string; exempt : table_*.das file's suffix is a role -// ARCHITECTURE.md sec.1.5's role table carries — a new role file adds its role to BOTH. +// ARCHITECTURE_GPU.md sec.1.5's role table carries — a new role file adds its role to BOTH. var private GPU_ROLES_METAL <- { "kernels", "common", "decode", "prefill", "gemm", "shapes", "tower", "asr_dec", "lens" } @@ -233,7 +234,7 @@ def private check_gpu_role_partition { let known = backend == "metal" ? key_exists(GPU_ROLES_METAL, role) : key_exists(GPU_ROLES_VULKAN, role) if (!known) { gate_finding("modules/dasLLAMA/dasllama/{name}", - "backend file with role '{role}' that the partition does not carry - add the role to ARCHITECTURE.md sec.1.5's role table and this gate's set in the same change") + "backend file with role '{role}' that the partition does not carry - add the role to ARCHITECTURE_GPU.md sec.1.5's role table and this gate's set in the same change") } } } diff --git a/modules/dasLLAMA/REVIEW.md b/modules/dasLLAMA/REVIEW.md index 6867d5ce84..74df4af13d 100644 --- a/modules/dasLLAMA/REVIEW.md +++ b/modules/dasLLAMA/REVIEW.md @@ -24,6 +24,11 @@ winners back. **A GPU kernel, driver, dispatch-class, or K/V-mirror change applies `REVIEW_GPU.md`.** +**A change to the image rail - `dasllama/dasllama_image.das`, or, wherever the diff puts it, +a `.dlim` mint (building a `.dlim` from a gguf), a `.dlim` load, an image identity, or a +flavor (the backend-and-layout variant an image is baked for, one part of its identity) - +applies `REVIEW_IMAGE.md`.** + **A change to `dasllama/dasllama_audio.das`, `dasllama/dasllama_audio_io.das`, `dasllama/dasllama_audio_embedder.das`, `dasllama/dasllama_asr.das`, `dasllama/dasllama_asr_types.das`, `dasllama/dasllama_vad.das`, or an ASR family file - one @@ -49,14 +54,15 @@ change, and a bump with no kernel work is the same defect.** Kernel work is what the generated kernel source, or the set of compiled pipeline variants (PSOs) built from it - a kernel body, a variant set, or a `[tune]` / `[tune_perm]` / `[tune_companion]` grid. A host-side bind or dispatch-argument change (an `@off` binding, a uniform value) is not, and -neither is `[tune_scope]` metadata (`covers=`, `tuner=`, `version_of=`). Equal versions mean -an equal kernel set and an interchangeable sidecar set - the exchange keys validity on -version and box. +neither is `[tune_scope]` metadata (`covers=`, `tuner=`, `version_of=`). **A value that cannot change between dispatches of one compiled kernel never reaches that kernel as a uniform, a kargs field, an `@off` bind offset, or a helper parameter.** A value that can change between dispatches goes in a uniform, a kargs field, or an `@off` bind offset. +**Never reorder or merge the float multiplies in a function that builds a RoPE angle table +(`dasllama/dasllama_rope.das`) - keep the multiply order the code already has.** A regrouping +moves the angles in the last bits and flips token-exact fixtures. **A diff that changes a kernel-selection predicate in `dasllama/` is based on timing that ran both variants interleaved in one process, under one instrument.** The same holds for a constant in `dasllama/` whose value was chosen by timing two candidates against each other. A @@ -71,7 +77,8 @@ wall-clock - and a stated decision.** mm) outside a parity or oracle rail, where a faster-format twin already serves the same weights and shape, is a defect - call that twin instead.** Weights with no faster twin (unquantized planes) are out of scope; a site that must stay f32 for another reason is ledgered on its own -file's charter line in `ARCHITECTURE.md` sec.1, not commented into compliance. +file's sec.1 charter line in `ARCHITECTURE_ENGINE.md`, `ARCHITECTURE_GPU.md`, or +`ARCHITECTURE_MEDIA.md`, not commented into compliance. **Platform-specific code in an engine file (`dasllama/`) lands only in that platform's backend file.** @@ -82,7 +89,7 @@ more of `dasllama/dasllama_common.das`.** **A NEW clock read paired with a print or log of the elapsed interval is a defect in an engine file (`dasllama/`) outside a cold one-shot load, bake, map, or tokenizer-build progress log** - instrumentation goes through the sanctioned rails, listed with their reasons in -`ARCHITECTURE.md` sec.2.10. +`ARCHITECTURE_MEASUREMENT.md` sec.2.10. **A clock value that changes what the program DOES - control flow, eviction, a generated name; not a reported wall or a best-of reduction over reported walls - is marked @@ -166,45 +173,6 @@ stage, and are not stage figures. The naming sits in the figure's own sentence, heading that covers the table's rows, or in a section-level provenance line that covers the paragraphs under it. -**A transform on the go-live path - repacking, quantizing, folding, permuting - is a defect; -it belongs to the mint.** Going live is `parse_image` pointing a live carrier's planes into the -mapped `.dlim`. - -**A missing `.dlim` is minted first, and the model is served from what was minted.** - -**A weight carrier becomes live only through `build_image` and `parse_image` in -`dasllama/dasllama_image.das`: reading weights into a live carrier, or releasing an image -backing, anywhere else is a defect - and a second mint path, per family, per format, or per -backend, is a defect even where its output is identical.** - -**A decoder mint never holds the whole model.** A decoder mint is the mint of an LLM decoder -model, not of a tower or embedder carrier. It sizes the image before writing the first byte and -writes each plane as it is produced. A mint that is slower in exchange for a lower peak is -correct. - -**A staged carrier mint (`cache_via_image_staged`) meeting a source file at or past 1 GiB -either refuses it or streams it the way a decoder mint does.** A refusal names that file. The -staged form holds source and image at once, and this rule caps that doubled peak. - -**A path that reinterprets a mismatched image, or widens an identity so that more files match, -is a defect.** `image_identity` names the box profile, the knobs, and the flavor a file was -baked for, and a mismatch declines loudly. - -**An image save drops AT MOST its own lane's dead siblings plus BROKEN/version-stale images in -any lane.** A lane is an identity's (quant, tag) pair. - -**Reaping an image whose identity the code cannot recompute - another flavor's, another -family's - is a defect.** - -**A plane split that follows the source FILE rather than a runtime knob takes ONE image tag**, -with the meta flags describing the layout - a per-tensor type split is not a second flavor. - -**A plane the target platform or config never reads is not written into the image - the mint -decides that, not the load.** - -**A flavor takes its image file through `image_path_for` and its tag through -`register_image_family_tag`.** - **A change to user-facing API updates every place it is shown: a tutorial source, `.rst` page, docstring, help string, `README.md`, or checked-in document still showing the old call, flag, or default is a defect of the change, not of the docs.** User-facing means anything a consumer @@ -219,21 +187,26 @@ path under `modules/dasLLAMA/` - is a defect:** a module added to its allowed se match dropped or narrowed, or an error text that no longer names the facade to require instead. The allowed set is the table in the lint. +**A `// nolint:STYLE037` or `// nolint:STYLE038` on a function a follow-up ledger entry calls +reducible is a defect - land the ledgered split instead.** The warning is what keeps the ledger +entry visible. + **`options _dasllama_internal` belongs only in a file whose job is to reach engine internals: an engine file under `dasllama/`, a test, harness, benchmark, or rig this module -owns, or a consumer `ARCHITECTURE.md` sec.1 names as ruled** - a symbol the facade lacks is -added to `dasllama/dasllama.das`, not escaped around. A `require ... public` that re-exports an -engine module OUT of an escaped file, beyond what that consumer's ruled charter -(`ARCHITECTURE.md` sec.1) grants, breaks this rule too. +owns, or a consumer `ARCHITECTURE_ENGINE.md` sec.1.8 names as ruled** - a symbol the facade +lacks is added to `dasllama/dasllama.das`, not escaped around. A `require ... public` that +re-exports an engine module OUT of an escaped file, beyond what that consumer's ruled charter +(`ARCHITECTURE_ENGINE.md` sec.1.8) grants, breaks this rule too. **Weakening `REVIEW.das` (beside this file) is a defect:** dropping a check, adding a name to a check's licensed set - the names that check does not flag - or a finding text that no longer names what failed. What the gate enforces is read from the gate itself; each check's finding text states its own rule. -**A new `REVIEW.das` check ships its `ARCHITECTURE.md` sec.1 line in the same change.** The -line names the check and the names it licenses. A licensed name is one that check does not -flag. When the check licenses no names, the line says so. +**A new `REVIEW.das` check ships its sec.1 charter line in the same change, on the charter of +the file it checks - in `ARCHITECTURE_ENGINE.md`, `ARCHITECTURE_MEDIA.md`, or +`ARCHITECTURE_GPU.md`.** The line names the check and the names it licenses. A licensed name +is one that check does not flag. When the check licenses no names, the line says so. **An upstream mechanism is described in our own terms, not attributed** - no "lifted/ported verbatim from" and no upstream symbol, header, or constant names in a `.md` @@ -260,13 +233,14 @@ a struct the renderer emits but the registry does not is caught by by retranscoding `$LCPP/src/unicode-data.cpp` (the reference checkout) instead.** **A diff that adds a file under `dasllama/`, moves code between files, or changes what a file -owns lands the sec.1 edit that keeps the charters true, in the same change.** A diff that adds +owns lands the sec.1 edit that keeps the charters true - in `ARCHITECTURE_ENGINE.md`, +`ARCHITECTURE_GPU.md`, or `ARCHITECTURE_MEDIA.md` - in the same change.** A diff that adds a file beside one that has its own sec.1 charter line lands that edit too. A module-root doc file - a ledger, a plan, `LAWS.md` - has no charter line and lands free. -**A per-file inventory restated in this checklist is a defect of the checklist.** -`ARCHITECTURE.md` sec.1 owns the per-file list. A rule naming what KIND of code lands in which -file is the checklist's own. +**A per-file inventory restated in this checklist is a defect of the checklist.** The sec.1 +charters - `ARCHITECTURE_ENGINE.md`, `ARCHITECTURE_GPU.md`, `ARCHITECTURE_MEDIA.md` - own the +per-file list. A rule naming what KIND of code lands in which file is the checklist's own. **A tensor format conversion lands in `dasllama/dasllama_convert.das`.** @@ -298,21 +272,8 @@ kind - `create_chat_` panics at create, not at render. files. A doc comment naming the family a helper was built for is fine. **A `dasllama/dasllama_tower.das` helper with one calling family lands in that family's -file** - a single-caller helper sanctioned as tower-worthy is ledgered on `ARCHITECTURE.md` -sec.1's tower charter line, not argued in review. - -**On the lane that serves the file's own planes, a weight plane's element type follows its -SOURCE tensors, per weight region - the set of source tensors a carrier stores in one plane -(a block stack, a merger/projector).** - -**A weight region - the set of source tensors a carrier stores in one plane - whose source -tensors disagree on element type is refused in a message naming the offending tensor and -both element types.** - -**A lane that PERSISTS a converted form of the file's planes is a separate flavor under its -own image identity.** A persisted form is one an image could carry. The load that picks such a -lane prints which lane it picked. A conversion made and dropped inside one forward pass -persists nothing and is not such a lane. +file** - a single-caller helper sanctioned as tower-worthy is ledgered on +`ARCHITECTURE_MEDIA.md` sec.1.7's tower charter line, not argued in review. **A harness that prints output for another tool to compare exits non-zero when its run ends without those comparison lines - wrong flags, failed load.** diff --git a/modules/dasLLAMA/REVIEW_AUDIO.md b/modules/dasLLAMA/REVIEW_AUDIO.md index bcce5111d8..e603c61d7f 100644 --- a/modules/dasLLAMA/REVIEW_AUDIO.md +++ b/modules/dasLLAMA/REVIEW_AUDIO.md @@ -6,7 +6,7 @@ shared encoder-tower home serves both), `dasllama/dasllama_audio.das`, `dasllama/dasllama_audio_io.das`, `dasllama/dasllama_audio_embedder.das`, `dasllama/dasllama_vad.das`, or an ASR family file - one `dasllama/dasllama_.das` holding a single speech model family - applies this list with `REVIEW.md`'s.** -`REVIEW_COMMON.md` (repo root) binds this file too. Architecture doc: `ARCHITECTURE.md`. +`REVIEW_COMMON.md` (repo root) binds this file too. Architecture doc: `ARCHITECTURE_MEDIA.md`. **In `dasllama/dasllama_asr.das` and `dasllama/dasllama_audio_embedder.das` - the union carriers that route each facade function to one family - a family arm that does anything but diff --git a/modules/dasLLAMA/REVIEW_GPU.md b/modules/dasLLAMA/REVIEW_GPU.md index fd43b0f27d..3485a2d1a0 100644 --- a/modules/dasLLAMA/REVIEW_GPU.md +++ b/modules/dasLLAMA/REVIEW_GPU.md @@ -1,7 +1,7 @@ # dasLLAMA GPU Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `ARCHITECTURE.md`. +doc: `ARCHITECTURE_GPU.md`. **Routed from `REVIEW.md`: a diff touching a GPU kernel, driver, dispatch class, or the K/V mirrors applies this list together with `REVIEW.md`.** @@ -26,7 +26,7 @@ buffer, or in the staging loop that reads it.** A float operand keeps the op off fast path. **A diff that stamps a kernel class `[metal_kernel(float_a_ok=true)]` lands its -`ARCHITECTURE.md` sec.2.2b ledger line in the same change.** +`ARCHITECTURE_GPU.md` sec.2.2b ledger line in the same change.** **Never threadgroup-stage a `matmul2d` operand whose staged form matches its stored form - stream it from device instead.** A dequant, a transpose, or a layout or element-type change @@ -109,8 +109,8 @@ builder instead.** Binding it separately adds a second place to get it wrong. layout the upload produces, in the key too.** A hit must cover the request. **Never put a backend-only capability outside its backend's file for the matching role - a -capability with no matching role gets its own role file.** `ARCHITECTURE.md` sec.1.5 carries -the role table. +capability with no matching role gets its own role file.** `ARCHITECTURE_GPU.md` sec.1.5 +carries the role table. **Never add a role's file to a backend that does not have the capability.** @@ -133,7 +133,7 @@ times both kernels on one queue and compares their outputs. **A diff that adds or removes a Metal-only or Vulkan-only hook, role, served path, or backend-only capability - anything that changes what one backend can serve and the other -cannot - lands its `ARCHITECTURE.md` sec.1.5 edit in the same change - including when +cannot - lands its `ARCHITECTURE_GPU.md` sec.1.5 edit in the same change - including when sec.1.5 already carries that class of asymmetry, and including sec.1.5's per-driver lists of registered hooks and borrowed kernels.** sec.1.5 is the closed list; an asymmetry it does not carry does not exist. diff --git a/modules/dasLLAMA/REVIEW_IMAGE.md b/modules/dasLLAMA/REVIEW_IMAGE.md new file mode 100644 index 0000000000..9a53c958b1 --- /dev/null +++ b/modules/dasLLAMA/REVIEW_IMAGE.md @@ -0,0 +1,76 @@ +# dasLLAMA image rail Code Review Checklist + +**Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture +doc: `ARCHITECTURE_IMAGE.md`. + +**Routed from `REVIEW.md`: a diff touching `dasllama/dasllama_image.das`, or a `.dlim` mint, +load, identity, or flavor concern wherever the diff puts it, applies this list together with +`REVIEW.md`.** + +**A transform on the go-live path - repacking, quantizing, folding, permuting - is a defect; +it belongs to the mint.** Going live is `parse_image` pointing a live carrier's planes into the +mapped `.dlim`. + +**A missing `.dlim` is minted first, and the model is served from what was minted.** + +**A weight carrier becomes live only through `build_image` and `parse_image` in +`dasllama/dasllama_image.das`: reading weights into a live carrier, or releasing an image +backing, anywhere else is a defect - and a second mint path, per family, per format, or per +backend, is a defect even where its output is identical.** + +**A decoder mint never holds the whole model.** A decoder mint is the mint of an LLM decoder +model, not of a tower or embedder carrier. It sizes the image before writing the first byte and +writes each plane as it is produced. A mint that is slower in exchange for a lower peak is +correct. + +**A staged carrier mint (`cache_via_image_staged`) meeting a source file at or past 1 GiB +either refuses it or streams it the way a decoder mint does.** A refusal names that file. The +staged form holds source and image at once, and this rule caps that doubled peak. + +**A path that reinterprets a mismatched image, or widens an identity so that more files match, +is a defect.** `image_identity` names the box profile, the knobs, and the flavor a file was +baked for, and a mismatch declines loudly. + +**An image save drops AT MOST its own lane's dead siblings plus BROKEN/version-stale images in +any lane.** A lane is an identity's (quant, tag) pair. + +**Reaping an image whose identity the code cannot recompute - another flavor's, another +family's - is a defect.** + +**A plane split that follows the source FILE rather than a runtime knob takes ONE image tag**, +with the meta flags describing the layout - a per-tensor type split is not a second flavor. + +**A plane the target platform or config never reads is not written into the image - the mint +decides that, not the load.** + +**A flavor takes its image file through `image_path_for` and its tag through +`register_image_family_tag`.** + +**A diff that changes the `.dlim` layout or serialization, or changes which tensors the gguf +loader puts into an image, bumps `IMAGE_VERSION` (`dasllama/dasllama_image.das`) in the same +change.** +Without the bump a stale image stays structurally valid and silently serves a different model. + +**Weakening the `serialize_image_meta` field-count tripwire (`IMAGE_META_FIELDS`, +`dasllama/dasllama_image.das`) is a defect** - raising the constant without adding the field to +`serialize_image_meta` leaves that field out of every image. + +**A filesystem or chunk-allocation decline while saving an image +(`dasllama/dasllama_image.das`) never fails the load - warn, and serve what is still whole: +the image already built in memory, or the carrier as loaded.** + +**A bounds check on an image section or the meta blob in `dasllama/dasllama_image.das` is +written `bytes > msize || off > msize - bytes`, never `off + bytes > msize`, which wraps.** + +**On the lane that serves the file's own planes, a weight plane's element type follows its +SOURCE tensors, per weight region - the set of source tensors a carrier stores in one plane +(a block stack, a merger/projector).** + +**A weight region - the set of source tensors a carrier stores in one plane - whose source +tensors disagree on element type is refused in a message naming the offending tensor and +both element types.** + +**A lane that PERSISTS a converted form of the file's planes is a separate flavor under its +own image identity.** A persisted form is one an image could carry. The load that picks such a +lane prints which lane it picked. A conversion made and dropped inside one forward pass +persists nothing and is not such a lane. diff --git a/modules/dasLLAMA/REVIEW_VISION.md b/modules/dasLLAMA/REVIEW_VISION.md index e6b0685bb4..756bf0cd29 100644 --- a/modules/dasLLAMA/REVIEW_VISION.md +++ b/modules/dasLLAMA/REVIEW_VISION.md @@ -1,7 +1,7 @@ # dasLLAMA Vision Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `ARCHITECTURE.md`. +doc: `ARCHITECTURE_MEDIA.md`. **This checklist governs `dasllama/dasllama_vision.das`, `dasllama/dasllama_vision_io.das`, `dasllama/dasllama_vision_embedder.das`, `dasllama/dasllama_tower.das` (the shared diff --git a/modules/dasLLAMA/benchmarks/REVIEW.md b/modules/dasLLAMA/benchmarks/REVIEW.md index e0beedb964..6c36663ea0 100644 --- a/modules/dasLLAMA/benchmarks/REVIEW.md +++ b/modules/dasLLAMA/benchmarks/REVIEW.md @@ -1,8 +1,8 @@ # dasLLAMA benchmarks Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `../ARCHITECTURE.md` (the benchmark rig is sec.2.5). Planned work: `../PERF_LEDGER.md` for -a performance followup, `../followup_general.md` for everything else. +doc: `../ARCHITECTURE_MEASUREMENT.md` (the benchmark rig is sec.2.5). Planned work: +`../PERF_LEDGER.md` for a performance followup, `../followup_general.md` for everything else. **A diff that adds or changes an instrument whose timed body runs a forward pass through a pipeline the model runtime selected also calls `tune_gate()` @@ -39,6 +39,10 @@ is one whose arms attribute cost across stages instead of selecting between two implementations. The same header line also names what its arms attribute. Without that text a reader takes the sweep's arms for an adoption decision it never made. +**A new instrument that times a served turn is a defect - add a board cell to +`../performance/gen_bench_records.das`, or a `lcpp_bench.das` cell with its own +`../PROFILE.md` section, instead.** A served turn is a whole prefill-plus-decode run. A +second instrument's numbers cannot be compared to any row the board already carries. **An out-of-process observer never measures what the benchmark process can measure about itself - that measurement goes inside the process instead.** An out-of-process observer is a script that measures a benchmark process from outside. diff --git a/modules/dasLLAMA/dasllama/dasllama_env.das b/modules/dasLLAMA/dasllama/dasllama_env.das index a57bc2aa3f..11868a0e7e 100644 --- a/modules/dasLLAMA/dasllama/dasllama_env.das +++ b/modules/dasLLAMA/dasllama/dasllama_env.das @@ -225,10 +225,10 @@ struct public MetalEnv { @clarg_doc = "Ceiling in MiB for the device-side KV mirror, clamped 64..4096." kv_mirror_mb : int = 4096 - @clarg_doc = "Pin the served working set (weight regions, planes, the recycled pool buffers) in an MTLResidencySet, macOS 15+: the per-commit residency pass stays a no-op, so the first submission after a CPU-only window stops repaying it (~15 ms/encode on the M1 Max released-rig image cell; ARCHITECTURE.md 2.12); 0 is the A/B rail." + @clarg_doc = "Pin the served working set (weight regions, planes, the recycled pool buffers) in an MTLResidencySet, macOS 15+: the per-commit residency pass stays a no-op, so the first submission after a CPU-only window stops repaying it (~15 ms/encode on the M1 Max released-rig image cell; ARCHITECTURE_RUNTIME.md 2.12); 0 is the A/B rail." residency : bool = true - @clarg_doc = "Seconds the residency-set heartbeat keeps re-requesting residency after the last served step (a requested set still un-wires over a long CPU-only window; ARCHITECTURE.md 2.12); 0 disables the heartbeat - the A/B rail." + @clarg_doc = "Seconds the residency-set heartbeat keeps re-requesting residency after the last served step (a requested set still un-wires over a long CPU-only window; ARCHITECTURE_RUNTIME.md 2.12); 0 disables the heartbeat - the A/B rail." heartbeat_s : int = 180 @clarg_doc = "Barrier at every dispatch instead of at tracker-detected hazards (correctness bisect)." diff --git a/modules/dasLLAMA/dasllama/dasllama_gemma3v.das b/modules/dasLLAMA/dasllama/dasllama_gemma3v.das index 0c4ce69ce5..d249f34f0c 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gemma3v.das +++ b/modules/dasLLAMA/dasllama/dasllama_gemma3v.das @@ -462,7 +462,7 @@ def stage_gemma3v_tower(path : string) : Gemma3vStaging { let swapped = !empty(m.tensors[dn].dims) && m.tensors[dn].dims[0] == d let names <- g3v_gemm_names(swapped) // this carrier serves ONE f32 plane (the widen is exact for these types, and ledgered in - // ARCHITECTURE.md); any other on-disk type is refused by name, never dequantized silently + // ARCHITECTURE_MEDIA.md sec.1.7b); any other on-disk type is refused by name, never dequantized silently for (l in range64(st.t.n_layer)) { for (g in range64(GEMMA3V_GEMMS)) { let name = "v.blk.{l}.{names[g]}.weight" diff --git a/modules/dasLLAMA/dasllama/dasllama_image.das b/modules/dasLLAMA/dasllama/dasllama_image.das index e409369653..1a7a5877f0 100644 --- a/modules/dasLLAMA/dasllama/dasllama_image.das +++ b/modules/dasLLAMA/dasllama/dasllama_image.das @@ -7,24 +7,24 @@ options _cyclomatic_complexity = 0 // image walk/GC bodies mirror the archive options _function_length = 0 // same require dasllama/dasllama_lint public -require dasllama/dasllama_ple // ple_check_table — the E-series table go-live tripwire +require dasllama/dasllama_ple require daslib/fio -require dasllama/dasllama_env // the g_env_* knob globals ([EnvConfig] declarations; ENVIRONMENT.md is generated from them) +require dasllama/dasllama_env require daslib/apply require strings -require daslib/strings_boost // split — the identity-lane parser -require dasllama/dasllama_tokenizer // Tokenizer — the bulk tokenizer serializer below +require daslib/strings_boost +require dasllama/dasllama_tokenizer require daslib/archive require daslib/array_boost -require dasllama/dasllama_plane // PlaneF/PlaneI8 + plane_bind — the borrowed-plane bind arm below +require dasllama/dasllama_plane require dasllama/dasllama_common -require dasllama/dasllama_gguf // GGUFMeta — the streamed cold build's block signature names it -require dasllama/dasllama_layout // convert_model_to_metal_blob — the blob-flavor image transform -require dasllama/dasllama_gpu_resident // the flavor bake this file drives (vulkan_bake_*) + the residency upload each load arms -require dasllama/dasllama_math // active_kernel_backend — half the image identity -require dasllama/dasllama_config // DlimConfiguration + dlim_identity — the identity formatter +require dasllama/dasllama_gguf +require dasllama/dasllama_layout +require dasllama/dasllama_gpu_resident +require dasllama/dasllama_math +require dasllama/dasllama_config require math -require dasllama/dasllama_load // the GGUF load walk (load_gguf/load_model_/load_checkpoint) +require dasllama/dasllama_load // The prepared-model image (spec: fmap-model-image plan, locked 2026-07-16): the post-load // Model dumped once — planes POST-repack, page-aligned — then mapped back with zero O(model) @@ -38,38 +38,29 @@ require dasllama/dasllama_load // the GGUF load walk (load_gguf/load_model_/lo // WhisperModel.enc) contribute their planes under dotted names ("enc.fblob"); string-array // fields ride the meta blob via serialize_strings — raw string pointers can't be planes. -// Bump on ANY change to the image's layout/serialization OR to WHAT THE LOADER PUTS IN IT — the -// identity below is computed before the gguf is parsed, so it can carry box/knob state but nothing -// config-derived. A loader change that alters which tensors load leaves a structurally VALID stale -// image that silently represents a different model; only this version catches it. Mismatch -// regenerates from the gguf, never migrates. -let IMAGE_VERSION = 13 // v13: qwen3v serves f32+ff_pad exact planes, mergers/taps f32 too. BUMP ON ANY SERVED-LAYOUT CHANGE: the identity hashes knobs+version+tag, never offset values, so a layout change without a bump maps stale images (three strikes) +let IMAGE_VERSION = 13 //! The metal (blob-only) flavor's identity tag: q8 planes ride the 34B block_q8_0 blob and the //! kq scale planes their GPU forms (convert_model_to_metal_blob) — flavors are per-config and //! self-sufficient, so the tag keeps the two flavors' files and identities apart. let METAL_IMAGE_TAG = "metal" -let IMAGE_PAGE = 16384l // Apple-Silicon page: the bytesNoCopy alignment contract -let IMAGE_HEADER_BYTES = 64l // magic..identity, inside the first page +let IMAGE_PAGE = 16384l +let IMAGE_HEADER_BYTES = 64l // magic..layout fingerprint, inside page 0 let IMAGE_MAGIC = 0x314D4C44u // 'DLM1' // public only for the save_image/load_image generics — they instantiate from caller modules struct ImgSection { name : string - off : uint64 // absolute byte offset, page-aligned + off : uint64 // absolute byte offset bytes : uint64 } -// the box + knob identity an image is valid for; folded into filename AND header. A PURE -// formatter of DlimConfiguration — the backend-select side effect lives in the config's cpu -// source, so callers need no ordering ritual. `tag` distinguishes kinds sharing one source path. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-identity-backend-order")] def image_identity(tag : string = ""; quant : string = "q8") : string { return dlim_identity(dlim_config_current(quant), IMAGE_VERSION, tag) } -// the image path for (gguf, current identity) — knob combos coexist as separate files -// (the full 64-bit hash keeps that deterministic in practice; the header check would catch -// a collision loudly, but two identities fighting over one path re-save forever) +// a 64-bit identity hash per file: a collision would leave two identities re-saving over one path forever def image_path_for(gguf_path : string; tag : string = ""; quant : string = "q8") : string { return "{gguf_path}.{hash(image_identity(tag, quant))}.dlim" } @@ -85,6 +76,7 @@ struct DlimPeek { config_json : string } +[arch(at = "../ARCHITECTURE_IMAGE.md#dlim-meta-head")] def image_peek(path : string) : DlimPeek { var p = DlimPeek() var msize = 0ul @@ -152,8 +144,6 @@ struct DlimImageInfo { identity : string } -// family flavor tags (whisper-q8, tower-f32, parakeet-q8, …) — each family module's [init] -// registers its own, so inventory/GC verdicts recognize those images as CURRENT var private g_family_tags : array var private g_image_frozen = false @@ -208,16 +198,13 @@ def dlim_identity_lane(identity : string) : string { return "{quant}#{tag}" } -// FOREIGN = this process cannot recompute the identity, so it may not judge it: GPU flavors -// (their bake context pins its own backend + winners) and family tags whose module is not -// loaded here. Only a lane this process can actually produce may be judged stale. def private dlim_lane_foreign(lane : string) : bool { let tag = slice(lane, find(lane, "#") + 1, length(lane)) if (empty(tag)) { - return false // the planar lane: every process produces it + return false } if (tag == METAL_IMAGE_TAG || (tag |> starts_with("vulkan"))) { - return true // pinned by its own bake context + return true } return !has_value(g_family_tags, tag) } @@ -293,7 +280,7 @@ def dlim_clean_(gguf_path : string; apply : bool; keep_other : bool = false) : t for (im in inv) { r.total++ if (im.verdict == "FOREIGN") { - foreign_kept++ // another bake context's lane — never judged from here + foreign_kept++ continue } continue if (im.verdict == "CURRENT" || (keep_other && im.verdict == "OTHER")) @@ -321,8 +308,7 @@ def dlim_wipe(gguf_path : string) : tuple { return r } let dirp = st.is_dir ? gguf_path : dir_name(gguf_path) - // image names are "..dlim" — requiring the dot keeps the reach exact - // (wiping "foo.gguf" must not take "foo.gguf2"'s images) + // the trailing dot keeps the reach exact: wiping "foo.gguf" must not take "foo.gguf2"'s images let base = st.is_dir ? "" : "{base_name(gguf_path)}." dir(dirp) $(name : string) { return if (!(name |> ends_with(".dlim")) || (base != "" && !(name |> starts_with(base)))) @@ -336,38 +322,38 @@ def dlim_wipe(gguf_path : string) : tuple { return r } +[arch(at = "../ARCHITECTURE_IMAGE.md#image-page-alignment")] def pad_to_page(cur : uint64) : uint64 { let page = uint64(IMAGE_PAGE) return (cur + page - 1ul) / page * page } -let private IMAGE_WRITE_BAND = 16l * 1024l * 1024l // the direct writer's staging band -let private SECTION_TABLE_SLACK = 65536ul // preallocation headroom for the section blob +let private DWRITE_STAGING_BAND = 16l * 1024l * 1024l +let private SECTION_TABLE_SLACK = 65536ul //! The mint sink: append-only and strictly ascending (what keeps NTFS from charging //! valid-data-length zero-fill on the preallocated file), one walk over two backings (memory //! chunk / direct file writer). Public so family streamers append planes from their hooks. struct ImgWriter { - h : void? = null // the direct file writer; null on the memory rail - mem : void? = null // the page-aligned chunk; null on the file rail - cap : uint64 = 0ul // the chunk's usable extent — an append past it is a sizing bug + dwrite_h : void? = null + chunk_base : void? = null + chunk_cap : uint64 = 0ul cur : uint64 = 0ul ok : bool = true } def private w_open(path : string; total : uint64) : ImgWriter { - let h = unsafe(dwrite_open(path, total, uint64(IMAGE_WRITE_BAND))) - return ImgWriter(h = h, ok = h != null) + let h = unsafe(dwrite_open(path, total, uint64(DWRITE_STAGING_BAND))) + return ImgWriter(dwrite_h = h, ok = h != null) } -// the memory sink. `total` must be image_total_bytes' answer — the chunk is exact, so an overrun -// means the sizing pass and the walk disagree, which is a bug and not a disk condition +[arch(at = "../ARCHITECTURE_IMAGE.md#image-sizing-exactness")] def private w_open_mem(total : uint64) : ImgWriter { let base = image_chunk_alloc(total, uint64(IMAGE_PAGE)) if (base == null) { to_log(LOG_WARNING, "dasLLAMA image: cannot allocate a {int64(total) >> 20l} MB image chunk\n") } - return ImgWriter(mem = base, cap = total, ok = base != null) + return ImgWriter(chunk_base = base, chunk_cap = total, ok = base != null) } def private mem_at(base : void?; off : uint64) : void? { @@ -379,16 +365,16 @@ def w_append(var w : ImgWriter; p : void?; nbytes : uint64) { if (!w.ok || nbytes == 0ul) { return } - if (w.mem != null) { - if (nbytes > w.cap - w.cur) { - to_log(LOG_ERROR, "dasLLAMA image: chunk overrun at {int64(w.cur)} + {int64(nbytes)} > {int64(w.cap)} - image sizing is wrong\n") + if (w.chunk_base != null) { + if (nbytes > w.chunk_cap - w.cur) { + to_log(LOG_ERROR, "dasLLAMA image: chunk overrun at {int64(w.cur)} + {int64(nbytes)} > {int64(w.chunk_cap)} - image sizing is wrong\n") w.ok = false return } unsafe { - memcpy(mem_at(w.mem, w.cur), p, nbytes) + memcpy(mem_at(w.chunk_base, w.cur), p, nbytes) } - } elif (!unsafe(dwrite_append(w.h, p, nbytes))) { + } elif (!unsafe(dwrite_append(w.dwrite_h, p, nbytes))) { w.ok = false // a dead save stays dead — don't keep pushing bytes at a full disk return } @@ -409,24 +395,22 @@ def w_zeros(var w : ImgWriter; n : uint64) { } def private w_close(var w : ImgWriter) : bool { - if (w.mem != null) { + if (w.chunk_base != null) { return w.ok // nothing to flush: the chunk IS the image } - if (w.h == null) { + if (w.dwrite_h == null) { return false } - let flushed = unsafe(dwrite_close(w.h)) - w.h = null + let flushed = unsafe(dwrite_close(w.dwrite_h)) + w.dwrite_h = null return flushed && w.ok } -// Page 0, which only the finished walk knows (section count, meta offset and size). The memory -// rail stores it in place; the file rail's direct writer is append-only, so it patches through an -// ordinary handle after close — which is why `path` is a parameter and not w's business. +// the direct writer is append-only, so page 0 is patched through an ordinary handle after close - hence the `path` parameter def private w_header(var w : ImgWriter; path : string; var hdr : array) : bool { - if (w.mem != null) { + if (w.chunk_base != null) { unsafe { - memcpy(w.mem, addr(hdr[0]), uint64(IMAGE_HEADER_BYTES)) + memcpy(w.chunk_base, addr(hdr[0]), uint64(IMAGE_HEADER_BYTES)) } return true } @@ -447,9 +431,7 @@ def private plane_end(cur : uint64; field : array) : uint64 { return pad_to_page(cur) + uint64(long_length(field)) * uint64(typeinfo sizeof(field[0])) } -// The image's final size, known before a byte goes out: planes contribute their sizes, the meta -// blob is already serialized. The writer preallocates from this, so the file lands as one -// contiguous run. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-sizing-exactness")] def private image_total_bytes(var t; meta_bytes : uint64) : uint64 { var cur = uint64(IMAGE_PAGE) apply(t) $ [unused_argument(name)] (name : string; var field) { @@ -507,8 +489,7 @@ def private read_u64(p : uint8 const?; at : int64) : uint64 { -// bulk POD-array serialize: ONE stream call for the payload. daslib/archive's per-element -// generic dispatch measured ~340us/element — a 128k-vocab tokenizer took 238s through it. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-bulk-serialize")] def private serialize_pod_array(var arch : Archive; var a : array) { var n = length(a) arch |> serialize_raw(n) @@ -531,6 +512,7 @@ def private serialize_pod_array(var arch : Archive; var a : array) { //! String arrays as [lengths][one blob] — two bulk payloads instead of per-string dispatch. //! Public because string-array fields cannot be raw planes (they hold pointers) — a type's //! serialize_image_meta must route them through this instead. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-bulk-serialize")] def serialize_strings(var arch : Archive; var a : array) { var lens : array var blob : array @@ -577,9 +559,7 @@ def serialize_strings(var arch : Archive; var a : array) { delete blob } -// the tokenizer, bulk: vocab/scores/token_type/merges as flat payloads; the lookup tables are -// DERIVED (piece -> index, merge -> rank-order index) and rebuild in milliseconds at load — -// exactly the "parse at load" carve-out the format spec allows +// derived state (piece -> index, merge -> rank) is never stored: it rebuilds in milliseconds at load def private serialize_tokenizer(var arch : Archive; var tk : Tokenizer) { arch |> serialize_raw(tk.kind) serialize_strings(arch, tk.spm.vocab) @@ -613,7 +593,7 @@ def private serialize_tokenizer(var arch : Archive; var tk : Tokenizer) { for (i, v in count(), merges) { tk.bpe.merge_rank[v] = int64(i) } - build_spm_derived(tk.spm) // the pair-merge engines are derived state, same as the lookups + build_spm_derived(tk.spm) build_bpe_derived(tk.bpe) } delete merges @@ -623,7 +603,7 @@ def private serialize_tokenizer(var arch : Archive; var tk : Tokenizer) { //! tripwire so the list grows with the struct (blocks re-binds in image_post_load instead). //! Public only because build/parse resolve `_::` at the instantiation ROOT — not a call-me API. def serialize_image_meta(var arch : Archive; var t : Model) { - verify(count_meta_fields(t) == IMAGE_META_FIELDS) // grew Model? extend this list + verify(count_meta_fields(t) == IMAGE_META_FIELDS) arch |> serialize(t.config) arch |> serialize(t.arch) arch |> serialize(t.chat_template_str) @@ -692,8 +672,7 @@ def serialize_image_meta(var arch : Archive; var t : Model) { arch |> serialize_raw(t.mtp_headnorm_off) } -// the tripwire: serialize_image_meta(Model) covers 66 fields; blocks + image_map + -// image_bytes are deliberate skips. A new non-array Model field trips this until BOTH agree. +// 66 serialized fields + 3 deliberate skips (blocks, image_map, image_bytes) let IMAGE_META_FIELDS = 66 + 3 //! Count of meta-carried fields of any struct: non-arrays plus string arrays (those cannot @@ -751,11 +730,7 @@ def image_post_load(var t : Model) { rebind_arch_blocks(t, t.arch) } -// ===== save ===== - -// one plane: pad to a page, bulk-append, account. The writer advances w.cur by exactly what it -// accepted, so the section table cannot drift from the file. long_ forms: a plane past 2 GiB -// (voxtral-mini qblob = 3.6 G elements) does not fit the int32 rail. +// long_ forms: a plane past 2 GiB (voxtral-mini qblob = 3.6 G elements) does not fit the int32 rail def private write_plane(var w : ImgWriter; var sections : array; name : string; var field : array) { if (empty(field) || !w.ok) { @@ -768,8 +743,7 @@ def private write_plane(var w : ImgWriter; var sections : array; unsafe { w_append(w, addr(field[0]), nbytes) } - // big planes get their own line: a uniform rate means the cost is systemic, a few slow ones - // mean stalls. Only >=256 MB, so small models stay quiet and this needs no env knob. + // >=256 MB only: a per-plane rate separates systemic cost from stalls, and small models stay quiet if (nbytes >= 256ul * 1024ul * 1024ul) { let ms = int64(get_time_usec(ts)) / 1000l let mb = int64(nbytes) >> 20l @@ -785,6 +759,8 @@ def private write_plane(var w : ImgWriter; var sections : array; //! THE image build — every prepared image in the process comes out of this walk, whichever sink it //! lands in: planes page-aligned raw (borrowed back at load), everything else archived. `plane` //! gets first refusal on each plane by name; `consume` frees each owned plane as it lands. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-page-alignment"), arch(at = "../ARCHITECTURE_IMAGE.md#dlim-meta-head"), + arch(at = "../ARCHITECTURE_IMAGE.md#image-sizing-exactness")] def private build_image(var t; dst : string; in_mem : bool; extra : uint64; tag : string; quant : string; consume : bool; layout : uint64; plane : block<(name : string; var w : ImgWriter; var sections : array) : bool>) : ImgWriter { @@ -798,8 +774,6 @@ def private build_image(var t; dst : string; in_mem : bool; extra : uint64; var meta <- new MemSerializer() var warch = Archive(reading = false, stream = meta) var sections : array - // meta first: it needs nothing from the planes, and having it sized up front is what lets - // the sink preallocate exactly (see image_total_bytes) let ts_meta = ref_time_ticks() _::serialize_image_meta(warch, t) to_log(LOG_INFO, "dasLLAMA image: meta serialize {get_time_usec(ts_meta) / 1000}ms\n") @@ -824,8 +798,7 @@ def private build_image(var t; dst : string; in_mem : bool; extra : uint64; } } } static_elif (typeinfo safe_has_field < image_planes > (field)) { - // a staging plane-group: the nested carrier's bytes under dotted names, with - // the hook getting the same first refusal the top level gives it + // a staging plane-group: the hook gets the same first refusal here as at the top level apply(field) $(iname : string; var ifield) { static_if (typeinfo is_array(ifield)) { if (!invoke(plane, "{name}.{iname}", w, sections)) { @@ -837,9 +810,6 @@ def private build_image(var t; dst : string; in_mem : bool; extra : uint64; } } } static_elif (typeinfo safe_has_field < image_map > (field)) { - // a nested weight-carrier on the image rail (marker: it declares image_map) — - // its planes join this image under dotted names; scalars ride the meta blob - // via the outer type's serialize_image_meta (e.g. WhisperModel.enc) apply(field) $(iname : string; var ifield) { static_if (typeinfo is_array(ifield)) { static_if (!typeinfo is_string(ifield[0])) { @@ -855,18 +825,14 @@ def private build_image(var t; dst : string; in_mem : bool; extra : uint64; let planes_ms = int64(get_time_usec(ts_planes)) / 1000l let planes_mb = int64(w.cur) >> 20l to_log(LOG_INFO, "dasLLAMA image: planes write {planes_ms}ms ({planes_mb} MB at {planes_ms > 0l ? planes_mb * 1000l / planes_ms : 0l} MB/s)\n") - if (w.h != null) { - // the file writer's own split — staging our bytes vs waiting on the device. Anything - // the two do not account for is time spent OUTSIDE the writer (the walk itself). - let copy_ms = int64(unsafe(dwrite_stat(w.h, 0))) / 1000000l - let write_ms = int64(unsafe(dwrite_stat(w.h, 1))) / 1000000l - let direct_mb = int64(unsafe(dwrite_stat(w.h, 2))) >> 20l - let bounce_mb = int64(unsafe(dwrite_stat(w.h, 3))) >> 20l + if (w.dwrite_h != null) { + // stage vs syscall vs unaccounted: what neither accounts for was spent outside the writer + let copy_ms = int64(unsafe(dwrite_stat(w.dwrite_h, 0))) / 1000000l + let write_ms = int64(unsafe(dwrite_stat(w.dwrite_h, 1))) / 1000000l + let direct_mb = int64(unsafe(dwrite_stat(w.dwrite_h, 2))) >> 20l + let bounce_mb = int64(unsafe(dwrite_stat(w.dwrite_h, 3))) >> 20l to_log(LOG_INFO, "dasLLAMA image: writer split: stage {copy_ms}ms ({bounce_mb} MB bounced), syscall {write_ms}ms ({direct_mb} MB direct), unaccounted {planes_ms - copy_ms - write_ms}ms\n") } - // meta = [identity string][config JSON][sections][the walk's scalar stream], one blob at - // the tail — the two strings lead so a mismatched load can print WHAT the image was - // baked for without parsing anything else var smeta <- new MemSerializer() var sarch = Archive(reading = false, stream = smeta) var m_ident = ident @@ -884,7 +850,6 @@ def private build_image(var t; dst : string; in_mem : bool; extra : uint64; w_append(w, addr(meta_blob[0]), meta_bytes) } } - // pad the image itself to a page multiple (the whole-image bytesNoCopy wrap needs it) fend = pad_to_page(w.cur) w_zeros(w, fend - w.cur) delete meta_blob @@ -917,7 +882,7 @@ def private build_image(var t; dst : string; in_mem : bool; extra : uint64; return w } -// tmp+rename: atomic on POSIX; on Windows a brief missing-file window costs a reader a regenerate +[arch(at = "../ARCHITECTURE_IMAGE.md#image-publish")] def private promote_tmp(tmp_path : string; path : string; built : bool; want_bytes : uint64) : bool { var ok = built if (ok) { @@ -936,9 +901,8 @@ def private promote_tmp(tmp_path : string; path : string; built : bool; want_byt return false } if (rename(tmp_path, path)) { - return true // the atomic replace (POSIX rename-over) + return true } - // rename-over-existing is not portable (Windows) — drop the stale image and retry remove(path) if (!rename(tmp_path, path)) { if (!remove(tmp_path)) { @@ -950,6 +914,7 @@ def private promote_tmp(tmp_path : string; path : string; built : bool; want_byt } // per-writer name: concurrent savers must not clobber each other; last rename wins, same content +[arch(at = "../ARCHITECTURE_IMAGE.md#image-publish")] def private tmp_name_for(path : string) : string { return "{path}.{ref_time_ticks()}.tmp" // clock: control — unique temp-name suffix, not timing } @@ -978,7 +943,6 @@ def private write_chunk(base : void?; bytes : uint64; path : string) : bool { //! Serialize a loaded weight-carrying struct as a prepared image on disk (see build_image). def save_image(var t; path : string; tag : string = ""; quant : string = "q8") : bool { - // no plane wants special handling — the walk writes them all from RAM return save_image_to_file(t, path, 0ul, tag, quant, layout_fingerprint(t)) $(name : string; var w : ImgWriter; var sections : array) { return false } @@ -990,9 +954,7 @@ def save_model_image(var t : Model; path : string; tag : string = ""; quant : st return save_image(t, path, tag, quant) } -// preallocation for a streamed build: the generic total counts streamed fields as empty (they -// are), so add each streamed plane's exact bytes + a page of padding, plus the pending-wscale -// qscales16 (absent until the epilogue). Over-reserve is free — dwrite_close truncates. +// streamed planes size as empty, so their bytes plus a page each are added here; over-reserve is free (dwrite_close truncates) def private stream_extra_bytes(t : Model; jobs : array) : uint64 { var extra = 0ul for (fname in ["qblob", "mblob", "k4q", "k5q", "k6q", "q40q", "q51q"]) { @@ -1018,8 +980,7 @@ def save_model_image_streaming(var t : Model; m : GGUFMeta; bytes : array var w : ImgWriter; var sections : array) { let want = stream_plane_bytes(t, jobs, name) if (want <= 0l) { - // a blob build's kq scale strips take their device form as each one's turn comes — - // its own fills are done by then and its bytes are not yet written + // a blob build's kq scale strips take their device form as each one's turn comes, before its bytes are written if (t.metal_blob) { blob_stream_scale_plane(t, name) } @@ -1044,9 +1005,7 @@ def save_model_image_streaming(var t : Model; m : GGUFMeta; bytes : array return ok } -// ===== load ===== - -// one plane: validate the section against the mapping, then borrow it in place +[arch(at = "../ARCHITECTURE_IMAGE.md#image-page-alignment")] def private borrow_plane(bp : uint8 const?; msize : uint64; path : string; by_name : table; var ok : bool&; name : string; var field : array) { @@ -1055,7 +1014,6 @@ def private borrow_plane(bp : uint8 const?; msize : uint64; path : string; } let sec = by_name?[name] ?? ImgSection() let esz = uint64(typeinfo sizeof(field[0])) - // bytes-then-offset ordering keeps the bounds math wrap-free (off + bytes could overflow) if (sec.off % uint64(IMAGE_PAGE) != 0ul || sec.bytes > msize || sec.off > msize - sec.bytes || sec.bytes % esz != 0ul) { to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - section '{name}' at {sec.off}+{sec.bytes} (esz {esz}) is malformed\n") ok = false @@ -1066,8 +1024,6 @@ def private borrow_plane(bp : uint8 const?; msize : uint64; path : string; } } -// one borrowed plane: validate the section against the mapping, then point the plane at it. -// The plane owns nothing, so a decline leaves it empty and the caller drops the whole carrier. def private bind_plane_section(bp : uint8 const?; msize : uint64; path : string; by_name : table; var ok : bool&; name : string; var pl) { @@ -1075,7 +1031,6 @@ def private bind_plane_section(bp : uint8 const?; msize : uint64; path : string; return } let sec = by_name?[name] ?? ImgSection() - // bytes-then-offset ordering keeps the bounds math wrap-free (off + bytes could overflow) if (sec.off % uint64(IMAGE_PAGE) != 0ul || sec.bytes > msize || sec.off > msize - sec.bytes) { to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - section '{name}' at {sec.off}+{sec.bytes} is malformed\n") ok = false @@ -1090,16 +1045,14 @@ def private bind_plane_section(bp : uint8 const?; msize : uint64; path : string; //! THE image parse — the only way a prepared image becomes a struct, mapped .dlim or chunk this //! process just built alike: plane fields borrow `base` in place, the archive blob restores //! everything else. false = decline (the caller owns `base` and releases it either way). +[arch(at = "../ARCHITECTURE_IMAGE.md#dlim-meta-head"), arch(at = "../ARCHITECTURE_RUNTIME.md#weights-epoch-on-load"), + arch(at = "../ARCHITECTURE_IMAGE.md#image-identity-backend-order")] def private parse_image(base : void?; msize : uint64; path : string; var t; tag : string; quant : string) : bool { - // the image's planes are packed for the load backend — activate it before any kernel runs - // (same select the gguf loader does; the identity check below then names that backend) select_matmul_backend_for_load_() var bp = unsafe(reinterpret(base)) if (read_u32(bp, 0l) != IMAGE_MAGIC || read_u32(bp, 4l) != uint(IMAGE_VERSION) || read_u32(bp, 8l) != uint(IMAGE_PAGE)) { - // every decline is loud (short of a missing file) — a silently regenerating cache is - // undebuggable. A wrong version means the meta layout is unknowable, so this arm - // cannot read the baked identity string the identity-mismatch arm below prints. + // a version mismatch makes the meta layout unknowable, so this arm names raw fields, not the baked identity to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - size {msize} magic {read_u32(bp, 0l)} version {read_u32(bp, 4l)} (want {IMAGE_VERSION}) page {read_u32(bp, 8l)}\n") return false } @@ -1107,9 +1060,7 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag let meta_off = read_u64(bp, 16l) let meta_bytes = read_u64(bp, 24l) let file_bytes = read_u64(bp, 32l) - // bytes-first ordering keeps meta_off + meta_bytes wrap-free; the INT_MAX cap bounds the - // int() casts the copy below takes (real meta blobs are a few MB, never empty — the - // section table always rides there, and addr(meta_blob[0]) needs a nonzero resize) + // the INT_MAX cap bounds the int() casts below; addr(meta_blob[0]) needs a nonzero resize if (file_bytes != msize || meta_bytes == 0ul || meta_bytes > msize || meta_off > msize - meta_bytes || meta_bytes > uint64(INT_MAX)) { to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - header says {file_bytes} bytes / meta {meta_off}+{meta_bytes}, mapped {msize}\n") return false @@ -1120,8 +1071,6 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag } let want_ident = image_identity(tag, quant) if (read_u64(bp, 40l) != hash(want_ident)) { - // same version ⇒ the meta head is readable: say WHAT the image was baked for, not - // just that a 64-bit hash missed (bounds were validated above) var head : array head |> resize(int(min(meta_bytes, 65536ul))) unsafe { @@ -1138,7 +1087,6 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag } return false } - // the meta blob is the ONE copied+parsed piece (small); planes stay lazy in the mapping var meta_blob : array meta_blob |> reserve(int(meta_bytes)) meta_blob |> resize(int(meta_bytes)) @@ -1153,9 +1101,7 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag rarch |> serialize(baked_cfg) var sections : array rarch |> serialize(sections) - // the header count must agree with the meta blob's own section list — a disagreement - // means header/meta corruption, and silently-absent sections would leave planes EMPTY - // (indistinguishable from legitimately-empty ones) + // silently-absent sections would leave planes EMPTY, indistinguishable from legitimately-empty ones if (length(sections) != nsections) { to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - header says {nsections} sections, meta carries {length(sections)}\n") delete sections @@ -1166,8 +1112,7 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag return false } var by_name <- {for (sec in sections); sec.name => sec} - // duplicate section names would silently overwrite table entries and borrow the wrong - // plane — a valid save never produces them, so a collision is corruption + // duplicate names would overwrite table entries and borrow the wrong plane if (length(by_name) != length(sections)) { to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - {length(sections)} sections carry duplicate names\n") delete by_name @@ -1179,9 +1124,7 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag return false } _::serialize_image_meta(rarch, t) - // a truncated/corrupt meta blob reads back as silent zero-progress (MemSerializer sets - // lastError, no panic) — decline BEFORE post_load, which would panic on garbage (e.g. an - // unreadable arch name), and before any plane borrows + // MemSerializer fails silently (lastError, no panic), so decline here - before post_load panics on garbage and before any plane borrows if (!meta->OK()) { to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - meta blob is truncated/corrupt ({meta->getLastError()})\n") delete t // half-deserialized, all owned — nothing is borrowed yet @@ -1203,8 +1146,7 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag } static_elif (typeinfo safe_has_field < borrowed_plane > (field)) { bind_plane_section(bp, msize, path, by_name, ok, name, field) } static_elif (typeinfo safe_has_field < image_map > (field)) { - // nested weight-carrier: borrow its planes back from the dotted sections (the - // nested image_map stays null — the OUTER struct owns this mapping) + // the nested image_map stays null: the OUTER struct owns this mapping apply(field) $(iname : string; var ifield) { static_if (typeinfo is_array(ifield)) { static_if (!typeinfo is_string(ifield[0])) { @@ -1223,16 +1165,12 @@ def private parse_image(base : void?; msize : uint64; path : string; var t; tag delete meta } if (!ok) { - // a malformed section table: forget whatever was borrowed (the finalizer) and leave the - // backing to the caller — image_map is not set, so the finalizer would not release it + // image_map is not set yet, so the finalizer forgets the borrows and leaves the backing to the caller delete t return false } t.image_map = base t.image_bytes = msize - // address-keyed weight caches (the Metal drivers' resident regions) flush on the weights - // epoch; this image may sit at a deleted model's recycled addresses, so it counts as a - // load exactly like load_gguf's own bump does bump_weights_epoch() return true } @@ -1250,8 +1188,7 @@ def load_image(path : string; var t; tag : string = ""; quant : string = "q8") : base = fmap_open(path, addr(msize)) } if (base == null || int64(msize) < IMAGE_HEADER_BYTES) { - // fmap_open declines with null (vanished file, mmap failure, >size_t on 32-bit) — - // treat it like any other decline; the header reads need a real mapping + // fmap_open declines with null (vanished file, mmap failure, >size_t on 32-bit) to_log(LOG_WARNING, "dasLLAMA image: '{path}' declined - cannot map ({msize} bytes)\n") if (base != null) { unsafe(fmap_close(base, msize)) @@ -1265,8 +1202,7 @@ def load_image(path : string; var t; tag : string = ""; quant : string = "q8") : return false } -// Parse an image chunk this process just built. On a decline the chunk is freed; on success the -// carrier owns it and its finalizer releases it exactly like a mapping. +// on a decline the chunk is freed; on success the carrier's finalizer releases it like a mapping def private adopt_image(base : void?; msize : uint64; src : string; var t; tag : string; quant : string) : bool { if (parse_image(base, msize, src, t, tag, quant)) { return true @@ -1275,28 +1211,23 @@ def private adopt_image(base : void?; msize : uint64; src : string; var t; tag : return false } -// The chunk rail's body: build into a chunk, tee it to `save_to`, adopt it into `out`. Both cold -// entries below are this with a different plane hook. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-sizing-exactness")] def private chunk_rail(var t; var out; save_to : string; extra : uint64; tag : string; quant : string; plane : block<(name : string; var w : ImgWriter; var sections : array) : bool>) : bool { let ts = ref_time_ticks() var w = build_image(t, "", true, extra, tag, quant, true, layout_fingerprint(out), plane) if (!w.ok) { - // failing to get the chunk is the only decline that precedes the walk — `t` is whole and - // the caller can still serve it. Failing after it means the sizing pass and the walk - // disagree, which is a bug in this file and not a condition to limp on from. - if (w.mem == null) { + if (w.chunk_base == null) { return false } - image_chunk_free(w.mem) + image_chunk_free(w.chunk_base) panic("dasLLAMA: the image build failed after it began consuming the model - see the error above") } to_log(LOG_INFO, "dasLLAMA image: built in memory in {get_time_usec(ts) / 1000} ms ({int64(w.cur) >> 20l} MB)\n") - // the save is a courtesy to the NEXT run — a full disk must not cost this one its model - if (save_to != "" && !write_chunk(w.mem, w.cur, save_to)) { + if (save_to != "" && !write_chunk(w.chunk_base, w.cur, save_to)) { to_log(LOG_WARNING, "dasLLAMA image: could not write '{save_to}' - this run still serves the image it built\n") } - if (!adopt_image(w.mem, w.cur, save_to != "" ? save_to : "", out, tag, quant)) { + if (!adopt_image(w.chunk_base, w.cur, save_to != "" ? save_to : "", out, tag, quant)) { panic("dasLLAMA: the image this process just built would not parse back - see the decline above") } return true @@ -1346,9 +1277,6 @@ def image_from_model_streaming(var t : Model; var out : Model; m : GGUFMeta; byt return ok } -// The vulkan flavor is a SECOND artifact baked off a model that is already SERVING: the GPU walk -// that just uploaded from its planes also collected the device-layout bytes. Warm map and cold -// stream both land here, which is why neither needs the model in RAM to bake one. def private vulkan_bake_flavor(var t : Model; path : string; vk_tag : string) { if (vk_tag == "" || !vulkan_bake_take(t)) { return @@ -1368,15 +1296,10 @@ def image_save_enabled : bool { return g_env_engine.image_save } -// ===== the metal flavor, derived in the writer ===== - -// ~36 MB of blob per band: big enough that the interleave amortizes and the writer sees bulk -// appends, small enough that the staging cost does not register against the model. +// ~36 MB per band: the interleave amortizes and appends stay bulk, with a staging cost that never registers let private METAL_BAND_BLOCKS = 1048576l -// Write mblob without it ever being a plane: interleave the planar planes band by band into a -// staging buffer the writer consumes before the next band refills it. Runs at QBLOB's position in -// the walk, not mblob's — on the consuming (chunk) rail qblob is freed the moment its field lands. +// band by band: the staging buffer holds one band, never the whole blob def private write_metal_blob_plane(var t : Model; var w : ImgWriter; var sections : array) { let sh = metal_blob_shape(t) if (sh.blocks <= 0l) { @@ -1421,12 +1344,10 @@ def private metal_derive_plane(var t : Model; name : string; var w : ImgWriter; return name == "qscales" || name == "qscales16" || name == "mblob" } -//! The cold half of a cached load, for any carrier: turn the freshly-loaded `m` into its prepared -//! image and put the image-served struct in `out`. With a target path the FILE is the handoff (`m` -//! is dropped between write and map); without one the image is built in memory. `m` is spent. +//! The cold half of a cached load, for any carrier: turn the freshly-loaded `m` into its prepared image and put the image-served struct in `out`. +//! With a target path the FILE is the handoff (`m` is dropped between write and map); without one the image is built in memory. `m` is spent. +//! True only when the FILE was written: only then may the caller reap its lane. def cache_via_image(var m; var out; img : string; tag : string = ""; quant : string = "q8") : bool { - // no plane wants special handling — the walk writes them all from RAM. - // True only when the FILE was written — only then may the caller reap its lane. return cache_via_image_(m, out, img, tag, quant) $(name : string; var w : ImgWriter; var sections : array) { return false } @@ -1451,7 +1372,7 @@ def cache_via_image_staged_hooked(var st; var out; img : string; extra : uint64; let ts = ref_time_ticks() if (save_image_to_file(st, img, extra, tag, quant, layout_fingerprint(out), plane)) { to_log(LOG_INFO, "dasLLAMA: prepared image written in {get_time_usec(ts) / 1000} ms - {img}\n") - delete st // before the map, not after: holding both is what makes a cold load peak twice + delete st if (!load_image(img, out, tag, quant)) { panic("dasLLAMA: '{img}' was just written and verified, and will not map back - see the decline above") } @@ -1465,8 +1386,6 @@ def cache_via_image_staged_hooked(var st; var out; img : string; extra : uint64; return false } -// cache_via_image with the walk's plane hook exposed — the metal flavor derives its blob plane -// from the planar ones here instead of materializing it first (see metal_derive_plane). def private cache_via_image_(var m; var out; img : string; tag : string; quant : string; plane : block<(name : string; var w : ImgWriter; var sections : array) : bool>) : bool { if (img == "" || !image_save_enabled()) { @@ -1482,16 +1401,14 @@ def private cache_via_image_(var m; var out; img : string; tag : string; quant : return false } to_log(LOG_INFO, "dasLLAMA: prepared image written in {get_time_usec(ts) / 1000} ms - {img}\n") - delete m // before the map, not after: holding both is what makes a cold load peak twice + delete m if (!load_image(img, out, tag, quant)) { panic("dasLLAMA: '{img}' was just written and verified, and will not map back - see the decline above") } return true } -// interp-load guard: the gguf path's O(model) das loops run ~10x slower interpreted (tinyllama -// load 53s vs 5.5s jitted; repack itself is native tune kernels, 0ms either way) — a 69GB -// hybrid extrapolates to an hour. Prepared images (the slice path) stay interp-friendly. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-interp-load")] def private guard_interp_gguf_load(path : string) { if (jit_enabled() || is_standalone_exe()) { return @@ -1512,9 +1429,7 @@ def load_model_image(path : string; var t : Model; tag : string = ""; quant : st if (!load_image(path, t, tag, quant)) { return false } - // AFTER the plane borrows (image_post_load runs before them, when every array is still - // empty): the PLE-table tripwire needs the bound planes, and the GPU MoE tier gathers - // from the PREPARED planes, so the mapped rail serves it too + // after the plane borrows: the PLE tripwire and the MoE gather both need bound planes ple_check_table(t, "prepared image '{path}'") moe_gpu_upload_resident(t) return true @@ -1523,24 +1438,21 @@ def load_model_image(path : string; var t : Model; tag : string = ""; quant : st //! The image-cached model load — ``load_model``'s engine half, public for callers bypassing the facade. ``.dlim`` maps directly (wrong identity panics); q8-mode gguf maps/saves a prepared image (``DASLLAMA_IMAGE=0`` disables); an fp32-mode gguf joins the rail only with ``image_fp32`` (planar flavor — the parity-pinned decoders, e.g. canary's). //! Under an active Metal mode the image is the BLOB-ONLY metal flavor (CPU inference against it panics) when servable, the planar CPU flavor otherwise; ``planar_only`` pins the planar CPU flavor regardless of armed GPU tiers (the ASR surface — its decode path reads CPU planes). //! Call inside ``with_job_que()``. +[arch(at = "../ARCHITECTURE_IMAGE.md#image-interp-load"), arch(at = "../ARCHITECTURE_IMAGE.md#image-identity-backend-order")] def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; planar_only : bool = false; image_fp32 : bool = false) : Model { - // vulkan-SPI worlds only: under an active Metal mode the GPU serves through the override - // registries, and "no backend registered" would be a false alarm + // under Metal the GPU serves through the override registries, so "no backend registered" would be a false alarm if (get_metal_mode() == MetalMode.off) { warn_gpu_backend_absent() } let qs = "{mode}" if (path |> ends_with(".dlim")) { - // a prepared image named directly — first-class once the source gguf is gone. EXPECTED - // flavor first (metal mode > armed vulkan tier > planar) so the normal case loads - // without the wrong-tag decline warning; a full mismatch panics + // expected flavor first (metal > armed vulkan > planar): the normal case loads with no wrong-tag decline warning apply_box_profile_runtime() let ts_img = ref_time_ticks() var t = Model() let metal_first = !planar_only && get_metal_mode() != MetalMode.off && has_decode_override("metal") - let vk_cfg = (planar_only || metal_first) ? "" : moe_gpu_bake_tag() - let vk_tag = vk_cfg // dlim_vulkan_tag already carries the "vulkan|" prefix ("" = tier absent) + let vk_tag = (planar_only || metal_first) ? "" : moe_gpu_bake_tag() var loaded = false if (vk_tag != "") { vulkan_bake_slice_begin() @@ -1566,11 +1478,9 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; to_log(LOG_INFO, "dasLLAMA: prepared image mapped in {get_time_usec(ts_img) / 1000} ms - {path}\n") return <- t } - // load-time flavor pick: blob-only metal when its drivers serve the config; the vulkan - // flavor (planar + baked GPU planes) when the vulkan tier arms; planar CPU otherwise let want_metal = !planar_only && mode == QuantMode.q8 && get_metal_mode() != MetalMode.off && has_decode_override("metal") if ((mode == QuantMode.q8 || (mode == QuantMode.fp32 && image_fp32)) && g_env_engine.image) { - apply_box_profile_runtime() // a backend pin must precede the identity (it names the backend) + apply_box_profile_runtime() let ts_img = ref_time_ticks() var t = Model() if (want_metal) { @@ -1580,10 +1490,8 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; return <- t } } - // the vulkan flavor: the GPU walk slices the baked device-layout planes off the mapping - // instead of re-gathering; ANY vulkan config switch changes the tag and rebakes below - let vk_cfg = (planar_only || want_metal || mode != QuantMode.q8) ? "" : moe_gpu_bake_tag() - let vk_tag = vk_cfg // dlim_vulkan_tag already carries the "vulkan|" prefix ("" = tier absent) + // the GPU walk slices baked device-layout planes off the mapping; any vulkan config switch changes the tag and rebakes + let vk_tag = (planar_only || want_metal || mode != QuantMode.q8) ? "" : moe_gpu_bake_tag() if (vk_tag != "") { let img_v = image_path_for(path, vk_tag) vulkan_bake_slice_begin() @@ -1605,8 +1513,7 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; to_log(LOG_INFO, "dasLLAMA: prepared image mapped in {get_time_usec(ts_img) / 1000} ms - {img}\n") return <- t } - // a planar image of a metal-servable config under metal mode — drop the mapping - // and regenerate the blob flavor (the planar file stays for CPU-mode loads) + // planar image, metal-servable config: drop it and rebake the blob flavor (the planar file stays for CPU loads) if (g_image_frozen) { panic("dasLLAMA: '{img}' is planar but this run needs the metal blob flavor, and artifacts are frozen - pre-mint {image_path_for(path, METAL_IMAGE_TAG)}") } @@ -1617,9 +1524,6 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; panic("dasLLAMA: no prepared image for '{path}' ({want}) and artifacts are frozen - pre-mint it") } guard_interp_gguf_load(path) - // The cold build streams: planes go from the gguf mapping straight into the image, never - // becoming model planes. Every flavor rides it — the metal blob assembles per tensor, and - // the vulkan bake collects off the mapping the stream just produced (as the warm path does) { let s_tag = want_metal ? METAL_IMAGE_TAG : "" let s_quant = want_metal ? "q8" : qs @@ -1631,8 +1535,7 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; let ts_s = ref_time_ticks() let save = image_save_enabled() let ran = load_gguf_streaming(path, mode, mt, jobs, want_metal) $(gm : GGUFMeta; gbytes : array) { - // the sizing pass answers whether the blob flavor was available; a metal run that - // got the planar answer must not write a planar image under the metal identity + // a metal run that got the planar answer must not write a planar image under the metal identity if (want_metal && !mt.metal_blob) { return false } @@ -1647,11 +1550,10 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; if (!load_image(s_img, streamed, s_tag, s_quant)) { panic("dasLLAMA: '{s_img}' was just streamed out and will not map back - see the decline above") } - dlim_gc_stale(path, s_tag, s_quant) // the fresh bake proves ITS LANE's siblings dead + dlim_gc_stale(path, s_tag, s_quant) } if (served) { - // the GPU MoE tier gathers from the PREPARED planes, same as the mapped rail — - // and that walk is what collects the vulkan bake, so the flavor costs no reload + // the MoE gather over the PREPARED planes is also what collects the vulkan bake, so the flavor costs no reload moe_gpu_upload_resident(streamed) vulkan_bake_flavor(streamed, path, vk_tag) return <- streamed @@ -1662,17 +1564,11 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; } } var m <- load_model_(path, mode) - // COMMIT to the metal flavor, do not build it: the blob plane is the planar planes - // interleaved, and the writer derives it band by band — materializing it here is what - // used to make a metal mint peak at two copies of the weights let metal_flavor = want_metal && metal_model_servable(m) && metal_blob_commit(m) if (!metal_flavor) { - // the stream declined for this model, so load_model_'s own GPU walk did the collecting; - // the image it goes on to serve must stay planar, so the flavor goes out and goes away + // the stream declined, so load_model_'s GPU walk did the collecting; the image it serves stays planar vulkan_bake_flavor(m, path, vk_tag) } - // and now the load's own flavor, on the one rail: built once in memory, written out for - // the next run, and served from the very bytes that were written var built = Model() let wrote = cache_via_image_(m, built, metal_flavor ? image_path_for(path, METAL_IMAGE_TAG) : img, metal_flavor ? METAL_IMAGE_TAG : "", metal_flavor ? "q8" : qs) $(name : string; @@ -1680,16 +1576,14 @@ def load_model_cached(path : string; mode : QuantMode = QuantMode.fp32; return metal_flavor && metal_derive_plane(m, name, w, sections) } if (metal_flavor && built.image_map == null) { - // the image declined and handed the carrier back committed but plane-less — finish - // the flavor in RAM, which is the one state that can still serve this run + // the carrier came back committed but plane-less: finishing the flavor in RAM is the only state that still serves convert_model_to_metal_blob(built) } delete m if (wrote) { - dlim_gc_stale(path, metal_flavor ? METAL_IMAGE_TAG : "", metal_flavor ? "q8" : qs) // this lane only; a failed/skipped write reaps nothing + dlim_gc_stale(path, metal_flavor ? METAL_IMAGE_TAG : "", metal_flavor ? "q8" : qs) } - // the GPU MoE tier gathered from planes the build just freed — re-gather from the image, - // exactly as the mapped rail does (a no-op when the build declined and `built` IS m) + // re-gather from the image: the build freed the planes the MoE tier gathered from (a no-op when it declined) moe_gpu_upload_resident(built) return <- built } diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_common.das b/modules/dasLLAMA/dasllama/dasllama_metal_common.das index d78cc6b5f9..6ae3573363 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_common.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_common.das @@ -561,7 +561,7 @@ def mirror_release(var mr : KVMirror) { //! re-initializes on the next served step; all mirrors rebuild from the CPU caches on demand. -// ===== the residency pin (the mechanism + figures: ARCHITECTURE.md 2.12) ===== +// ===== the residency pin (the mechanism + figures: ARCHITECTURE_RUNTIME.md 2.12) ===== // The first submission after a CPU-only window repays the per-commit residency pass over the // resident bytes; a committed+requested MTLResidencySet keeps it a no-op. var private g_rset : MetalResidencySet? @@ -609,7 +609,7 @@ def residency_flush { if (g_env_metal.heartbeat_s > 0) { // keep-alive kick: a requested set still un-wires over a long CPU-only window, and the // next submission repays the whole tracked-set pass — the background re-request keeps - // the pages wired for heartbeat_s after the last served step (ARCHITECTURE.md 2.12) + // the pages wired for heartbeat_s after the last served step (ARCHITECTURE_RUNTIME.md 2.12) metal_residency_set_heartbeat(g_rset, g_env_metal.heartbeat_s) } } diff --git a/modules/dasLLAMA/dasllama/dasllama_tower.das b/modules/dasLLAMA/dasllama/dasllama_tower.das index f229763054..4b23aaeaca 100644 --- a/modules/dasLLAMA/dasllama/dasllama_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_tower.das @@ -13,7 +13,7 @@ require dasllama/dasllama_gguf // read_clamp reads sidecar scalar tensors // The family-neutral encoder-tower home: the row math, activations, GEMM wrappers, clamp // sidecars, patch geometry, blocked bidirectional attention, and the encode-stage prof rail -// that every audio AND vision tower composes (ARCHITECTURE.md §1.7). One home: a family +// that every audio AND vision tower composes (ARCHITECTURE_MEDIA.md §1.7). One home: a family // file that re-implements a shape found here is a defect; nothing here names a family type. // erfc rational approximation (Numerical Recipes 6.2), |fractional error| < 1.2e-7 — diff --git a/modules/dasLLAMA/followup_general.md b/modules/dasLLAMA/followup_general.md index 01a382ed86..73ac38a890 100644 --- a/modules/dasLLAMA/followup_general.md +++ b/modules/dasLLAMA/followup_general.md @@ -543,7 +543,7 @@ 44. **The qwen25v (Qwen2.5-Omni/VL window ViT) CPU encode stays ~4.6x behind mtmd's clip (released `lcpp_bench --image` vs patched llama-mtmd-cli, CPU, --image-think, r=3, t=8, M1 Max) - - the tower is ruled exact-only (ARCHITECTURE.md 1.7b: per-32-block activation requant + the tower is ruled exact-only (ARCHITECTURE_MEDIA.md 1.7b: per-32-block activation requant cannot represent its outlier rows; a q8q8 lane measured 2.0 x rms where a deleted layer measures less).** Two honest paths if that encode ever matters: the Metal tower for the qwen ViT families (the same slice the qwen3v towers await), or outlier-aware activation @@ -612,3 +612,11 @@ caller (the LINT023 class). Done = the signature becomes `var buf : MetalBuffer?&` and the body nulls after the push, with every call site audited for lvalue-ness in the same change - an API pass over das_metal_boost, not a babysit patch. + +52. **`build_image`'s staging plane-group arm deletes nested fields unguarded** - the + `image_planes` arm frees with `if (consume)` alone while its top-level and `image_map` + siblings guard on `lock_count(field) == 0`. Unreachable today: `AudioTowerPlanes` embeds + only in the mint-side staging structs (`Qwen3aStaging.core`, `WhisperStaging.enc`), + which are always fresh-built and never parse targets, so no borrowed view can reach the + arm. Done = the arm carries the same `lock_count` guard as its siblings, added when that + arm is next touched for real work. diff --git a/modules/dasLLAMA/harness/residency_ramp_probe.das b/modules/dasLLAMA/harness/residency_ramp_probe.das index 93f4531a18..5111d78a07 100644 --- a/modules/dasLLAMA/harness/residency_ramp_probe.das +++ b/modules/dasLLAMA/harness/residency_ramp_probe.das @@ -2,7 +2,7 @@ options gen2 options stack = 524288 // every dasLLAMA program root takes this budget (options stack does not unify up from libs) options _dasllama_internal = true -// ATTRIBUTION SWEEP - the post-CPU-window residency-ramp ladder (ARCHITECTURE.md 2.12): its +// ATTRIBUTION SWEEP - the post-CPU-window residency-ramp ladder (ARCHITECTURE_RUNTIME.md 2.12): its // arms attribute the first-submission cost across the drv/queue/gpu split (das_metal_boost's // metal_submit_trace) and across CPU-window lengths; it selects between no implementations. // A/B rail: DASLLAMA_METAL_HEARTBEAT_S=0 reproduces the ramp, the default heartbeat holds it. diff --git a/modules/dasLLAMA/tests/test_dlim_wipe.das b/modules/dasLLAMA/tests/test_dlim_wipe.das index 8302dfa2c4..7e8b5421de 100644 --- a/modules/dasLLAMA/tests/test_dlim_wipe.das +++ b/modules/dasLLAMA/tests/test_dlim_wipe.das @@ -5,7 +5,7 @@ require daslib/fio require dasllama/dasllama_image // dlim_wipe is verdict- and lane-blind BY DESIGN (the batch-lifecycle owner carve-out, -// ARCHITECTURE.md 2.1) — but its reach must be exact: a model path wipes only that model's +// ARCHITECTURE_IMAGE.md 2.1) — but its reach must be exact: a model path wipes only that model's // images, a directory path wipes every image in it, and non-.dlim files are never touched. def private plant(root, name : string) { diff --git a/modules/dasPUGIXML/daslib/linq_fold_xml.das b/modules/dasPUGIXML/daslib/linq_fold_xml.das index 8a9d70b5a4..e299fc9f4d 100644 --- a/modules/dasPUGIXML/daslib/linq_fold_xml.das +++ b/modules/dasPUGIXML/daslib/linq_fold_xml.das @@ -10,7 +10,7 @@ module linq_fold_xml shared public //! ``from_xml_node(node[, name], type)`` chain emit an inlined DOM child-element walk (mirroring //! ``PUGIXML_boost``'s generator body) instead of driving the generator + an intermediate array. //! Optional companion to ``daslib/linq_fold`` — pulled in via ``require ?pugixml`` and gated by -//! ``static_if (typeinfo builtin_module_exists(pugixml))``. See daslib/ARCHITECTURE.md sec. 37. +//! ``static_if (typeinfo builtin_module_exists(pugixml))``. See daslib/ARCHITECTURE_LINQ.md sec. 37. require pugixml public require pugixml/PUGIXML_boost // nolint:STYLE030 — build_xml_row / read_xml_field / peek_xml_field are emitted via qmacro, invisible to the require-usage scan @@ -64,7 +64,7 @@ def private try_xml_under_guard_materialize(var body : Expression?; bindName : s // Fills `prelude` (the hoisted default-row decl, flatten path only) + `inner` (per-element, ends with // `body`): allUsed -> full build_xml_row; no field refs -> body only; pruned -> one `let xf_` read -// per referenced field with body's `it.` rewritten to it (struct dropped). See daslib/ARCHITECTURE.md sec. 37. +// per referenced field with body's `it.` rewritten to it (struct dropped). See daslib/ARCHITECTURE_LINQ.md sec. 37. [macro_function] def private build_xml_materializer(var body : Expression?; bindName : string; declT : TypeDeclPtr; defName, chName : string; at : LineInfo; diff --git a/skills/comment_style_hygiene.md b/skills/comment_style_hygiene.md index a1d6a901e2..cf656a604b 100644 --- a/skills/comment_style_hygiene.md +++ b/skills/comment_style_hygiene.md @@ -66,8 +66,10 @@ cannot tell the two apart, so this boundary is the reviewer's to hold. **C and C++: no NEW comments.** Comments go stale, code does not - new C-family code says it in a name, a shape, or a test, or documents itself in the module's `.md`. Kept when one -is earned: `//!` and `/** */` docs on public API, `NOLINT` / `clang-format` suppressions, -the leading header block. Existing comments answer to the deletion test. +is earned: `//!` and `/** */` docs on a contract a CALLER must know - public API always, a +non-public helper when callers inside the module need the contract - `NOLINT` / +`clang-format` suppressions, the leading header block. Existing comments answer to the +deletion test. **The deletion test settles any comment no kept set covers: remove it and re-read.** If the WHY goes dark it was load-bearing - restore it, rewritten in present tense; otherwise diff --git a/skills/internal/make_pr.md b/skills/internal/make_pr.md index 0281bdb587..18dbd0e423 100644 --- a/skills/internal/make_pr.md +++ b/skills/internal/make_pr.md @@ -15,13 +15,17 @@ failing test slice, a scratch probe - minutes, not tens of minutes), say so in y and let CI validate the tip. Later fix commits - its own findings, Copilot/CI rounds (`skills/internal/babysit.md`) - do NOT re-trigger it. +**The tree is FROZEN while the chain runs** - no commits, no reconfigure, no `cmake +--install`, and never `ci/smoke_test_bundle.sh` (it hides `lib/` for its whole run, which +kills the chain's JIT loads, AOT links, and spawned tools mid-suite). + ## The checklist | Step | Gate / tool | Fix policy | |---|---|---| | 0 Sync | make-pr `sync` | Red = behind origin/master: rebase (never onto local `master`), re-run. A listed PR-set file you did not edit = the rebase went wrong; a conflict on a file also changed upstream keeps origin/master's. Squash only AFTER the rebase - `git reset --soft master` on a stale `master` bakes other PRs in; already pushed: rebase + `git push --force-with-lease`. Re-read any `skills/*.md` / `REVIEW*.md` the rebase changed | | 0 Untracked | preflight `untracked` gate | Empty at PR time - commit, delete, or ignore each (`.gitignore`; `.git/info/exclude` for box-local) | -| 0a0 Comment drain | MCP `format_file` on all changed `.das` in ONE batched call (same engine and `.lint_config` policy as step 5) | Working comments are welcome while building the PR; this row is where they settle. A `rescue_advisory` in the result = comments were deleted: spawn ONE `rescue-bot` per module root the strip touched, over that root's strip diff (report-only rescue ledger). YOU rule on every entry - apply accepted renames / REVIEW.md / ARCHITECTURE.md entries, treat TODO signals as possibly unfinished PR work, surface lint candidates per CLAUDE.md's lint-opportunities rule, drop the rest. Sits before the audits because rescues re-enter the diff | +| 0a0 Comment harvest | the diff's ADDED comments, per touched module root | Working comments are welcome while building the PR; this row is where they settle. When the diff adds comments beyond the hygiene skill's kept set, spawn ONE `harvester` per touched module root, scoped to the comments the diff adds (a full-file harvest is the on-first-touch sweep, not a PR gate). YOU rule on its ledger - RENAME first (the strongest resolution), RULE/FACT proposals land in the folder's REVIEW.md / ARCHITECTURE*.md, KEEP one-liners are `//!` contract comments, TODO signals are possibly unfinished PR work, lint candidates surface per CLAUDE.md's lint-opportunities rule. Sits before the audits because landings re-enter the diff | | 0a REVIEW audit | make-pr `review-md` | Red = a discovered `REVIEW.das` gate failed - fail-fix, no agents until green. Then one `review-md-auditor` per checklist. Discovered rules bind on top of this file; checklist defects fixed in the same batch | | 0a TDD audit | one `tdd-auditor` over the whole diff, REVIEW.md folders or not (`skills/tdd_audit.md`) | UNTESTED branch -> test in the same change, never a follow-up promise. UNPROVEN -> run its named settling gate or state the claim in the PR body. RETUNED/WEAKENED test edit -> restore the expectation/instrument or state the reason | | 0a2 Style hygiene | `style-hygiene-auditor` (`skills/comment_style_hygiene.md`) | Mandatory run, non-blocking findings: fix each or consciously decline it | @@ -37,7 +41,7 @@ and let CI validate the tip. Later fix commits - its own findings, Copilot/CI ro | 3 AOT build | kill by path first (below), then `cmake --build build --config Release --target test_aot -j 64 -- /nodeReuse:false` with `timeout: 0` (2-25 min) | Doesn't build: register new test directories in `tests/aot/CMakeLists.txt` (`skills/internal/aot_testing.md`); `error[50101]` is a hash desync (`skills/internal/aot_hash_desync_debugging.md`) | | 3 AOT tests | `bin/Release/test_aot.exe -use-aot dastest/dastest.das -- --use-aot --color --failures-only --timeout 1800 --test tests` (the `-use-aot` / `--use-aot` doubling matches CI) | Same triage as step 2. PR CI builds only `test_aot_subset` - this run and the nightly cron are the only full-AOT checks | | 4 Docs | see below | Skip when the PR only changes examples, tests, or non-public code | -| 5 Format | MCP `format_file` on all changed `.das` in ONE batched call | Only files in the PR, every era - it handles gen1 and `.das_project`, and CI fails on unformatted gen1; comment stripping follows the folder's `.lint_config` `[format]` policy (teaching folders opt out, swept trees opt in - no per-call flag). Verify they still compile. A `rescue_advisory` here means scaffolding was written after 0a0 - re-run the drain (rescue-bot over the new strip diff) before continuing. CI's `utils/das-fmt/dasfmt.das -- --path ./ --verify` wraps the same engine and the same policy | +| 5 Format | MCP `format_file` on all changed `.das` in ONE batched call | Only files in the PR, every era - it handles gen1 and `.das_project`, and CI fails on unformatted gen1; no folder arms comment stripping any more - comments are harvested (row 0a0), never formatter-deleted. Verify the files still compile. New comments written after 0a0 re-run that row before continuing. CI's `utils/das-fmt/dasfmt.das -- --path ./ --verify` wraps the same engine and the same policy | | 5 `.md` stop | `git diff --name-only origin/master..HEAD \| grep '\.md$'` | Any match: STOP, list the changes, ask the user to review BEFORE push | | 6 PR | GitHub MCP `create_pull_request` or `gh pr create` | Body follows the two-layer template below. On a squashed branch every later fix is `git commit --amend --no-edit` + force-push, never a new commit | | 6a Babysit | continue into `skills/internal/babysit.md`; triage every comment per `skills/internal/review_triage.md` | Creating the PR does not end the workflow; the stop rule and merge gate are babysit sec.0's | @@ -45,7 +49,7 @@ and let CI validate the tip. Later fix commits - its own findings, Copilot/CI ro ## 0a0-0a3. Agent topology -**0a0** - ONE `rescue-bot` (`.claude/agents/rescue-bot.md`) per module root the strip touched, over that root's strip diff; report-only, the session rules on every ledger entry. Same registry caveat as 0a. +**0a0** - ONE `harvester` (`.claude/agents/harvester.md`) per touched module root, scoped to the comments the diff adds; it edits only the source files (deletions, `//!` compressions) and PROPOSES every document landing - the session rules on the ledger and lands what it accepts. Same registry caveat as 0a. **0a** - ONE `review-md-auditor` (`.claude/agents/review-md-auditor.md`) per discovered checklist, each auditing only its own under the self-review rule, plus ONE `tdd-auditor` (`.claude/agents/tdd-auditor.md`) for the whole diff; merge the reports. **Registry caveat: agent definitions snapshot at session start - a just-pulled or just-edited definition only exists in the NEXT session.** A non-trivial change runs the full round (`skills/internal/review_round.md`) on top; these instances are its surfacing phase, not a repeat. diff --git a/skills/mcp_tools.md b/skills/mcp_tools.md index 842c77ece9..b04bdb1246 100644 --- a/skills/mcp_tools.md +++ b/skills/mcp_tools.md @@ -28,6 +28,8 @@ The daslang MCP server (`utils/mcp/main.das`) exposes compiler diagnostics, prog | `describe_type` | Reading source to understand type fields, methods, and values | | `grep_usage` | Built-in Grep across `.das` files (parse-aware via ast-grep + tree-sitter) | | `outline` | Manually scanning files for function/struct/enum declarations | +| `arch_of` | Opening a file's cited architecture document and counting headings to find the section - returns the section text per `[arch]` citation | +| `arch_sites` | Grepping for a document's anchors to learn which code answers for a section - returns the citation census, grouped by anchor | | `cpp_grep_usage` | Built-in Grep across `.cpp/.h/.hpp/.cc` files (parse-aware via ast-grep + tree-sitter-cpp) | | `cpp_find_symbol` | Searching for C++ symbol DECLARATIONS by name + kind (function/class/struct/enum/union/typedef/namespace/macro) | | `cpp_outline` | Manually scanning C++ files for top-level declarations | @@ -64,6 +66,8 @@ The daslang MCP server (`utils/mcp/main.das`) exposes compiler diagnostics, prog **`with_cpp_source` redirect.** `find_symbol` and `goto_definition` accept an optional `with_cpp_source` boolean. When `true`, results that have a C++ implementation (builtin functions, handled types via `addExtern`/`MAKE_TYPE_FACTORY`) get a resolved C++ source location appended via the lazily-built cpp index. First call costs ~2s (one full scan); subsequent calls cost ~150ms (a git-state staleness signature: `rev-parse HEAD` + filtered `git status` + per-file mtimes + `cpp_search_config.das` mtime). The index rebuilds automatically when relevant `.cpp/.cc/.h/.hpp` files change, when HEAD moves, or when the search config is edited. Default off - opt in when the question is "where is X *actually* implemented", not when just enumerating symbols. +**`[arch]` tools.** `arch_of` (code -> document) and `arch_sites` (document -> code) resolve citations exactly as LINT026 does: the path in `[arch(at=".md#")]` resolves against the CITING file's folder, an anchor is a `{#name}` heading suffix, a match with `//` earlier on its line is not a citation, and a failure reports lint's own reason (`no such file` / `no such anchor` / `anchor appears N times` / `malformed citation`). Both read source rather than the AST, so they answer for a file this environment cannot compile. `arch_of` returns the cited section - its heading line through the last line before the next heading of the same or a higher level. `arch_sites` searches the subtree of the folder that owns the document, skipping `_`- and `.`-prefixed names the way lint's reverse pass does. + **Live tools.** `live_*` interact with a running `daslang-live` instance via its REST API. `live_launch` starts one if not already running (sets working directory to the script's folder). All live tools accept an optional `port` parameter (default 9090). When a compilation error is active, `live_command` and `live_pause` return HTTP 503 with the error - use `live_reload` to fix. Hitting any unknown endpoint returns JSON help with all endpoints + curl examples. **`shutdown` tool.** Shuts down the MCP server process. Claude Code auto-restarts it, picking up code changes to `.das` tool files. Tool registration changes (adding/removing tools) still require a manual MCP restart. diff --git a/src/builtin/REVIEW.md b/src/builtin/REVIEW.md index d5ea84145e..f154981bf4 100644 --- a/src/builtin/REVIEW.md +++ b/src/builtin/REVIEW.md @@ -5,7 +5,9 @@ - **A plain-value bind added or changed under this folder uses `addExternInline` or `addExternInlineEx` when its module is an Inline module, and an `addExtern...` entry point - whose name does not contain `Inline` in every other module.** The Inline modules are `$` + whose name does not contain `Inline` in every other module.** A bind is a C++ function + registered into a module with an `addExtern...` or `addInterop` entry point; annotation, + type, and structure registrations are not binds. The Inline modules are `$` (builtin), `math`, `strings` and `jit`. A plain-value bind returns neither a reference nor a result the callee writes into the caller's result slot, is not an interop bind (`addInterop`), and is not a generic container or equality helper (`das_vector_*`, diff --git a/src/builtin/module_builtin_runtime.cpp b/src/builtin/module_builtin_runtime.cpp index 2e84c48b95..b52a590b8e 100644 --- a/src/builtin/module_builtin_runtime.cpp +++ b/src/builtin/module_builtin_runtime.cpp @@ -161,6 +161,38 @@ namespace das }; }; + // [arch(at=".md#")] — binds this function to a section of an architecture + // document; the LINT026 pass in utils/lint/main.das resolves the citation against the tree. + // Same doctrine as the hot_path family above: a citation must not drag its checker into every + // build of the cited code, so the marker is registered here and costs nothing. Compile owns + // the citation's SHAPE only — whether it RESOLVES is the lint driver's question. Repeat the + // annotation to cite more than one section. + // + // A das structure or class takes the same spelling: a module holds one annotation per NAME + // (Module::addAnnotation is keyed by it), so "arch" cannot also be a StructureAnnotation, and + // the structure path — which silently ignores an annotation that is not a StructureAnnotation — + // accepts the citation unvalidated. LINT026 shape-checks structure citations in its stead. + static bool verifyArchCitation ( const AnnotationArgumentList & args, string & err ) { + if ( args.size()!=1 || args[0].name!="at" || args[0].type!=Type::tString ) { + err = "expecting exactly one argument, at=\".md#\""; + return false; + } + const string & at = args[0].sValue; + auto hash = at.find('#'); + if ( hash==string::npos || hash==0 || hash==at.length()-1 ) { + err = "citation \"" + at + "\" is not \".md#\" — a document path and an anchor, both non-empty, separated by #"; + return false; + } + return true; + } + + struct ArchFunctionAnnotation : MarkFunctionAnnotation { + ArchFunctionAnnotation() : MarkFunctionAnnotation("arch") { } + virtual bool apply(const FunctionPtr &, ModuleGroup &, const AnnotationArgumentList & args, string & err) override { + return verifyArchCitation(args, err); + }; + }; + struct RequestJitFunctionAnnotation : MarkFunctionAnnotation { RequestJitFunctionAnnotation() : MarkFunctionAnnotation("jit") { } virtual bool apply(const FunctionPtr & func, ModuleGroup &, const AnnotationArgumentList &, string &) override { @@ -2172,6 +2204,7 @@ namespace das addAnnotation(new HotPathFunctionAnnotation("no_env")); addAnnotation(new HotPathFunctionAnnotation("no_io")); addAnnotation(new HotPathFunctionAnnotation("cold_path")); + addAnnotation(new ArchFunctionAnnotation()); addAnnotation(new HintFunctionAnnotation()); addAnnotation(new RequestJitFunctionAnnotation()); addAnnotation(new RequestNoJitFunctionAnnotation()); diff --git a/tests/dasPUGIXML/parity_xml_order_distinct_group.das b/tests/dasPUGIXML/parity_xml_order_distinct_group.das index 7cc580ba51..8add907a31 100644 --- a/tests/dasPUGIXML/parity_xml_order_distinct_group.das +++ b/tests/dasPUGIXML/parity_xml_order_distinct_group.das @@ -220,7 +220,7 @@ def parity_order_by_take(t : T?) { } } -// ===== Deferred-materialization (handle-buffering) bounded-heap variants — see daslib/ARCHITECTURE.md sec. 37 ===== +// ===== Deferred-materialization (handle-buffering) bounded-heap variants — see daslib/ARCHITECTURE_LINQ.md sec. 37 ===== [test] def parity_order_by_descending_take(t : T?) { diff --git a/tests/language/_arch_annotation_doc.md b/tests/language/_arch_annotation_doc.md new file mode 100644 index 0000000000..e91e58dba6 --- /dev/null +++ b/tests/language/_arch_annotation_doc.md @@ -0,0 +1,21 @@ +# Architecture document fixture + +Support file for `arch_annotation.das`. Its sections exist so that the `[arch]` citations in +that test resolve under LINT026 (`utils/lint/main.das`) the way a real citation does - the +`_` prefix keeps it out of every tree walk that collects sources. + +## Tokenizer {#tokenizer} + +The section a cited function answers for. + +## Buffers {#buffers} + +The second section, so one function can be shown citing two. + +## Token {#token} + +The section a cited structure answers for. + +## Lexer {#lexer} + +The section a cited class answers for. diff --git a/tests/language/arch_annotation.das b/tests/language/arch_annotation.das new file mode 100644 index 0000000000..71c4c8b3ad --- /dev/null +++ b/tests/language/arch_annotation.das @@ -0,0 +1,61 @@ +options gen2 +require dastest/testing_boost public + +// [arch(at=".md#")] binds code to a section of an architecture document. It is an +// inert marker registered in the builtin module, so citing a document costs a build nothing and +// needs no require; whether the citation RESOLVES is LINT026's question, not the compiler's. +// This file is the compile-side contract: the shapes that must be accepted, everywhere. + +[arch(at="_arch_annotation_doc.md#tokenizer")] +def cited_function : int { + return 17 +} + +// one function may answer for more than one section +[arch(at="_arch_annotation_doc.md#tokenizer"), arch(at="_arch_annotation_doc.md#buffers")] +def twice_cited_function : int { + return 4 +} + +// the marker composes with the annotations it sits beside +[never_inline, arch(at="_arch_annotation_doc.md#tokenizer")] +def cited_beside_another : int { + return 9 +} + +[arch(at="_arch_annotation_doc.md#token")] +struct CitedStruct { + value : int +} + +[arch(at="_arch_annotation_doc.md#lexer")] +class CitedClass { + count : int + def bump { + count ++ + } +} + + +[test] +def test_arch_annotation_compiles(t : T?) { + t |> run("a cited function is an ordinary function") @(t : T?) { + t |> equal(cited_function(), 17, "the marker changes no behavior") + t |> equal(twice_cited_function(), 4, "two citations are legal on one function") + t |> equal(cited_beside_another(), 9, "the marker composes with other annotations") + } + t |> run("a cited structure is an ordinary structure") @(t : T?) { + var s = CitedStruct(value = 5) + s.value ++ + t |> equal(s.value, 6, "fields behave as declared") + } + t |> run("a cited class is an ordinary class") @(t : T?) { + var c = new CitedClass() + c->bump() + c->bump() + t |> equal(c.count, 2, "methods behave as declared") + unsafe { + delete c + } + } +} diff --git a/tests/typer_errors/_arch_empty_citation.das b/tests/typer_errors/_arch_empty_citation.das new file mode 100644 index 0000000000..49431703be --- /dev/null +++ b/tests/typer_errors/_arch_empty_citation.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — [arch] citing the empty string. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(at="")] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_no_anchor_half.das b/tests/typer_errors/_arch_no_anchor_half.das new file mode 100644 index 0000000000..6140dcace3 --- /dev/null +++ b/tests/typer_errors/_arch_no_anchor_half.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — a document with an empty anchor half. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(at="ARCHITECTURE.md#")] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_no_argument.das b/tests/typer_errors/_arch_no_argument.das new file mode 100644 index 0000000000..4dbbc0b1f9 --- /dev/null +++ b/tests/typer_errors/_arch_no_argument.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — [arch] with no citation at all. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_no_doc_half.das b/tests/typer_errors/_arch_no_doc_half.das new file mode 100644 index 0000000000..9a0ea7fa7b --- /dev/null +++ b/tests/typer_errors/_arch_no_doc_half.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — an anchor with no document half. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(at="#x")] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_no_hash.das b/tests/typer_errors/_arch_no_hash.das new file mode 100644 index 0000000000..bfac1bb3e8 --- /dev/null +++ b/tests/typer_errors/_arch_no_hash.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — a document with no anchor half. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(at="ARCHITECTURE.md")] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_two_args.das b/tests/typer_errors/_arch_two_args.das new file mode 100644 index 0000000000..551e4a7683 --- /dev/null +++ b/tests/typer_errors/_arch_two_args.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — a second argument. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(at="ARCHITECTURE.md#x", extra="y")] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_wrong_name.das b/tests/typer_errors/_arch_wrong_name.das new file mode 100644 index 0000000000..4453051ea9 --- /dev/null +++ b/tests/typer_errors/_arch_wrong_name.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — a named argument that is not at. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(doc="ARCHITECTURE.md#x")] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/_arch_wrong_type.das b/tests/typer_errors/_arch_wrong_type.das new file mode 100644 index 0000000000..a02756f34f --- /dev/null +++ b/tests/typer_errors/_arch_wrong_type.das @@ -0,0 +1,15 @@ +// Fixture for test_arch_annotation_errors.das — a non-string argument. +// `_`-prefix keeps dastest from auto-running it; the `expect` directive makes lint +// skip it (a non-compiling program never reaches the lint passes). +expect 20800 + +options gen2 + +[arch(at=13)] +def cited { +} + +[export] +def main { + cited() +} diff --git a/tests/typer_errors/test_arch_annotation_errors.das b/tests/typer_errors/test_arch_annotation_errors.das new file mode 100644 index 0000000000..394840baf1 --- /dev/null +++ b/tests/typer_errors/test_arch_annotation_errors.das @@ -0,0 +1,69 @@ +// [arch] owns the SHAPE of a citation and nothing else: exactly one `at=` string, carrying a +// document path and an anchor, both non-empty, separated by '#'. Whether the citation resolves +// to a real section is LINT026's question (utils/lint/main.das), which is why a well-shaped +// citation to a document that is not there still compiles. +options gen2 +options no_aot // invokes the compiler at runtime (compile_file) — not AOT-linkable + +require daslib/ast +require strings +require dastest/testing_boost public + + +def compile_issues(file_path : string) : string { + var issuesText = "" + var inscope access <- make_file_access("") + using() $(var mg : ModuleGroup) { + using() $(var cop : CodeOfPolicies) { + cop.threadlock_context = true + compile_file(file_path, access, unsafe(addr(mg)), cop) $(_ok, _program, issues) { + issuesText := issues + } + } + } + return issuesText +} + + +def expect_issue(t : T?; fixture, message : string) { + let issues = compile_issues("{get_das_root()}/tests/typer_errors/{fixture}") + t |> success(find(issues, message) >= 0, + "expected '{message}' compiling {fixture}, got: {issues}") +} + + +[test] +def test_arch_citation_shape_errors(t : T?) { + t |> run("[arch] with no citation") @(t : T?) { + t |> expect_issue("_arch_no_argument.das", + "expecting exactly one argument, at=\".md#\"") + } + t |> run("[arch] citing the empty string") @(t : T?) { + t |> expect_issue("_arch_empty_citation.das", + "citation \"\" is not \".md#\"") + } + t |> run("[arch] citing a document with no anchor") @(t : T?) { + t |> expect_issue("_arch_no_hash.das", + "citation \"ARCHITECTURE.md\" is not \".md#\"") + } + t |> run("[arch] with a named argument that is not at") @(t : T?) { + t |> expect_issue("_arch_wrong_name.das", + "expecting exactly one argument, at=\".md#\"") + } + t |> run("[arch] with a non-string argument") @(t : T?) { + t |> expect_issue("_arch_wrong_type.das", + "expecting exactly one argument, at=\".md#\"") + } + t |> run("[arch] with a second argument") @(t : T?) { + t |> expect_issue("_arch_two_args.das", + "expecting exactly one argument, at=\".md#\"") + } + t |> run("[arch] citing an anchor with no document half") @(t : T?) { + t |> expect_issue("_arch_no_doc_half.das", + "citation \"#x\" is not \".md#\"") + } + t |> run("[arch] citing a document with an empty anchor half") @(t : T?) { + t |> expect_issue("_arch_no_anchor_half.das", + "citation \"ARCHITECTURE.md#\" is not \".md#\"") + } +} diff --git a/utils/common/arch_citations.das b/utils/common/arch_citations.das new file mode 100644 index 0000000000..b6a8a4c074 --- /dev/null +++ b/utils/common/arch_citations.das @@ -0,0 +1,212 @@ +// The `[arch]` citation scanner: one implementation of LINT026's resolution semantics, shared by +// the rule that enforces them (`utils/lint/main.das`) and the tools that read them +// (`utils/mcp/tools/arch_of`, `utils/mcp/tools/arch_sites`). The failure strings live here too, +// so a citation lint reports and a citation a tool refuses cannot describe themselves +// differently. +// +// Everything here reads SOURCE rather than the AST: the annotation is an inert builtin marker, so +// citing a document costs a build nothing, and a citation must be checkable in a file the reading +// environment cannot compile. + +options gen2 + +module arch_citations + +require daslib/fio public +require strings public +require daslib/strings_boost public +require daslib/regex +require daslib/regex_boost + + +// The citation as written in a .das source. A match with `//` earlier on its line is prose or a +// commented-out annotation and is skipped; one inside a string literal or a block comment is not, +// so a fixture that plants citation text assembles it rather than spelling it out. +var private ARCH_CITE <- %regex~arch\s*\(\s*at\s*=\s*"([^"]*)"%% + + +// One citation, split at the first '#'. `doc` is normalized against the CITING file's folder, so +// it compares equal to the same document reached from anywhere else in the tree; `anchor` is +// empty when the citation carries no '#' at all. +struct public ArchCitation { + line : int + doc : string + anchor : string + raw : string +} + + +// One `{#anchor}` a heading declares: the name, the line that declares it, and the heading level +// that bounds its section. +struct public MdAnchor { + name : string + line : int + level : int +} + + +// The anchors of every document a scan has resolved so far. `counts` is keyed "#" so +// it answers the duplicate question in one lookup; `known` records existence separately, because +// a citation naming a missing file must not read as a missing anchor. +struct public ArchDocs { + counts : table + known : table +} + + +// The one spelling a path is compared under: lexical, so it works on a document that is not +// there, and '/'-folded, so a Windows walk and a citation agree. +def public arch_path(path : string) : string { + return normalize(path) |> replace("\\", "/") +} + + +// Where a match sits: its 1-based line, and the offset that line starts at - which is what lets +// the scan tell an annotation from the same text inside a `//` comment. +struct private SrcPos { + line : int + line_start : int +} + + +// Byte offsets (ascending) to source positions in one pass over the bytes. Counting newlines with +// find() per citation would re-walk the source from the top each time. +def private positions_of_offsets(text : string; offsets : array) : array { + var out : array + out |> reserve(length(offsets)) + peek_data(text) $(d) { + var line = 1 + var start = 0 + var at = 0 + for (off in offsets) { + while (at < off) { + if (int(d[at]) == '\n') { + line ++ + start = at + 1 + } + at ++ + } + out |> push(SrcPos(line = line, line_start = start)) + } + } + return <- out +} + + +// Every citation `text` makes, in file order. `file` is only read for the folder a relative +// citation resolves against, so it may name a file that is no longer on disk. +def public collect_arch_citations(file, text : string) : array { + var out : array + var raws : array + var offsets : array + regex_foreach(ARCH_CITE, text) $(at) { + offsets |> push(at.x) + raws |> push(regex_group(ARCH_CITE, 1, text)) + return true + } + return <- out if (empty(raws)) + let positions <- positions_of_offsets(text, offsets) + let folder = dir_name(file) + for (raw, off, pos in raws, offsets, positions) { + let slashes = find(text, "//", pos.line_start) + continue if (slashes >= 0 && slashes < off) + let hash = find(raw, "#") + if (hash <= 0 || hash == length(raw) - 1) { + // shape, not resolution: the compiler rejects this on a function, but a structure + // takes the marker unvalidated (one annotation per name), so the scan owns it there + out |> push(ArchCitation(line = pos.line, raw = raw)) + continue + } + out |> push(ArchCitation(line = pos.line, raw = raw, anchor = slice(raw, hash + 1), + doc = arch_path(path_join(folder, slice(raw, 0, hash))))) + } + return <- out +} + + +// The ATX heading level of a line, 0 when the line is not a heading. One to six '#' whose run ends +// the token - so `#!shebang` and `#include` are prose. A line carrying an anchor suffix passes +// whatever follows the run, because that is the shape an anchor is read under. +def public heading_level(line : string) : int { + let t = line |> strip + return 0 if (!(t |> starts_with("#"))) + var n = 0 + peek_data(t) $(d) { + let len = length(d) + while (n < len && int(d[n]) == '#') { + n ++ + } + } + return 0 if (n > 6) + let rest = slice(t, n) + return n if (empty(rest) || first_character(rest) == ' ' || first_character(rest) == '\t' + || t |> ends_with("}")) + return 0 +} + + +// The anchor a heading line declares, name empty when it declares none. Only a heading SUFFIX +// counts - `## {#anchor}` at any level, the brace group last on the line. A `{#...}` +// elsewhere is prose about anchors, not a section identity. +def public anchor_of_heading(line : string) : MdAnchor { + var out : MdAnchor + let t = line |> strip + return out if (!(t |> starts_with("#")) || !(t |> ends_with("}"))) + let open = rfind(t, "\{#") + return out if (open < 0) + let name = slice(t, open + 2, length(t) - 1) + return out if (empty(name)) + out.name = name + out.level = heading_level(t) + return out +} + + +// Every anchor the document at `path` declares, in file order. A repeated name is kept twice - +// the duplicate is what makes a citation ambiguous. +def public collect_md_anchors(path : string) : array<MdAnchor> { + var out : array<MdAnchor> + let text = fread(path) + if (!empty(text)) { + var lineno = 0 + for (line in split(text, "\n")) { + lineno ++ + var a = anchor_of_heading(line) + continue if (empty(a.name)) + a.line = lineno + out |> push(a) + } + } + return <- out +} + + +// True when `doc` exists, indexing its anchors on first sight. +def private arch_doc_exists(var docs : ArchDocs; doc : string) : bool { + if (!key_exists(docs.known, doc)) { + let st = stat(doc) + let ok = st.is_valid && st.is_reg + docs.known[doc] = ok + if (ok) { + for (a in collect_md_anchors(doc)) { + let key = "{doc}#{a.name}" + docs.counts[key] = (docs.counts ?[key] ?? 0) + 1 + } + } + } + return docs.known[doc] +} + + +// Why a citation does not resolve, in the words LINT026 reports; "" when it resolves. A citation +// names a document section that must be there, exactly once: an anchor declared twice is worse +// than a missing one - the citation reads as precise and points at whichever section the reader +// finds first. +def public citation_failure_reason(var docs : ArchDocs; c : ArchCitation) : string { + return "malformed citation (expected <doc>.md#<anchor>)" if (empty(c.anchor)) + return "no such file" if (!arch_doc_exists(docs, c.doc)) + let n = docs.counts ?["{c.doc}#{c.anchor}"] ?? 0 + return "no such anchor" if (n == 0) + return "anchor appears {n} times" if (n > 1) + return "" +} diff --git a/utils/das-fmt/dasfmt.das b/utils/das-fmt/dasfmt.das index afce8e4d77..3fabb9af31 100644 --- a/utils/das-fmt/dasfmt.das +++ b/utils/das-fmt/dasfmt.das @@ -360,7 +360,7 @@ def main() { run_format_jobs(jobs, threadsNum, verify, verbose, args, verified, strip_failures, stripped_count) if (stripped_count > 0) { - log::info("comments were deleted from {stripped_count} file(s) — before committing, run the rescue-bot agent over their strip diff (the '(comments stripped)' files above)") + log::info("comments were deleted from {stripped_count} file(s) — before committing, run the harvester agent over their strip diff (the '(comments stripped)' files above)") } if (strip_failures > 0) { diff --git a/utils/detect-dupe/main.das b/utils/detect-dupe/main.das index 7c627a0905..2bf6b5ca5e 100644 --- a/utils/detect-dupe/main.das +++ b/utils/detect-dupe/main.das @@ -435,16 +435,24 @@ def main() { // nolint:STYLE037,STYLE038 — flat CLI validation ladder plus one var compile_files : array<string> compile_files |> reserve(length(files)) var n_skipped = 0 + var n_agent = 0 for (f in files) { if (has_expect_directive(f)) { n_skipped++ continue } + if (installs_debug_agent(f)) { + n_agent++ + continue + } compile_files |> push(f) } if (n_skipped > 0) { to_log(LOG_INFO, "{n_skipped} file(s) skipped (expect-directive)\n") } + if (n_agent > 0) { + to_log(LOG_INFO, "{n_agent} file(s) skipped (debug-agent installer - two in one scan process panic at the second install)\n") + } files_scanned = length(files) // Parallel export fast-path: skip the in-process compile loop diff --git a/utils/detect-dupe/pipeline.das b/utils/detect-dupe/pipeline.das index 6d5f8c4401..2886a8b1f6 100644 --- a/utils/detect-dupe/pipeline.das +++ b/utils/detect-dupe/pipeline.das @@ -70,6 +70,25 @@ def public has_expect_directive(file : string) : bool { } +//! True for a file whose COMPILATION installs a thread-local debug agent (`options debugger` +//! / `options profiler`, or a direct installer call). The agent outlives compile_file, so a +//! scan process that compiles two such files panics at the second install +//! (`debugger.das: thread local debug agent already installed`) - they never enter a corpus. +def public installs_debug_agent(file : string) : bool { + let text = fread(file) + if (empty(text)) return false + let lines <- text |> split("\n") + for (line in lines) { + let trimmed = line |> strip + if (trimmed |> starts_with("options debugger") || trimmed |> starts_with("options profiler") + || (find(trimmed, "install_new_thread_local_debug_agent") >= 0 && !(trimmed |> starts_with("//")))) { + return true + } + } + return false +} + + def public scan_das_files(path : string; var files : array<string>; var cache : table<string; void?>) { let st = stat(path) if (!st.is_valid) return diff --git a/utils/lint/README.md b/utils/lint/README.md index 7453882a41..02a9b7f50a 100644 --- a/utils/lint/README.md +++ b/utils/lint/README.md @@ -10,12 +10,15 @@ Run: Exit 2 on any warning - CI's changed-files gate and the nightly full sweep both key on it. The SDK also ships a prebuilt `bin/lint.exe`. -One rule is the runner's own, because it is about folders rather than code: **LINT025**. A -directory whose `.lint_config` carries `[docs] rule_docs_only = true` may hold only rule -documents (`REVIEW*.md`, `ARCHITECTURE*.md`, `LAWS.md`); any other `.md` beside the sources -is reported once per invocation, from a walk of the directory roots the run was given. The -key is a folder property - it never cascades, unlike `[format]`. Fixture: `tests/lint025_*`, -which drives the CLI over a planted tree. +Three rules are the runner's own, because they are about folders rather than code. Each runs +once per invocation, over a walk of the directory roots the run was given. A directory whose +`.lint_config` carries `[docs] rule_docs_only = true` may hold only rule documents +(`REVIEW*.md`, `ARCHITECTURE*.md`, `LAWS.md`); any other `.md` beside the sources is +**LINT025**. `[docs] enforce_arch = true` adds **LINT026**'s reverse direction: every +`{#anchor}` in the folder's `.md` must be cited by an `[arch]` in its `.das`. Either tag arms +**LINT027**, which caps each `REVIEW*.md` / `ARCHITECTURE*.md` there at 300 lines. The keys +are folder properties - they never cascade, unlike `[format]`. Fixtures: `tests/lint025_*`, +`tests/lint026_*` and `tests/lint027_*`, each driving the CLI over a planted tree. Design: the runner stays thin - rules live in the daslib modules (authoring rails: `skills/internal/perf_lint_authoring.md`, `skills/internal/style_lint_authoring.md`); diff --git a/utils/lint/REVIEW.md b/utils/lint/REVIEW.md index 2ce292db35..6e9c256414 100644 --- a/utils/lint/REVIEW.md +++ b/utils/lint/REVIEW.md @@ -4,8 +4,7 @@ `README.md`. **A diff that removes `lint` from `DAS_UTILS_SHIPPED_EXES` in `utils/CMakeLists.txt` is a -defect.** The removal drops the prebuilt lint exe from the release bundle, and the gate -cannot see a removed entry. +defect.** **A diff that shrinks the set of rule ids `REVIEW.das` (beside this file) scans is a defect** - whether by editing the gate or by deleting an id's last scannable spelling. The @@ -13,4 +12,5 @@ gate checks that every emitted rule id has a fixture and a `doc/source/reference (repo root) section, so a shrunk set retires those checks with no warning. **A diff that makes a file emit a rule id the gate `REVIEW.das` (beside this file) does not -scan adds that file to the gate's module set, in the same change.** +scan adds that file to `RULE_MODULES` in the gate, in the same change.** A file emits an id +when its own code prints it; a fixture asserting on that text does not emit. diff --git a/utils/lint/main.das b/utils/lint/main.das index 3cb9a223cd..e64c1a5943 100644 --- a/utils/lint/main.das +++ b/utils/lint/main.das @@ -16,6 +16,7 @@ require strings require daslib/strings_boost require daslib/rtti require ../common/parallel_workers.das +require ../common/arch_citations.das let JSON_PREFIX = "##lint##\n" @@ -175,6 +176,9 @@ def scan_das_files(path : string; var files : array<string>; var cache : table<s } } +// LINT027's cap on one rule document, in lines. +let RULE_DOC_MAX_LINES = 300 + // The .md basenames a `[docs] rule_docs_only = true` folder may carry: the folder's // checklists (REVIEW.md and its routed companions), its architecture doc (and companion // splits), and the rulings sidecar. Everything else is a ledger, a plan or a note. @@ -183,30 +187,64 @@ def is_rule_doc_name(name : string) : bool { || name |> starts_with("REVIEW_") || name |> starts_with("ARCHITECTURE_")) } -// Directories under `path` whose OWN .lint_config sets [docs] rule_docs_only = true. -// Folder property, no cascade - each directory answers for itself. Skip rules mirror -// scan_das_files so both passes see the same tree. -def scan_docs_folders(path : string; var folders : array<string>; var seen : table<string; void?>) { +// A rule runs unless --disable names it, or --enable put the run in whitelist mode without it. +def rule_is_on(code : string; disabled_codes, enabled_codes : table<string>) : bool { + return !key_exists(disabled_codes, code) && (empty(enabled_codes) || key_exists(enabled_codes, code)) +} + +// Directories under `path` whose OWN .lint_config arms a [docs] flag: rule_docs_only feeds +// LINT025, enforce_arch feeds LINT026's reverse direction, and either one feeds LINT027's +// line gate. Folder property, no cascade - each directory answers for itself. Skip rules +// mirror scan_das_files so all passes see one tree. +def scan_docs_folders(path : string; want_rule_docs, want_arch, want_doc_lines : bool; + var folders, arch_folders, line_folders : array<string>; var seen : table<string; void?>) { let st = stat(path) return if (!st.is_valid || !st.is_dir || seen |> key_exists(path)) seen |> insert(path, null) - if (rule_docs_only_at("{path}/.lint_config")) { + let cfg = "{path}/.lint_config" + let armed_rule_docs = (want_rule_docs || want_doc_lines) && rule_docs_only_at(cfg) + let armed_arch = (want_arch || want_doc_lines) && enforce_arch_at(cfg) + if (want_rule_docs && armed_rule_docs) { folders |> push(path) } + if (want_arch && armed_arch) { + arch_folders |> push(path) + } + if (want_doc_lines && (armed_rule_docs || armed_arch)) { + line_folders |> push(path) + } fio::dir(path) $(name) { return if (name == "." || name == ".." || name |> starts_with("_") || is_skip_dir(name)) let full = "{path}/{name}" let fst = stat(full) if (fst.is_valid && fst.is_dir) { - scan_docs_folders(full, folders, seen) + scan_docs_folders(full, want_rule_docs, want_arch, want_doc_lines, folders, arch_folders, line_folders, seen) } } } -// The .md files in `folder` (non-recursive) that are not rule documents. -def collect_stray_rule_docs(folder : string; var out : array<string>) { +// One walk of the run's scan roots for all three folder rules - the tree is walked once +// however many of them are on, and not at all when none is. +def collect_docs_folders(roots : array<string>; want_rule_docs, want_arch, want_doc_lines : bool; + var folders, arch_folders, line_folders : array<string>&) { + return if (!want_rule_docs && !want_arch && !want_doc_lines) + var seen : table<string; void?> + for (root in roots) { + scan_docs_folders(root, want_rule_docs, want_arch, want_doc_lines, folders, arch_folders, line_folders, seen) + } + folders |> sort + arch_folders |> sort + line_folders |> sort +} + +// The .md files in `folder` (non-recursive) on one side of `is_rule_doc_name`: the strays +// LINT025 reports, or the rule documents LINT027 sizes. LAWS.md is append-only ruling +// provenance, never read end to end, so it stays out of the rule-document side. +def collect_folder_mds(folder : string; rule_docs : bool; var out : array<string>) { fio::dir(folder) $(name) { - return if (name == "." || name == ".." || !(name |> ends_with(".md")) || is_rule_doc_name(name)) + return if (name == "." || name == ".." || !(name |> ends_with(".md")) + || is_rule_doc_name(name) != rule_docs + || (rule_docs && name == "LAWS.md")) let full = "{folder}/{name}" let fst = stat(full) if (fst.is_valid && fst.is_reg) { @@ -215,22 +253,14 @@ def collect_stray_rule_docs(folder : string; var out : array<string>) { } } -// Folder-docs pass: one sweep per invocation over the directory roots the run was given. +// Folder-docs pass: one sweep per invocation over the folders the walk found armed. // A ledger or plan parked beside shipped code is read by nobody and goes stale unnoticed, // so a tagged folder admits rule documents only. Returns the finding count. -def run_rule_docs_pass(roots : array<string>; disabled_codes, enabled_codes : table<string>) : int { - return 0 if (key_exists(disabled_codes, "LINT025") - || (!empty(enabled_codes) && !key_exists(enabled_codes, "LINT025"))) - var folders : array<string> - var seen : table<string; void?> - for (root in roots) { - scan_docs_folders(root, folders, seen) - } - folders |> sort +def run_rule_docs_pass(folders : array<string>) : int { var found = 0 for (folder in folders) { var strays : array<string> - collect_stray_rule_docs(folder, strays) + collect_folder_mds(folder, false, strays) strays |> sort for (md in strays) { print("{md}: LINT025: stray .md beside shipped code - rule documents only (REVIEW*.md, ARCHITECTURE*.md, LAWS.md); ledgers and plans live at the module root or plans/ (repo root)\n") @@ -240,6 +270,121 @@ def run_rule_docs_pass(roots : array<string>; disabled_codes, enabled_codes : ta return found } +// Newline count, plus the last line when the file does not end in one. +def count_text_lines(text : string) : int { + return 0 if (empty(text)) + var lines = 0 + var from = 0 + while (true) { + let at = find(text, "\n", from) + break if (at < 0) + lines ++ + from = at + 1 + } + return from < length(text) ? lines + 1 : lines +} + +// Folder-docs pass: a rule document nobody reads end to end has stopped being a contract, so a +// tagged folder's checklists and architecture docs are capped. The fix is always a companion +// split, never a trim - every criterion survives. Returns the finding count. +def run_doc_lines_pass(folders : array<string>) : int { + var found = 0 + for (folder in folders) { + var docs : array<string> + collect_folder_mds(folder, true, docs) + docs |> sort + for (md in docs) { + let lines = count_text_lines(fread(md)) + continue if (lines <= RULE_DOC_MAX_LINES) + print("{md}: LINT027: rule document exceeds {RULE_DOC_MAX_LINES} lines ({lines}) - split it: ARCHITECTURE_<CONCERN>.md companions for architecture docs, routed REVIEW_<KIND>.md companions for checklists\n") + found ++ + } + } + return found +} + +// Forward direction: one finding per citation the shared scanner +// (`utils/common/arch_citations.das`) cannot resolve, in that scanner's words. +def private report_arch_citation(file : string; c : ArchCitation; var docs : ArchDocs) : int { + let reason = citation_failure_reason(docs, c) + return 0 if (empty(reason)) + print("{file}:{c.line}: LINT026: [arch] citation does not resolve - {reason} (\"{c.raw}\")\n") + return 1 +} + +// Both halves of the reverse check in one walk of an armed folder: the .md files that declare +// anchors and the .das files that may cite them. Same pruning as scan_das_files. +def private scan_arch_folder(path : string; var mds, dases : array<string>) { + fio::dir(path) $(name) { + return if (name == "." || name == ".." || name |> starts_with("_") || is_skip_dir(name)) + let full = "{path}/{name}" + let fst = stat(full) + return if (!fst.is_valid) + if (fst.is_dir) { + scan_arch_folder(full, mds, dases) + } elif (fst.is_reg && name |> ends_with(".md")) { + mds |> push(full) + } elif (fst.is_reg && name |> ends_with(".das") && !is_skip_file(name)) { + dases |> push(full) + } + } +} + +// Reverse direction, for a folder whose .lint_config sets [docs] enforce_arch = true: an anchor +// is a promise that some code answers for the section, so an uncited one is either code that +// forgot to say so or a section that was never anyone's contract. +def private run_arch_reverse(folder : string) : int { + var mds : array<string> + var dases : array<string> + scan_arch_folder(folder, mds, dases) + mds |> sort + dases |> sort + var cited : table<string> + for (f in dases) { + let text = fread(f) + continue if (empty(text)) + for (c in collect_arch_citations(f, text)) { + continue if (empty(c.anchor)) + cited |> insert("{c.doc}#{c.anchor}") + } + } + var found = 0 + for (md in mds) { + let doc = arch_path(md) + for (a in collect_md_anchors(md)) { + continue if (cited |> key_exists("{doc}#{a.name}")) + print("{md}:{a.line}: LINT026: anchor \{#{a.name}\} has no [arch] citation - cite it from the owning code, or strip the anchor to demote the section to narrative\n") + found ++ + } + } + return found +} + +// [arch] citation pass: forward over every .das under the run's scan roots, reverse over the +// folders that armed it. A folder pass like LINT025 - once per invocation, not once per file. +def run_arch_pass(roots, arch_folders : array<string>) : int { + var files : array<string> + var cache : table<string; void?> + var skipped = 0 + for (root in roots) { + scan_das_files(root, files, cache, skipped) + } + files |> sort + var found = 0 + var docs : ArchDocs + for (f in files) { + let text = fread(f) + continue if (empty(text)) + for (c in collect_arch_citations(f, text)) { + found += report_arch_citation(f, c, docs) + } + } + for (folder in arch_folders) { + found += run_arch_reverse(folder) + } + return found +} + // Validates a rule code: ^(LINT|PERF|STYLE)\d{3}$. Returns true if shape matches. def is_valid_rule_code(code : string) : bool { let n = length(code) @@ -707,6 +852,31 @@ def tally_result(result : LintResult; var issues : int&; var errors : int&; var } } +// Folder policy, not file policy: the three folder rules answer for the run's scan ROOTS rather +// than its file list, so they run once per invocation, in both driver modes. Returns the +// combined finding count. +def run_folder_passes(docs_roots : array<string>; disabled_codes, enabled_codes : table<string>) : int { + let want_rule_docs = rule_is_on("LINT025", disabled_codes, enabled_codes) + let want_arch = rule_is_on("LINT026", disabled_codes, enabled_codes) + let want_doc_lines = rule_is_on("LINT027", disabled_codes, enabled_codes) + var rule_docs_folders : array<string> + var arch_folders : array<string> + var line_folders : array<string> + collect_docs_folders(docs_roots, want_rule_docs, want_arch, want_doc_lines, + rule_docs_folders, arch_folders, line_folders) + var found = 0 + if (want_rule_docs) { + found += run_rule_docs_pass(rule_docs_folders) + } + if (want_arch) { + found += run_arch_pass(docs_roots, arch_folders) + } + if (want_doc_lines) { + found += run_doc_lines_pass(line_folders) + } + return found +} + // Prints the summary block; returns the process exit code (0 clean, 1 errors, 2 issues). def print_summary(files_count, total_issues, total_errors, total_skipped, total_excluded : int) : int { print("\n--- Summary ---\n") @@ -810,8 +980,7 @@ def main() : int { } } - // Folder policy, not file policy: runs once per invocation, in both driver modes. - total_issues += run_rule_docs_pass(docs_roots, disabled_codes, enabled_codes) + total_issues += run_folder_passes(docs_roots, disabled_codes, enabled_codes) return print_summary(length(files), total_issues, total_errors, total_skipped, total_excluded) } diff --git a/utils/lint/tests/lint026_arch_citations.das b/utils/lint/tests/lint026_arch_citations.das new file mode 100644 index 0000000000..afd3abfaf0 --- /dev/null +++ b/utils/lint/tests/lint026_arch_citations.das @@ -0,0 +1,317 @@ +options gen2 +options indenting = 4 +options no_unused_function_arguments = false + +require dastest/testing_boost public + +require strings +require daslib/fio + +// LINT026 is a FOLDER rule the runner enforces with its own tree walk, not a compile pass, so it +// cannot ride the `expect 50503:N` fixture form the rule fixtures here use - the check is driven +// through the CLI over a planted tree. +// +// Every citation this file PLANTS is assembled from `ANN` rather than spelled out. The forward +// pass reads source and skips only `//` comments, so a citation spelled out inside one of the +// string literals below would be a citation made by this file, against a document +// utils/lint/tests does not carry. + +let ANN = "arch" + +var private _tmp_dir : string +var private _tmp_dir_inited = false + + +def private tmp_root() : string { + if (!_tmp_dir_inited) { + let r = create_temp_directory_result("lint026_arch_citations") + // Fail-fast: an empty path would resolve against the working directory and plant the + // fixture tree in the worktree. + if (!(r is value)) { + panic("could not create temp dir for the LINT026 test: {unsafe(r.error)}") + } + _tmp_dir = unsafe(r.value) + _tmp_dir_inited = true + } + return _tmp_dir +} + + +[finalize] +def cleanup_temp_dir { + if (_tmp_dir_inited && !empty(_tmp_dir)) { + rmdir_rec(_tmp_dir) + } +} + + +// One `[arch]` citation to plant, and the function that carries it. +struct private Cite { + doc : string + anchor : string + fn : string +} + + +def private plant(dir, name, content : string) { + fopen(path_join(dir, name), "wb") $(f) { + if (f != null) { + fwrite(f, content) + } + } +} + + +// A lint-clean .das carrying `cites`, one per function, every function called from main so none +// of them reads as unused. Citation N lands on line 4N - 1. +def private plant_code(dir, name : string; cites : array<Cite>) { + plant(dir, name, build_string() $(var w) { + w |> write("options gen2\n\n") + for (c in cites) { + w |> write("[{ANN}(at=\"{c.doc}#{c.anchor}\")]\ndef {c.fn} \{\n\}\n\n") + } + w |> write("[export]\ndef main \{\n") + for (c in cites) { + w |> write(" {c.fn}();\n") + } + w |> write("\}\n") + }) +} + + +// A .md declaring `anchors`, one heading each - repeat a name to declare it twice. Anchor N +// lands on line 3N. +def private plant_doc(dir, name : string; anchors : array<string>) { + plant(dir, name, build_string() $(var w) { + w |> write("# {name}\n\n") + for (a in anchors) { + w |> write("## Section {a} \{#{a}\}\ntext\n\n") + } + }) +} + + +// argv[0] is the running interpreter - dastest itself is spawned as +// `daslang(.exe) dastest/dastest.das ...`, so it names the binary to spawn. +def private das_exe() : string { + let args <- get_command_line_arguments() + return empty(args) ? "" : args[0] +} + + +def private run_lint_argv(extra : array<string>; root : string; var output : string&) : int { + var argv <- [das_exe(), "-dasroot", get_das_root(), "{get_das_root()}/utils/lint/main.das", "--", "-q"] + argv |> push_from(extra) + argv |> push(root) + return unsafe(popen_argv(argv, 0.0, $(f) { + if (f != null) { + output = fread(f) + } + })) +} + + +def private run_lint(root : string; var output : string&) : int { + let none : array<string> + return run_lint_argv(none, root, output) +} + + +def private count_of(text, sub : string) : int { + var n = 0 + var from = 0 + while (true) { + let at = find(text, sub, from) + break if (at < 0) + n ++ + from = at + length(sub) + } + return n +} + + +// One document, and code citing it four ways: resolving, dangling anchor, dangling document, +// and an anchor the document declares twice. +def private build_forward(name : string) : string { + let dir = path_join(tmp_root(), name) + mkdir_rec(dir) + plant_doc(dir, "ARCHITECTURE.md", ["alpha", "beta", "beta"]) + plant_code(dir, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "alpha", fn = "good"), + Cite(doc = "ARCHITECTURE.md", anchor = "nope", fn = "bad_anchor"), + Cite(doc = "MISSING.md", anchor = "alpha", fn = "bad_file"), + Cite(doc = "ARCHITECTURE.md", anchor = "beta", fn = "dup_anchor")]) + return dir +} + + +// An armed folder whose document declares one cited anchor and one orphan, beside an unarmed +// folder whose anchors nobody owes a citation for. +def private build_reverse(name : string) : string { + let root = path_join(tmp_root(), name) + let armed = path_join(root, "armed") + mkdir_rec(armed) + plant(armed, ".lint_config", "[docs]\nenforce_arch = true\n") + plant_doc(armed, "ARCHITECTURE.md", ["cited", "orphan"]) + plant_code(armed, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "cited", fn = "one"), + Cite(doc = "ARCHITECTURE.md", anchor = "cited", fn = "two")]) + let plain = path_join(root, "plain") + mkdir_rec(plain) + plant_doc(plain, "NOTES.md", ["loose"]) + plant(plain, "ok.das", "options gen2\n\n[export]\ndef main \{\n print(\"ok\\n\")\n\}\n") + return root +} + + +[test] +def test_arch_forward(t : T?) { + t |> run("each unresolved citation fires once, and the resolving one is silent") @(t : T?) { + let dir = build_forward("fwd") + var out : string + let rc = run_lint(dir, out) + t |> equal(count_of(out, "LINT026"), 3, "one finding per unresolved citation\n{out}") + t |> success(find(out, "ok.das:7: LINT026: [arch] citation does not resolve - no such anchor") >= 0, + "the dangling anchor is named at its own line\n{out}") + t |> success(find(out, "ok.das:11: LINT026: [arch] citation does not resolve - no such file") >= 0, + "the dangling document is named at its own line\n{out}") + t |> success(find(out, "ok.das:15: LINT026: [arch] citation does not resolve - anchor appears 2 times") >= 0, + "a twice-declared anchor is not a resolution\n{out}") + t |> success(find(out, "#alpha\")") < 0 || count_of(out, "ARCHITECTURE.md#alpha") == 0, + "the citation that resolves says nothing\n{out}") + t |> equal(rc, 2, "a finding sets the issue exit code\n{out}") + } + t |> run("a citation resolves against the CITING file's folder, not the run root") @(t : T?) { + let root = path_join(tmp_root(), "nested") + let sub = path_join(root, "sub") + mkdir_rec(sub) + plant_doc(root, "ARCHITECTURE.md", ["up"]) + plant_code(sub, "ok.das", [Cite(doc = "../ARCHITECTURE.md", anchor = "up", fn = "only")]) + var out : string + let rc = run_lint(root, out) + t |> equal(count_of(out, "LINT026"), 0, "the relative path walks up from sub/\n{out}") + t |> equal(rc, 0, "and the run is clean\n{out}") + } +} + + +[test] +def test_arch_forward_control(t : T?) { + t |> run("negative control - the forward finding follows the citation") @(t : T?) { + let dir = path_join(tmp_root(), "fwdctl") + mkdir_rec(dir) + plant_doc(dir, "ARCHITECTURE.md", ["alpha"]) + plant_code(dir, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "nope", fn = "only")]) + var out : string + run_lint(dir, out) + t |> equal(count_of(out, "LINT026"), 1, "the dangling anchor fires\n{out}") + // mutate: the same citation, pointed at a section the document declares + plant_code(dir, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "alpha", fn = "only")]) + var fixed : string + let rc = run_lint(dir, fixed) + t |> equal(count_of(fixed, "LINT026"), 0, "a resolving citation is silent\n{fixed}") + t |> equal(rc, 0, "and the run is clean\n{fixed}") + // restore + plant_code(dir, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "nope", fn = "only")]) + var again : string + run_lint(dir, again) + t |> equal(count_of(again, "LINT026"), 1, "restoring the citation brings the finding back\n{again}") + } + t |> run("a commented-out citation is not a citation") @(t : T?) { + let dir = path_join(tmp_root(), "commented") + mkdir_rec(dir) + plant_doc(dir, "ARCHITECTURE.md", ["alpha"]) + plant(dir, "ok.das", build_string() $(var w) { + w |> write("options gen2\n\n") + w |> write("// [{ANN}(at=\"GONE.md#vanished\")]\n") + w |> write("[export]\ndef main \{\n print(\"ok\\n\")\n\}\n") + }) + var out : string + let rc = run_lint(dir, out) + t |> equal(count_of(out, "LINT026"), 0, "a `//` line is prose, not a citation\n{out}") + t |> equal(rc, 0, "and the run is clean\n{out}") + } + t |> run("a structure's citation is shape-checked by lint - the compiler never sees it") @(t : T?) { + let dir = path_join(tmp_root(), "malformed") + mkdir_rec(dir) + plant_doc(dir, "ARCHITECTURE.md", ["alpha"]) + // a structure takes the marker unvalidated (one annotation per NAME, so `arch` is the + // function annotation), which is why lint owns the shape on this side + plant(dir, "ok.das", build_string() $(var w) { + w |> write("options gen2\n\n") + w |> write("[{ANN}(at=\"ARCHITECTURE.md\")]\nstruct S \{\n a : int;\n\}\n\n") + w |> write("[export]\ndef main \{\n var s : S;\n s.a = 1;\n print(\"\{s.a\}\\n\");\n\}\n") + }) + var out : string + let rc = run_lint(dir, out) + t |> equal(count_of(out, "LINT026"), 1, "the anchor-less citation fires\n{out}") + t |> success(find(out, "ok.das:3: LINT026: [arch] citation does not resolve - malformed citation") >= 0, + "reported at the annotation, as malformed\n{out}") + t |> equal(rc, 2, "a finding, not a compile error\n{out}") + } +} + + +[test] +def test_arch_reverse(t : T?) { + t |> run("an uncited anchor fires, and only in an armed folder") @(t : T?) { + let root = build_reverse("rev") + var out : string + let rc = run_lint(root, out) + t |> equal(count_of(out, "LINT026"), 1, "only the orphan is a finding\n{out}") + t |> success(find(out, "armed/ARCHITECTURE.md:6: LINT026: anchor \{#orphan\} has no [arch] citation") >= 0, + "the finding names the anchor at its heading line\n{out}") + t |> success(find(out, "NOTES.md") < 0, "an unarmed folder's anchors stay narrative\n{out}") + t |> success(find(out, "\{#cited\}") < 0, "two citations satisfy one anchor\n{out}") + t |> equal(rc, 2, "a finding sets the issue exit code\n{out}") + } + t |> run("negative control - dropping the citation orphans the anchor it covered") @(t : T?) { + let root = build_reverse("revctl") + let armed = path_join(root, "armed") + var before : string + run_lint(root, before) + t |> equal(count_of(before, "LINT026"), 1, "only the orphan\n{before}") + // mutate: the code stops citing `cited`, so both anchors are owed + plant_code(armed, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "orphan", fn = "one")]) + var swapped : string + run_lint(root, swapped) + t |> equal(count_of(swapped, "\{#cited\}"), 1, "the anchor it stopped citing is now owed\n{swapped}") + t |> equal(count_of(swapped, "\{#orphan\}"), 0, "and the one it took up is covered\n{swapped}") + // restore + plant_code(armed, "ok.das", [Cite(doc = "ARCHITECTURE.md", anchor = "cited", fn = "one"), + Cite(doc = "ARCHITECTURE.md", anchor = "cited", fn = "two")]) + var again : string + run_lint(root, again) + t |> equal(count_of(again, "\{#orphan\}"), 1, "restoring the citations restores the finding\n{again}") + } +} + + +[test] +def test_arch_modes(t : T?) { + t |> run("--disable LINT026 silences the forward direction") @(t : T?) { + let dir = build_forward("fwddis") + var out : string + let rc = run_lint_argv(["--disable", "LINT026"], dir, out) + t |> equal(count_of(out, "LINT026"), 0, "the disabled rule reports nothing\n{out}") + t |> equal(rc, 0, "and the run is clean\n{out}") + } + t |> run("--disable LINT026 silences the reverse direction") @(t : T?) { + let root = build_reverse("revdis") + var out : string + let rc = run_lint_argv(["--disable", "LINT026"], root, out) + t |> equal(count_of(out, "LINT026"), 0, "the armed folder reports nothing\n{out}") + t |> equal(rc, 0, "and the run is clean\n{out}") + } + t |> run("--enable whitelist mode without LINT026 silences it") @(t : T?) { + let dir = build_forward("fwdwl") + var out : string + run_lint_argv(["--enable", "LINT001"], dir, out) + t |> equal(count_of(out, "LINT026"), 0, "whitelist mode without LINT026 reports nothing\n{out}") + } + t |> run("a file positional arms its folder - the PR lint shape") @(t : T?) { + let dir = build_forward("fwdfile") + var out : string + let rc = run_lint(path_join(dir, "ok.das"), out) + t |> equal(count_of(out, "LINT026"), 3, "linting one file still checks its folder\n{out}") + t |> equal(rc, 2, "a finding sets the issue exit code\n{out}") + } +} diff --git a/utils/lint/tests/lint027_rule_doc_lines.das b/utils/lint/tests/lint027_rule_doc_lines.das new file mode 100644 index 0000000000..0d1198642b --- /dev/null +++ b/utils/lint/tests/lint027_rule_doc_lines.das @@ -0,0 +1,187 @@ +options gen2 +options indenting = 4 +options no_unused_function_arguments = false + +require dastest/testing_boost public + +require strings +require daslib/fio + +// LINT027 is a FOLDER rule the runner enforces with its own tree walk, not a compile pass, so +// like LINT025 it is driven through the CLI over a planted tree rather than an `expect` header. +// +// The tree carries four folders: `tagged` (rule_docs_only = true), `arch` (enforce_arch = true, +// the other tag that arms the gate), `off` (both tags false) and `plain` (no config). Two +// findings may come back - one per oversized doc in an armed folder. + +var private _tmp_dir : string +var private _tmp_dir_inited = false + + +def private tmp_root() : string { + if (!_tmp_dir_inited) { + let r = create_temp_directory_result("lint027_rule_doc_lines") + // Fail-fast: an empty path would resolve against the working directory and plant + // the fixture tree in the worktree. + if (!(r is value)) { + panic("could not create temp dir for the LINT027 test: {unsafe(r.error)}") + } + _tmp_dir = unsafe(r.value) + _tmp_dir_inited = true + } + return _tmp_dir +} + + +[finalize] +def cleanup_temp_dir { + if (_tmp_dir_inited && !empty(_tmp_dir)) { + rmdir_rec(_tmp_dir) + } +} + + +def private plant(dir, name, content : string) { + fopen(path_join(dir, name), "wb") $(f) { + if (f != null) { + fwrite(f, content) + } + } +} + + +// A markdown body of exactly `n` lines, each newline-terminated. +def private md_of_lines(n : int) : string { + return build_string() $(var w) { + for (i in range(n)) { + w |> write("line {i}\n") + } + } +} + + +// One folder: its own .lint_config (skipped when `config` is empty), a lint-clean .das so the +// run has something to lint, and the .md files the folder carries, each at its own length. +def private plant_folder(root, name, config : string; docs : array<string>; lengths : array<int>) : string { + let dir = path_join(root, name) + mkdir_rec(dir) + if (!empty(config)) { + plant(dir, ".lint_config", config) + } + plant(dir, "ok.das", "options gen2\n\n[export]\ndef main \{\n print(\"ok\\n\")\n\}\n") + for (doc, n in docs, lengths) { + plant(dir, doc, md_of_lines(n)) + } + return dir +} + + +def private build_tree() : string { + let root = path_join(tmp_root(), "tree") + mkdir_rec(root) + // 301 fires, 300 is at the cap, and LAWS.md is provenance rather than a checklist + plant_folder(root, "tagged", "[docs]\nrule_docs_only = true\n", + ["ARCHITECTURE.md", "REVIEW.md", "LAWS.md"], [301, 300, 400]) + plant_folder(root, "arch", "[docs]\nenforce_arch = true\n", ["REVIEW_GPU.md"], [301]) + plant_folder(root, "off", "[docs]\nrule_docs_only = false\nenforce_arch = false\n", + ["ARCHITECTURE.md"], [400]) + plant_folder(root, "plain", "", ["ARCHITECTURE.md"], [400]) + return root +} + + +// argv[0] is the running interpreter - dastest itself is spawned as +// `daslang(.exe) dastest/dastest.das ...`, so it names the binary to spawn. +def private das_exe() : string { + let args <- get_command_line_arguments() + return empty(args) ? "" : args[0] +} + + +def private run_lint_argv(extra : array<string>; root : string; var output : string&) : int { + var argv <- [das_exe(), "-dasroot", get_das_root(), "{get_das_root()}/utils/lint/main.das", "--", "-q"] + argv |> push_from(extra, [root]) + return unsafe(popen_argv(argv, 0.0, $(f) { + if (f != null) { + output = fread(f) + } + })) +} + + +def private run_lint(root : string; var output : string&) : int { + let none : array<string> + return run_lint_argv(none, root, output) +} + + +def private count_of(text, sub : string) : int { + var n = 0 + var from = 0 + while (true) { + let at = find(text, sub, from) + break if (at < 0) + n ++ + from = at + length(sub) + } + return n +} + + +[test] +def test_rule_doc_line_gate(t : T?) { + t |> run("an oversized rule document fires, once per armed folder") @(t : T?) { + let root = build_tree() + var out : string + let rc = run_lint(root, out) + t |> equal(count_of(out, "LINT027"), 2, "one per oversized doc in an armed folder\n{out}") + t |> success(find(out, "tagged/ARCHITECTURE.md: LINT027: rule document exceeds 300 lines (301)") >= 0, + "the finding names the document and its size\n{out}") + t |> success(find(out, "arch/REVIEW_GPU.md: LINT027") >= 0, + "enforce_arch arms the gate too\n{out}") + t |> equal(rc, 2, "a finding sets the issue exit code\n{out}") + } + t |> run("--enable LINT027 alone still arms the folder walk") @(t : T?) { + let root = build_tree() + var out : string + let rc = run_lint_argv(["--enable", "LINT027"], root, out) + t |> equal(count_of(out, "LINT027"), 2, "the line gate runs with LINT025/026 whitelisted off\n{out}") + t |> equal(rc, 2, "a finding sets the issue exit code\n{out}") + } + t |> run("the cap itself is silent, and so is LAWS.md at any length") @(t : T?) { + let root = build_tree() + var out : string + run_lint(root, out) + t |> success(find(out, "REVIEW.md: LINT027") < 0, "300 lines exactly is at the cap\n{out}") + t |> success(find(out, "LAWS.md: LINT027") < 0, "the rulings sidecar carries no line gate\n{out}") + } + t |> run("an untagged folder is silent however long its documents are") @(t : T?) { + let root = build_tree() + var out : string + run_lint(root, out) + t |> success(find(out, "off/ARCHITECTURE.md") < 0, "both tags false reports nothing\n{out}") + t |> success(find(out, "plain/ARCHITECTURE.md") < 0, "no .lint_config reports nothing\n{out}") + } + t |> run("the tag is a folder property - a subfolder does not inherit it") @(t : T?) { + let root = build_tree() + let nested = path_join(path_join(root, "tagged"), "nested") + mkdir_rec(nested) + plant(nested, "ARCHITECTURE.md", md_of_lines(400)) + var out : string + run_lint(root, out) + t |> success(find(out, "nested/ARCHITECTURE.md") < 0, "no cascade into an untagged subfolder\n{out}") + } + t |> run("--disable LINT027 silences the pass") @(t : T?) { + let root = build_tree() + var out : string + let rc = run_lint_argv(["--disable", "LINT027"], root, out) + t |> equal(count_of(out, "LINT027"), 0, "the disabled rule reports nothing\n{out}") + t |> equal(rc, 0, "and the run is clean\n{out}") + } + t |> run("--enable whitelist mode silences an unlisted LINT027") @(t : T?) { + let root = build_tree() + var out : string + run_lint_argv(["--enable", "LINT001"], root, out) + t |> equal(count_of(out, "LINT027"), 0, "whitelist mode without LINT027 reports nothing\n{out}") + } +} diff --git a/utils/lint/tests/style014_long_comment.das b/utils/lint/tests/style014_long_comment.das index 9f7ba15bca..949a291901 100644 --- a/utils/lint/tests/style014_long_comment.das +++ b/utils/lint/tests/style014_long_comment.das @@ -2,6 +2,11 @@ options gen2 options auto_inline_functions = false // lint fixtures assert SOURCE shapes; splices rewrite them options _comment_hygiene = true +// two firing arms: the 4-line // block and the 4-line //! public docstring +expect 31209:2 + +require daslib/style_lint + //! Module-level header docstring — always allowed regardless of length. //! Even though this `//!` block is more than three lines, STYLE014 must //! treat it as the module-leading docstring and skip it. This is the diff --git a/utils/lint/tests/style015_long_comment_private.das b/utils/lint/tests/style015_long_comment_private.das index 812e64c04a..86f7c9215c 100644 --- a/utils/lint/tests/style015_long_comment_private.das +++ b/utils/lint/tests/style015_long_comment_private.das @@ -2,9 +2,13 @@ options gen2 options auto_inline_functions = false // lint fixtures assert SOURCE shapes; splices rewrite them options _comment_hygiene = true -//! STYLE015: comment block longer than 1 line inside a `def private`. -//! Reason: private functions don't get public docs, so multi-line prose -//! there is dead weight. +//! STYLE015: comment block inside a `def private` over its cap. +//! A `//!` contract doc caps at 3 lines like a public doc; a plain `//` +//! block caps at one WHY line. + +expect 31209:3 + +require daslib/style_lint // good: 1-line `//` inside `def private` is at the cap (== 1, not >). def private good_one_liner() { @@ -29,6 +33,23 @@ def private bad_long_block() { let _ = 1 } +// good: 3-line `//!` contract doc inside `def private` is at the doc cap (== 3, not >). +def private good_private_contract_doc() { + //! Returns nothing. + //! Takes nothing. + //! Never fails. + let _ = 1 +} + +// bad: 4-line `//!` contract doc inside `def private` — over the 3-line doc cap. +def private bad_private_contract_doc() { + //! Returns nothing. + //! Takes nothing. + //! Never fails. + //! Fourth line — exceeds the doc cap. + let _ = 1 +} + // good: same long block, suppressed with `// nolint:STYLE015` on first line. def private good_suppressed() { // nolint:STYLE015 diff --git a/utils/mcp/README.md b/utils/mcp/README.md index 99d0a2d741..ad5b8ca63a 100644 --- a/utils/mcp/README.md +++ b/utils/mcp/README.md @@ -29,6 +29,8 @@ A [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that e | `grep_usage` | Parse-aware symbol search across `.das` files using ast-grep + tree-sitter. Finds identifier occurrences excluding comments and strings. Conditional on `sg` CLI | | `outline` | List all declarations (functions, structs, classes, enums, bitfields, variants, globals, typedefs) in a file or set of files using tree-sitter. Works on broken/incomplete code - no compilation needed. Conditional on `sg` CLI | | `aot` | Generate AOT (ahead-of-time) C++ code for a `.das` file or a single function. Without `function`, returns full AOT output. With `function`, extracts that function's C++ only. Overloaded names return a disambiguation list with mangled names for exact selection | +| `arch_of` | The `[arch]` citations a `.das` file makes, each resolved to its document plus the section text under the cited anchor heading. Optional `symbol` to keep only one declaration's citations. Unresolved citations report LINT026's own reason. Reads source, so a file that does not compile still answers | +| `arch_sites` | The census of an architecture document: every `.das` under the folder that owns it citing it, with file, line and annotated symbol. Grouped under the document's own headings (uncited anchors included) unless the query names an anchor | ### C++ Source & Build Tools diff --git a/utils/mcp/REVIEW.md b/utils/mcp/REVIEW.md index d5115a37a7..50f6bf0c8f 100644 --- a/utils/mcp/REVIEW.md +++ b/utils/mcp/REVIEW.md @@ -1,7 +1,7 @@ # mcp Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture doc: -`README.md`. +`README.md`. Planned work: `ROADMAP.md`. **Never add `mcp` to `DAS_UTILS` or `DAS_UTILS_SHIPPED_EXES` (`utils/CMakeLists.txt`) - run it interpreted through `.mcp.json` instead.** diff --git a/utils/mcp/ROADMAP.md b/utils/mcp/ROADMAP.md index 52d80f6e07..ebf6eb5803 100644 --- a/utils/mcp/ROADMAP.md +++ b/utils/mcp/ROADMAP.md @@ -6,6 +6,8 @@ Future tools for the daslang MCP server, organized by priority and difficulty. | Tool | Description | |---|---| +| `arch_of` | A .das file's [arch] citations, each with the cited section's text | +| `arch_sites` | An architecture document's citation census - every citing symbol, dead anchors and dangling citations included | | `compile_check` | Compile file(s), return errors or success. Supports single file, comma-separated list, or glob pattern | | `list_functions` | List all functions after macro expansion | | `list_types` | List structs, classes, enums, type aliases | diff --git a/utils/mcp/cpp-mcp-setup.md b/utils/mcp/cpp-mcp-setup.md index d1f5d22976..309cb52b5f 100644 --- a/utils/mcp/cpp-mcp-setup.md +++ b/utils/mcp/cpp-mcp-setup.md @@ -15,7 +15,7 @@ cpp-mcp/ bin/cpp-mcp[.exe] # the server (statically linked; AOT-compiled hot paths) utils/mcp/*.das # server sources (compiled at startup) utils/mcp/tools/*.das - utils/common/git_signature.das + utils/common/*.das # shared modules the tools require utils/mcp/daslang-mcp-msvc.cmd # Windows vcvars launcher tree-sitter-daslang/*.yml # ast-grep rule files (loaded at runtime) daslib/ # daslang standard library (sources) diff --git a/utils/mcp/registry_das.das b/utils/mcp/registry_das.das index b370bf5487..3294c02cef 100644 --- a/utils/mcp/registry_das.das +++ b/utils/mcp/registry_das.das @@ -27,6 +27,8 @@ require tools/export_corpus public require tools/judge_duplicates public require tools/find_dupe public require tools/live public +require tools/arch_of public +require tools/arch_sites public def build_das_tools(var reg : array<ToolDef>) { // nolint:STYLE038 — flat tool-registration table, one emplace per tool reg |> emplace(ToolDef( @@ -74,7 +76,7 @@ def build_das_tools(var reg : array<ToolDef>) { // nolint:STYLE038 — flat tool reg |> emplace(ToolDef( tool <- make_tool( "format_file", - "Format .das file(s) using the daslang code formatter. Comment policy comes from the file's folder: each `.lint_config` `[format]` key is taken from the nearest file that declares it (`force_clean_comments = true` strips comments outside the kept set — leading header block, `//!` documentation, `//fmt:` directives, `nolint:` suppressions — fail-closed: a strip must pass compile_check or the original is restored; `format_enabled = false` exempts the tree). No per-call override. Supports single file, comma-separated list, or glob pattern (e.g. 'daslib/*.das'). Returns JSON array with file, status ('formatted' | 'already_formatted' | 'reverted' | 'skipped' | 'error' | 'rescue_advisory'), message, and comments_stripped; when any file in the batch had comments stripped, a final 'rescue_advisory' entry (no file) says to run the rescue-bot agent over the strip diff of the comments_stripped entries before committing.", + "Format .das file(s) using the daslang code formatter. Comment policy comes from the file's folder: each `.lint_config` `[format]` key is taken from the nearest file that declares it (`force_clean_comments = true` strips comments outside the kept set — leading header block, `//!` documentation, `//fmt:` directives, `nolint:` suppressions — fail-closed: a strip must pass compile_check or the original is restored; `format_enabled = false` exempts the tree). No per-call override. Supports single file, comma-separated list, or glob pattern (e.g. 'daslib/*.das'). Returns JSON array with file, status ('formatted' | 'already_formatted' | 'reverted' | 'skipped' | 'error' | 'rescue_advisory'), message, and comments_stripped; when any file in the batch had comments stripped, a final 'rescue_advisory' entry (no file) says to run the harvester agent over the strip diff of the comments_stripped entries before committing.", { "file" => PropertySchema(_type = "string", description = "Path to .das file, comma-separated paths, or glob pattern (e.g. 'dir/*.das')"), "project" => PROJECT_PROP, @@ -384,6 +386,29 @@ def build_das_tools(var reg : array<ToolDef>) { // nolint:STYLE038 — flat tool required <- ["paths"], arg_names <- ["paths", "out", "model", "threshold", "max_clusters", "dry_run"], handler = @@(arg1, arg2, arg3, arg4, arg5, arg6, project, project_root : string; load_modules : array<string>) => do_find_dupe(arg1, arg2, arg3, arg4, arg5, arg6))) + reg |> emplace(ToolDef( + tool <- make_tool( + "arch_of", + "Read direction of the [arch] annotation: the architecture-document citations a .das file makes, each resolved to its document and the SECTION TEXT under the cited anchor heading (heading line through the last line before the next heading of the same or a higher level). A citation that does not resolve reports LINT026's own reason ('no such file', 'no such anchor', 'anchor appears N times', 'malformed citation'). Reads source, not the AST — works on a file that does not compile.", + { + "file" => PropertySchema(_type = "string", description = "Path to the .das file (repo-relative or absolute)"), + "symbol" => PropertySchema(_type = "string", description = "Optional function/struct/class name — show only the citations annotating it") + }, + ["file"]), + required <- ["file"], + arg_names <- ["file", "symbol"], + handler = @@(arg1, arg2, arg3, arg4, arg5, arg6, project, project_root : string; load_modules : array<string>) => do_arch_of(arg1, arg2))) + reg |> emplace(ToolDef( + tool <- make_tool( + "arch_sites", + "Write direction of the [arch] annotation: the census of an architecture document — every .das citing it, with file, line, and the annotated symbol. Searched from the folder that OWNS the document (its own subtree), the scope LINT026's reverse pass walks. Without an anchor, the census is grouped under the document's own headings in document order, uncited anchors included; with one, only that section's citations are listed.", + { + "doc" => PropertySchema(_type = "string", description = "Path to the .md document, optionally with an anchor: 'utils/lint/ARCHITECTURE.md' or 'utils/lint/ARCHITECTURE.md#citations'") + }, + ["doc"]), + required <- ["doc"], + arg_names <- ["doc"], + handler = @@(arg1, arg2, arg3, arg4, arg5, arg6, project, project_root : string; load_modules : array<string>) => do_arch_sites(arg1))) reg |> emplace(ToolDef( tool <- make_tool( "live_status", diff --git a/utils/mcp/test_tools.das b/utils/mcp/test_tools.das index 2f34440828..493255d34a 100644 --- a/utils/mcp/test_tools.das +++ b/utils/mcp/test_tools.das @@ -40,6 +40,8 @@ require tools/program_log public require tools/live public require tools/detect_duplicates public require tools/export_corpus public +require tools/arch_of public +require tools/arch_sites public def parse_result(result : string; var text : string&; var is_error : bool&) : bool { var err : string @@ -736,7 +738,7 @@ def test_format_file_comment_policy(t : T?) { t |> success(ok, "parse result JSON") t |> success(find(text, "comments stripped") >= 0, "opted-in folder stripped -- text={text}") t |> success(find(text, "rescue_advisory") >= 0, "strip batch appends the rescue advisory -- text={text}") - t |> success(find(text, "rescue-bot") >= 0, "advisory names the rescue-bot agent") + t |> success(find(text, "harvester") >= 0, "advisory names the harvester agent") let after = fread(dst) t |> success(find(after, "narration") < 0, "comments removed on disk") t |> success(find(after, "//! doc comment survives") >= 0, "docs kept on disk") @@ -3321,3 +3323,147 @@ def test_project_root_invalid_dir(t : T?) { "should still report missing prereq -- text={text}") } } + +// Every citation these fixtures plant is assembled from ARCH_ANN rather than spelled out: the +// LINT026 forward pass reads source and skips only `//` comments, so a citation spelled out in +// one of the string literals below would be a citation made by THIS file, against a document +// utils/mcp does not carry. +let ARCH_ANN = "arch" + +// A temp folder holding a document and the code citing it. Both sides have to live under one +// root because arch_sites walks the subtree the DOCUMENT owns. `code.das` cites twice: once +// into a section the document declares, once into an anchor it does not. +def private plant_arch_fixture(name : string; var err : string&) : string { + let dir = create_temp_directory(name, err) + return "" if (empty(dir)) + fwrite(path_join(dir, "ARCHITECTURE.md"), build_string() $(var w) { + w |> write("# Doc\n\n## Tokenizer \{#tokenizer\}\n\nThe tokenizer section body.\n\n") + w |> write("### Details \{#details\}\n\nA subsection stays inside its parent.\n\n") + w |> write("## Buffers \{#buffers\}\n\nThe buffers section body.\n") + }) + fwrite(path_join(dir, "code.das"), build_string() $(var w) { + w |> write("options gen2\n\n") + w |> write("[{ARCH_ANN}(at=\"ARCHITECTURE.md#tokenizer\")]\ndef scan_tokens \{\n\}\n\n") + w |> write("[{ARCH_ANN}(at=\"ARCHITECTURE.md#gone\")]\ndef dangling \{\n\}\n") + }) + return dir +} + +[test] +def test_arch_of(t : T?) { + t |> run("a resolving citation carries the cited section text") <| @(t : T?) { + var terr : string + let dir = plant_arch_fixture("mcp_arch_of_test", terr) + if (empty(dir)) { + t |> failure("temp dir: {terr}") + return + } + var text : string + var is_error = false + let ok = parse_result(do_arch_of(path_join(dir, "code.das")), text, is_error) + t |> success(ok, "parse result JSON") + t |> success(!is_error, "a readable file is not a tool failure -- text={text}") + t |> success(find(text, "code.das:3:") >= 0, "the citation is reported at its own line -- text={text}") + t |> success(find(text, "symbol: scan_tokens") >= 0, "the annotated function is named -- text={text}") + t |> success(find(text, "## Tokenizer \{#tokenizer\}") >= 0, "the section opens at its heading -- text={text}") + t |> success(find(text, "The tokenizer section body.") >= 0, "the section body is returned -- text={text}") + t |> success(find(text, "A subsection stays inside its parent.") >= 0, + "a deeper heading does not end the section -- text={text}") + t |> success(find(text, "The buffers section body.") < 0, + "the next same-level heading ends it -- text={text}") + rmdir_rec(dir) + } + t |> run("a dangling citation reports LINT026's reason") <| @(t : T?) { + var terr : string + let dir = plant_arch_fixture("mcp_arch_dangling_test", terr) + if (empty(dir)) { + t |> failure("temp dir: {terr}") + return + } + var text : string + var is_error = false + parse_result(do_arch_of(path_join(dir, "code.das")), text, is_error) + t |> success(find(text, "does not resolve - no such anchor") >= 0, + "lint's own wording, so one tool's answer reads as the other's -- text={text}") + t |> success(find(text, "symbol: dangling") >= 0, "the dangling citation still names its symbol -- text={text}") + rmdir_rec(dir) + } + t |> run("the symbol filter keeps one citation") <| @(t : T?) { + var terr : string + let dir = plant_arch_fixture("mcp_arch_symbol_test", terr) + if (empty(dir)) { + t |> failure("temp dir: {terr}") + return + } + var text : string + var is_error = false + parse_result(do_arch_of(path_join(dir, "code.das"), "scan_tokens"), text, is_error) + t |> success(find(text, "symbol: scan_tokens") >= 0, "the named symbol's citation is kept -- text={text}") + t |> success(find(text, "symbol: dangling") < 0, "the other symbol's citation is dropped -- text={text}") + rmdir_rec(dir) + } + t |> run("a missing file is a tool failure") <| @(t : T?) { + var text : string + var is_error = false + parse_result(do_arch_of("nonexistent_arch_file_12345.das"), text, is_error) + t |> success(is_error, "should be error -- text={text}") + t |> success(find(text, "not found") >= 0, "should mention file not found -- text={text}") + } + t |> run("a file citing nothing says so") <| @(t : T?) { + var text : string + var is_error = false + parse_result(do_arch_of(fixture_path("_fixture_valid.das")), text, is_error) + t |> success(!is_error, "no citations is not a failure -- text={text}") + t |> success(find(text, "No [arch] citations") >= 0, "should report an empty scan -- text={text}") + } +} + +[test] +def test_arch_sites(t : T?) { + t |> run("the census finds the citations planted in the document's subtree") <| @(t : T?) { + var terr : string + let dir = plant_arch_fixture("mcp_arch_sites_test", terr) + if (empty(dir)) { + t |> failure("temp dir: {terr}") + return + } + var text : string + var is_error = false + let ok = parse_result(do_arch_sites(path_join(dir, "ARCHITECTURE.md")), text, is_error) + t |> success(ok, "parse result JSON") + t |> success(!is_error, "an existing document is not a tool failure -- text={text}") + t |> success(find(text, "\{#tokenizer\} (line 3) - 1 citation(s)") >= 0, + "the cited anchor is grouped at its heading line -- text={text}") + t |> success(find(text, "code.das:3 scan_tokens") >= 0, + "the site names file, line and symbol -- text={text}") + t |> success(find(text, "\{#buffers\} (line 11) - no citations") >= 0, + "an uncited anchor is part of the census -- text={text}") + t |> success(find(text, "\{#gone\}") >= 0, + "a citation naming an undeclared anchor is surfaced, not dropped -- text={text}") + t |> success(find(text, "2 citation(s) in 1 file(s).") >= 0, "the tally counts both -- text={text}") + rmdir_rec(dir) + } + t |> run("an anchor query narrows the census to that section") <| @(t : T?) { + var terr : string + let dir = plant_arch_fixture("mcp_arch_anchor_test", terr) + if (empty(dir)) { + t |> failure("temp dir: {terr}") + return + } + var text : string + var is_error = false + let doc = path_join(dir, "ARCHITECTURE.md") + parse_result(do_arch_sites("{doc}#tokenizer"), text, is_error) + t |> success(find(text, "code.das:3 scan_tokens") >= 0, "the section's own site is listed -- text={text}") + t |> success(find(text, "1 citation(s) in 1 file(s).") >= 0, "and only it -- text={text}") + t |> success(find(text, "\{#buffers\}") < 0, "other anchors stay out -- text={text}") + rmdir_rec(dir) + } + t |> run("a missing document is a tool failure") <| @(t : T?) { + var text : string + var is_error = false + parse_result(do_arch_sites("nonexistent_arch_doc_12345.md"), text, is_error) + t |> success(is_error, "should be error -- text={text}") + t |> success(find(text, "Document not found") >= 0, "should mention the missing document -- text={text}") + } +} diff --git a/utils/mcp/tools/arch_common.das b/utils/mcp/tools/arch_common.das new file mode 100644 index 0000000000..671ae2f3b5 --- /dev/null +++ b/utils/mcp/tools/arch_common.das @@ -0,0 +1,102 @@ +options gen2 + +module arch_common + +require common public +require outline public +require ../../common/arch_citations.das public +require daslib/fio +require strings +require daslib/strings_boost + +//! What the `arch_of` and `arch_sites` tools add to the shared `[arch]` scanner: the text under a +//! cited anchor, the symbol a citation annotates, and the .das tree the census walks. +//! +//! Resolution itself - the citation regex, the failure strings, the anchor rule - is +//! `utils/common/arch_citations.das`, the same code LINT026 runs, so a tool and the rule never +//! disagree about whether a citation resolves. + +// The cited section: the anchor's heading line through the last line before the next heading of +// the same or a higher level, trailing blank lines dropped. "" when the document declares no +// such anchor. +def section_text(doc, anchor : string) : string { + let text = fread(doc) + return "" if (empty(text)) + let lines <- split(text, "\n") + var start = -1 + var level = 0 + for (i in range(length(lines))) { + let a = anchor_of_heading(lines[i]) + if (a.name == anchor) { + start = i + level = a.level + break + } + } + return "" if (start < 0) + var stop = length(lines) + for (i in range(start + 1, length(lines))) { + let lv = heading_level(lines[i]) + if (lv > 0 && lv <= level) { + stop = i + break + } + } + while (stop > start + 1 && empty(strip(lines[stop - 1]))) { + stop -- + } + return build_string() $(var w) { + for (i in range(start, stop)) { + // split() leaves the '\r' of a CRLF document on the line + let raw = lines[i] + w |> write(raw |> ends_with("\r") ? slice(raw, 0, length(raw) - 1) : raw) + w |> write("\n") + } + } +} + +// The name a declaration line introduces, "" when the line declares nothing. Keyword plus the +// modifiers `outline` already knows how to skip, which is all a navigation aid needs. +def private declared_name(t : string) : string { + return extract_name_from_text(t, "function") if (t |> starts_with("def ")) + return extract_name_from_text(t, "struct") if (t |> starts_with("struct ") || t |> starts_with("class ")) + return "" +} + +// The symbol a citation on `line` (1-based) annotates, "" when the source does not say. The +// marker sits ABOVE its declaration, so the answer is the first declaration below it, past the +// rest of the annotation block; a citation found anywhere else - a comment, a string literal - +// falls back to the declaration above it, which is the enclosing one. +def annotated_symbol(lines : array<string>; line : int) : string { + var i = line - 1 + while (i < length(lines)) { + let t = lines[i] |> strip + let name = declared_name(t) + return name if (!empty(name)) + break if (!(empty(t) || t |> starts_with("//") || t |> starts_with("[") || t |> ends_with("]") || t |> ends_with(","))) + i ++ + } + var j = line - 2 + while (j >= 0) { + let name = declared_name(lines[j] |> strip) + return name if (!empty(name)) + j -- + } + return "" +} + +// Every .das under `path`, pruned the way LINT026's reverse walk prunes: `_`-prefixed and +// dot-prefixed names are fixtures and tooling, and `builtin.das` is not source anyone cites. +def scan_das_tree(path : string; var out : array<string>) { + fio::dir(path) $(name) { + return if (name == "." || name == ".." || name |> starts_with("_") || name |> starts_with(".")) + let full = "{path}/{name}" + let fst = stat(full) + return if (!fst.is_valid) + if (fst.is_dir) { + scan_das_tree(full, out) + } elif (fst.is_reg && name |> ends_with(".das") && name != "builtin.das") { + out |> push(full) + } + } +} diff --git a/utils/mcp/tools/arch_of.das b/utils/mcp/tools/arch_of.das new file mode 100644 index 0000000000..ff36b383be --- /dev/null +++ b/utils/mcp/tools/arch_of.das @@ -0,0 +1,69 @@ +options gen2 + +module arch_of + +require common public +require arch_common public +require daslib/fio +require strings +require daslib/strings_boost + +//! `arch_of` - the `[arch]` citations one file makes, each with the document section it cites. +//! +//! The read direction of the annotation: given code, show the architecture text it answers for, +//! so the section can be read without opening the document and counting headings. Resolution is +//! LINT026's, failures included - a citation this tool cannot follow is one lint will report. + +// Indent a section so it reads as quoted text rather than as part of the report. +def private write_section(var w : StringBuilderWriter; body : string) { + for (line in split(body, "\n")) { + w |> write(empty(line) ? "\n" : " {line}\n") + } +} + +def private write_citation(var w : StringBuilderWriter; var docs : ArchDocs; + rel : string; c : ArchCitation; symbol : string) { + w |> write("{rel}:{c.line}: arch(at=\"{c.raw}\")\n") + if (!empty(symbol)) { + w |> write(" symbol: {symbol}\n") + } + let reason = citation_failure_reason(docs, c) + if (!empty(reason)) { + w |> write(" does not resolve - {reason}\n") + if (!empty(c.doc)) { + w |> write(" doc: {c.doc}\n") + } + return + } + w |> write(" doc: {c.doc}\n") + let body = section_text(c.doc, c.anchor) + w |> write(" section:\n") + write_section(w, body) +} + +def do_arch_of(file : string; symbol : string = "") : string { + return make_tool_result("'file' argument is required", true) if (empty(file)) + let resolved = resolve_path(file) + let st = stat(resolved) + return make_tool_result("File not found: {file}", true) if (!(st.is_valid && st.is_reg)) + let text = fread(resolved) + return make_tool_result("File is empty or unreadable: {file}", true) if (empty(text)) + let cites <- collect_arch_citations(arch_path(resolved), text) + let rel = make_relative_path(resolved, get_das_root()) + return make_tool_result("No [arch] citations in {rel}") if (empty(cites)) + let lines <- split(text, "\n") + var docs : ArchDocs + var shown = 0 + let body = build_string() $(var w) { + for (c in cites) { + let owner = annotated_symbol(lines, c.line) + continue if (!empty(symbol) && owner != symbol) + shown ++ + write_citation(w, docs, rel, c, owner) + } + } + if (shown == 0) { + return make_tool_result("No [arch] citation on '{symbol}' in {rel} ({length(cites)} citation(s) in the file)") + } + return make_tool_result("{body}{shown} citation(s) shown.") +} diff --git a/utils/mcp/tools/arch_sites.das b/utils/mcp/tools/arch_sites.das new file mode 100644 index 0000000000..11bf4d7b05 --- /dev/null +++ b/utils/mcp/tools/arch_sites.das @@ -0,0 +1,130 @@ +options gen2 + +module arch_sites + +require common public +require arch_common public +require daslib/fio +require strings +require daslib/strings_boost + +//! `arch_sites` - the census of an architecture document: who cites it, and from where. +//! +//! The write direction of the annotation, and the question to ask before editing a section: +//! which code answers for it. The search subtree is the folder that OWNS the document, which is +//! the scope a citation's relative path can reach and the scope LINT026's reverse pass walks. + +// One citation of the queried document. +struct private Site { + file : string + line : int + symbol : string + anchor : string +} + +def private write_sites(var w : StringBuilderWriter; sites : array<Site>; which : array<int>) { + for (i in which) { + let s = sites[i] + let tail = empty(s.symbol) ? "" : " {s.symbol}" + w |> write(" {s.file}:{s.line}{tail}\n") + } +} + +// Every citation of `doc` (of its `anchor` alone, when one was named) made by a .das under +// `folder`, in file then line order. +def private collect_sites(folder, doc, anchor : string) : array<Site> { + var out : array<Site> + var files : array<string> + scan_das_tree(folder, files) + files |> sort + let root = get_das_root() + for (f in files) { + let text = fread(f) + continue if (empty(text)) + let cites <- collect_arch_citations(arch_path(f), text) + continue if (empty(cites)) + let lines <- split(text, "\n") + let rel = make_relative_path(f, root) + for (c in cites) { + continue if (c.doc != doc || (!empty(anchor) && c.anchor != anchor)) + out |> emplace(Site(file = rel, line = c.line, anchor = c.anchor, + symbol = annotated_symbol(lines, c.line))) + } + } + return <- out +} + +// The census grouped under the document's own headings, in document order, followed by the +// citations naming an anchor the document does not declare - a rename that got away. +def private write_census(var w : StringBuilderWriter; doc : string; sites : array<Site>) { + var by_anchor : table<string; array<int>> + for (i in range(length(sites))) { + by_anchor[sites[i].anchor] |> push(i) // nolint:PERF006 - one bucket per anchor cited, so no bucket's size is known up front + } + var declared : table<string> + for (a in collect_md_anchors(doc)) { + continue if (declared |> key_exists(a.name)) + declared |> insert(a.name) + // guarded: reading a mutable table by key would INSERT the anchor as a bucket of its own + if (!(by_anchor |> key_exists(a.name))) { + w |> write("\{#{a.name}\} (line {a.line}) - no citations\n") + continue + } + let count = length(by_anchor[a.name]) + w |> write("\{#{a.name}\} (line {a.line}) - {count} citation(s)\n") + write_sites(w, sites, by_anchor[a.name]) + } + var orphans : array<string> + for (s in sites) { + continue if (declared |> key_exists(s.anchor) || find_index(orphans, s.anchor) >= 0) + orphans |> push(s.anchor) + } + return if (empty(orphans)) + w |> write("\ncitations naming an anchor this document does not declare:\n") + for (name in orphans) { + w |> write("\{#{name}\}\n") + write_sites(w, sites, by_anchor[name]) + } +} + +def private anchor_declared(doc, name : string) : bool { + for (a in collect_md_anchors(doc)) { + return true if (a.name == name) + } + return false +} + +def do_arch_sites(doc : string) : string { + return make_tool_result("'doc' argument is required", true) if (empty(doc)) + let hash = find(doc, "#") + let anchor = hash >= 0 ? slice(doc, hash + 1) : "" + let doc_arg = hash >= 0 ? slice(doc, 0, hash) : doc + return make_tool_result("'doc' names an anchor but no document: {doc}", true) if (empty(doc_arg)) + let resolved = arch_path(resolve_path(doc_arg)) + let st = stat(resolved) + return make_tool_result("Document not found: {doc_arg}", true) if (!(st.is_valid && st.is_reg)) + let folder = dir_name(resolved) + let sites <- collect_sites(folder, resolved, anchor) + var files : array<string> + for (s in sites) { + continue if (find_index(files, s.file) >= 0) + files |> push(s.file) + } + let title = empty(anchor) ? resolved : "{resolved} \{#{anchor}\}" + let searched = make_relative_path(folder, get_das_root()) + return make_tool_result(build_string() $(var w) { + w |> write("{title}\nsearched: {searched}\n") + if (empty(anchor)) { + w |> write("\n") + write_census(w, resolved, sites) + } else { + if (!anchor_declared(resolved, anchor)) { + w |> write("note: this document declares no such anchor - a citation naming it does not resolve\n") + } + w |> write("\n") + let which <- [for (i in range(length(sites))); i] + write_sites(w, sites, which) + } + w |> write("\n{length(sites)} citation(s) in {length(files)} file(s).\n") + }) +} diff --git a/utils/mcp/tools/format_file.das b/utils/mcp/tools/format_file.das index 899e73d487..d966cdfd3b 100644 --- a/utils/mcp/tools/format_file.das +++ b/utils/mcp/tools/format_file.das @@ -107,7 +107,7 @@ def do_format_file(file : string; project : string = ""; project_root : string = } if (stripped_count > 0) { results |> push(FormatResult(status = "rescue_advisory", - message = "comments were deleted from {stripped_count} file(s) — before committing, run the rescue-bot agent over the strip diff of the entries with comments_stripped = true")) + message = "comments were deleted from {stripped_count} file(s) — before committing, run the harvester agent over the strip diff of the entries with comments_stripped = true")) } return make_tool_result(sprint_json(results, false), any_errors) }