Reconcile qwen4exp NextN/MTP draft head: adopt gs qwen4exp-mtp lineage over hand-rolled cdd11021b (#763) by ddvnguyen · Pull Request #120 · ddvnguyen/llama.cpp

ddvnguyen · 2026-09-12T14:49:26Z

STATUS: OPEN FOR REVIEW. The two -Werror breaks are fixed in c101306ef; issue #121 (grouped-MoE target verification on the plain kv-cache splitter) is fixed in f6301d4ee. Local build clean; tools/server/tests/unit/test_speculative.py 10/10 PASS on the rig; test-moe-cache, test-batch-alloc (34/34) and test-arg-parser PASS; both GPU E2E probes PASS (qwen35 production-shaped 51.4% accept, qwen4exp shared-draft borrow 44.7% accept). CI re-running on f6301d4ee.

Update 2026-09-12: #121 fixed in `f6301d4ee`

d659bcefa added a target-verification validator that requires a span-grouped ubatch
(n_seq_tokens == verification_span). The server marks that span for every spec method, but the
plain llama_kv_cache path splits with split_simple, which reports one sequence per row. So
single-stream models failed ubatch does not match the validated execution intent and the server
returned HTTP 500. The grouped CUDA certificate imposes the same grouped-row shape.

Fix (localized, no qwen4exp-specific branches):

src/llama-batch.{h,cpp}: add a verification_span hint to the batch allocr. split_simple
delegates to split_equal when the batch is an atomic target-verification span, keeping the rows
grouped by sequence (the same shape the hybrid memory path already produces).
src/llama-context.cpp: set the span hint and require/forward the MAIN target-verification intent
and certificate only when grouped MoE execution is active
(required_grouped_execution_flags(moe_expert_cache_slots, moe_required_grouped_execution_supported) != NONE).
Non-grouped backends keep the plain pre-d659bcefa split. DRAFT/MTP intents are unchanged.

Blast radius: only grouped MoE execution changes; non-grouped backends behave exactly as before.

Unrelated observation

A local LLAMA_FATAL_WARNINGS=ON build (GCC 15) trips pre-existing warnings in
ggml/src/ggml-cuda/moe-cache.cu and staged-input.cu; the CI CUDA job uses a GCC 13 container.
Not caused by this change; worth a separate issue if CI reproduces it.

What this is

baseline-flash-next and this branch (fork/763-qwen4exp-mtp, tip efd26f235) are two competing answers to the same question: how to give qwen4exp (Qwen3.8-Flash-Next) a working NextN/MTP speculative draft head for the baseline-flash-next line.

Base, baseline-flash-next (cdd11021b) = the old Hydra Simplify to include lower-case windows.h always, fix compile on mingw32 ggml-org/llama.cpp#747 chain plus one hand-rolled feat(qwen4exp): add NextN/MTP draft-head graph with shared-tensor borrowing commit.
This branch = the GenerelSchwerz/llama.cpp qwen4exp-mtp lineage (CUDA MoE expert cache, grouped MoE drafting, Flash-Next MTP, upstream merges) plus the same 5 Hydra Simplify to include lower-case windows.h always, fix compile on mingw32 ggml-org/llama.cpp#747 commits replayed, plus this reconciliation merge.

This PR reconciles the two and proposes the gs qwen4exp-mtp lineage as the basis.

Review as a base/lineage adoption, not a small patch: 486 commits, 406 files, +64,978 / -8,687 vs baseline-flash-next.

Decision: primary basis = the gs `qwen4exp-mtp` lineage

Diff findings

The two draft-head implementations are the same code:

identical LLM_TENSOR_NEXTN_HC_HEAD_{NORM,DOWN,UP} enum values and "blk.%d.nextn.hc_head_*" names,
identical llama_layer_nextn.hc_head_{norm,down,up} fields,
identical llama_model_qwen4exp::graph shape (protected, no_build_t tag ctor, graph_mtp : public graph),
the qwen4exp.cpp draft graph is the same.

The only real difference is how a draft borrows the target's token embeddings / LM head:

	gs lineage	`cdd11021b`
when	model load	graph build
how	opt-in GGUF key `{arch}.nextn_shared_target_tensors` + `llama_model_params.model_shared`, resolved by `llama_model_loader::borrow_shared_tensor()`	`qwen4exp_shared_model(cparams.ctx_other, ...)`, qwen4exp-only
scope	`token_embd`, `output`, `output_norm`	qwen4exp only
safety	gated; shape-checked; refuses standalone load	relies on `ctx_other`; `token_embd` marked optional for `mtp_only`
wire-up	`--mtp-shared-embd` (convert) + `mparams.model_shared = model_tgt` (common)	none; graph-time

What was ported from `cdd11021b`: nothing

Every cdd delta is either already present in gs or superseded by it:

src/llama-model.cpp mtp_on_hybrid_qwen QWEN4EXP and the NEXTN_HC_HEAD_* arch changes are already in gs unchanged.
The 3 conflicts are comment-only (src/llama-model.h, src/models/models.h) or gs-correct (src/models/qwen4exp.cpp: token_embd flags=0, n_ff_exp(il), loader-based borrow).
common/speculative.cpp is_mem_shared gemma gate and src/llama-context.cpp QWEN4EXP ctx_other registration are workarounds for cdd's graph-time borrow and are not carried.

Why the is_mem_shared gate is not needed here. gs resets cparams.ctx_other = nullptr in llama_context and only sets it for Gemma4Assistant / EAGLE3 / DFlash (src/llama-context.cpp:791-808). For qwen4exp it stays null, so llama_get_ctx_other(ctx_dft) == ctx_tgt is false and is_mem_shared is already correct. cdd's gate exists only because cdd does set ctx_other for qwen4exp (to power its graph-time borrow) and then has to gate is_mem_shared back off. Carrying the gate into gs would be redundant and would change behavior for gemma4/eagle3/dflash.

Resulting tree

git rev-parse HEAD^{tree} == git rev-parse efd26f235^{tree} (df72c2736). The merge takes the gs side in full; cdd11021b is recorded as an ancestor (not silently dropped) but contributes no tree change.

Behavior differences a reviewer should check

Base adoption, not a patch — 486 commits. Review as "replace baseline-flash-next's draft head with the gs lineage".
Draft-sidecar loading — gs requires the exporter to set nextn_shared_target_tensors (convert_hf_to_gguf.py --mtp-shared-embd) for a draft that does not ship token_embd/output. A self-contained draft (Hydra's current Qwen3.8-27B-*-MTP.gguf ship token_embd) loads unchanged. cdd's auto-detecting graph-time fallback is not carried, so any existing draft-only GGUF without the flag is now refused at load instead of silently borrowed.
MTP memory sharing — qwen4exp keeps gs's is_mem_shared=false single-head path. If anyone expected cdd's gated behavior, please confirm.
Scope — supersedes cdd11021b's graph-time borrow entirely.

Not fully confident in / did not verify

No end-to-end run. The rig is occupied; verification was build + static analysis only. I did not run qwen4exp MTP on this tree.
The load-time borrow path (nextn_shared_target_tensors) is gs's and is untested by me against Hydra's GGUF exports.
The is_mem_shared judgment comes from reading gs's ctx_other lifecycle, not a runtime test. If a reviewer knows of a qwen4exp path where ctx_other is non-null in gs, revisit.
cdd's original motivation (a specific loop/crash on the old base) is not documented in its commit; I found no case where gs needs cdd's mechanism, but absence of evidence is not proof.

Verification done

Host build of efd26f235 (CUDA 13.2.2, CMAKE_CUDA_ARCHITECTURES=86;120): llama-server links; --help shows --moe-expert-cache-size, --spec-draft-moe-expert-cache-size, --load-mode, --parallel-ctx-threshold.
Merge tree is byte-identical to that built tree.
git merge-tree conflict probe + full git merge inspection (3 conflicts, all resolved to gs).

CI status

Head c101306ef fixes -Werror=sign-compare (ggml/src/ggml-cuda/ggml-cuda.cu:3053) and -Wstringop-overflow (ggml/src/ggml-cuda/ggml-cuda.cu:5509); verified by compiling ggml-cuda.cu with the exact CUDA CI flags (gcc 13, LLAMA_FATAL_WARNINGS=ON) -> 0 errors.
Still red: test_speculative.py::test_with_and_without_draft (HTTP 500) - blocked by [engine] draft-simple 500: d659bcefac target-verification intent rejects split_simple ubatch (blocks PR #120) #121.
Non-code failure: Windows HF 429 rate-limit while downloading test models.
Do not merge until [engine] draft-simple 500: d659bcefac target-verification intent rejects split_simple ubatch (blocks PR #120) #121 is resolved and CI is green.

Assisted-by: Codex

Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.

Adds --spec-type draft-mtp support for Qwen3.8-Flash-Next. The MTP head folds the next token's embedding into the trunk's wide hyper-connection residual, runs one trunk-style block (dense attention + MoE) over it, and collapses the result with its own mixer before reusing the trunk's LM head. - read nextn_predict_layers so n_layer() excludes the MTP block - load the trailing block through the existing trunk path: is_recr() and is_ple() are already false past the trunk, so it needs no special casing - eh_proj fuses the checkpoint's fc_embedding and fc_hidden side by side, so one matmul computes fc_embedding@e + fc_hidden@h - the head carries its own hyper-connection mixer, mirroring the trunk's hc_head_*, which stands in for the output norm qwen4exp does not have - export the wide pre-collapse residual as t_h_nextn from both graphs, so the driver can feed it back for the next draft step - route MTP contexts to a plain KV cache filtered to the trailing layer The draft block attends densely for now: the trunk's QSA only prunes context past a 2048-token budget, so dense is a numerical superset and drafts are verified either way. Indexer tensors are still loaded.

The MTP block is one trunk-shaped block (dense attention + MoE wrapped in hyper-connections) plus a head-level combiner, so once _QwenMtpMixin renames mtp.layers.0.* to the trailing block index its tensors ride the existing qwen4exp mappings unchanged. Two head-level pieces need handling: - fc_embedding and fc_hidden fuse into the eh_proj the shared NextN code expects, since W_e@e + W_h@h == [W_e|W_h] @ concat(e, h) - mtp.hyper_connection_mixer.* is the head's own copy of the trunk's hc_head_* output mixer, unindexed in the checkpoint and per-block in the GGUF compress_ratios is read with length block_count, so it gains a trailing 0 for the MTP block, which attends densely. --no-nextn drops the head; --mtp exports it on its own.

A NextN/MTP draft exported with --mtp carries the token embeddings, output norm and lm head so it can be loaded as a standalone model. For every current sidecar those three tensors are most of the file: ggml-org/Qwen3.8-27B-GGUF mtp-Qwen3.8-27B-Q4_0.gguf is 1.565 GiB, of which 1.332 GiB (85%) is the copy, against 0.223 GiB for the MTP block itself. Add an opt-in --mtp-shared-embd that leaves them out and marks the file with nextn_shared_target_tensors. The loader then resolves those names against the already loaded target model. The graph side needs no change: the nextn blocks of twelve archs already fall back to model.tok_embd and model.output. The borrow is gated on the new key, so a sidecar published before this change cannot reach it and keeps its current behaviour. Shapes are checked against the target and a mismatch is refused, as is loading such a file on its own.

The graph cache is keyed on cgraph->nodes[0] alone, so two evaluations that share a first node but differ in shape collide on one entry. Warmup needs two consecutive calls with unchanged node properties, so a workload whose batch shape varies resets warmup on nearly every call and falls back to eager launch. Speculative decoding is exactly that workload. The qwen4exp verify batch is distributed 2:13 percent, 3:11 percent, 4:75 percent as the accepted count varies, where qwen35 sits at 4:98 percent and is effectively constant. Host launch time for the qwen4exp target decode was 1.52 ms with the draft head disabled and 12.35 ms with it enabled, while GPU time was unchanged, so the regression was entirely host side. The key now mixes the first node, the last node and the node count. This is O(1) rather than a walk over every node: the existing uid early return fires on 127 of 128 decodes, so the hot path must not touch node data. An earlier all-nodes hash reintroduced exactly the per-node walk a CUDA graph exists to avoid. Measured overhead against the previous key is 0.2 to 0.6 percent, with both variants built into one binary to avoid comparing across runs. Capture churn on Qwen3.8-27B UD-Q2_K_XL drops from 52 captures and 50 destroys to 4 and 0, with identical output md5 and an unchanged speculative ratio. Across 14 distinct prefill shapes the cache instantiates 16 entries against 14 before, with no destroys and no growth, and is capped at 64 by LRU on top of the existing sweep. test-backend-ops passes 13646 of 13646 on CUDA0, and Llama-3.2-1B-Instruct Q8_0 is byte identical with no throughput change.

_QwenMtpMixin is not a ModelBase subclass, so it re-declares the attributes it reads off cls for the type checker. filter_tensors reads cls.mtp_shared_embd without a matching declaration, which ty reports as unresolved-attribute. The declaration is a bare annotation, matching no_mtp and mtp_only above it. That creates no class attribute, so it cannot shadow ModelBase.mtp_shared_embd even though the mixin precedes the model class in the MRO; a default value here would have. Assisted-by: Claude

The previous key returned cgraph->nodes[0] without dereferencing it, so an empty graph was harmless. The shape-aware key reads nodes[0]->ne[], which is not, and neither call site checks n_nodes. Assisted-by: Claude

A draft-only export declares the full block count but ships the MTP block alone, so the trunk tensors load as null and only the MTP graph is buildable. Context reservation builds the trunk graph, which walked those nulls and segfaulted. A shared-embedding draft is caught earlier by the borrow check, since it has no token_embd of its own. A self-contained draft keeps one, so it passed that check and reached here. Assisted-by: Claude

Assisted-by: Codex

Assisted-by: Claude Opus 5

Assisted-by: Codex

* server: refactor subproc handling * fix Windows build * download: keep concurrent downloads of one blob apart Every process writes the same path + .downloadInProgress, so a second download of the same blob finds that file, takes it for its own partial transfer and asks for the bytes after it, which produces a corrupt result. The in-progress file now carries the pid of the process writing it. std::rename also replaces an existing destination on POSIX but fails on Windows, so a download whose blob appeared in the meantime is dropped after every retry and an etag rewrite silently keeps the old value. std::filesystem::rename has the POSIX behaviour everywhere, and the error now carries the reason reported by the system. * Revert "download: keep concurrent downloads of one blob apart" This reverts commit 917b83f. * tests: serialize the router tests that download the same model Parallel workers share one cache, so the two tests fetch the same blob into the same in-progress file and race to rename it. They now take a file lock around the download, like the session fixture does for the preset models. * Revert "tests: serialize the router tests that download the same model" This reverts commit c368a4a. --------- Co-authored-by: Pascal <admin@serveurperso.com>

Assisted-by: Codex # Conflicts: # src/CMakeLists.txt

Assisted-by: Codex

Advise read-mostly + prefetch to the allocating device for unified-memory buffers so lazily-migrated pages settle on the owning GPU. Co-Authored-By: opencode <noreply@opencode.ai> (cherry picked from commit ba2c46f)

Defer queued requests when resident-ctx-sum + candidate prompt would reach the threshold; deferred tasks retry on slot release. Auto-slot path only (explicit id_slot bypasses). Vanilla slot accounting: max(prompt-cache tokens, full task prompt length) per processing slot. Env: LLAMA_ARG_PARALLEL_CTX_THRESHOLD. Co-Authored-By: opencode <noreply@opencode.ai> (cherry picked from commit d065374)

Mirror the defer SRV_INF with an admit line (resident + candidate < threshold) so boundary tests can show the exact accounting either way. Co-Authored-By: opencode <noreply@opencode.ai> (cherry picked from commit 0ed2ac3)

… exceeding threshold alone The gate deferred any request with resident + candidate >= threshold, including resident == 0. A lone request bigger than the threshold on an otherwise idle pool then defers forever: nothing is resident, so no slot release ever retries it. The threshold guards combined oversubscription between concurrent requests, not a single request's own size (that is bounded by the per-slot cap). Defer now requires resident > 0; a lone request always admits. Admit log distinguishes the lone case so the printed comparison stays accurate. Co-Authored-By: opencode <noreply@opencode.ai> (cherry picked from commit 3ab0fde)

… one task A release event re-posted a single deferred task (FIFO head, or one explicitly requesting the slot). If that task re-deferred - e.g. the hydra#747 threshold gate deferring a large candidate - the release was burned and every other waiter stayed queued even with a slot idle, while brand-new arrivals kept getting served from the main queue (priority inversion, silent client hangs). Re-post the entire deferred FIFO in order (explicit-slot matches first) to the front of the main queue so every waiter is re-evaluated against the freed capacity before newer arrivals; tasks that still cannot proceed re-defer to the back, preserving FIFO order. Deterministic repro + production case: hydra#747 task-12239 hang (docs/investigations/740-results-report.md). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> (cherry picked from commit 1d3c4a8)

…t head Reconciles cdd1102's hand-rolled qwen4exp NextN/MTP draft head with the gs/qwen4exp-mtp lineage already carried by this branch. Findings: the two NextN/MTP draft-head implementations are the same code (same LLM_TENSOR_NEXTN_HC_HEAD_* enum/names, same llama_layer_nextn.hc_head_* fields, same graph_mtp / no_build_t shape, same qwen4exp.cpp graph). The only real difference is how a draft borrows the target's token embeddings and LM head: * gs lineage: at model-load time, opt-in via the GGUF key {arch}.nextn_shared_target_tensors + llama_model_params.model_shared, resolved in llama_model_loader::borrow_shared_tensor(). Graph side just asserts the tensor exists. Wired end to end (converter --mtp-shared-embd, common passes mparams.model_shared = model_tgt). * cdd1102: at graph-build time, qwen4exp-only, via qwen4exp_shared_model(cparams.ctx_other, ...), with token_embd marked TENSOR_NOT_REQUIRED for mtp_only files. Resolution: keep the gs loader-level mechanism everywhere. Rationale below. Conflicts resolved to the gs side: * src/llama-model.h comment only (hc_head_* fields already identical) * src/models/models.h comment only (no_build_t / graph_mtp already identical) * src/models/qwen4exp.cpp token_embd flags (loader guarantees it is present or borrowed, so flags=0 is correct); n_ff_exp(il) (gs API); borrow paths (handled by the loader, not the graph) Auto-merged cdd1102 deltas reverted to the gs side: * common/speculative.cpp is_mem_shared gemma4 gate. Unnecessary in gs: llama_context resets cparams.ctx_other to null and only sets it for Gemma4Assistant/EAGLE3/DFlash, so llama_get_ctx_other(ctx_dft) is null for qwen4exp and is_mem_shared is already false. The gate existed only to counteract cdd's own ctx_other use for graph-time borrow. * src/llama-context.cpp QWEN4EXP added to the EAGLE3/DFLASH ctx_other check. Inert with loader-level borrow (qwen4exp tok_embd/output are never null), so it is dropped rather than carried as dead relaxation. cdd1102's remaining changes (llama-model.cpp mtp_on_hybrid_qwen QWEN4EXP, llama-arch.* NEXTN_HC_HEAD_*) are already present unchanged in the gs lineage. Result: the merge tree is byte-identical to efd26f2; no unique cdd code is lost and no cdd divergence is retained.

ddvnguyen · 2026-09-12T15:39:16Z

Empirical E2E of the resolved tree — 2026-09-12

Rig freed, production stopped. Built the resolved tree (efd26f235, CUDA 13.2.2, CMAKE_CUDA_ARCHITECTURES=86;120) and ran two E2Es.

1) Production-shaped self-contained model — PASS

-m /mnt/SSD/Qwen3.8-27B-UD-Q5_K_S.gguf --spec-type draft-mtp, dual-GPU via RPC (-dev RPC0,CUDA0 -ts 27,38), UM on, production KV types.

Loaded clean, /health ok. Note: this model declares general.architecture=qwen35 (not qwen4exp), so it uses the generic same-file MTP head (has_draft=false); the borrow path is not involved. Hydra's production pin 130 (Qwen3.8-27B-UD-Q5_K_S.gguf + qwen3.8-sharp.jinja) is qwen35, not qwen4exp.
400-token generation: prefill 134.7 tok/s, decode 32.57 tok/s, draft_n=533 / draft_n_accepted=221 → 41.5% acceptance, 178 verify steps. Output coherent (all tokens landed in reasoning_content, none in content under this template).
Conclusion: no regression for production's actual pairing.

2) qwen4exp shared-draft borrow + MTP — PASS

Real qwen4exp pairing:

target /mnt/SSD/qwen3.8-flash-next-apex-mini/Qwen3.8-Flash-Next-APEX-I-Mini-*-of-00006.gguf (qwen4exp, 48 blocks, n_embd 2560, no nextn tensors),
sidecar /mnt/SSD/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (qwen4exp, 49 blocks, nextn_shared_target_tensors=True, ships no token_embd/output/output_norm).
Assumption (a) — load-time borrow: CONFIRMED. The sidecar is hard-refused standalone — borrow_shared_tensor: this model is a draft head without its own 'token_embd.weight'; load it as a draft of its target model, not on its own — but loads cleanly as --spec-draft-model against the target. The only mechanism that permits this is borrow_shared_tensor resolving those tensors from model_shared.
- Minor caveat: the "tensor … taken from the target model" INFO line was not captured in the server log (verbosity 3). The functional contrast (refused standalone vs. loads paired) is unambiguous.
Assumption (b) — is_mem_shared false for qwen4exp / no double-borrow corruption: CONFIRMED. MTP decode: draft_n=97 / draft_n_accepted=62 → 63.9% acceptance, 33 verify steps (accepted-per-position 29+19+14 = 62), coherent correct output ("A speculative decoding acceptance rate measures the fraction of draft tokens that the target model accepts as correct…"). Decode 5.72 tok/s — the 74 GB target runs via UM across 16+12 GB, so this is memory-bound throughput, not a correctness signal.
Load took ~3m50s for the 74 GB target; host RAM peaked near full before settling.

Correction to my own earlier reading

While setting up I briefly concluded from common/speculative.cpp:3002 (llama_model_load_from_file(params.model.path.c_str(), mparams)) that -md loads the target. That was wrong. A probe with a non-existent -md path shows the loader honors the draft path (gguf_init_from_file: failed to open … NONEXISTENT-draft.gguf), i.e. it resolves mparams.path. No bug; not a finding.

Verdict

No PR-blocking finding. Both flagged risk assumptions hold empirically. The only substantive observation is that Hydra's production model is qwen35, so the qwen4exp draft head is exercised by the Flash-Next pairing, not by pin 130.

Stack shut down cleanly; both GPUs back to 1 MiB used.

Fix two -Werror build failures hit by the CUDA CI job (gcc 13, LLAMA_FATAL_WARNINGS=ON) on the qwen4exp lineage: - ggml_cuda_mul_mat_id_grouped_host_staged: cast ids->ne[0] to size_t before comparing against the SIZE_MAX bound (sign-compare). - ggml_cuda_try_fuse: reserve the topk-moe op vector up front. gcc 13 flags the growing std::vector<ggml_op> insert with a -Wstringop-overflow false positive. Assisted-by: opencode

ddvnguyen · 2026-09-12T15:55:42Z

CI status (honest)

Head updated to c101306ef, which fixes the two -Werror build breaks on this lineage under the CUDA CI job (gcc 13, LLAMA_FATAL_WARNINGS=ON):

-Werror=sign-compare at ggml/src/ggml-cuda/ggml-cuda.cu:3053 (ggml_cuda_mul_mat_id_grouped_host_staged) -> cast ids->ne[0] to size_t.
-Werror=stringop-overflow at ggml/src/ggml-cuda/ggml-cuda.cu:5509 (ggml_cuda_try_fuse) -> reserve the topk-moe op vector up front (gcc 13 false positive on the growing std::vector<ggml_op> insert).

Verified locally by compiling ggml-cuda.cu with the exact CI flags (nvcc 13.2.86 -ccbin g++-13 -O3 -DNDEBUG -Werror ...): exit 0, 0 errors.

Remaining red (blocks merge)

tools/server/tests/unit/test_speculative.py::test_with_and_without_draft still fails with HTTP 500. Blocked by #121.

Cause is d659bcefa ("speculative: support grouped MoE drafting"), introduced by this lineage (not reachable from the common base 5266f24da): the new target-verification execution-intent check requires a span-grouped ubatch (n_seq_tokens == span), but llama_kv_cache::init_batch uses split_simple for single-stream batches (n_seq_tokens = 1, n_seqs = n_tokens). Fixing this is a design decision in the grouped-MoE-drafting feature, so it is tracked in #121 rather than patched here.

Reproduced deterministically (4/4) on efd26f235.

Other CI failures (not caused by this change)

Windows Server / router tests: HuggingFace 429 rate-limit while downloading test models (failed to download model ... status code: 429) - infrastructure flake, not code.

Do not merge until #121 is resolved and the CI matrix is green.

opened cdd11021b's hand-rolled qwen4exp NextN/MTP draft head and the gs qwen4exp-mtp lineage are the same code; only the embedding/LM-head borrow mechanism differs. Resolved to gs's loader-level mechanism; merge tree is byte-identical to efd26f235. PR ddvnguyen/llama.cpp#120 open against baseline-flash-next (review only, not merged). Refs #763

Rig window (production stopped, GPUs 1 MiB): - qwen35 production model + same-file MTP: PASS, 41.5% acceptance, 32.57 tok/s, coherent. Note production is qwen35 (not qwen4exp). - qwen4exp apex-mini + shared draft sidecar (nextn_shared_target_tensors): PASS. Standalone-refused, paired-loads -> loader borrow confirmed; 63.9% acceptance, coherent -> is_mem_shared false / no corruption. No PR-blocking finding. Detail in ddvnguyen/llama.cpp#120 comment. Stack shut down; GPUs back to 1 MiB. Refs #763

The target-verification intent was validated for every batch. But the plain kv-cache splitter (split_simple) describes each row as its own sequence, so grouped MoE target verification failed for single-stream models with "ubatch does not match the validated execution intent". Give the batch allocr a verification span hint. When the span is an atomic target verification, split_simple groups the rows by sequence, matching split_equal. Only require the grouped intent and certificate when grouped MoE execution is active, so non-grouped backends keep the plain split. Fixes #121 Assisted-by: opencode

ddvnguyen · 2026-09-12T16:53:03Z

#121 fixed in f6301d4ee; PR body updated from BLOCKED to OPEN FOR REVIEW.

Verification on the rig:

tools/server/tests/unit/test_speculative.py 10/10 PASS (incl. test_with_and_without_draft, previously HTTP 500, and test_parallel_verification_spans_are_atomic).
test-moe-cache, test-batch-alloc (34/34), test-arg-parser PASS; changed TUs clean under -Werror.
GPU probes PASS: production-shaped qwen35 draft-mtp 36.8 tok/s, 51.4% accept; qwen4exp shared-draft borrow 44.7% accept, coherent.

Fix groups the target-verification ubatch by sequence only when grouped MoE execution is active (required_grouped_execution_flags(...) != NONE); non-grouped backends keep the plain split. No qwen4exp-specific code. See #121 for details.

ddvnguyen · 2026-09-12T17:29:05Z

Corrected qwen4exp MTP benchmark — the earlier 0.79x was a UM config error

Root cause: the previous run set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, which on discrete GPUs makes --fit skip the layer split (49/49 layers on GPU -> host-as-device-free lie -> UM paging collapse). Per docs/arms/fit-solver-um-regression.md, that config caps decode at ~5-8 tok/s. With UM unset the same build lands in the documented 22-30 tok/s band. Not a code regression.

Common config (all arms): fork f6301d4ee (this PR), APEX-I-Mini, --split-mode layer --fit on --fit-target 1536,3072 -c 8192 --parallel 1 --flash-attn on --jinja, UM unset, greedy (temp 0, seed 4242), 3 prompts x 256 tok, cache_prompt=false; MTP = shared Q8 head + --spec-draft-n-max 2. Every boot gated by PR ggml-org#765 test-suite.sh --smoke (PASS).

Arm	no-draft tok/s	MTP tok/s	accept	MTP uplift
single-process 2-GPU, `-t 8`	22.31	26.56	67.4%	+19.0%
single-process 2-GPU, `-t 20`	21.52	23.76	67.4%	+10.4%
RPC-split, `-t 8`	22.97	26.39	62.1%	+14.9%
RPC-split, `-t 20`	—	25.34	62.1%	—

RPC-split = ggml-rpc-server on CUDA1/3060 :50052 + llama-server on CUDA0/5060Ti with --rpc, UM unset on both.

Findings

MTP is net-positive on qwen4exp with a GPU-resident draft: +15-19% at -t 8 (matches the can llama do other task except text-generate,like translate ggml-org/llama.cpp#761 arms' +11-20% chat).
RPC-split and single-process dual-GPU fit are equivalent for this model (26.39 vs 26.56, within noise). Topology is not the lever here; draft presence is.
Best measured: 26.6 tok/s (single-process MTP, -t 8).
Depth check (checkpoint-seed harness, 6.7K-token prefix, checkpoint-replay --sweep-max-tokens): decode 24.0-25.0 tok/s over 100-800-token budgets.

Harness finding (for ggml-org#765)

test-suite.sh --smoke hard-fails on FAIL: /props reports no build_info for locally rebuilt fork binaries whose build-info.cpp was not regenerated (empty string) — despite /health=200, model loaded, and llama-server --version reporting the build. Recommend downgrading that check to a warning.

ddvnguyen · 2026-09-12T18:23:28Z

Deep + concurrent arm: qwen4exp apex at ~74.5K depth x 2 slots

Production-relevant shape (per CLAUDE.md multi-agent design): --parallel 2, two concurrent ~80K sessions, both MTP-off and MTP-on.

Fit / config (verified before committing to the run)

-m .../Qwen3.8-Flash-Next-APEX-I-Mini-00001-of-00006.gguf
--split-mode layer --fit on --fit-target 1536,3072
-c 196608 --parallel 2 -t 8 --flash-attn on --jinja   # 2 x 98304 ctx/slot
-ctk q8_0 -ctv q8_0                                    # UM UNSET

Load (single-process dual-GPU fit, no RPC): n_slots=2, n_ctx_slot=98304, kv_unified=false.
VRAM at load — no-draft CUDA0 13.8 / CUDA1 8.7 GiB; MTP-on CUDA0 13.7 / CUDA1 11.7 GiB (draft on CUDA1). Host RAM 78 GiB available. It fits (q8_0 KV is what buys the headroom; f16 KV fits at 163840 but leaves MTP only ~600 MiB on CUDA1).

Method (PR ggml-org#765 checkpoint-seed harness)

Grow 2 concurrent sessions to depth via multiturn-growth-test.sh 2 6 15000 200 --checkpoint-dir, then immediately re-issue the frozen last turn on both transcripts concurrently (checkpoint-replay.sh --sweep-max-tokens 200,400) against the live KV (prefix-cache hit, cached ~= 74520/74525). Concurrency overlap check PASS (99.9% of run span overlapped).

Actual resident depth reached: 74,525 / 74,641 tokens (target 80K; harness word->token ratio ran ~7% short). Both slots ≈74.5K.

Decode results (aggregate = sum of both concurrent slots' tokens / run span)

Budget	no-draft aggregate	MTP-on aggregate	MTP uplift	accept
200 tok	12.22 tok/s	15.24 tok/s	+25%	70.4%
400 tok	12.39 tok/s	18.19 tok/s	+47%	70.4%

Per-slot (200 tok): no-draft 6.11 + 6.11; MTP 7.92 + 8.88. MTP acceptance over the run: 1323/1879 draft tokens = 70.4%.

Notes

At this depth/concurrency, aggregate decode is ~12 tok/s no-draft vs ~15-18 tok/s MTP — MTP stays net-positive at production shape (matches the +25-30% shallow uplift). The 400-tok MTP figure is inflated by one slot hitting EOS early and freeing the other; the 200-tok row (both generated maximally) is the conservative comparison.
Both GPUs sit at ~13.7-14.2 GiB / 8.7-11.7 GiB; no OOM, no paging (UM unset throughout).
Draft acceptance 70.4% at depth tracks the shallow 67-70%.
Interpretation: 80K x 2 fits on this rig only via q8_0 KV + single-process dual-GPU fit with classic CPU overflow; MTP remains worth keeping at production depth.

ddvnguyen · 2026-09-13T03:14:47Z

Deep + concurrent arm — production KV pin (K=q8_0, V=q5_1)

Re-run of the ~74.5K x 2 / --parallel 2 shape with the production KV pin. Same config otherwise: single-process dual-GPU --fit-target 1536,3072 -t 8, UM unset, apex-I-Mini, -c 196608 (2 x 98304/slot), MTP shared Q8 head + n-max 2. Fits: MTP CUDA0 13.6 / CUDA1 11.7 GiB. Method: grow 2 concurrent sessions (checkpoint-seed), then replay both frozen transcripts concurrently (cache hit ~74510/74556); overlap PASS 99.9%. Actual depth 74,514 / 74,558.

Budget	no-draft aggregate	MTP-on aggregate	uplift	accept
200 tok	12.49 tok/s (6.25+6.25)	17.33 tok/s (8.66+8.94)	+38.8%	75.5%
400 tok	12.52 tok/s (6.26+6.26)	19.61 tok/s (11.03+9.41)	+56.6%	75.5%

vs q8_0/q8_0 (previous comment)

	no-draft	MTP (200)	accept
K=q8_0 V=q8_0	12.22	15.24	70.4%
K=q8_0 V=q5_1	12.49	17.33	75.5%

No-draft essentially unchanged (~+2%), MTP aggregate +14% and acceptance +5 pts with V=q5_1 at this depth/concurrency. MTP remains strongly net-positive at production shape. No OOM/paging; UM unset throughout.

ddvnguyen · 2026-09-13T09:36:33Z

`--moe-expert-cache-size` cost/benefit sweep (apex, plain decode)

Measured while investigating GGML_CUDA_MOE_EARLY_ROUTER (PR #123 lineage). Build bdd54f475.

Config: APEX-I-Mini (qwen3.8-flash-next-apex-mini), --split-mode layer --fit on --fit-target 1536,3072 -c 8192 --parallel 1 --flash-attn on --jinja -t 8, UM unset, greedy (temp 0, seed 4242), 3 prompts x 128 tok, cache_prompt=false. 3/3 prompts completed per arm.

`--moe-expert-cache-size`	tok/s (mean)	vs off
0 (disabled)	20.40	—
2	9.11	-55.3%
4	9.70	-52.4%
8	10.13	-50.3%
16	13.62	-33.2%
32	16.09	-21.1%

Per-prompt: N0 [19.96, 20.57, 20.67], N2 [8.97, 9.30, 9.04], N4 [9.71, 9.81, 9.58], N8 [10.18, 10.29, 9.94], N16 [13.94, 13.92, 13.00], N32 [16.68, 15.95, 15.63].

Finding: on this 2-GPU rig the expert cache is a net loss at this shape. The fit solver keeps experts resident, so enabling the cache routes them through the PCIe LRU path and costs ~55% at N=2; it recovers monotonically with size but never reaches the no-cache baseline up to N=32. Recommendation: leave --moe-expert-cache-size off for apex at these shapes and use it only for memory-constrained profiles (larger ctx / higher parallelism) that would otherwise evict experts.

Correction: an earlier "MTP 10.15 -> 16.68 tok/s" comparison was not MTP. With --spec-type draft-mtp there was 0 draft acceptance (no draft acceptance log line; no draft fields in the responses) and the main graph hit required grouped execution failed: grouped plan unavailable 63-95x before falling back. The two points were just this cache curve at a non-default --fit-target, inside the slow cache regime. Proper apex plain-decode baseline is 20.4 tok/s. The MTP interaction is tracked separately (see linked issue).

Assisted-by: opencode

ddvnguyen · 2026-09-13T09:37:19Z

Follow-up to the cache-size sweep above: the MTP interaction (no draft acceptance + repeated required grouped execution failed) is tracked in #124.

Assisted-by: opencode

ddvnguyen · 2026-09-13T09:56:53Z

APEX model memory layout: 28.8 GB of the 78.7 GB is a sparse PLE n-gram table that stays on disk

Verification while looking at the ~20 tok/s decode ceiling. The model is 78.7 GB on disk but the real working set is ~50 GB.

GGUF tensor split (all 6 shards):

TOTAL tensor bytes          = 78.697 GB
per_layer_token_embd.weight = 28.800 GB   IQ4_NL   (160, 320001536)   [shard 00002]
everything else             = 49.897 GB

What the 28.8 GB tensor is: a qwen4exp PLE (per-layer embedding) n-gram table. Metadata ple.ngram_size=3, ple.heads_per_ngram=8 -> ple_n_heads=16; 16 head tables totalling 320,001,536 rows; head_dim=160. Per decode token src/llama-staged-input.cpp:98-124 hashes the last n-gram and reads exactly 16 rows:

mixed = context[0]*mult[0] ^ context[1]*mult[1] ^ ...   (per n in 2..3)
row   = mixed % ple_head_vocab_sizes[h] + ple_head_offsets[h]
read_row(model.per_layer_tok_embd, row, ...)   // table->data + row*nb[1]

16 rows x 160 dims x ~90 bytes/row = ~1.4 KB/token, random access. readable() requires ggml_backend_buffer_is_host(...), so the table must stay in host memory, not VRAM.

Live measurement (mmap default, t8, fit-target 1536,3072, c8192):

VmRSS after a decode            = 31.4 GB
PLE shard 00002 Rss             =  5.8 MB   <- the 28.8 GB table is ~all on disk
shard 00005 / 00004 / 00006 rss = 14.1 / 10.1 / 4.5 GB

Consequences:

--load-mode none is a bad fit for this model: it is global, so it would pull all 78.7 GB (including the 28.8 GB PLE table) into anonymous RAM, slowing startup and inflating RSS for a table read at ~1.4 KB/token. Keep mmap (default) so the table stays disk-backed.
--ple-prefetch (default off, common/arg.cpp:2329) is the intended knob: it madvise(WILLNEED)s the exact PLE row pages before the CPU GET_ROWS (llama-mmap.cpp:706). Worth an A/B on the decode path.
The binding constraint stays the ~49.9 GB of real weights vs 28 GB VRAM, so the earlier levers still apply (MTP without the MoE expert cache, lower --fit-target, threads, draft placement). The PLE table is not part of the memory problem.

Assisted-by: opencode

GenerelSchwerz and others added 30 commits September 1, 2026 22:39

speculative: support grouped MoE drafting

d659bce

Assisted-by: Codex

cuda: pad partial mapped MMQ tiles

c52448a

Assisted-by: Codex

cuda: pad partial mapped MMQ tiles

7fc7981

Assisted-by: Codex

gguf-py : register the qwen4exp NextN tensors

bd428de

Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.

qwen4exp: allow loading a draft-only MTP export

2914435

qwen4exp mtp: trim comments

0572cfa

ggml-cuda: guard the graph key against an empty graph

68bba65

The previous key returned cgraph->nodes[0] without dereferencing it, so an empty graph was harmless. The shape-aware key reads nodes[0]->ne[], which is not, and neither call site checks n_nodes. Assisted-by: Claude

speculative: honor per-sequence MTP draft limits

bfc2871

Assisted-by: Codex

speculative: honor zero draft limits

6de9e8c

Assisted-by: Codex

speculative: retain one-step MTP draft state

6451562

Assisted-by: Codex

server: clear deferred MTP state on release

47d1eaf

Assisted-by: Codex

server: batch retained MTP suffix updates

55aeee8

Assisted-by: Codex

Merge upstream master into llama/main

bb3da59

Assisted-by: Codex

Merge upstream master into llama/dev

dc3b3ef

Assisted-by: Codex

Merge llama/main into llama/dev

9dc93fd

Assisted-by: Codex

Merge llama/dev into moe-cache

0c332d7

Assisted-by: Codex

Merge llama/dev into moe-cache-experimental

44f924f

Assisted-by: Codex

Merge llama/dev into moe-cache-drafting

532f8a4

Assisted-by: Codex

docs : link CLAUDE.md to AGENTS.md

f29324d

Assisted-by: Claude Opus 5

Merge moe-cache into moe-cache-drafting

0b74594

Assisted-by: Codex

Merge Qwen3.8 Flash Next MTP support

adfdf1e

Assisted-by: Codex

ggml-cuda: retain grouped speculative graph variants

15e501c

Assisted-by: Codex

common: configure draft MoE cache size separately

e3371bf

Assisted-by: Codex

ggml-cuda: retain grouped speculative graph variants

babfa43

Assisted-by: Codex

ngxson and others added 14 commits September 12, 2026 00:53

Merge moe-cache-drafting into qwen4exp-mtp

d2328fc

Assisted-by: Codex # Conflicts: # src/CMakeLists.txt

Merge upstream master into llama/main

0358897

Assisted-by: Codex

Merge llama/main into llama/dev

58a6df8

Assisted-by: Codex

Merge llama/dev into moe-cache

67da2ef

Assisted-by: Codex

Merge moe-cache into moe-cache-drafting

915f175

Assisted-by: Codex

Merge moe-cache-drafting into qwen4exp-mtp

f61cac1

Assisted-by: Codex

cuda: add MoE previous-layer prefetch

77b733d

Assisted-by: Codex

ddvnguyen mentioned this pull request Sep 12, 2026

[engine] draft-simple 500: d659bcefac target-verification intent rejects split_simple ubatch (blocks PR #120) #121

Open

This was referenced Sep 13, 2026

cuda : MoE look-ahead expert prefetch (PR-A consumer) + LFRU eviction guard #126

Open

MoE look-ahead PR-P1: producer (NOT LANDABLE, see body) #127

Open

Navigation Menu

Reconcile qwen4exp NextN/MTP draft head: adopt gs qwen4exp-mtp lineage over hand-rolled cdd11021b (#763) - #120

Reconcile qwen4exp NextN/MTP draft head: adopt gs qwen4exp-mtp lineage over hand-rolled cdd11021b (#763)#120
ddvnguyen wants to merge 488 commits into
baseline-flash-nextfrom
feat/763-reconcile-qwen4exp-mtp

ddvnguyen commented Sep 12, 2026 •

edited

Loading

Uh oh!

ddvnguyen commented Sep 12, 2026

Uh oh!

ddvnguyen commented Sep 12, 2026

Uh oh!

ddvnguyen commented Sep 12, 2026

Uh oh!

ddvnguyen commented Sep 12, 2026

Uh oh!

ddvnguyen commented Sep 12, 2026

Uh oh!

ddvnguyen commented Sep 13, 2026

Uh oh!

ddvnguyen commented Sep 13, 2026

Uh oh!

ddvnguyen commented Sep 13, 2026

Uh oh!

ddvnguyen commented Sep 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

ddvnguyen commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update 2026-09-12: #121 fixed in f6301d4ee

Unrelated observation

What this is

Decision: primary basis = the gs qwen4exp-mtp lineage

Diff findings

What was ported from cdd11021b: nothing

Resulting tree

Behavior differences a reviewer should check

Not fully confident in / did not verify

Verification done

CI status

Uh oh!

ddvnguyen commented Sep 12, 2026

Empirical E2E of the resolved tree — 2026-09-12

1) Production-shaped self-contained model — PASS

2) qwen4exp shared-draft borrow + MTP — PASS

Correction to my own earlier reading

Verdict

Uh oh!

ddvnguyen commented Sep 12, 2026

CI status (honest)

Remaining red (blocks merge)

Other CI failures (not caused by this change)

Uh oh!

ddvnguyen commented Sep 12, 2026

Uh oh!

ddvnguyen commented Sep 12, 2026

Corrected qwen4exp MTP benchmark — the earlier 0.79x was a UM config error

Findings

Harness finding (for ggml-org#765)

Uh oh!

ddvnguyen commented Sep 12, 2026

Deep + concurrent arm: qwen4exp apex at ~74.5K depth x 2 slots

Fit / config (verified before committing to the run)

Method (PR ggml-org#765 checkpoint-seed harness)

Decode results (aggregate = sum of both concurrent slots' tokens / run span)

Notes

Uh oh!

ddvnguyen commented Sep 13, 2026

Deep + concurrent arm — production KV pin (K=q8_0, V=q5_1)

vs q8_0/q8_0 (previous comment)

Uh oh!

ddvnguyen commented Sep 13, 2026

--moe-expert-cache-size cost/benefit sweep (apex, plain decode)

Uh oh!

ddvnguyen commented Sep 13, 2026

Uh oh!

ddvnguyen commented Sep 13, 2026

APEX model memory layout: 28.8 GB of the 78.7 GB is a sparse PLE n-gram table that stays on disk

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ddvnguyen commented Sep 12, 2026 •

edited

Loading

Update 2026-09-12: #121 fixed in `f6301d4ee`

Decision: primary basis = the gs `qwen4exp-mtp` lineage

What was ported from `cdd11021b`: nothing

`--moe-expert-cache-size` cost/benefit sweep (apex, plain decode)