Conversation
GGML_CUDA_MOE_MM_DEBUG=1 enables the existing mm-debug telemetry without calling ggml_backend_cuda_moe_set_debug_mm() from a test. configure_early_router now prints a one-time reason when it stays dormant. The server stats dump uses the backend proc-address because the previous #ifdef GGML_USE_CUDA block is dead in dynamic-backend builds. Assisted-by: opencode
Assisted-by: opencode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Small, opt-in debug tooling for the MoE expert cache and the experimental
GGML_CUDA_MOE_EARLY_ROUTERpath. No behavior change when the env vars are unset.GGML_CUDA_MOE_MM_DEBUG=1now enables the existing mm-debug telemetry.Previously
g_moe_cache_mm_debugcould only be set through the C APIggml_backend_cuda_moe_set_debug_mm(), which only the tests call, somoe-cache:/moe-grouped-decode:/moe-cache-tensor-*output wasunreachable from
llama-server.configure_early_router()one-time dormant diagnostics. When the earlyrouter is enabled but does not reach
DECODE_GROUPED, it now prints thereason once (e.g.
outcome-not-decode-grouped outcome=0, ordormant awaiting captured graph).print_timings()stats call was behind#ifdef GGML_USE_CUDA, which is not defined in dynamic-backend builds(the CUDA backend is a shared object), so
ggml_backend_cuda_moe_log_and_reset_stats()was never invoked from the server. It now uses the backend proc-address,
matching
llama_model_free().Why
Needed to answer "why doesn't
GGML_CUDA_MOE_EARLY_ROUTERengage?". With thesechanges the telemetry shows the answer directly.
Usage
GGML_CUDA_MOE_EARLY_ROUTER=1 GGML_CUDA_MOE_MM_DEBUG=1 \ llama-server -m MODEL --moe-expert-cache-size N ... # grep the server log for: moe-early-router:, moe-grouped-decode:, moe-cache:Findings from the investigation (for reference, not fixed here)
configure_early_router()requiresexecution->outcome() == DECODE_GROUPED(
moe-cache.cu:8780).flags = NONE(llama-context.cpp:4155-4178only stampsREQUIRED_GROUPEDfor DRAFT/MTP, or the MAIN
execution_intentbranch used by speculativeverification), so the plan is classified
PREFILL_LEGACYand the earlyrouter can never engage.
required_unsupported=95at--moe-expert-cache-size 2, still63at--moe-expert-cache-size 32;covered=0andcalls=0, so no grouped planis built.
from 10.15 tok/s (slots=2) to 16.68 tok/s (slots=32).
Base:
feat/763-reconcile-qwen4exp-mtp(the early-router code is not present onbaseline-flash-next), so this PR is only the debug tooling.Assisted-by: opencode