Skip to content

cuda : add opt-in MoE cache telemetry env and early-router diagnostics - #123

Open
ddvnguyen wants to merge 2 commits into
feat/763-reconcile-qwen4exp-mtpfrom
feat/moe-early-router-debug
Open

ddvnguyen wants to merge 2 commits into
feat/763-reconcile-qwen4exp-mtpfrom
feat/moe-early-router-debug

Conversation

@ddvnguyen

Copy link
Copy Markdown
Owner

What

Small, opt-in debug tooling for the MoE expert cache and the experimental
GGML_CUDA_MOE_EARLY_ROUTER path. No behavior change when the env vars are unset.

  1. GGML_CUDA_MOE_MM_DEBUG=1 now enables the existing mm-debug telemetry.
    Previously g_moe_cache_mm_debug could only be set through the C API
    ggml_backend_cuda_moe_set_debug_mm(), which only the tests call, so
    moe-cache: / moe-grouped-decode: / moe-cache-tensor-* output was
    unreachable from llama-server.
  2. configure_early_router() one-time dormant diagnostics. When the early
    router is enabled but does not reach DECODE_GROUPED, it now prints the
    reason once (e.g. outcome-not-decode-grouped outcome=0, or
    dormant awaiting captured graph).
  3. Server stats dump fix. The print_timings() stats call was behind
    #ifdef GGML_USE_CUDA, which is not defined in dynamic-backend builds
    (the CUDA backend is a shared object), so ggml_backend_cuda_moe_log_and_reset_stats()
    was never invoked from the server. It now uses the backend proc-address,
    matching llama_model_free().

Why

Needed to answer "why doesn't GGML_CUDA_MOE_EARLY_ROUTER engage?". With these
changes the telemetry shows the answer directly.

Usage

GGML_CUDA_MOE_EARLY_ROUTER=1 GGML_CUDA_MOE_MM_DEBUG=1 \
llama-server -m MODEL --moe-expert-cache-size N ...
# grep the server log for: moe-early-router:, moe-grouped-decode:, moe-cache:

Findings from the investigation (for reference, not fixed here)

  • configure_early_router() requires execution->outcome() == DECODE_GROUPED
    (moe-cache.cu:8780).
  • On plain decode the main-context execution certificate leaves
    flags = NONE (llama-context.cpp:4155-4178 only stamps REQUIRED_GROUPED
    for DRAFT/MTP, or the MAIN execution_intent branch used by speculative
    verification), so the plan is classified PREFILL_LEGACY and the early
    router can never engage.
  • With MTP (grouping required) the candidate groups fail capability:
    required_unsupported=95 at --moe-expert-cache-size 2, still 63 at
    --moe-expert-cache-size 32; covered=0 and calls=0, so no grouped plan
    is built.
  • Side effect observed: the legacy cache scales with size — apex + MTP went
    from 10.15 tok/s (slots=2) to 16.68 tok/s (slots=32).

Base: feat/763-reconcile-qwen4exp-mtp (the early-router code is not present on
baseline-flash-next), so this PR is only the debug tooling.

Assisted-by: opencode

GGML_CUDA_MOE_MM_DEBUG=1 enables the existing mm-debug telemetry without
calling ggml_backend_cuda_moe_set_debug_mm() from a test. configure_early_router
now prints a one-time reason when it stays dormant. The server stats dump uses
the backend proc-address because the previous #ifdef GGML_USE_CUDA block is
dead in dynamic-backend builds.

Assisted-by: opencode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant