From 18211d799adccb9ca595c0db7b3fede2778089a7 Mon Sep 17 00:00:00 2001 From: Minxi Jin Date: Mon, 20 Jul 2026 14:27:18 +0800 Subject: [PATCH] docs: add Ascend NVIDIA timeline comparison skill --- .../ascend-nv-timeline-compare/SKILL.md | 307 ++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100644 .claude/skills/ascend-nv-timeline-compare/SKILL.md diff --git a/.claude/skills/ascend-nv-timeline-compare/SKILL.md b/.claude/skills/ascend-nv-timeline-compare/SKILL.md new file mode 100644 index 00000000..2a8a7d38 --- /dev/null +++ b/.claude/skills/ascend-nv-timeline-compare/SKILL.md @@ -0,0 +1,307 @@ +--- +name: ascend-nv-timeline-compare +description: Compare inference timelines between Ascend NPU and NVIDIA GPU for the same lmdeploy model/scenario. Part 1 (done) — how to pull an NVIDIA (CUDA backend) decode timeline the same way the ascend-timeline skill pulls an Ascend one, including the CUDA-side audit-marker patch, delay selection on fast GPUs, and phase/batch/KV validation. Part 2 (TODO) — how to actually compare NV vs Ascend performance. +--- + +# Ascend vs NVIDIA Timeline Comparison + +Goal: capture the **same scenario** (model, batch, context, parallelism, decode/prefill +phase) on both Ascend NPU and NVIDIA GPU, then compare. This skill is the NV-side +companion to `ascend-timeline`. Read `ascend-timeline` first — the profiling concepts, +deliverables, and analysis methods there apply verbatim; this skill only records what is +**different on NVIDIA** and how to keep the two runs comparable. + +> Status: **Part 1 (pull NV timeline) is written and validated.** Part 2 (NV↔Ascend +> performance comparison) is a TODO stub at the bottom, to be filled in later. + +--- + +## Part 0 — What is the same, what is different + +The profiling machinery in lmdeploy is **device-agnostic**. `profiler.py` +(`AgentProfiler`) uses `torch.profiler.profile(activities=[CPU, CUDA])` and +`export_chrome_trace()` on every backend. So all of this is identical to Ascend: + +- env vars: `LMDEPLOY_PROFILE_CPU/CUDA/DELAY/DURATION/OUT_PREFIX` +- output naming: `.json`, Chrome Trace JSON, one file per rank +- delay/duration semantics: `asyncio.sleep(delay)` → `profiler.start()` → + `asyncio.sleep(duration)` → `dump()` +- phase markers: `forward_eager` (prefill/eager), `forward_cudagraph` (graph decode) +- the fixed-real-batch / KV-context audit **concept** + +What differs on NVIDIA: + +| Aspect | Ascend | NVIDIA | +|--------|--------|--------| +| Backend | dlinfer vendor ops patched into lmdeploy | lmdeploy **native CUDA backend**, no dlinfer | +| `device_type` in `PytorchEngineConfig` | `"ascend"` | omit / `"cuda"` (default) | +| Profiler routing | `torch_npu.profiler` | stock `torch.profiler` CUDA | +| Audit-marker patch site | `dlinfer/framework/lmdeploy_ext/cudagraph/ascend_cudagraph.py` → `AscendSingleGraphRunner.forward` | `lmdeploy/pytorch/backends/cuda/graph_runner.py` → `CUDASingleGraphRunner.forward` | +| `HCCL_OP_EXPANSION_MODE=AI_CPU` | required | **not used** | +| Collective / compute kernel names in trace | `HcclAllReduce`, `GroupedMatmul`, `FusedInferAttentionScore`, `MatMulV2`, AI CPU kernels | `ncclDevKernel_AllReduce*`, cutlass/`sm90_*`/`ampere_*` gemm, flash-attn kernels, `void at::native::*` | +| Port-conflict cause between runs | Ray holds **HCCL** ports ~45 s | Ray holds GCS/worker ports; still wait between serial tp runs | + +Keep everything else (model path, `batch_size`, `input_len`, `output_len`, +`num_speculative_tokens`, `block_size`, `max_batch_size`, `cache_max_entry_count`, target +phase, target real batch, target KV-context band) **identical across the two devices** — +that is the whole point of a comparison run. + +--- + +## Part 1 — Pull an NVIDIA decode timeline + +Worked example that produced validated timelines: **metamoe (Qwen3.5 MoE + qwen3_5_mtp, +4 spec tokens), bs=64, KV context 150–250, decode, tp=2 and tp=1, on H200**. + +### 1.1 Environment layout (container vs host) + +On the reference machine, lmdeploy + the Python env (`/opt/py3`, torch 2.10+cu128) live +**inside a container**; you attach via `tmux attach -t `. `ssh ` lands on +the **host**, where that env is absent. But the code tree and model live on a **shared +mount** (`/mnt/...`) visible from both. + +Consequences (generalize per machine, don't hard-code paths): + +- **Edit code from the host** (shared mount) with normal tools; the container's editable + lmdeploy install picks it up. +- **Run lmdeploy inside the container** via `tmux send-keys -t "" Enter` + and read progress from a log file on the shared mount (`tmux capture-pane` or + `tail` the log). +- Put all scripts / logs / timelines in a shared-mount workdir. Watch **ownership**: a + dir created by container-root may not be writable by the host user — create the workdir + as the host user (or `chmod 777`) so `scp` from the host works. +- **Never** broadly `kill -9` by grep pattern on a shared host — you can kill other users' + Ray jobs. Rely on clean process exit + a wait between serial runs. + +### 1.2 Audit-marker patch (CUDA single graph runner) + +Same purpose as the Ascend patch, different file. Add a **zero-device-sync** marker +emitter to `CUDASingleGraphRunner.forward` in +`lmdeploy/pytorch/backends/cuda/graph_runner.py`. Back up the file first (`.bak`). + +Near the top of the module (after `logger = get_logger('lmdeploy')`): + +```python +# --- decode fixed-batch / KV-context audit (skill: ascend-nv-timeline-compare) --- +import os as _os + +_AUDIT_BS = _os.getenv('LMDEPLOY_AUDIT_DECODE_BATCH_SIZE') == '1' +_AUDIT_KV = _os.getenv('LMDEPLOY_AUDIT_KV_CONTEXT') == '1' + + +def _emit_decode_audit_markers(context): + """CPU-side record_function markers for the fixed-real-batch decode audit. + + All values come from CPU scalars: batch = q_seqlens tensor SHAPE (no sync), + max/sum = StepContext ints. No .item() on a device tensor in the replay + path, so the measured timeline is not perturbed. For a uniform-length batch + mean == max, hence min == max. + """ + if not (_AUDIT_BS or _AUDIT_KV): + return + if not getattr(context, 'is_decoding', False): + return + q_seqlens = getattr(context, 'q_seqlens', None) + if q_seqlens is None: + return + bs = int(q_seqlens.size(0)) # real scheduler batch, pre graph-padding + if _AUDIT_BS: + with record_function(f'decode_actual_batch_size={bs}'): + pass + if _AUDIT_KV and bs > 0: + max_kv = int(getattr(context, 'max_kv_seqlen', 0) or 0) + sum_kv = int(getattr(context, 'sum_kv_seqlen', 0) or 0) + mean_kv = sum_kv / bs + min_kv = max_kv # no CPU per-seq min scalar; uniform batch -> min==max + with record_function( + f'decode_kv_context_min={min_kv},max={max_kv},mean={mean_kv:.1f}'): + pass +``` + +Then call it inside `forward`, right after the context is fetched / cudagraph context is +updated and **before** `self._graph.replay()`: + +```python + @record_function('forward_cudagraph') + def forward(self, **kwargs): + padded_kwargs = self.model.fill_buffers_cudagraph(self.meta, **kwargs) + context = self.ctx_mgr.current_context() + self.model.update_context_cudagraph(self.meta, context) + _emit_decode_audit_markers(context) # <-- inserted + if self.USE_GRAPH: + ... +``` + +Why this is zero-sync and safe (same rule the Ascend skill states — do **not** `.item()` +an NPU/GPU KV tensor in the replay path): + +- `q_seqlens.size(0)` is a **tensor shape**, read on CPU, no device sync. It is the real + scheduler batch **before** graph padding (padding goes into buffers, not into + `context.q_seqlens`). This is the definition of "real batch". +- `context.max_kv_seqlen` and `context.sum_kv_seqlen` are **CPU Python ints** already + carried on `StepContext` (computed at input construction in + `pytorch/engine/inputs_maker.py`, advanced each step). No sync. + +Caveats to record in the report (observed with MTP): + +- `min` is reported as `= max` because there is no CPU-side per-sequence min scalar + without threading one through `ModelInputs`/`StepContext`. For a **uniform-length** + profiling workload (identical prompts, lockstep decode) `min == max` exactly, and this + is **self-verifying**: if `mean == max` on a marker, that context is uniform. +- The `mean` field can diverge from `max` on **MTP draft-head** replays, because the + draft-head `StepContext` has different `sum_kv_seqlen`/batch semantics. Treat the + `min==max` value (per-sequence KV context) as the reliable number; treat `mean` as + reliable only when it equals `max`. +- If you need an exact per-seq `min` for a **non-uniform** batch, thread a + `min_kv_seqlen` int through `inputs_maker.py` (next to the existing + `kv_seqlens.max()/.sum()` — those already run on a CPU tensor, no GPU sync) → `ModelInputs` + (field + `.step()` + `.clone`) → `StepContext.new()`, then read it here. Higher risk; + skip it unless the workload is non-uniform. + +Verify after patching, from the **container** Python: + +```python +import lmdeploy.pytorch.backends.cuda.graph_runner as g +assert hasattr(g, '_emit_decode_audit_markers') +``` + +### 1.3 Profiling script (CUDA) + +Identical to the `ascend-timeline` template except `device_type` is omitted (defaults to +`cuda`) and there is no `HCCL_OP_EXPANSION_MODE`. Must use the `lmdeploy.pipeline` API +(not raw `Engine` + threaded `stream_infer`). Key knobs from the worked example: + +```python +backend = PytorchEngineConfig( + tp=args.tp, cache_max_entry_count=0.6, max_batch_size=64, + block_size=128, eager_mode=False, # eager_mode=False => graph decode +) +spec = SpeculativeConfig(method='qwen3_5_mtp', num_speculative_tokens=4) +pipe = lmdeploy.pipeline(model_path, backend_config=backend, + speculative_config=spec, trust_remote_code=True) +gen = GenerationConfig(do_sample=False, max_new_tokens=OUT, ignore_eos=True) +``` + +Add **wall-clock logging** in the harness (`print(f"[WALL {time.time():.3f}] ...")`) at +engine-ready (t0) and per round. You correlate these wall timestamps with the profiler's +`"Profiler start on rank[...]"` log line to confirm the window landed where you intended. + +### 1.4 Delay selection on a fast GPU (the crux) + +The profiler delay clock starts at `agent.start()` (≈ when `pipeline()` returns, i.e. t0), +**not** at first request. On an H200, MTP decode is **very fast** (~150 KV tokens/s per +sequence), so KV context grows ~150 within any 1 s window — you **cannot** keep a full 1 s +decode window inside a 100-wide context band like [150,250], because +`LMDEPLOY_PROFILE_DURATION` is an **integer ≥ 1 s**. + +Strategy that works (keeps every captured replay in-band and makes the window easy to hit): + +1. **Short `output_len` so each round's whole decode stays in the band.** With prompt + ≈ 150 tokens and `output_len=64`, per-sequence context runs 154→~218 across the entire + round — every decode replay is in [150,250]. (Tokenizer note: this model is + ~5.36 chars/token; `input_len=166` gave 153 tokens. Confirm from the log's + `input_tokens=` field, not from the char heuristic.) +2. **Many back-to-back rounds** (`total_rounds=30`) so the decode phase spans ~30 s and + the window lands in decode regardless of small timing error. +3. **Pick `delay` mid-rounds.** Dry-run once (no profiling env vars) to get the cadence: + in the example, warmup done ~0.7–2.9 s, rounds ~1.0–1.4 s each. `delay=15` landed the + 1 s window around round 10–14 — solidly in decode, big margin either side. +4. `LMDEPLOY_PROFILE_DURATION=1`. + +If the whole 1 s window physically cannot fit the requested band, that is fine: capture +early decode with the band starting at the prompt length, and **report the exact per-replay +context from the markers** rather than claiming a tighter band than the physics allow. + +### 1.5 Run env vars + +```bash +export LMDEPLOY_PROFILE_CPU=1 +export LMDEPLOY_PROFILE_CUDA=1 +export LMDEPLOY_PROFILE_DELAY=15 # measured, mid-rounds +export LMDEPLOY_PROFILE_DURATION=1 +export LMDEPLOY_PROFILE_OUT_PREFIX=/metamoe_tp2_decode_ +export LMDEPLOY_AUDIT_DECODE_BATCH_SIZE=1 +export LMDEPLOY_AUDIT_KV_CONTEXT=1 +# NO HCCL_OP_EXPANSION_MODE on NVIDIA +``` + +### 1.6 Serial tp runs + +Run tp=2 and tp=1 (or any parallel settings you also captured on Ascend) **serially**. +Between runs: let the previous `pipeline` process exit cleanly, confirm GPUs are free +(`nvidia-smi`, 0 MiB), and wait for Ray ports to release. Do **not** broad-kill on a shared +host. Each run reloads the model, so minimize iterations. + +### 1.7 Validation (mandatory before trusting a file) + +Per rank JSON, grep the markers (works fine on ~80–90 MB files): + +```bash +grep -o "forward_cudagraph" f.json | wc -l # decode graph replays present? +grep -o "forward_eager" f.json | wc -l # prefill events (mixed window OK) +grep -oE "decode_actual_batch_size=[0-9]+" f.json | sort | uniq -c +grep -oE "decode_kv_context_min=[0-9]+" f.json | grep -oE "[0-9]+" \ + | awk '{if($1>=150&&$1<=250)a++;else b++} END{print "in:"a" out:"b}' +``` + +What the worked example showed, and what to check: + +- **Phase**: `forward_cudagraph` present ⇒ decode-phase window captured (the required + marker for a decode timeline). `forward_eager` may also appear; a mixed window is valid + as long as the decode marker is present. +- **Real batch**: `decode_actual_batch_size=64` is the **main-model** decode batch + (tp2: 128 markers; tp1: 83). A second value `=16` also appears (tp2: 33; tp1: 58) — these + are the **MTP draft-head** replays that every MTP decode step contains. So an MTP timeline + is *not* "purely 64"; report both, and state that the main decode is 64. +- **KV context**: min==max on every marker (uniform batch confirmed), range 154–226, + **all markers in [150,250]** on both ranks. +- **Rank agreement**: both ranks reported identical marker counts. +- **Profiler-stop artifact**: the round active when `dump()` fires is abnormally slow + (example: round 14 = 2.7 s vs ~1.1 s neighbors). This is the known + `npu_profile.stop()`/`profiler.stop()` stall — do **not** treat that round as workload + perf. Same behavior on both devices. + +Rigor note: counting markers + rank agreement + in-window presence is a *good* check, but +the **gold-standard** qualification (from `ascend-timeline`) is stronger — verify by +timestamp containment that each `decode_actual_batch_size=B` interval is nested 1:1 inside a +distinct `forward_cudagraph` interval (same pid/tid, start/end containment). Do that pass +when the conclusion must be airtight. + +### 1.8 Finding bs=64 in the trace (Perfetto) + +1. Open the rank JSON in [ui.perfetto.dev](https://ui.perfetto.dev) (>200 MB → Perfetto; + <200 MB → chrome://tracing also works). +2. Search `decode_actual_batch_size=64`; jump between hits with Enter. These are CPU-thread + `record_function` slices on the forward thread. +3. Click one → its **parent** slice is `forward_cudagraph` (confirms graph decode), and its + **sibling** is `decode_kv_context_min=…,max=…` (the KV context of that replay). +4. Box-select that interval → the device/CUDA-stream kernels underneath are what that + bs=64 decode replay actually ran. +5. To jump precisely: each marker has a microsecond `ts` in the raw JSON; extract with a + tiny `json.load` script and enter the `ts` in Perfetto. + +### 1.9 Deliverables (same as ascend-timeline) + +Preserve raw per-rank JSONs, the stdout/stderr log (used to pick delay and diagnose), and a +report covering: window/params, files analyzed, phase validation, real-batch + KV-context +audit (qualified replay count, exact B, KV min/max/mean or ranges, rank agreement), and any +anomalies. `scp` the raw JSONs + logs back for offline analysis. + +--- + +## Part 2 — Compare NV vs Ascend performance (TODO — to be added) + +