[codex] add qwen3.5-4b inference example - #48
Conversation
|
@Fatemanx Hello! It looks like the CI is failing on this PR. Could you take a look? |
|
@Fatemanx Thanks a lot for your contribution! We really appreciate it. We'll consider merging this PR, and let's work together to get it across the finish line. Could you rebase your branch onto main? |
8717c60 to
24bd0ff
Compare
I rebase to main already. |
| nvtx.switch_profile(iter_id, 0, PROFILE_CNT) | ||
|
|
||
|
|
||
| def make_text_ids(model: Qwen35ForConditionalGeneration, seq_len: int, device: torch.device) -> torch.Tensor: |
There was a problem hiding this comment.
The current harness hardcodes batch=1 ((1, seq_len) / (1, 1)). For a compiler example this is too narrow — MagiCompiler needs to see a real batch dimension.
There was a problem hiding this comment.
infer.sh already wraps nsys. Please run each mode on a real GPU and paste the nsys screenshots into the PR (timeline / CUDA HW view is enough)
🗂️ PR Category
📝 Description
Add a forward-only Qwen3.5-4B inference example under
example/inference/qwen3.5-4b.What changed
infer.py,infer.sh,modeling.py, and a short README for the new example.text_prefill,text_decode,image_prefill,image_decode, andall.MODEL_PATHfrom the environment; no local weight path, GPU id, or Cutlass default is baked in.Validation
python -m py_compile example/inference/qwen3.5-4b/infer.py example/inference/qwen3.5-4b/modeling.pybash -n example/inference/qwen3.5-4b/infer.shpytest tests/example_inference/test_qwen35_4b_infer.py -q-> 6 passed