Skip to content

[codex] add qwen3.5-4b inference example - #48

Open
Fatemanx wants to merge 2 commits into
SandAI-org:mainfrom
Fatemanx:feat/qwen35-4b-inference-example
Open

[codex] add qwen3.5-4b inference example#48
Fatemanx wants to merge 2 commits into
SandAI-org:mainfrom
Fatemanx:feat/qwen35-4b-inference-example

Conversation

@Fatemanx

@Fatemanx Fatemanx commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

🗂️ PR Category

  • ✨ New Feature

📝 Description

Add a forward-only Qwen3.5-4B inference example under example/inference/qwen3.5-4b.

What changed

  • Adds infer.py, infer.sh, modeling.py, and a short README for the new example.
  • Keeps only the minimal compiled entrypoints: text_prefill, text_decode, image_prefill, image_decode, and all.
  • Requires MODEL_PATH from the environment; no local weight path, GPU id, or Cutlass default is baked in.
  • Follows the same minimal VAE-style wrapper pattern already used by the repo's inference examples.

Validation

  • python -m py_compile example/inference/qwen3.5-4b/infer.py example/inference/qwen3.5-4b/modeling.py
  • bash -n example/inference/qwen3.5-4b/infer.sh
  • pytest tests/example_inference/test_qwen35_4b_infer.py -q -> 6 passed

@Fatemanx Fatemanx changed the title [codex] add qwen3.5-4b inference example add qwen3.5-4b inference example Jul 21, 2026
@Fatemanx Fatemanx changed the title add qwen3.5-4b inference example [codex] add qwen3.5-4b inference example Jul 21, 2026
@Fatemanx
Fatemanx marked this pull request as ready for review July 21, 2026 12:31
@wtr0504

wtr0504 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

@Fatemanx Hello! It looks like the CI is failing on this PR. Could you take a look?

@Fatemanx
Fatemanx marked this pull request as draft August 12, 2026 01:23
@cennn cennn closed this Aug 12, 2026
@cennn cennn reopened this Aug 12, 2026
@wtr0504

wtr0504 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

@Fatemanx Thanks a lot for your contribution! We really appreciate it. We'll consider merging this PR, and let's work together to get it across the finish line. Could you rebase your branch onto main?

@Fatemanx
Fatemanx force-pushed the feat/qwen35-4b-inference-example branch from 8717c60 to 24bd0ff Compare August 13, 2026 09:59
@Fatemanx

Copy link
Copy Markdown
Contributor Author

@Fatemanx Thanks a lot for your contribution! We really appreciate it. We'll consider merging this PR, and let's work together to get it across the finish line. Could you rebase your branch onto main?

I rebase to main already.

@Fatemanx
Fatemanx marked this pull request as ready for review August 13, 2026 11:01
nvtx.switch_profile(iter_id, 0, PROFILE_CNT)


def make_text_ids(model: Qwen35ForConditionalGeneration, seq_len: int, device: torch.device) -> torch.Tensor:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current harness hardcodes batch=1 ((1, seq_len) / (1, 1)). For a compiler example this is too narrow — MagiCompiler needs to see a real batch dimension.

@wtr0504 wtr0504 Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

infer.sh already wraps nsys. Please run each mode on a real GPU and paste the nsys screenshots into the PR (timeline / CUDA HW view is enough)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants