ci(e2e): run Qwen and DS4 end to end on lucebox3

[Graffioh]

What

A new Model e2e workflow (.github/workflows/model-e2e.yml) runs a real model on lucebox3 and checks whether a change crashes it or changes its behaviour.

• Qwen3.8-27B (IQ4_XS + dflash2 draft) runs on the R9700 (gfx1201).
• DeepSeek V4 Flash (STRIX mix, sparse prefill) runs on the Strix Halo (gfx1151).

Each job builds luce_server for its GPU, loads the model, sends 16 fixed prompts with greedy decoding (.github/ci/e2e/prompts.json), stops the server, and compares with the last good run on main (the baseline). The prompts cover arithmetic, lists, code, JSON, translation, unicode, multi-turn, a tool call, a 2.5K-token needle, streaming, thinking, a longer story, and a repeated request.

Result	When
❌ fail	crash · hang · server won't load or stop · GPU errors in the kernel log · a request fails or times out · the suite runs out of time · a model or the draft is missing (a PR skips instead) · 2+ checks that passed on the baseline now fail · under half the checks pass
⚠️ warn	output text differs from the baseline (with the character where it first differs) · a single regressed check · a repeated request answers differently · decode speed −15% · load time +50% · the kernel log can't be read, or lost messages during the run

When it runs

• Only when a maintainer adds the e2e label, like compact-commits in ci(commits): check conventional commit style and dispatch the compaction bot #755. It runs the models the PR touches (select_models.py), or both when the PR touches neither DS4, Qwen nor shared server code. To test new pushes, remove the label and add it again.
• Every merge to main: runs the models whose code changed since their baseline (find_baseline.py --only-changed), and a run that doesn't fail becomes the new baseline. The diff is measured from the baseline's commit, not the previous one, so a merge whose run was skipped, replaced in the queue or failed is covered by the next merge. Manual dispatch refreshes the baseline when update_baseline is ticked on main (e.g. after a ROCm upgrade).

lucebox3 only

• Routing: jobs use runs-on: [self-hosted, lucebox3], next to gpu-tests-amd. The runner runs one job at a time, so e2e jobs never share the GPUs with each other or with gpu-tests-amd. Spreading jobs over more luceboxes can come later.
• One table per model: select_models.py holds each model's GPU, HIP index (R9700 = 0, Strix Halo = 1, as in gpu-tests-amd), model files and server flags. The job reads them from its matrix. Models live in vars.LUCEBOX_MODELS_DIR, default /opt/models.
• Baseline: baseline runs on main upload their result as the artifact model-e2e-baseline-<model>-<device> (kept 90 days). find_baseline.py gives each job the newest one, counting only model-e2e runs from pushes to or dispatches on this repo's main, so PR code can't replace it. Results record the ROCm version, and the report warns when the baseline ran on a different one.
• Someone using the GPU by hand: the job waits up to 4 minutes, then skips with a warning. To keep jobs off while you benchmark, stop lucebox3's runner service.
• Seeing every user's GPU processes: the job reads the kernel's KFD process list (/sys/class/kfd/kfd/proc, readable by any user; it's what rocm-smi --showpids uses) and fails if it can't.

Staying bounded

• 35-minute job cap, plus caps per step: build 12 min, suite 15 min, model load 480 s (DS4 takes ~185 s from a cold page cache, ~30 s warm), 120 s per request, 300 s for the whole suite.
• Skips repeats: a clean pass (not a warning, which must show again) is remembered by tree hash, configuration, model files, ROCm version and the baseline's digest. The suite is judged against a snapshot of the baseline taken with the key, so a new baseline always re-runs it. A skipped run shows the report of the run it reused.
• Builds incrementally: the build folder persists on lucebox3 and uses ccache if installed.

Queue fix (separate commit)

By default a concurrency group keeps only one waiting job, so a second PR's waiting GPU job cancelled the first one's. queue: max (GitHub, May 2026) keeps up to 100 waiting, first in first out, on gpu-tests, gpu-tests-amd and speed-profile. actionlint doesn't know the key yet, but GitHub's docs confirm it at job level with cancel-in-progress: false.

Measured on lucebox5 (same R9700 + Strix Halo hardware)

	cold build	suite incl. load	load	decode
Qwen3.8-27B IQ4_XS + dflash2	100 s	~30 s	6–18 s	~105 tok/s
DS4 STRIX mix, sparse prefill	95 s	~80 s	28–31 s warm, 185 s cold	~19 tok/s

• Deterministic: both models gave identical text on all 16 prompts across two runs.
• Workflow steps replayed: I ran the job's shell steps on lucebox5 with Qwen: persistent build, suite (16/16), pass marker, a cached skip, a re-run once a baseline appeared (same text on all prompts), and writing the new baseline. That was before the switch to lucebox3 only, which dropped the GPU-index lookup and model search from those steps. gpu_wait.sh reports busy when another process holds /dev/kfd. find_baseline.py ran against the live API; the artifact download itself can only run once a baseline exists on main.
• Why DS4 uses sparse prefill: exact prefill runs at about 21 tok/s on the Strix Halo, and the 2.5K-token needle alone took 120 s.
• DS4 arithmetic, for reviewers: exact prefill says 60/1.5 = 10, sparse says 17×23 = 340. Both are deterministic, and Qwen gets both right. That's why a single regressed check only warns. It may be a real accuracy issue worth checking separately.

Before enabling

• Put the three models in /opt/models on lucebox3 (or set LUCEBOX_MODELS_DIR): Qwen3.8-27B-UD-IQ4_XS.gguf, qwen38-dflash2-q8_0.gguf, DeepSeek-V4-Flash-0731-ROCMFPX-MIX-STRIX.gguf.
• Make the kernel log readable for the runner user on lucebox3: passwordless sudo dmesg or kernel.dmesg_restrict=0. Without it every run warns that GPU errors weren't checked.
• After merging, dispatch the workflow on main with update_baseline ticked to create the first baseline for both models (the first merge also does it).
• Create the e2e label.
• New kernel-log lines are matched by their boot timestamp, not by line count, so ring-buffer rollover can't hide a fault. When the log after the run no longer reaches back to the messages from before it, the report says so and warns.

🤖 Generated with Claude Code

[davide221]

The commit compaction bot stopped.

The pull request is still a draft.

Maintainers: see the lucebox_bots run log.

[davide221]

The commit compaction bot stopped.

Pushing to Graffioh/lucebox-hub failed (the branch moved, the fork blocks maintainer edits, or the token lacks the workflow scope). The branch is unchanged.

Maintainers: see the lucebox_bots run log.

[davide221]

Folded 2 commits into 2 (same final code; the tree hash was checked before pushing).

New commit	From
158ac7ab ci(workflows): queue GPU jobs instead of replacing waiting ones	341c98c
7ae026fb ci(e2e): run Qwen and DS4 end to end on any free lucebox	437f06c

Previous head: 437f06c2ce7c5c7b036de33f0e0d43782cf7495c.

[davide221]

Folded 3 commits into 2 (same final code; the tree hash was checked before pushing).

New commit	From
2b7e938d ci(workflows): queue GPU jobs instead of replacing waiting ones	875bd28
4eff4ad8 ci(e2e): run Qwen and DS4 end to end on any free lucebox	09f3658, 1121a5e

Previous head: 1121a5e57bd44975931025e6812e143b505e5347.

[cubic-dev-ai]

All reported issues were addressed across 12 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[davide221]

Folded 3 commits into 2 (same final code; the tree hash was checked before pushing).

New commit	From
94389b05 ci(workflows): queue GPU jobs instead of replacing waiting ones	c84282f
e5d1ffd9 ci(e2e): run Qwen and DS4 end to end on lucebox3	9bc0f99, 96e2c82

Previous head: 96e2c825e711e543ce3fe7ae577d2df12ae282ad.

[cubic-dev-ai]

Review completed against the latest diff

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>