Eval bug: WebGPU hy_v4 crashes on macOS in test-llama-archs and test-save-load-state

[22elix3r]

Name and Version

version: 0.4.0-dev (build 10902, commit df03399)

affected commit: df03399

Operating systems

Mac

GGML backends

WebGPU

Hardware

GitHub-hosted macOS ARM64 runner
Apple M1 (Virtual)
WebGPU backend using Dawn
Runner image: macos-26-arm64
Runner image version: 20260831.0337.3
Operating system: macOS 26.6.2 (25G83)

Models

Synthetic Hy4 MoE fixture generated by test-llama-archs: build/tests/test-models/hy_v4-moe.gguf

It is a test-generated GGUF fixture.

Problem description & steps to reproduce

The macOS WebGPU CI crashes when testing the hy_v4 architecture.

The failure occurs during architecture execution in test-llama-archs and during the initial model evaluation in test-save-load-state.

• test-llama-archs terminates with SIGSEGV.
• test-save-load-state terminates with SIGBUS.

The Ubuntu WebGPU job passes the same test suite. The observed failure is specific to the hosted macOS WebGPU job.

The issue is still present on master at commit df03399b885831b2a1603b3abb0d8c156808e363.

The WebGPU workflow has failed on all 65 master runs since Hy4 was introduced. The first and current failures both occur on hy_v4.

Steps to reproduce

Use the same configuration as the macOS WebGPU CI job.

After the workflow's Dawn Dependency step, build and test with:

export CMAKE_PREFIX_PATH=dawn

cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_WEBGPU=ON \
  -DGGML_METAL=OFF \
  -DGGML_BLAS=OFF

cmake --build build

cd build
ctest -L main --verbose --timeout 900

Expected behavior

Hy4 execution should complete, fall back to a supported backend, or be skipped as unsupported. It should not terminate the process.

First Bad Commit

49c0dc8
model : add Tencent Hy 4 (hy_v4) preview architecture support (#28127)

The preceding WebGPU run passed, while the run at this commit failed.

Relevant log output

Logs

Start 27: test-llama-archs
Test command: /Users/runner/work/llama.cpp/llama.cpp/build/bin/test-llama-archs

|     Model arch.|            Device|Config|   NMSE vs. CPU|Roundtrip|
|           hy_v3|            WebGPU|   MoE|  OK (8.82e-08)|       OK|
|           hy_v3|Apple M1 (Virtual)|   MoE|  OK (0.00e+00)|       OK|
|           hy_v4|            WebGPU|   MoE|
Test #27: test-llama-archs ...........................***Exception: SegFault

Start 33: test-save-load-state
Test command: /Users/runner/work/llama.cpp/llama.cpp/build/bin/test-save-load-state --models /Users/runner/work/llama.cpp/llama.cpp/build/tests/test-models/

main: model /Users/runner/work/llama.cpp/llama.cpp/build/tests/test-models/hy_v4-moe.gguf
load: adding 128 dummy tokens
run_save_load_tests_for_model: no prompt provided, generating 100 (n_batch) random tokens
run_save_load_tests_for_model: the input prompt is 100 tokens
Test #33: test-save-load-state .......................Bus error***Exception

96% tests passed, 2 tests failed out of 54

The following tests FAILED:
    27 - test-llama-archs (SEGFAULT)
    33 - test-save-load-state (Bus error)

Errors while running CTest
Process completed with exit code 8.

[fairydreaming]

Current status: the test-llama-archs issue is fixed (HY_V4 arch tests disabled for WebGPU), but webgpu macos CI still crashes in:

The following tests FAILED:
	 30 - test-recurrent-state-rollback-dsv4 (SIGTRAP)      main
	 32 - test-save-load-state (SIGTRAP)                    main

From what I see in logs both crashes happen when testing DeepSeek V4.

[fairydreaming]

I did some more research by rebasing #28855 (HY_V4 arch tests disabled for WebGPU) on various commits and checking if WebGPU CI passes. From this it looks like commits up to (and including) 1dfe94e pass, while the next one 3bcfeb7 and following ones fail. The failing commit is from #28091.

From a brief look the PR was about precompiled headers so no idea why WebGPU tests would fail because of this. But there are some commits with fixes later, so maybe there were problems with this.

@danbev Any idea why #28091 would cause MacOS WebGPU CI to fail?

[ggerganov]

@fairydreaming I think this was resolved with #28882