Skip to content

breeze: enable bf16 activation rounding on Metal - #554

Open
gqf2008 wants to merge 2 commits into
0xShug0:mainfrom
gqf2008:breeze-bf16-metal
Open

gqf2008 wants to merge 2 commits into
0xShug0:mainfrom
gqf2008:breeze-bf16-metal

Conversation

@gqf2008

@gqf2008 gqf2008 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

BreezeTTS rounds activations to bf16 on CUDA/HIP/Vulkan to match the reference implementation. Metal was excluded from that policy; this enables it there and uses a bf16 KV cache as well.

The branch also adds the two missing bf16<->f16 copy kernels to the Metal backend, so a graph that copies between those types can be scheduled on Metal instead of falling back.

Changes

ggml-metal: add bf16<->f16 copy kernels

  • CPY, SET, DUP and CONT accepted f32 conversions and same-type copies on Metal, but not the two cross combinations of bf16 and f16.
  • Instantiate kernel_cpy_f16_bf16 and kernel_cpy_bf16_f16 (contiguous and strided) under GGML_METAL_HAS_BF16, and accept the F16<->BF16 pairs in ggml_metal_device_supports_op.

breeze: enable bf16 activation rounding on Metal

  • Include Metal in the activation-cast policy and use a bf16 KV cache.
  • Unlike CUDA/HIP/Vulkan, Metal has no fused round-to-bf16 unary op, so fused_round stays disabled and the cast is a separate graph node.

Verification

On macOS (Apple M4, Metal 4) with both targets built (audiocpp_cli, audiocpp_server):

  • audiocpp_cli --task tts --family breeze_tts --model breeze-tts-2-q8_0.gguf --backend metal runs to completion and produces audio.
  • The bf16 path is active in the run: the Metal library loads kernel_cpy_bf16_bf16, kernel_cpy_bf16_f32, kernel_cpy_f32_bf16 and kernel_set_rows_bf16_i32.
  • The two new kernels are not on the breeze path — the graph converts through f32 rather than directly between bf16 and f16. They are included as a backend capability (same shape as the Vulkan change in ggml-vulkan: add bf16<->f32/f16 cpy pipelines), not as a requirement of the BreezeTTS change: enabling the policy alone builds and runs identically.

CPY, SET, DUP and CONT accepted f32 conversions and same-type copies on
Metal, but not the two cross combinations of bf16 and f16, so a graph
copying between them could not be scheduled on the backend.

Instantiate kernel_cpy_f16_bf16 and kernel_cpy_bf16_f16 (contiguous and
strided) under GGML_METAL_HAS_BF16, and accept the F16<->BF16 pairs in
ggml_metal_device_supports_op.
BreezeTTS rounds activations to bf16 on CUDA/HIP/Vulkan to match the
reference implementation. Enable the same policy on Metal and use a bf16
KV cache there as well.

Unlike CUDA/HIP/Vulkan, Metal has no fused round-to-bf16 unary op, so
fused_round stays disabled and the cast is a separate graph node.
@0xShug0 0xShug0 added the ggml label Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants