Conversation
CPY, SET, DUP and CONT accepted f32 conversions and same-type copies on Metal, but not the two cross combinations of bf16 and f16, so a graph copying between them could not be scheduled on the backend. Instantiate kernel_cpy_f16_bf16 and kernel_cpy_bf16_f16 (contiguous and strided) under GGML_METAL_HAS_BF16, and accept the F16<->BF16 pairs in ggml_metal_device_supports_op.
BreezeTTS rounds activations to bf16 on CUDA/HIP/Vulkan to match the reference implementation. Enable the same policy on Metal and use a bf16 KV cache there as well. Unlike CUDA/HIP/Vulkan, Metal has no fused round-to-bf16 unary op, so fused_round stays disabled and the cast is a separate graph node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BreezeTTS rounds activations to bf16 on CUDA/HIP/Vulkan to match the reference implementation. Metal was excluded from that policy; this enables it there and uses a bf16 KV cache as well.
The branch also adds the two missing bf16<->f16 copy kernels to the Metal backend, so a graph that copies between those types can be scheduled on Metal instead of falling back.
Changes
ggml-metal: add bf16<->f16 copy kernelsCPY,SET,DUPandCONTaccepted f32 conversions and same-type copies on Metal, but not the two cross combinations of bf16 and f16.kernel_cpy_f16_bf16andkernel_cpy_bf16_f16(contiguous and strided) underGGML_METAL_HAS_BF16, and accept theF16<->BF16pairs inggml_metal_device_supports_op.breeze: enable bf16 activation rounding on Metalfused_roundstays disabled and the cast is a separate graph node.Verification
On macOS (Apple M4, Metal 4) with both targets built (
audiocpp_cli,audiocpp_server):audiocpp_cli --task tts --family breeze_tts --model breeze-tts-2-q8_0.gguf --backend metalruns to completion and produces audio.kernel_cpy_bf16_bf16,kernel_cpy_bf16_f32,kernel_cpy_f32_bf16andkernel_set_rows_bf16_i32.ggml-vulkan: add bf16<->f32/f16 cpy pipelines), not as a requirement of the BreezeTTS change: enabling the policy alone builds and runs identically.