Conversation
added 8 commits
September 10, 2026 00:39
Measured per-device bandwidth via llama-bench single-device probes (5060 Ti Q4 27B tg 25.95 => 427 GB/s, 3060 9B proxy 60.74 => 345 GB/s, not spec sheets). Q5 27B does not fit single 5060 Ti (spills to CPU, pp collapses 978->70), so 3060 uses 9B/4B proxies. Roofline predicts 37.0 t/s raw (ts-pinned) and 41.4 t/s raw balanced, both >=48 t/s with conservative 1.3x MTP, so GO for PR104.1. Assisted-by: opencode/muse-spark-1.2-contributor
- per-device slice_type[] on ggml_tensor_extra_gpu (common.cuh:1222) - LLAMA_ARG_SPLIT_ROW_QUANT=q4_k,q6_k helper with call_once parsing, lower-case ggml_type_name matching, default/fallback, quantized guard, GGML_LOG_INFO active log + static init to make rebuild-free A/B observable - build verified with CUDACXX 13.2.2 arch 86;120a FA all quants (ggml-cuda 417s, llama-server link ok) - live single-GPU verification: env active log, VRAM 14137 MiB, health ok, no bytes change yet (split buffer not yet reintroduced; cuda split only in sycl). Comparison real 25.9 vs projection 53.8-78.7 pending full split buffer wiring for PR104.2 (get_row_rounding/get_row_split/ ggml_nbytes_split + init/set/mul_mat dispatch from slice_type) - docs: append PR104.1 Spike Results
Storage/rebalancing proven via VRAM (9B 2747/2929, 27B 8283/9001 vs stub 14137 flat) from ported CUDA split buffer (get_row_rounding/split, nbytes_split, split_buffer_type, init/set/get, contiguous_data). Compute path blocked: illegal access at common.cuh:1641 cudaFuncGetAttributes during graph capture, reproduces even vanilla row-split (isolated to split-src MUL_MAT -> scale), needs compute-sanitizer + MMQ peer-async port. Recommendation defer; rig clean 1/1, not restarting production for PR105.0. Updates PR #113 on fork/pr104-mixed-quant-arm.
- gate CUDA graph capture off for graph nodes with split-buffer srcs (mid-capture false returns were silently skipped in release builds, leaving malformed captured graphs) - harden the contiguous-gather guard: abort on contiguous copy on the wrong compute device instead of reading across devices without peer access - gate fusion windows in try_fuse so fused kernels (which bypass the per-op gather and read split tensor->data directly) never dispatch with split-backed srcs - document why split get_base must stay a pseudo pointer
…needs per-op split dispatch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arm series spec for per-device quantization of row-split weight slices: Q4_K rows on the 3060, Q5/Q6_K rows on the 5060 Ti. Row ratios unchanged; bytes-per-token rebalanced onto bandwidth-proportional lines. Attacks the rig's actual wall (3060 bandwidth x VRAM-forced share) instead of porting htsglang-style asymmetric TP.
What this PR contains
Docs only (
docs/arms/pr104-mixed-quant-row-sharding.md):LLAMA_ARG_SPLIT_ROW_QUANT=q4_k,q6_k(default = model type, rebuild-free A/B); bars >= 43 ship-min / >= 45 targetggml_tensor_extra_gpu,init_tensor/set_tensorchanges (ggml-cuda.cu:929/979),mul_matdispatch fromslice_type[id]; no graph/kernel/ggml-core changesTest plan
Sequencing: PR104.0 can run in parallel with PR103.0 coding; PR104.1/2 build on PR103.0's landed base.
AI usage disclosure: YES — arm spec drafted by AI (opencode) from local row-split source verification (split buffer single-type limitation at ggml-cuda.cu:929/950/979); implementation code to be human-authored/owned.