fix: stable fgb.ps dot product and N-split for 2048 harts#21
Open
CodeDoes wants to merge 12 commits into
Open
Conversation
Reorder from contiguous write to read with atomic stores.
…ecovery (aifoundry-org#15) This PR adds Q4_K MUL_MAT to the ET (ETSOC-1) backend — a scalar kernel plus a matrix-engine kernel for prefill — improves the existing Q4_0/F16/F32 matrix-engine kernels, and fixes a Q8_0 generation regression in the uberkernel path. Verified on Llama-3.2-1B-Instruct on ETSOC-1 with flash attention enabled.
…sor-engine & vectorized dots, plus uberkernel support (aifoundry-org#16) This PR adds full ET-backend support for the Q2_K, Q3_K, Q5_K, and Q6_K super-block quantizations (previously only Q4_0/Q8_0/Q4_K existed), and substantially improves generation throughput across all K-quants — including the pre-existing Q4_K — by (a) vectorizing the generation dot product and (b) enabling the K-quant matmuls to run inside the uberkernel. For prefill, every K-quant gets a tensor-unit (matrix-engine) kernel delivering very large token/sec. Everything is validated on test-backend-ops (MUL_MAT + GET_ROWS) and by coherent end-to-end generation on Llama-3.2-1B models.
This was referenced Jul 21, 2026
CodeDoes
force-pushed
the
clean-pr19-kernel
branch
from
July 21, 2026 17:38
9195bf9 to
c118878
Compare
- fix: replace fg32b.ps with fgb.ps in q8_dot_tile and q8_dot_compute_x2_aligned fg32b.ps requires 32-byte aligned base, but block_q8_0.qs starts at offset 2 after fp16 d, causing intermittent Code 0 Type 4 stream error. - perf: add N-split path for generation (M<=4) to utilize all 2048 harts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the intermittent
Code 0 Type 4 mcause=4crash in the ET Q8_0 multiply path and keeps all 2048 harts busy during generation.This is a squashed cleanup of the kernel part of
fix-q8-dot(what the reviewer asked to keep). The converter was split out to PR #22, and the uberkernel-default-ON change is split out to PR #23.Changes
1. Fix Q8_0 gather alignment (
block_ops.h)fg32b.psrequires its address to be 32-byte aligned, butblock_q8_0.qsstarts at offset 2 in the struct, so alignment was unpredictable block-by-block. The old code computed aqs_alignedbase + running offsets and split the 32-byte block into up to 4 chunks, patching each address withqs_aligned | ((qs_addr + off) & 31). That required 4 bigif/elsebranches in bothq8_dot_tileandq8_dot_compute_x2_aligned.The fix is to use
fgb.psunconditionally: it gathers 4 bytes by byte offset and has no alignment restriction. Every load becomesfgb.ps f..., f31(%[apN])with&blk->qs[0/8/16/24]. This removes the alignment math, eliminates the branches, and stops the crash from ever firing.2. Two-row scale path (
block_ops.h,q8_dot_compute_x2_aligned)The K-block to fp32 scale multiply-accumulate was also streamlined:
fbcx.psnow feeds directly intofcvt.ps.f16in the same cycle, saving a couple of cycles per K-block versus the older extract-then-convert sequence.3. N-split decode path (
mul_mat_Q8_0.c)Old
M-splitdivided rows across harts:m = hart_id; m < M; m += STRIDE_M. During generationMis typically 1, which meant 2047 of 2048 harts were idle.N-splitflips the outer loop so each hart gets a slice of output columns instead: it processes allMrows for its[n_start, n_end)column range. The fast path is enabled whenM <= 4 && N > M && N >= 32; the original M-split loop is preserved for large-M workloads in theelsebranch.Note: The Q4_0/Q4_K N-split scaffolding from PR #20 also gates on
NSPLIT_MIN_N = 10000so the test frameworks small-N correctness checks still pass. This PR uses the more aggressiveM<=4 && N>=32gate for Q8_0 because the N-split path in Q8_0 duplicates the outer loop rather than wrapping each sub-path.\n\n## Why the diff looks large\nMost of the churn is inblock_ops.h: ~300 lines of alignment-branch boilerplate were deleted and replaced by a single concise gather sequence in each function. The actual new instruction logic is smaller and faster; the big headline is simplification, not expansion.\n\n## Review notes\n- Behavior change: the Q8_0 gather method changed from alignment-branchedfg32b.psto unconditionalfgb.ps. Correctness hinges onfgb.psmatching or exceeding the gather bandwidth of the old path.\n- Performance: N-split should improve generation throughput whenMis small (1-4). Benchmarks on real hardware (DGX-Spark / 2048-hart runtime) are the best way to validate the perf claim.\n- Risk: the duplicate loop nest added for N-split inmul_mat_Q8_0.cis straightforward, but double-check then_start/n_endedge handling whenNis not evenly divisible by 2048.\n\n## Split-out dependencies\n- PR #22 has the SpatialLM Qwen converter (was originally bundled with this kernel work; reviewer asked forconversion/*revert).\n- PR #23 has theGGML_ET_UBERKERNELdefault-ON flip, kept out of this PR so reviewers can approve the kernel correctness and the default-behavior change independently.