Skip to content

fix: stable fgb.ps dot product and N-split for 2048 harts#21

Open
CodeDoes wants to merge 12 commits into
aifoundry-org:etfrom
CodeDoes:clean-pr19-kernel
Open

fix: stable fgb.ps dot product and N-split for 2048 harts#21
CodeDoes wants to merge 12 commits into
aifoundry-org:etfrom
CodeDoes:clean-pr19-kernel

Conversation

@CodeDoes

@CodeDoes CodeDoes commented Jul 21, 2026

Copy link
Copy Markdown

Fixes the intermittent Code 0 Type 4 mcause=4 crash in the ET Q8_0 multiply path and keeps all 2048 harts busy during generation.

This is a squashed cleanup of the kernel part of fix-q8-dot (what the reviewer asked to keep). The converter was split out to PR #22, and the uberkernel-default-ON change is split out to PR #23.

Changes

1. Fix Q8_0 gather alignment (block_ops.h)

fg32b.ps requires its address to be 32-byte aligned, but block_q8_0.qs starts at offset 2 in the struct, so alignment was unpredictable block-by-block. The old code computed a qs_aligned base + running offsets and split the 32-byte block into up to 4 chunks, patching each address with qs_aligned | ((qs_addr + off) & 31). That required 4 big if/else branches in both q8_dot_tile and q8_dot_compute_x2_aligned.

The fix is to use fgb.ps unconditionally: it gathers 4 bytes by byte offset and has no alignment restriction. Every load becomes fgb.ps f..., f31(%[apN]) with &blk->qs[0/8/16/24]. This removes the alignment math, eliminates the branches, and stops the crash from ever firing.

2. Two-row scale path (block_ops.h, q8_dot_compute_x2_aligned)

The K-block to fp32 scale multiply-accumulate was also streamlined: fbcx.ps now feeds directly into fcvt.ps.f16 in the same cycle, saving a couple of cycles per K-block versus the older extract-then-convert sequence.

3. N-split decode path (mul_mat_Q8_0.c)

Old M-split divided rows across harts: m = hart_id; m < M; m += STRIDE_M. During generation M is typically 1, which meant 2047 of 2048 harts were idle. N-split flips the outer loop so each hart gets a slice of output columns instead: it processes all M rows for its [n_start, n_end) column range. The fast path is enabled when M <= 4 && N > M && N >= 32; the original M-split loop is preserved for large-M workloads in the else branch.

Note: The Q4_0/Q4_K N-split scaffolding from PR #20 also gates on NSPLIT_MIN_N = 10000 so the test frameworks small-N correctness checks still pass. This PR uses the more aggressive M<=4 && N>=32 gate for Q8_0 because the N-split path in Q8_0 duplicates the outer loop rather than wrapping each sub-path.\n\n## Why the diff looks large\nMost of the churn is in block_ops.h: ~300 lines of alignment-branch boilerplate were deleted and replaced by a single concise gather sequence in each function. The actual new instruction logic is smaller and faster; the big headline is simplification, not expansion.\n\n## Review notes\n- Behavior change: the Q8_0 gather method changed from alignment-branched fg32b.ps to unconditional fgb.ps. Correctness hinges on fgb.ps matching or exceeding the gather bandwidth of the old path.\n- Performance: N-split should improve generation throughput when M is small (1-4). Benchmarks on real hardware (DGX-Spark / 2048-hart runtime) are the best way to validate the perf claim.\n- Risk: the duplicate loop nest added for N-split in mul_mat_Q8_0.c is straightforward, but double-check the n_start/n_end edge handling when N is not evenly divisible by 2048.\n\n## Split-out dependencies\n- PR #22 has the SpatialLM Qwen converter (was originally bundled with this kernel work; reviewer asked for conversion/* revert).\n- PR #23 has the GGML_ET_UBERKERNEL default-ON flip, kept out of this PR so reviewers can approve the kernel correctness and the default-behavior change independently.

vidas and others added 11 commits July 13, 2026 07:35
Reorder from contiguous write to read with atomic stores.
…ecovery (aifoundry-org#15)

This PR adds Q4_K MUL_MAT to the ET (ETSOC-1) backend — a scalar kernel plus a matrix-engine kernel for prefill — improves the existing Q4_0/F16/F32 matrix-engine kernels, and fixes a Q8_0 generation regression in the uberkernel path. Verified on Llama-3.2-1B-Instruct on ETSOC-1 with flash attention enabled.
…sor-engine & vectorized dots, plus uberkernel support (aifoundry-org#16)

This PR adds full ET-backend support for the Q2_K, Q3_K, Q5_K, and Q6_K super-block quantizations (previously only Q4_0/Q8_0/Q4_K existed), and substantially improves generation throughput across all K-quants — including the pre-existing Q4_K — by (a) vectorizing the generation dot product and (b) enabling the K-quant matmuls to run inside the uberkernel. For prefill, every K-quant gets a tensor-unit (matrix-engine) kernel delivering very large token/sec.

Everything is validated on test-backend-ops (MUL_MAT + GET_ROWS) and by coherent end-to-end generation on Llama-3.2-1B models.
- fix: replace fg32b.ps with fgb.ps in q8_dot_tile and q8_dot_compute_x2_aligned
  fg32b.ps requires 32-byte aligned base, but block_q8_0.qs starts at offset 2
  after fp16 d, causing intermittent Code 0 Type 4 stream error.
- perf: add N-split path for generation (M<=4) to utilize all 2048 harts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants