Skip to content

ggml-cpu: add AVX2 vec_dot for Q2_0 x Q8_0 on x86 - #164

Open
nicode1990 wants to merge 1 commit into
Anbeeld:mainfrom
nicode1990:q2_0-avx2
Open

nicode1990 wants to merge 1 commit into
Anbeeld:mainfrom
nicode1990:q2_0-avx2

Conversation

@nicode1990

Copy link
Copy Markdown

Overview

Adds an AVX2 ggml_vec_dot_q2_0_q8_0 for x86. The x86 build had no kernel for it: arch-fallback.h mapped it to the generic scalar loop (only ARM has NEON; ggml-org#26348 upstream covers VNNI only, not AVX2-only CPUs like Zen 2/3).

Q2_0 stores weight 4*b + j in bits [2j, 2j+1] of byte b. The 8 raw bytes are broadcast into four 64-bit lanes, lane j is shifted by 2j, which gives order 8*j + b; the Q8_0 block is permuted to the same order, then the existing mul_sum_i8_pairs_float + FMA pattern of the Q4_0 kernel applies.

Additional information

Ryzen 5 5600X (AVX2, no VNNI), RTX 3060 12 GB, 32 GB DDR4, gcc 13.3, -DGGML_CUDA=ON -DGGML_NATIVE=ON. Model: Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf (ISTA-DASLab, all routed experts Q2_0). Before = the same build without this commit (scalar fallback).

./build/bin/llama-bench -m Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf -p 0 -n 64 -r 2 -ngl 99 -ncmoe 38 -fa on -lzm on -t 6

tg64: 4.17 -> 10.15 tok/s (64 tokens are warm-up dominated on 32 GB; -n 256 gives 16.03 +/- 4.26, llama-server 15-20 tok/s).

./build/bin/test-backend-ops test -b CUDA0 -o MUL_MAT -p "type_a=q2_0,"     # OK
./build/bin/test-backend-ops test -b CUDA0 -o MUL_MAT_ID -p "type_a=q2_0,"  # 75/75
./build/bin/test-quantize-fns                                               # q2_0 dot product ok, 0 failed

Uses FMA like the neighbouring kernels, so not bit-identical to the scalar loop (same as noted in ggml-org#26348).

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - kernel and this description written with Claude Code from my problem statement and measurements; reviewed and tested by me.

Q2_0 had no x86 kernel: arch-fallback.h mapped ggml_vec_dot_q2_0_q8_0 to
the generic scalar loop (only ARM had a NEON path). On a MoE model with all
routed experts in Q2_0 (Qwen3.8-Flash-Next GSQ-RCO) this made CPU-side expert
layers the bottleneck: ~4.2 tok/s at -ncmoe 38 on a 6-core box, versus 15-20
tok/s with this kernel.

The AVX2 path broadcasts 8 raw Q2_0 bytes into four 64-bit lanes, shifts
lane j by 2j, masks to 2 bits and subtracts 1, then permutes the Q8_0 block
into the same (8*j + b) order and uses mul_sum_i8_pairs_float.

Verified with test-backend-ops against CUDA0: MUL_MAT (type_a=q2_0) all OK,
MUL_MAT_ID 75/75.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant