Vectorize the fp32 RoPE rotate_pairs kernel with AVX2/FMA

[codspeed-hq]

Summary

Adds an AVX2/FMA fast path to rotate_pairs in ggml/src/ggml-cpu/ops.cpp, the kernel that applies rotary position embedding (RoPE) to the query/key projections of every transformer layer.

Analysis

Flamegraph analysis of the rope benchmark (558 µs baseline) showed 84% of self time inside rotate_pairs<float>, split roughly evenly between instructions and memory. The kernel was a purely scalar loop rotating one pair of elements per iteration, with per-element conversions and no SIMD, despite the build targeting AVX2/FMA.

Change

For the fp32, scale == 2 case (GGML_ROPE_TYPE_NEOX / MROPE / IMROPE / VISION), the index ic = i0/2 advances contiguously, so src[ic] and src[ic + n_offset] are two contiguous fp32 spans, and the cos/sin values live interleaved in cache[2*ic] / cache[2*ic + 1].

The new path processes 8 pairs per iteration:

• loads the two contiguous operand spans into __m256 vectors,
• loads the 16 interleaved cache floats and deinterleaves them into cos/sin vectors via _mm256_shuffle_ps + _mm256_permutevar8x32_ps,
• computes dst[ic] = x0*cos - x1*sin and dst[ic+n_offset] = x0*sin + x1*cos with FMA (_mm256_fmsub_ps / _mm256_fmadd_ps),
• and stores both result spans.

A scalar tail handles the remainder. The path is guarded by #if defined(__AVX2__) && defined(__FMA__) and a runtime scale == 2 && std::is_same<T, float> check, so:

• the fp16 (ggml_fp16_t) instantiation is unchanged,
• the GGML_ROPE_TYPE_NORMAL (scale == 1) path is unchanged,
• non-AVX2 builds keep the original scalar loop.

Correctness

Verified via the repository's own tests:

• tests/test-rope — passes.
• tests/test-backend-ops test -o ROPE -b CPU — 448/448 ROPE tests passed (274 of them fp32), 0 failures, across all modes (NEOX, MROPE, IMROPE, VISION) and shapes including non-multiple-of-8 tails. The CPU output is compared against the reference implementation.

Performance (CodSpeed, CPU simulation)

Benchmark	Base	Head	Change
rope	558.2 µs	379.7 µs	+47%

14 other benchmarks unchanged, 0 regressions.

[codspeed-hq]

Merging this PR will improve performance by 47.33%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 14 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	rope	558.6 µs	379.2 µs	+47.33%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-vectorize-the-fp32-rope-rotate-pairs-kernel-with-a-1785125858504 (3a274d0) with master (048bb65)}