ggml-cpu: fuse the row-max scan into the mask pass in ggml_compute_forward_soft_max_f32

[codspeed-hq]

Summary

Optimizes ggml_compute_forward_soft_max_f32 in ggml/src/ggml-cpu/ops.cpp — the CPU soft-max kernel that normalizes attention scores, exercised by the soft_max benchmark.

Analysis

Flamegraph analysis of soft_max (baseline cpuTotal ~1.83 ms, ~72% instruction-bound) showed the kernel does several separate full passes over the per-row work buffer wp: copy, scale, additive-mask, then a standalone ggml_vec_max_f32 scan to find the row maximum for the numerically-stable exponentiation. That separate max scan re-reads the entire row that the mask pass has just finished writing.

Change

When an additive mask is present (the common attention case), the row maximum is now accumulated inside the same loop that applies the mask:

for (int i = 0; i < ne00; ++i) {
    wp[i] += slope*mp[i];
    max = MAX(max, wp[i]);   // fused, was a separate ggml_vec_max_f32 pass
}

The dedicated ggml_vec_max_f32(ne00, &max, wp) scan is kept only for the no-mask path, so the total number of full passes over the row drops by one whenever a mask is used.

Correctness

The fused reduction is identical to ggml_vec_max_f32: same max = -INFINITY seed, same max = MAX(max, wp[i]) update, same iteration order, computed over the same final wp[i] values (mask already applied). The subsequent sinks correction (max = MAX(max, sk[i02])) is unchanged.

Verified with test-backend-ops test -o SOFT_MAX -b CPU: 212/212 cases pass, covering f32/f16 masks, ALiBi (max_bias), sinks, broadcast, and no-mask rows.

Performance (CodSpeed, CPU simulation)

Measured locally through the CodSpeed CLI against a clean baseline built from the default branch:

Metric	Baseline	Optimized	Δ
cpuTotal	0.0018312 s	0.0016858 s	−7.9%
instructions	0.0013175 s	0.0011721 s	−11%
cache	0.00003468 s	0.00003467 s	~0
memory	0.00047906 s	0.00047897 s	~0

The instruction-count drop of ~11% corresponds exactly to the eliminated read pass; cache and memory time are unchanged. The other 14 benchmarks are unaffected — no regressions.

[codspeed-hq]

Merging this PR will improve performance by 11.1%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 14 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	soft_max	1.8 ms	1.6 ms	+11.1%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-fuse-the-row-max-scan-into-the-mask-pass-in-ggml-c-1785123776772 (f642bed) with master (048bb65)}