Comprehensive FFTA-vs-FFTW benchmark suite with committed report

[pankgeorg]

This adds a second, broader benchmark suite next to the existing run_benchmarks.jl, plus a committed markdown report with results from an aarch64 machine.

What it measures (benchmark/suite.jl)

• 1D sizes from 8 to 2^22 in four classes: powers of two, other smooth sizes (2^a·3^b·5^c·7^d), primes (both sides of the Bluestein cutoff), and prime × small factor.
• 2D (square and non-square), 3D, and batched transforms along one dimension of a matrix (dims=1 contiguous, dims=2 strided).
• ComplexF64/ComplexF32 fft and Float64/Float32 rfft.
• Plan construction time and planned execution time separately, plus the one-shot (plan + execute) cost; FFTW with ESTIMATE and, for powers of two, MEASURE.
• Bytes allocated per planned execution.
• FFTW multi-threaded vs single-threaded (FFTA is single-threaded).
• Every FFTA result is checked against FFTW (rel. err column).

How — both packages are loaded in one process; FFTA's plan_* methods are reached with invoke because FFTW's StridedArray methods are more specific (this is also why the existing suite uses two processes). A small custom timing loop is used instead of @benchmark, whose per-call-site compilation dominated a ~500-case sweep. benchmark/report.jl renders REPORT.md and SVG plots with no plotting dependency.

Results — see benchmark/REPORT.md. Headline (aarch64 Neoverse-N1, Julia 1.12.6, FFTW 3.3.11, ComplexF64, planned execution, FFTA / FFTW):

type	pow2	smooth	prime	prime×small	2D	3D	batched dims=1	batched dims=2
ComplexF64 fft	2.6× (1.0–7.1)	7.3× (2.6–14.7)	5.8× (3.1–10.6)	5.6× (3.4–11.4)	5.2× (1.8–13.3)	7.6× (3.5–15.1)	3.1× (2.0–5.7)	2.2× (1.2–5.8)
ComplexF32 fft	3.8× (1.6–7.1)	10.2× (3.2–21.5)	7.5× (3.8–13.4)	7.5× (4.8–16.5)	6.1× (1.9–15.1)	11.6× (8.1–17.1)	5.1× (3.1–9.2)	2.9× (1.4–9.7)
Float64 rfft	3.3× (1.4–7.3)	8.7× (3.1–25.3)	6.3× (2.6–11.9)	6.2× (3.6–16.7)	11.8× (5.7–25.5)	unsupported	4.9× (2.9–9.7)	3.1× (1.4–10.3)
Float32 rfft	4.5× (2.8–7.2)	10.5× (3.4–24.8)	6.6× (2.8–10.9)	7.7× (5.2–17.1)	13.2× (7.5–21.9)	unsupported	6.4× (4.6–9.0)	3.5× (1.6–9.4)

(geometric mean and range of FFTA/FFTW execution time, FFTW planned with ESTIMATE, single thread.) Other observations from the report:

• Powers of two converge to ~1.0–1.3× of FFTW-ESTIMATE at n ≥ 2^20, but FFTW-MEASURE plans are 2.2–2.8× faster than ESTIMATE there, so against tuned FFTW the large-size gap is ~2.5–3×.
• Plan creation is much cheaper in FFTA (0.1–30 µs vs 2 µs–190 ms for FFTW ESTIMATE; large primes are slow to plan in FFTW), so one-shot fft(x) on primes is closer than the planned numbers suggest.
• Planned complex mul! allocates nothing except through Bluestein (12 KiB–200 MiB per call for primes ≥ 73, also inside composites) and ND transforms (pencil buffers, 0.4–66 KiB); real plans only support *, so they always allocate; rfft along dims allocates per column via mapslices.
• ComplexF32 is no faster than ComplexF64 in FFTA (FFTW: 1.6–1.8× faster).
• FFTW with 8 threads is 7–20× faster than FFTA on n ≥ 2^16, 2D ≥ 256² and batched transforms.

The report is from an aarch64 machine; an x86-64 run would be useful and only needs cd benchmark; julia --project=. -t 8 suite.jl; julia --project=. report.jl results_suite.json.

Two things noticed while writing this, which I'll open separately: with FFTW.jl and FFTA.jl both loaded, plan_rfft(::Vector{Float64}, ::Int) (hence rfft(x)) is a method ambiguity; and the profile of composite sizes is dominated by sincospi inside singleton_params, i.e. twiddles are recomputed per execution rather than stored in the plan.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.80%. Comparing base (7aeb327) to head (fcc7ecb).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #128   +/-   ##
=======================================
  Coverage   98.80%   98.80%           
=======================================
  Files           5        5           
  Lines         585      585           
=======================================
  Hits          578      578           
  Misses          7        7           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dannys4]

I'm fine adding different/more benchmarks into the package, but I'm not going to add images into the git package. It makes the repo unnecessarily large---if people want to generate them, they are more than welcome to run whatever benchmark code is provided.