fix: use CPU-aware prover scheduler

[jotabulacios]

This PR fixes the CPU proving regression introduced by the table scheduler.

The GPU scheduler uses external driver threads to admit and overlap table work. On CPU, that caused table tasks to launch nested Rayon work from outside the Rayon pool, leading to oversubscription, cache contention, and memory-bandwidth contention.

The CPU path now:

• schedules table work through the existing Rayon pool;
• limits table concurrency in bounded chunks; and
• separates auxiliary-trace construction/commitment from rounds 2–4 with a CPU phase barrier.

The CUDA path and proof format are unchanged.

[jotabulacios]

/bench

[github-actions]

Benchmark — real block (ethrex_mainnet_25368371.bin) (median of 3)

_{continuations · epoch 2^22 · 10 epochs}

Metric	main	PR	Δ
Peak heap	47344 MB	52784 MB	+5440 MB (+11.5%) 🔴
Prove time	136.925s	121.870s	-15.055s (-11.0%) 🟢

🎉 Improvement on the real block — prove time down 11.0%.

_{Prove-time spread 2.6% (121.870s / 123.141s / 119.938s)}

Memory Growth

_{ethrex distinct-account transfers · default parallelism · 1 sample per point}

Transfers	main (MB)	PR (MB)	Δ
4	12242	14631	+2389 MB (+19.5%)
8	14928	18288	+3360 MB (+22.5%)
12	19774	21514	+1740 MB (+8.8%)
16	21972	25555	+3583 MB (+16.3%)
20	23838	29814	+5976 MB (+25.1%)

Growth rate: 941 MB / transfer (main: 756, Δ: +24.5%)
Fit: R² = 0.9973 (main: 0.9717)

⚠️ Memory scaling regression — growth rate increased by +24.5%

_{Commit: 5bba8d4 · Baseline: cached · Runner: self-hosted bench}

[diegokingston]

/bench

[MauroToscano]

/bench-growth

[MauroToscano]

Feedback incoming

[jotabulacios]

/ai-review

[github-actions]

Codex Code Review

No actionable issues found in the PR diff.

[claude]

Review: CPU-aware prover scheduler

Read the diff against crypto/stark/src/prover.rs plus prover/src/auto_storage.rs. The cfg split is sound — I checked all four (cuda x debug-checks) combinations and exactly one run_admitted and one table_results is defined in each, with no gap or duplicate. The correctness-critical parts hold up: aux_outs comes back in index order so staged[idx] pairs correctly, each index appears once in peak_order so the take() never sees None, every slot is filled so the expects are unreachable, and transcript/proof ordering is untouched. No unsafe, no new panic paths on valid input.

No Critical or High findings. Everything below is Medium or lower.

Medium

1. The chunk barrier comes back. order.chunks(workers) reintroduces precisely what the CUDA sibling's doc comment says it replaced: 'the fixed chunks this replaces made every table wait for the slowest of its chunk.' Because order is heaviest-first, chunk 0 contains the long pole and the other k-1 lanes idle on it. rayon::scope plus the existing atomic cursor gets you pool-resident drivers and a k cap and no barrier — see the inline comment for the ~10-line version. If chunking measured faster than that, the comment should say so.

2. Peak host RAM rises on CPU. staged holds all N tables' Lde at once where the fused path held k; the aux half is ext3, so this is roughly (N - k) * aux_cols * 24 * lde_size of extra live heap. Not a storage misprediction (persistent_per_table already counts aux for every table, so peak_bytes goes from over-estimate to exact), but two doc comments now state the opposite invariant and should be updated here: struct Lde in prover.rs and peak_bytes in auto_storage.rs. The trade-off is also worth a line in the PR description.

Low

3. The Vec<Mutex<Option<T>>> in the new parallel arm is unnecessary — all writes happen on the calling thread after collect() joins the chunk. A plain Vec<Option<T>> works, as the not(parallel) fallback already does. Also, both new arms size results by order.len() but index by idx taken from order; estimates.len() is the invariant that actually holds.

4. The new CPU block is a near-copy of the debug-checks block below it. Gating on any(not(feature = "cuda"), feature = "debug-checks") with an inline cfg'd run_debug_checks collapses three definitions of table_results into two.

5. On CPU builds VramGate::acquire/VramPermit are now fully dead, kept alive by three allow(dead_code) attributes, and _gate is a vestigial parameter. Either gate the type on cuda or document that it is a CUDA-only mechanism. Separately, the + Send added to the CUDA task bound isn't required — the workers capture &task, so Sync suffices.

Nothing here blocks merge; (1) and (2) are the two I would want addressed, at minimum the stale comments in (2).

[github-actions]

AI Review

PR #897 · 1 changed files

Findings

Status	Sev	Location	Finding	Found by
uncertain	low	crypto/stark/src/prover.rs:740	impl Drop for VramPermit missing the allow(dead_code) its siblings got on non-cuda	glm openrouter/z-ai/glm-5.2
confirmed	low	crypto/stark/src/prover.rs:842	heaviest_first uses unstable sort — non-deterministic scheduling for equal estimates	nemotron openrouter/nvidia/nemotron-3-ultra-550b-a55b

Status column reflects the verdict from the verifier: deepseek-verifier (openrouter/deepseek/deepseek-v4-pro).

AI-005: `impl Drop for VramPermit` missing the `allow(dead_code)` its siblings got on non-cuda

• Status: uncertain
• Severity: low
• Location: crypto/stark/src/prover.rs:740
• Found by: glm:openrouter/z-ai/glm-5.2
• Verified by: -
• Rejected by: -

Claim

On non-cuda builds VramGate::acquire is never called (the CPU/sequential run_admitted variants take _gate: &VramGate and ignore it), so VramPermit is never constructed and its Drop::drop is never invoked. The PR silenced dead-code on struct VramGate, struct VramPermit, and impl VramGate with #[cfg_attr(not(feature = "cuda"), allow(dead_code))], but the same attribute is missing on impl Drop for VramPermit, so that impl may emit a dead_code warning on the default (non-cuda) build.

Evidence

Lines 705/712/718 carry #[cfg_attr(not(feature = "cuda"), allow(dead_code))]; line 740 impl Drop for VramPermit<'_> has no such attribute. acquire (line 728) is the only constructor of VramPermit and it is dead on non-cuda, so VramPermit is never instantiated there.

Suggested fix

Add #[cfg_attr(not(feature = "cuda"), allow(dead_code))] above impl Drop for VramPermit to match the other three items, or hoist a single #[cfg_attr(not(feature = "cuda"), allow(dead_code))] module/region attribute over the whole VramGate/VramPermit block.

AI-009: heaviest_first uses unstable sort — non-deterministic scheduling for equal estimates

• Status: confirmed
• Severity: low
• Location: crypto/stark/src/prover.rs:842
• Found by: nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b
• Verified by: deepseek-verifier:openrouter/deepseek/deepseek-v4-pro
• Rejected by: -

Claim

order.sort_by_key(|&i| std::cmp::Reverse(estimates[i])) uses unstable sort. Tables with identical VRAM estimates get non-deterministic order, affecting scheduling and potentially proof generation timing (though not correctness).

Evidence

Lines 839-844: fn heaviest_first(estimates: &[u64]) -> Vec<usize> { let mut order: Vec<usize> = (0..estimates.len()).collect(); order.sort_by_key(|&i| std::cmp::Reverse(estimates[i])); order }

Suggested fix

Use sort_by with a tie-breaker on index: order.sort_by(|&a, &b| estimates[b].cmp(&estimates[a]).then(a.cmp(&b))) for deterministic ordering.

Reviewer Lanes

Lane	Model	Prompt	Status	Findings
glm	openrouter/z-ai/glm-5.2	general	success	2
kimi	openrouter/moonshotai/kimi-k2.7-code	general	success	2
minimax	minimax/MiniMax-M3	general	error: opencode failed (provider/auth/runtime error) and no findings were submitted	0
moonmath	zro/minimax-m3	general	error: opencode failed (provider/auth/runtime error) and no findings were submitted	0
nemotron	openrouter/nvidia/nemotron-3-ultra-550b-a55b	general	success	6

Verification Lanes

Lane	Model	Status	Confirmed	Rejected	Uncertain
deepseek-verifier	openrouter/deepseek/deepseek-v4-pro	success	1	6	1

Native Codex and Claude reviews run separately and post their own comments. They are not included in this structured provenance report.

Discarded candidates (6) — rejected by the verifier

• CPU scheduler buffers all tables' LDEs, invalidating RAM estimate (crypto/stark/src/prover.rs:4042, found by kimi:openrouter/moonshotai/kimi-k2.7-code, glm:openrouter/z-ai/glm-5.2) — The auto_storage::peak_bytes function counts all tables' main+aux LDEs as 'persistent' (all-N-live) via persistent_per_table, which already includes the aux LDE for every table summed across all specs. The code comment at lines 246-250 explicitly says this 'is exact on cuda and an over-estimate on CPU rather than an unsound bound.' The CPU scheduler storing all LDEs in staged between passes does not create an underestimate — the persistent_total already assumes all-N-live for LDEs. The claim that the estimator underestimates and may OOM is incorrect.
• VramGate admits oversized requests when idle but provides no backpressure for subsequent tasks (crypto/stark/src/prover.rs:730, found by nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b) — The behavior is intentional and documented: the doc comment on lines 699-700 states 'An oversized request is admitted alone (when nothing else holds bytes), so tables larger than the whole budget still prove.' The scheduler uses heaviest_first ordering, so the oversized table is always admitted first and deterministically. There is no 'burst' scenario — the order is fixed at call time. This is a deliberate design choice, not a bug.
• CPU run_admitted uses fixed chunk barriers (crypto/stark/src/prover.rs:808, found by kimi:openrouter/moonshotai/kimi-k2.7-code, nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b) — The fixed-chunk pattern is an intentional design tradeoff. The doc comment at lines 790-795 explains that keeping work inside the Rayon pool preserves work-stealing behavior and avoids losing it from nested Rayon work launched from OS threads. Lines 4034-4039 explicitly state that interleaving CPU-heavy regions 'regresses the host prover through cache and memory-bandwidth contention.' This is a deliberate phase-barrier design, not an unintentional reintroduction of a problem.
• VramGate::acquire notifies all waiters on each release (thundering herd) (crypto/stark/src/prover.rs:745, found by nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b) — notify_all() is the correct and safe primitive for a condition variable when the condition is re-checked in a loop (as it is at lines 730-736). The number of waiting threads is bounded by workers (capped at order.len()). The 'thundering herd' is at most a few threads doing one mutex lock+check+unlock each. notify_one() would risk missed wakeups if a thread is notified but cannot acquire (e.g., its bytes don't fit yet) — the spec for Condvar does not guarantee which waiter is woken. This is safe, correct, and the overhead is negligible.
• Potential deadlock if CUDA task panics while holding VramPermit and other tasks wait (crypto/stark/src/prover.rs:776, found by nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b) — If task(idx) panics at line 777, Rust's unwinding drops the VramPermit (which has a Drop impl releasing the budget), so no deadlock on the gate. The result slot is left empty, but std::thread::scope propagates the panic, causing run_admitted to panic and the caller never reaches the expect(). This is correct standard panic behavior — the proof fails, not deadlocks. The claim's hypothetical about catch_unwind is unsupported: grep confirms zero uses of catch_unwind in the crate. No actual deadlock path exists.
• Non-CUDA builds instantiate and use VramGate with u64::MAX budget causing unnecessary overhead (crypto/stark/src/prover.rs:3422, found by nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b) — The overhead of creating one VramGate (one Mutex<u64> + one Condvar, stack-allocated on Linux with the standard library) with a u64::MAX budget is negligible — on the order of tens of bytes. The code keeps the same API surface across all build configurations to avoid conditional compilation complexity in the call site. The CPU run_admitted already takes _gate: &VramGate as an unused parameter, so the gate is truly inert. This is a deliberate simplification with no measurable cost.

Raw lane outputs, candidates, final issues, and model metrics are uploaded as workflow artifacts.

[jotabulacios]

/bench

[jotabulacios]

/bench-growth