Let the guest's memcpy use unaligned word loads - #980
Conversation
compiler_builtins enables its mem-unaligned feature for x86, aarch64 and bpf only, so on RISC-V every misaligned word inside memcpy and memmove is reassembled with a load, two shifts, an or and a store. This VM does not need that: #864 already builds guests with +unaligned-scalar-mem because the executor byte-assembles an unaligned doubleword host-side at the same one-instruction cost as an aligned one, so the rest of the guest has been emitting unaligned loads since July and only these two functions avoided them. memcpy is the largest leaf in an ethrex block profile at 25.15% of guest cycles, and its biggest caller is rkyv deserializing the witness trie. On the real mainnet block 25368371 this is -8.33% guest cycles, 8 epochs down to 7, -6.33% proving time, -8.57% proof size, -9.47% verify and -4.78% base-field-equivalent cells; keccak and ecsm call counts are unchanged. tooling/ethrex-tests passes 6/6 including the two --ignored cases, and both arms of every A/B verify OK. Unaligned accesses land on the more expensive MEMW table rather than MEMW_A, so prover area was measured rather than assumed. The key is scoped to the guest target, so host builds read the file and take nothing from it. Cargo joins rustflags arrays across ancestor config files, so each guest crate keeps its own flags and gains this one. The feature cannot be reached through -Z build-std-features: that chain only forwards compiler-builtins-mem and no compiler-builtins-mem-unaligned exists.
|
/bench |
Benchmark — real block (
|
| Metric | main | PR | Δ |
|---|---|---|---|
| Peak heap | 46886 MB | 52041 MB | +5155 MB (+11.0%) 🔴 |
| Prove time | 110.675s | 103.297s | -7.378s (-6.7%) 🟢 |
❓ -6.7% — beyond what 3 runs resolve. Use
/bench-abbafor a paired test of the same block (default 12 pairs, ~72 min, resolves ~1%).Prove-time spread 1.2% (103.582s / 103.297s / 102.329s)
Commit: d97faba · Baseline: cached · Runner: self-hosted bench
|
/bench-verify |
|
⏳ Benchmark started on the bench server. Two verifier arms (monolithic + continuations over an ethrex 20-tx block), then the recursion-guest cycle comparison, which adds guest builds on top — longer on a cold runner. The bench server is occupied until it finishes. |
Verifier benchmark —
|
| Metric | main | PR | Δ |
|---|---|---|---|
| Verify time (ABBA, 20 pairs) | 2.515s | 2.508s | -0.30% 🟢 |
| Proof size (exact, 1 reading) | 102.33 MiB | 102.33 MiB | +0.00% ⚪ |
pairs: 20 mean A (PR): 2.508s mean B (main): 2.515s
[parametric] paired-t mean -0.30% sd 0.58% se 0.13%
95% CI: [-0.57%, -0.03%] (t df=19 = 2.093)
[robust] median -0.34% Wilcoxon W+=50 W-=160 p(exact)=0.0400 (z=-2.03)
run-to-run jitter: A CV 0.39% B CV 0.43% (lower = steadier)
within-session drift: -0.08% over the run, 1st->2nd half +0.05%
🟢 REAL IMPROVEMENT — PR verifies ~0.30% faster (paired-t and Wilcoxon agree).
ethrex 20-tx block · continuations, epoch 2^20 (3 epochs) · blowup=2, 219 queries
| Metric | main | PR | Δ |
|---|---|---|---|
| Verify time (ABBA, 8 pairs) | 3.161s | 3.160s | -0.01% ⚪ |
| Proof size (exact, 1 reading) | 177.07 MiB | 177.07 MiB | +0.00% ⚪ |
pairs: 8 mean A (PR): 3.160s mean B (main): 3.161s
[parametric] paired-t mean -0.01% sd 0.68% se 0.24%
95% CI: [-0.58%, +0.56%] (t df=7 = 2.365)
[robust] median +0.08% Wilcoxon W+=19 W-=17 p(exact)=0.9453 (z=+0.07)
run-to-run jitter: A CV 0.31% B CV 0.50% (lower = steadier)
within-session drift: +0.48% over the run, 1st->2nd half +0.17%
⚪ INCONCLUSIVE — effect not separable from 0 at n=8 (point estimate ~+0.08%). Add pairs to resolve.
Verify-time rows only: drift-free interleaved A/B/B/A, with paired-t and exact Wilcoxon — trust the verdict when the two agree. Proof sizes are single exact readings (no averaging). - = PR faster.
Recursion guest cycles — verifier running INSIDE the VM (main vs PR)
empty program · monolithic · blowup=2, 1 query (diagnostic — NOT a real verifier cost)
Single exact reading per ref — no ABBA: guest cycles are deterministic for a fixed
(guest ELF, input blob), so there is no machine drift to cancel.
| Metric | main | PR | Δ |
|---|---|---|---|
| Guest cycles | 32.1M | 31.9M | -0.2M (-0.71%) |
| Keccak calls | 3029 | 3029 | 0 |
baseline origin/main 99d7567afe guest=recursion-min.elf
PR d97faba020465838251c02d7965f4bf537804e53 d97faba020 guest=recursion-min.elf
note: cycles reproduce to ~±100k (build codegen + proof nondeterminism);
treat sub-100k deltas as noise, not signal.
raw (exact integer counts)
ref_b_sha=99d7567afec95c672e78065bc2bacb4416fb7577 ref_b_elf=recursion-min.elf ref_b_cycles=32137866 ref_b_keccak=3029 ref_b_execute_wall_s=1
ref_a_sha=d97faba020465838251c02d7965f4bf537804e53 ref_a_elf=recursion-min.elf ref_a_cycles=31909978 ref_a_keccak=3029 ref_a_execute_wall_s=1
delta_cycles=-227888 delta_keccak=0
ethrex 20-tx block · continuations, epoch 2^21 (2 epochs) · blowup=2, 219 queries (128-bit)
Single exact reading per ref — no ABBA: guest cycles are deterministic for a fixed
(guest ELF, input blob), so there is no machine drift to cancel.
| Metric | main | PR | Δ |
|---|---|---|---|
| Guest cycles | 1910.0M | 1893.9M | -16.1M (-0.84%) |
| Keccak calls | 3296811 | 3232935 | -63876 |
baseline origin/main 99d7567afe guest=recursion-cont-blowup2.elf
PR d97faba020465838251c02d7965f4bf537804e53 d97faba020 guest=recursion-cont-blowup2.elf
note: cycles reproduce to ~±100k (build codegen + proof nondeterminism);
treat sub-100k deltas as noise, not signal.
raw (exact integer counts)
ref_b_sha=99d7567afec95c672e78065bc2bacb4416fb7577 ref_b_elf=recursion-cont-blowup2.elf ref_b_cycles=1909967824 ref_b_keccak=3296811 ref_b_execute_wall_s=31
ref_a_sha=d97faba020465838251c02d7965f4bf537804e53 ref_a_elf=recursion-cont-blowup2.elf ref_a_cycles=1893870267 ref_a_keccak=3232935 ref_a_execute_wall_s=30
delta_cycles=-16097557 delta_keccak=-63876
|
/ai-review |
Codex Code Review
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # | ||
| # Cargo joins `rustflags` arrays across ancestor config files, so each guest | ||
| # crate's own `.cargo/config.toml` keeps its flags and gains this one. | ||
| rustflags = ["--cfg", "feature=\"mem-unaligned\""] |
There was a problem hiding this comment.
Low — worth one more sentence in the comment: this sets the cfg for every crate in the guest graph, and it can silently become a no-op.
Two things the comment doesn't say:
-
--cfg feature="mem-unaligned"is not scoped tocompiler_builtins— it's set on every crate compiled for this target (core/alloc/std,libc, and all of the ethrex guest's third-party deps). Today onlycompiler-builtinsreads that name, so it's harmless, but if some dep ever declares a feature by that name it gets enabled without Cargo activating its dependencies. Also, since the cfg is unexpected for every crate that doesn't declare it, expectunexpected_cfgswarnings across the guest graph; pairing it with"--check-cfg", "cfg(feature, values(\"mem-unaligned\"))"in the same array unions the value into the expected set and keeps the build output clean. -
Nothing fails loudly if the flag stops taking effect, and the failure mode is the 8% cycle regression coming back with a green build. Three ways that happens: the pinned
nightly-2026-02-01sysroot'scompiler-builtinsrenames/drops the knob (it's build.rs-internal, no stability guarantee — worth naming the nightly pin inMakefileas the coupling);RUSTFLAGS/CARGO_ENCODED_RUSTFLAGSis set in the environment, which replacestarget.<triple>.rustflagsrather than joining with it (unlike the ancestor-config joining the comment describes); or a guest is ever built from a cwd outside this repo tree.
Review — no blocking issues13-line, guest-only build flag. I checked the mechanism and the correctness exposure it widens; both hold up. One inline note (Low) on the config comment. Mechanism — verified
Correctness exposure — the interesting part. This makes 8-byte unaligned accesses common inside
GPU/device memory. Benchmark methodology (three-run ranges, prover area measured rather than assumed, second box with a different clang, 262 ELFs + 827 prover tests) is more than sufficient for the claim. 🤖 Generated with Claude Code |
AI ReviewPR #980 · 1 changed files Findings
Status column reflects the verdict from the verifier: deepseek-verifier (openrouter/deepseek/deepseek-v4-pro). AI-003: CI cache keys ignore workspace .cargo/config.toml
Claim The rust-elf and recursion artifact cache keys do not include the new root .cargo/config.toml, so changes to global guest rustflags will restore stale cached artifacts that Cargo must rebuild unnecessarily. Evidence The cache key at line 106 (and other similar keys in the workflow) hashes executor/programs/rust/, executor/programs/riscv64im-lambda-vm-elf.json, syscalls/, and Makefile, but omits .cargo/config.toml, which is now an input to every guest build. Suggested fix Add '.cargo/config.toml' to the hashFiles(...) expressions for rust-elf-artifacts and recursion-elf-artifacts cache keys. Reviewer Lanes
Verification Lanes
Native Codex and Claude reviews run separately and post their own comments. They are not included in this structured provenance report. Discarded candidates (2) — rejected by the verifier
Raw lane outputs, candidates, final issues, and model metrics are uploaded as workflow artifacts. |
|
/bench |
|
/bench |
|
/bench-verify |
|
⏳ Benchmark started on the bench server. Two verifier arms (monolithic + continuations over an ethrex 20-tx block), then the recursion-guest cycle comparison, which adds guest builds on top — longer on a cold runner. The bench server is occupied until it finishes. |
2 participants
compiler_builtinsenables itsmem-unalignedfeature for x86, aarch64 and bpf only, so on RISC-Vmemcpy/memmovereassemble every misaligned word with a load, two shifts, an or and a store. ThisVM doesn't need that: #864 already builds guests with
+unaligned-scalar-mem, since the executorbyte-assembles an unaligned doubleword host-side at the same one-instruction cost as an aligned one.
The rest of the guest has been emitting unaligned loads since July — these two functions were the
last holdouts.
memcpyis the largest leaf in an ethrex block at 25% of guest cycles.The feature isn't reachable through
-Z build-std-features(that chain only forwardscompiler-builtins-mem), hence the--cfg. The key is scoped to the guest target, so host buildstake nothing from it.
Real mainnet block 25368371, continuations at 2^22 (what
benchmark-pr.ymluses), EPYC 9454P48c/96t, this branch against its base:
Ranges don't overlap (base 109.854/109.262/110.288, patched 101.623/101.425/102.249). Unaligned
accesses land on the pricier
MEMWtable rather thanMEMW_A, so prover area was measured ratherthan assumed — it still drops.
Correctness:
ethrex-tests6/6 including the two--ignored; all 262 guest ELFs build; local proversuite 827/827;
verifyOK at two epoch sizes (2^22 and 2^21) — deliberate, since a known latentissue can break the L2G table when a guest build moves the static layout. Measured on a second box
with a different clang: identical absolute saving of 2,539,031 cycles. All 29 Rust guests and the
four ethrex fixtures were measured both ways; nothing regresses.
The GPU job was not run locally — no GPU on the benchmark boxes.