Skip to content

Load tracing and iteration benchmarks - #2465

Open
sergerad wants to merge 1 commit into
nextfrom
sergerad-bench-load
Open

Load tracing and iteration benchmarks#2465
sergerad wants to merge 1 commit into
nextfrom
sergerad-bench-load

Conversation

@sergerad

@sergerad sergerad commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Relates to #1697.

Why:

  • Node startup on testnet takes ~16s, but the instrumented spans only account for ~11.9s. The
    main suspects for the gap are the three RocksDB opens (account tree, nullifier tree, account
    state forest), which were invisible in traces and can pay multi-second WAL recovery after a
    run without a clean flush.
  • There was also no reproducible way to measure startup time locally; the load-state bench
    ran a single load, which cannot distinguish first-open WAL recovery from a clean warm restart.

How:

  • Added open_tree_storage and open_forest_storage spans to the RocksDB create
    implementations in the store loader, with a path field to distinguish the account tree,
    nullifier tree, and forest opens.
  • Extended the benchmark-store load-state endpoint with --load-iterations (default 3): the
    first iteration surfaces WAL recovery / cold-cache cost, later iterations measure a clean
    warm restart. A percentile summary is printed when more than one iteration runs. A dedicated
    flag is used instead of the parent --iterations (default 10000) to avoid accidentally
    running 10000 full loads.
  • Added an opt-in tracing-forest feature to the stress-test crate that renders the per-phase
    timing tree. Opt-in because feature unification would otherwise switch the log format of
    every binary in a workspace-wide build.

Results:

load [ 36.1ms ]
┕━ load_with_database_options [ 36.1ms ]
   ┝━ load_with_pool_size [ 8.47ms | 23.5% ]
   ┝━ load_mmr [ 1.27ms ]
   ┝━ open_tree_storage [ 10.3ms | 28.7% ] path: "accounttree"
   ┝━ load_account_tree [ 2.19ms ]
   ┝━ open_tree_storage [ 6.38ms | 17.7% ] path: "nullifiertree"
   ┝━ load_nullifier_tree [ 822µs ]
   ┝━ open_forest_storage [ 5.98ms | 16.6% ] path: "accountstateforest"
   ┝━ load_account_state_forest [ 74.4µs ]
   ┕━ verify_account_state_forest_consistency [ 458µs ]

Even at this toy scale the result backs the issue's hypothesis: the three RocksDB opens are
~63% of load time and were entirely absent from the spans before. Iteration 0 also ran ~2×
slower than iterations 1–2 (38ms vs 18–20ms), consistent with WAL recovery/cold cache on
first open. On testnet data this instrumentation should now attribute most of the ~4.4s gap
directly.

Run it with:

cargo run -p miden-node-stress-test --release --features tracing-forest -- \
  benchmark-store -d <data-dir> load-state --load-iterations 3

Changelog

changelog = "none"
reason    = "Internal change only."

@sergerad
sergerad requested review from Mirko-von-Leipzig and kkovaacs and removed request for Mirko-von-Leipzig August 11, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant