From f3166780d0e6850617ec520651325e7c1b0e633d Mon Sep 17 00:00:00 2001 From: Dhruv Kumar Singh Date: Tue, 4 Aug 2026 18:24:49 +0530 Subject: [PATCH] Rerun the benchmarks against the current tree --- CHANGELOG.md | 2 +- README.md | 8 ++++---- benchmarks/README.md | 33 ++++++++++++++++----------------- 3 files changed, 21 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c7ffd7..e064dce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -129,7 +129,7 @@ the syncs Raft owes before it answers an RPC; drop the sync before an append is acknowledged and the first seed loses a committed entry. - Campaign evidence at scale, regenerable via `benchmarks/campaign.py`: - 100,000 seeds of jobqueue chaos green in under six minutes on a laptop, + 100,000 seeds of jobqueue chaos green in just over six minutes on a laptop, every ablation caught with its failure density recorded, and 20 sampled failing seeds replaying with identical trace hashes across 100 re-runs apiece. A small nightly CI sweep keeps the numbers honest. diff --git a/README.md b/README.md index b83d350..391c252 100644 --- a/README.md +++ b/README.md @@ -272,12 +272,12 @@ the sharpest case. The table lives in ## Performance -Simulation is cheap: SimLoop schedules a task step in ~4.4 µs (trace -recording included — about 3.6× faster than the stock loop, which pays a +Simulation is cheap: SimLoop schedules a task step in ~4.5 µs (trace +recording included — about 3.7× faster than the stock loop, which pays a selector syscall per iteration), compresses sleep-heavy workloads ~2,000× against wall clock, and with `--simloop-jobs` fanning seeds across -processes, the full jobqueue chaos scenario sweeps 100,000 seeds in under -six minutes (~300 seeds/second) on an M4 MacBook Air. Methodology, numbers, +processes, the full jobqueue chaos scenario sweeps 100,000 seeds in just +over six minutes (~260 seeds/second) on an M4 MacBook Air. Methodology, numbers, and the campaign results: [benchmarks/README.md](https://github.com/dhruvl/simloop/blob/main/benchmarks/README.md). diff --git a/benchmarks/README.md b/benchmarks/README.md index 30c4bfe..4902176 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -21,12 +21,12 @@ hand-off — no I/O, no timers. | loop | median | per hop | |---|---|---| -| stock asyncio | 0.319 s | 16.0 µs | -| SimLoop | 0.088 s | 4.4 µs | +| stock asyncio | 0.332 s | 16.6 µs | +| SimLoop | 0.090 s | 4.5 µs | -SimLoop comes out about **3.6× faster per scheduling step**, trace recording +SimLoop comes out about **3.7× faster per scheduling step**, trace recording included, and the ratio holds from 10×2000 to 500×200 task/round shapes -(0.26–0.30× across the sweep, widening with the task count). That is not +(0.26–0.31× across the sweep, widening with the task count). That is not because simloop is a faster event loop in any general sense — it is because a simulated loop never touches the OS. Profiling the stock run shows about half its time inside `select.kqueue.control`: the real loop pays a selector @@ -69,17 +69,16 @@ The jobqueue chaos campaign runs one full distributed scenario per seed — a broker, 3 workers, and 2 clients submitting 8 jobs (some poisoned) under randomized partitions and a worker crash, then settles for up to 600 simulated seconds and checks every invariant. 300 seeds complete in -**5.3–6.3 s** across nine runs (median 5.8 s), about **52 seeds/second**, -in one process. A thousand-seed overnight search is a 19-second coffee +**4.9–5.4 s** across nine runs (median 5.1 s), about **59 seeds/second**, +in one process. A thousand-seed overnight search is a 17-second coffee break. -That is slower than the ~55 seeds/second (5.4–6.0 s) published for 0.1.0, -and the cost is a feature: 0.2.0 records a trace event for every packet -delivery, not -just for every send, which is what lets a timeline draw both ends of a -crossing — and a scenario this network-heavy pays for the extra events -directly. The wire is where this benchmark spends its time, which is why -it is the number that moved. +That is on par with — in fact a touch faster than — the ~55 seeds/second +(5.4–6.0 s) published for 0.1.0, even though 0.2.0 records a trace event for +every packet delivery, not just for every send, which is what lets a +timeline draw both ends of a crossing. The extra event volume a +network-heavy scenario like this one pays for no longer shows up as a +wall-clock cost against the 0.1.0 baseline. ## Campaigns @@ -116,13 +115,13 @@ default `campaign-{green,ablations}.json`) and resume from it with the failing seeds out of the `ablations` checkpoint. These files are scratch, not repository content — they are gitignored. -Results, recorded 2026-08-01 on the M4 MacBook Air (10 jobs): +Results, recorded 2026-08-04 on the M4 MacBook Air (10 jobs): | campaign | scale | result | |---|---|---| -| green | 100,000 seeds, 5.6 min, 300.1 seeds/s | green — no invariant violated | -| ablations | 6 mutations × 10,000 seeds, 2.5 min | every ablation caught, densities below | -| replay stability | 20 failing seeds × 100 re-runs, 13.5 s | identical trace hash on every run | +| green | 100,000 seeds, 6.3 min, 263.1 seeds/s | green — no invariant violated | +| ablations | 6 mutations × 10,000 seeds, 2.7 min | every ablation caught, densities below | +| replay stability | 20 failing seeds × 100 re-runs, 14.2 s | identical trace hash on every run | Per-ablation failure density: