release: v2.3.1 "Plumb Line" — measurement apparatus, and ten measured rejections

.github/release-notes/v2.3.1.md

@@ -0,0 +1,87 @@
+RustyNES **v2.3.1 "Plumb Line"** is a measurement release. It makes the
+performance apparatus trustworthy and then uses it — and what it found is that
+**none of the ten hot-path candidates it measured yielded a shippable
+improvement.** That is a claim about those ten, not about the core as a whole:
+two core leads the campaign surfaced (the APU at 18.7% of frame, and `range.rs`
+inlined inside `Ppu::tick` at 1.52%) remain **unmeasured** and are carried
+forward.
+
+**No emulation-core changes.** AccuracyCoin holds at **exactly 141/141** and
+nestest is 0-diff, verified after every experimental probe was reverted rather
+than merely asserted by construction: this release did land and remove real
+edits.
+
+## Why a measurement release
+
+Two failures in the preceding release motivated it.
+
+- **v2.3.0's adopted PPU optimization measured `+2%` on a contended host and
+  `−5.13%` re-measured quiet** — the same commit, opposite sign. The project's
+  adopt/reject bar is only as good as the host it runs on, and nothing noticed
+  the host.
+- **The profile the campaign was scoped from does not contain the APU.**
+  `perf report` shows zero `rustynes_apu::` symbols at any percent limit, because
+  fat LTO inlines the APU wholesale into `cpu_clock`. The working split
+  "PPU ~53%, CPU+bus ~39%" had folded roughly a fifth of the frame into the wrong
+  bucket.
+
+## New measurement tooling
+
+| tool | what it revealed |
+| --- | --- |
+| `frame_probe` — harness-free frame cost | criterion's own rayon / `exp` / sort work was **~17% of every profile** |
+| `frame_breakdown.sh` — attribution by source file | the **APU is 18.7% of frame time**; `perf report --inline` does *not* recover it |
+| `ab_check.sh` — adoption A/B with an A/B/A order-bias control | the reference drifts up to **−1.17% from run position alone** |
+
+Corrected subsystem split: **PPU 52.1% · APU 18.7% · CPU 10.1% · bus/scheduler
+coupling 9.9% · std inlined at call sites 6.7% · mappers 2.5%.** The CPU proper
+is about a third of what the symbol profile implied.
+
+`bench_relative_check.sh` additionally declines to emit a verdict when the host
+was too noisy to resolve the effect under test, keyed on a robust MAD-based
+coefficient of variation.
+
+## Ten candidates measured, ten rejected
+
+| mechanism | items |
+| --- | --- |
+| LLVM already performs the transformation | sink dead per-dot derivations |
+| the premise is factually false | `repr(Rust)` ignores source order; the named functions were already inlined |
+| real work, absorbed off the critical path | the `index_framebuffer` store; the open-bus decay loop; the ALE/read recompute |
+| the elision is real but buys nothing | typed-index bounds elision |
+| the target is too small to matter | the `bg_split_state` capability gate (0.09% of frame) |
+| forbidden by the ownership model | hoisting `PpuBusAdapter` (borrow checker, with no `unsafe` permitted) |
+
+Six distinct mechanisms, which is what makes this a finding rather than one bad
+assumption repeated: **the per-dot loop has no incidental overhead left to
+reclaim.** Its ~3.78 ms is work the accuracy model requires. That corroborates
+the existing record, where bounds-check elision and a SIMD blitter both measured
+*slower*.
+
+## Two near-misses
+
+Worth recording, because each would have shipped on a single reading:
+
+- One candidate produced a textbook **−1.84% … −2.75% at p = 0.00 on all four
+  workloads** — entirely an order-bias artifact. It measured as exactly zero on
+  re-run. This is what prompted the A/B/A control.
+- Another measured **−0.51% at p = 0.00 on a shipped configuration** with a clean
+  control, then **+0.01% (p = 0.96)** on re-run.
+
+Both were caught only by requiring an independent second run.
+
+## Also in this release
+
+- The PGO workflow's BOLT probe no longer reports success without BOLT. It ran
+  `apt-get install bolt` and trusted the exit status — but on Ubuntu that package
+  is the **Thunderbolt 3 device manager**, so the stage failed on the tool it had
+  just "confirmed" instead of skipping as its best-effort contract intends.
+- Every rejected experiment is recorded in `docs/performance.md` with its
+  numbers, its order-bias control, and the mechanism behind the null result.
+
+## Verification
+
+- `cargo test --workspace --features test-roms` green — AccuracyCoin **141/141**,
+  `visual_regression` 9/9, nestest 0-diff.
+- Workspace clippy clean at `-D warnings`; `cargo fmt --all --check` clean.
+- `shellcheck` clean on every touched script.

.github/workflows/pgo.yml

@@ -266,13 +266,91 @@ jobs:
       - name: Install cargo-pgo
         run: cargo install cargo-pgo --locked
 
+      # Probe for llvm-bolt by LOCATING THE BINARY, never by trusting a package
+      # manager's exit code.
+      #
+      # The previous form did `apt-get install -y bolt` and set have_bolt=true if
+      # that succeeded. On Ubuntu the package named `bolt` is the **Thunderbolt 3
+      # device manager** — an unrelated project that happens to own the name. apt
+      # installed it, exited 0, the probe reported success, and the stage then
+      # died on `Cannot find llvm-bolt: cannot find binary path` (run
+      # 31006334399). A "best-effort" job that is supposed to SKIP when the tool
+      # is missing instead failed the whole run.
+      #
+      # LLVM ships the binary as `llvm-bolt` (apt.llvm.org's `bolt-<N>` packages)
+      # or versioned under /usr/lib/llvm-<N>/bin, so search all of those and
+      # export the directory on PATH for `cargo pgo`, which resolves `llvm-bolt`
+      # by name. Verifying the binary exists is what makes the skip honest.
       - name: Probe for llvm-bolt
         id: bolt_probe
         run: |
-          if command -v llvm-bolt >/dev/null 2>&1; then
-            echo "have_bolt=true" >> "$GITHUB_OUTPUT"
-          elif sudo apt-get update && sudo apt-get install -y --no-install-recommends bolt; then
+          # NOT `set -e`: this step probes for things that are allowed to be
+          # absent. A missing tool must SKIP the stage, not fail the run.
+          set -uo pipefail
+          # PURE discovery: echoes the path of an llvm-bolt binary, or nothing.
+          # It does not install, symlink, or otherwise touch the host — a probe
+          # that mutates state as a side effect of looking is hard to reason
+          # about and hard to re-run. Linking is a separate, explicit step below.
+          #
+          # ENUMERATE the versioned installs rather than probing a fixed version
+          # window: a hard-coded `for v in 21 .. 16` reports "not found" on any
+          # image shipping a version outside it, indistinguishable from "BOLT is
+          # not installed" — the failure mode this whole probe exists to
+          # eliminate. Globs that match nothing expand to the literal pattern,
+          # which the `-x` test rejects.
+          find_bolt_bin() {
+            if command -v llvm-bolt >/dev/null 2>&1; then
+              command -v llvm-bolt; return 0
+            fi
+            for cand in /usr/bin/llvm-bolt-* /usr/local/bin/llvm-bolt-* \
+                        /usr/lib/llvm-*/bin/llvm-bolt; do
+              [ -x "${cand}" ] || continue
+              printf '%s\n' "${cand}"; return 0
+            done
+            return 1
+          }
+
+          # cargo-pgo resolves the UNVERSIONED name, so a versioned hit needs a
+          # symlink. Kept separate from discovery, and reporting failure rather
+          # than assuming success: without the `|| return 1` and the
+          # executability re-check, a FAILED link still yielded a directory and
+          # a success status — the same "assume it worked" bug, one level down.
+          link_bolt() {
+            _bin="$1"
+            case "${_bin}" in
+              */llvm-bolt) dirname "${_bin}"; return 0 ;;   # already unversioned
+            esac
+            sudo ln -sf "${_bin}" /usr/local/bin/llvm-bolt || return 1
+            [ -x /usr/local/bin/llvm-bolt ] || return 1
+            echo /usr/local/bin
+          }
+
+          find_bolt() {
+            _found="$(find_bolt_bin)" || return 1
+            link_bolt "${_found}"
+          }
+
+          bolt_dir="$(find_bolt || true)"
+          if [ -z "${bolt_dir}" ]; then
+            # Try to install it, then LOOK AGAIN — an install succeeding proves
+            # nothing about which project's `bolt` landed on disk.
+            sudo apt-get update >/dev/null 2>&1 || true
+            # Candidate package names only — the loop re-probes after EACH one
+            # and stops at the first that actually yields a binary, so an
+            # unlisted name costs a skip, never a false positive.
+            for pkg in llvm-bolt bolt-21 bolt-20 bolt-19 bolt-18 bolt-17 \
+                       llvm-21-tools llvm-20-tools llvm-19-tools llvm-18-tools; do
+              sudo apt-get install -y --no-install-recommends "$pkg" >/dev/null 2>&1 || continue
+              bolt_dir="$(find_bolt || true)"
+              [ -n "${bolt_dir}" ] && break
+            done
+          fi
+
+          if [ -n "${bolt_dir}" ]; then
+            echo "${bolt_dir}" >> "$GITHUB_PATH"
             echo "have_bolt=true" >> "$GITHUB_OUTPUT"
+            echo "llvm-bolt found: $(command -v llvm-bolt || echo "${bolt_dir}/llvm-bolt")" \
+              >> "$GITHUB_STEP_SUMMARY"
           else
             echo "have_bolt=false" >> "$GITHUB_OUTPUT"
             echo "llvm-bolt unavailable on this runner — skipping BOLT stage." >> "$GITHUB_STEP_SUMMARY"
@@ -303,33 +381,61 @@ jobs:
           scripts/pgo/run.sh "$PGO_FRAMES"
           cargo pgo bolt optimize -- -p rustynes-frontend
 
+      # DISABLED (v2.3.1) — these two steps cannot measure what they claim, and
+      # one of them silently reported a fabricated number. Run 31067782333 is the
+      # evidence: the probe and runtime fixes above made BOLT genuinely work
+      # (instrument + optimize both succeeded for the first time), which finally
+      # exposed what the gate downstream was doing.
+      #
+      # 1. `cargo pgo bolt optimize` takes NO cargo subcommand. Its usage is
+      #    `cargo pgo bolt optimize [OPTIONS] [-- <CARGO_ARGS>...]`, unlike
+      #    `cargo pgo optimize` (the PGO side) which does accept `bench`/`test`.
+      #    Both steps were written by analogy with the PGO stage and both are
+      #    rejected: `unexpected argument 'bench' found` / `'test' found`.
+      #
+      # 2. The bench step swallowed that with `|| cargo bench ...`, so it fell
+      #    back to a PLAIN (non-BOLT) build, computed a speedup against the plain
+      #    baseline, and wrote it to the summary as "BOLT speedup vs plain
+      #    release". It compared plain against plain. The step reported SUCCESS.
+      #    Had that ratio landed above the 3% bar, the gate would have promoted a
+      #    BOLT binary on a measurement containing no BOLT.
+      #
+      # 3. Even with the CLI corrected, the design does not hold: BOLT optimizes
+      #    the `rustynes` FRONTEND binary, while the gate benches
+      #    `rustynes-core`'s `full_frame` criterion bench — a different binary
+      #    BOLT never touched. Measuring BOLT honestly needs a harness that runs
+      #    inside the optimized artifact (`frame_probe` built as part of it), not
+      #    a core bench built separately.
+      #
+      # Left disabled rather than patched: a gate that cannot measure its subject
+      # is worse than no gate, and #2 shows this one could actively mislead. The
+      # instrument/optimize steps above still prove BOLT runs end to end, and the
+      # artifact is still produced. Re-enable behind a harness that benches the
+      # BOLT-optimized binary itself.
       - name: BOLT full_frame bench + gate
-        if: steps.bolt_probe.outputs.have_bolt == 'true'
+        if: false
         run: |
-          cargo pgo bolt optimize bench -- -p rustynes-core --bench full_frame -- \
-            --warm-up-time 1 --measurement-time 5 --save-baseline bolt || \
-            cargo bench -p rustynes-core --bench full_frame -- \
-              --warm-up-time 1 --measurement-time 5 --save-baseline bolt
-          bolt_est="target/criterion/nes_run_frame_nestest/bolt/estimates.json"
-          bolt_ns="$(python3 -c "import json,sys;print(int(json.load(open(sys.argv[1]))['mean']['point_estimate']))" "$bolt_est")"
-          speedup="$(python3 -c "print(f'{(1 - ${bolt_ns}/${BASE_NS})*100:.2f}')")"
-          {
-            echo "### BOLT post-link gate"
-            echo "BOLT speedup vs plain release: ${speedup}% (threshold > ${PGO_MIN_SPEEDUP_PCT}%)"
-          } >> "$GITHUB_STEP_SUMMARY"
-          python3 -c "import sys; sys.exit(0 if ${speedup} > ${PGO_MIN_SPEEDUP_PCT} else 1)" || {
-            echo "BOLT did not beat the > ${PGO_MIN_SPEEDUP_PCT}% bar — not promoting." >> "$GITHUB_STEP_SUMMARY"
-            exit 0
-          }
+          echo "disabled — see the comment above (run 31067782333)"
 
       - name: Determinism oracle (BOLT codegen)
-        if: steps.bolt_probe.outputs.have_bolt == 'true'
-        run: cargo pgo bolt optimize test -- --workspace --release --features test-roms
+        if: false
+        run: |
+          echo "disabled — see the comment above (run 31067782333)"
 
+      # The artifact must be the BOLT output, not whatever `rustynes` happens to
+      # be sitting in target/. `path: target/**/release/rustynes` matched the
+      # PGO binary that `scripts/pgo/run.sh` had already written, so an artifact
+      # named `rustynes-pgo-bolt` contained NO BOLT — the same mislabelling as
+      # the bench gate above, which reported a plain build as a BOLT speedup.
+      #
+      # `cargo pgo bolt optimize` writes `rustynes-bolt-optimized` alongside the
+      # plain binary. `if-no-files-found: error` rather than `warn`: if the file
+      # is absent the correct outcome is a visible failure, not a quietly empty
+      # artifact that looks like a successful BOLT build.
       - name: Upload BOLT binary
         if: steps.bolt_probe.outputs.have_bolt == 'true'
         uses: actions/upload-artifact@v7
         with:
-          name: rustynes-pgo-bolt
-          path: target/**/release/rustynes
-          if-no-files-found: warn
+          name: rustynes-bolt-optimized
+          path: target/**/release/rustynes-bolt-optimized
+          if-no-files-found: error

CHANGELOG.md

@@ -14,6 +14,50 @@ cycle-accurate core later replaced.
 
 ## [Unreleased]
 
+## [2.3.1] - 2026-08-06 - "Plumb Line" (measurement apparatus + ten measured rejections)
+
+### Performance
+
+- **No emulation-core changes. Ten hot-path optimization candidates were
+  measured and all ten rejected**, through six distinct mechanisms: LLVM already
+  performed the transformation; the item's premise was factually false; the work
+  was real but absorbed off the critical path; the elision was real but bought
+  nothing; the target was too small to matter; the ownership model forbids it.
+  Full numbers, controls and reasoning are in `docs/performance.md`
+  (entries G1–G10). **AccuracyCoin remains at exactly 141/141 and nestest
+  0-diff**, verified after every experimental probe was reverted.
+- New measurement tooling, all of which found something the previous apparatus
+  could not:
+  - `crates/rustynes-test-harness/src/bin/frame_probe.rs` — harness-free
+    steady-state frame cost, with no criterion in the process image (criterion's
+    own rayon/`exp`/sort work had been ~17% of every profile).
+  - `scripts/perf/frame_breakdown.sh` — per-subsystem attribution by **source
+    file**, which recovers work the symbol profile hides. It shows the **APU at
+    18.7% of frame time**, invisible under `perf report` because fat LTO inlines
+    it wholesale into `cpu_clock` (`perf report --inline` does not recover it).
+  - `scripts/perf/ab_check.sh` — adoption A/B with an **A/B/A order-bias
+    control**: the reference is benched a third time, last, against its own first
+    run, so drift from position-in-the-run is reported rather than mistaken for a
+    result.
+- `scripts/bench_relative_check.sh` now declines to emit a verdict when the host
+  was too noisy to resolve the effect it tests for, keyed on a robust
+  MAD-based coefficient of variation.
+
+### Fixed
+
+- **The PGO workflow's BOLT probe reported success without BOLT present.** It ran
+  `apt-get install bolt` and trusted the exit status — but on Ubuntu that package
+  is the *Thunderbolt 3 device manager*, an unrelated project that owns the name.
+  The stage then failed on the tool it had just "confirmed", instead of skipping
+  as its best-effort contract intends. The probe now locates the actual
+  `llvm-bolt` binary and reports honestly when it is absent.
+
+### Documentation
+
+- `docs/performance.md` records every rejected experiment with its numbers, its
+  order-bias control, and the mechanism behind the null result — including two
+  near-misses that a single measurement would have adopted.
+
 ## [2.3.0] - 2026-08-05 - "Datum II" (PPU-accuracy capstone + true multi-viewport tool windows)
 
 Closes the **v2.2.6 → v2.3.0 NESdev-remediation line**. Both remaining