feat(xtest): benchmark two named refs via workflow_dispatch

.github/workflows/check.yml

@@ -34,6 +34,19 @@ jobs:
           uv run ruff format --check .
           uv run pyright
         working-directory: xtest
+      # The benchmark harness's own tests: statistics, measurement, and the
+      # CLI command builders. No platform and no SDK builds required, so the
+      # part of the gate that has to be *correct* is checked on every PR
+      # rather than only when the nightly benchmark runs.
+      # --frozen --no-build: resolve nothing and build nothing, so a
+      # dependency cannot slip in an unlocked version or a setup script on a
+      # runner that already has everything installed from the step above.
+      - name: Test xtest benchmark harness
+        run: >-
+          uv run --frozen --no-build pytest --no-header -q
+          test_bench_stats.py test_bench_measure.py test_bench_runner.py
+          test_bench_arms.py test_bench_report.py test_sdk_commands.py
+        working-directory: xtest
       - name: Lint and test otdf-local
         run: |
           uv sync

.github/workflows/pr-lint.yaml

@@ -29,6 +29,7 @@ jobs:
             java
             web
             xtest
+            perf
             ci
             dependabot
         env:

.gitignore

@@ -13,6 +13,7 @@ vulnerability/tilt_modules/
 /xtest/node_modules/
 /xtest/tilt_modules/
 /xtest/tmp/
+/xtest/test-results/
 /xtest/sdk/js/web/dist/
 /xtest/.helm
 

otdf-sdk-mgr/src/otdf_sdk_mgr/resolve.py

@@ -300,7 +300,14 @@ def _resolve_against(
                 "alias": version,
                 "head": True,
                 "sha": sha,
-                "tag": version,
+                # Flattened the same way _classify_sha_match flattens a branch
+                # it reached by SHA: the tag becomes a single dist/<tag>/ and
+                # src/<tag>/ path component. A slash here nests those
+                # directories, and every consumer walks them one level deep --
+                # xtest's all_versions_of() lists dist/*/ and the go Makefile
+                # finds src/*/, so "feat/x" is discovered as a "feat" build
+                # with no cli.sh in it.
+                "tag": version.replace("/", "--"),
             }
 
         if infix and version.startswith(f"{infix}/"):

otdf-sdk-mgr/tests/test_resolve.py

@@ -76,7 +76,24 @@ def test_refs_heads_non_main_branch(self):
             result = resolve("js", "refs/heads/release/sdk-v0.17", None)
         assert is_resolve_success(result)
         assert "head" in result and result["head"] is True
-        assert result["tag"] == "release/sdk-v0.17"
+        assert result["tag"] == "release--sdk-v0.17"
+        assert result["sha"] == SHA40
+
+    def test_branch_by_name_flattens_slashes(self):
+        # Same flattening the SHA path applies, and for the same reason: the
+        # tag is one path component under dist/ and src/. Reached by name
+        # rather than by SHA, which is the shape a workflow_dispatch input
+        # arrives in.
+        ls = make_ls_remote(
+            (SHA40, "refs/heads/feat/DSPX-2604-createtdf-chunked"),
+            ("d" * 40, "refs/heads/main"),
+        )
+        with patch_git(ls):
+            result = resolve("go", "feat/DSPX-2604-createtdf-chunked", None)
+        assert is_resolve_success(result)
+        assert result.get("head") is True
+        assert result["tag"] == "feat--DSPX-2604-createtdf-chunked"
+        assert result["alias"] == "feat/DSPX-2604-createtdf-chunked"
         assert result["sha"] == SHA40
 
 

spec/DSPX-4372.md

@@ -0,0 +1,256 @@
+---
+ticket: DSPX-4372
+title: Statistically valid SDK performance regression benchmarks
+status: draft
+authors: [dmihalcik@virtru.com]
+branches: [opentdf/tests:DSPX-4372]
+prs: []
+created: 2026-08-13
+updated: 2026-08-13
+---
+
+# DSPX-4372 — Statistically valid SDK performance regression benchmarks
+
+## Summary
+
+A nightly, per-SDK CI job measures the branch build against the newest release
+**on the same runner, in the same interleaved round**, and fails the job on a
+confirmed wall-clock or peak-RSS regression. CPU time is measured and reported
+but never gates.
+
+Nothing is compared to history. Absolute timings from a GitHub-hosted runner
+are not comparable to timings from any other runner, so the only quantity the
+job reports is a *ratio between two builds measured under identical
+conditions*.
+
+## Problem / Motivation
+
+Nothing in this repo measures the cost of an SDK operation. `tdfs.SDK.encrypt`
+and `decrypt` return `None`, there is no timing code, and CI produces no
+machine-readable durations. A performance regression in any SDK ships
+silently, and is found by a customer rather than by us.
+
+The ticket asks for "performance metric tests ... (memory usage, wall clock
+time, cpu usage)" and already points at the design that makes them work:
+*"comparing main to release of the SDKs ... separate jobs from the existing job
+matrix so we can have encrypt from different versions running on the same
+instance."*
+
+The naive alternative — record timings, store them, compare to last week — does
+not work here. GitHub-hosted runners vary in CPU model, are shared tenancy, and
+suffer unbounded steal time. Run-to-run variation on identical code exceeds any
+regression worth catching, so a historical gate produces false alarms until
+people mute it, at which point it is worse than nothing.
+
+## Proposed Solution
+
+### Paired A/B on one runner
+
+A **cell** is one operation at one payload size for one SDK, e.g.
+`go-encrypt-1MiB`. Each cell runs *rounds*; each round runs both arms once:
+
+- **baseline** — the newest installed release (`go@v0.36.0`)
+- **candidate** — the branch build (`go@main`)
+
+Runner speed, thermal state, and noisy neighbours are shared within a round and
+cancel in the per-round ratio. Order within the round is randomized from a
+seeded RNG so neither arm systematically inherits the other's cache state.
+
+### Statistics
+
+Per round, on the log scale: `d_i = ln(candidate_i) - ln(baseline_i)`.
+
+| Quantity | Method |
+|---|---|
+| Point estimate | median of `d_i`, exponentiated |
+| Interval | BCa percentile bootstrap, 95%, 10 000 resamples |
+| Test | one-sided Wilcoxon signed-rank (`alternative="greater"`) |
+| Multiplicity | Benjamini–Hochberg across the run's gated cells |
+
+**Decision rule: regression iff `ci_low > threshold` AND BH-adjusted
+`p < 0.05`.** Default threshold 1.15 (+15%). The conjunction is deliberate and
+neither clause is redundant: the interval clause cannot fire on pure noise —
+that would require excluding an effect that is not there — and the p clause
+cannot fire on a real-but-trivial effect surviving by luck across ~14 cells.
+Symmetrically, `ci_high < 1/threshold` reports an improvement, which is
+informational and never fails.
+
+### The A/A control
+
+Each SDK runs one extra cell comparing the **baseline against itself** through
+the identical pipeline. Its true ratio is 1.0 by construction, so whatever it
+reports is the harness's own error. Two things come out of it:
+
+- If it *trips* — reports an effect past the threshold — the runner is too noisy
+  or the harness is biased. The whole run is downgraded: verdicts are reported,
+  the build is not failed.
+- Its interval width is the run's empirical **noise floor**. If that is not
+  tighter than the threshold, the run had no power to detect the effect being
+  gated on, and no cell may report `PASS` — only `INCONCLUSIVE`. "We could not
+  tell" must never be reported as "no regression".
+
+Controls run **first** in each SDK's cell list. A run that overruns its budget
+loses whatever is at the end; losing one comparison leaves the rest
+trustworthy, losing the control leaves nothing trustworthy at all.
+
+### Stopping rule
+
+Rounds continue until the bootstrap CI half-width falls below
+`ln(threshold)/3`, bounded below by `--bench-min-rounds` (20), above by
+`--bench-max-rounds` (60), and by a shared wall-clock budget.
+
+**Stopping is on precision, never on significance.** Peeking at the p-value and
+stopping when it drops below alpha is optional stopping: it inflates the
+false-positive rate well past nominal, because each round is a fresh chance to
+cross the line and the loop only ever stops on the lucky side. Attained CI width
+is driven by dispersion rather than location, so it is approximately ancillary
+to the effect being tested. This is easy to "optimize away" — stopping on
+significance finishes sooner — and doing so silently invalidates every number
+the job produces.
+
+## Inputs / Outputs / Contracts
+
+### New modules
+
+| Path | Responsibility |
+|---|---|
+| `xtest/perf/measure.py` | one invocation → wall ns, CPU s, peak RSS bytes |
+| `xtest/perf/_launcher.py` | forks the measured command from an empty process, so its RSS is its own |
+| `xtest/perf/stats.py` | log-ratios, BCa CI, Wilcoxon, BH, decision rule |
+| `xtest/perf/runner.py` | paired round loop, warm-up, stopping rule, budget |
+| `xtest/perf/cells.py` | the experiment matrix |
+| `xtest/perf/report.py` | JSON artifact + `$GITHUB_STEP_SUMMARY` markdown |
+| `xtest/fixtures/bench.py` | arm selection, payloads, ciphertexts, comparability guards |
+| `xtest/test_benchmarks.py` | the cells (needs a platform) |
+| `xtest/test_bench_stats.py` | statistics, offline |
+| `xtest/test_bench_measure.py` | measurement primitive, offline |
+| `xtest/test_bench_runner.py` | round loop and gate, offline |
+| `xtest/test_sdk_commands.py` | the `XT_WITH_*` CLI contract, offline |
+
+`xtest/tdfs.py` gains `SDK.encrypt_command` / `SDK.decrypt_command` — argv+env
+builders extracted from the existing `encrypt`/`decrypt`, whose behaviour is
+unchanged — and `SDK.semver()`. The benchmark drives the CLI through the same
+builders the functional tests use, so the `XT_WITH_*` contract cannot drift
+between them.
+
+### Measurement primitive
+
+```python
+@dataclass(frozen=True, slots=True)
+class Sample:
+    wall_ns: int          # perf_counter_ns around the call
+    cpu_s: float          # ru_utime + ru_stime
+    max_rss_bytes: int    # ru_maxrss, unit-normalized
+    exit_code: int
+    rss_floor_bytes: int  # RSS of the process that forked it
+```
+
+`os.wait4`, not `resource.getrusage(RUSAGE_CHILDREN)`: the latter's
+`ru_maxrss` is a process-lifetime high-water mark, so deltas are meaningless.
+rusage folds in reaped descendants, so the `java`/`node` process behind each
+`cli.sh` shim is counted. `ru_maxrss` is KiB on Linux and bytes on macOS;
+normalized on `sys.platform`.
+
+The command is **not** forked from the pytest process. On Linux a child
+inherits the parent's resident-set accounting and `execve` does not clear it,
+so `ru_maxrss` comes back as `max(the child's true peak, the parent's RSS at
+fork time)`. Measured from a pytest process holding numpy, scipy and a
+session's worth of samples, every SDK invocation reported *pytest's* footprint
+— about 165 MiB on a CI runner — instead of its own. That does not look
+broken; it looks like a stable ratio of 1.000, which reads as "no regression"
+forever. `posix_spawn` and `sh -c 'exec …'` were measured and are equally
+contaminated: an exec is too late, the accounting is already latched. So
+`perf/_launcher.py` runs as a small `python -I -S` process holding nothing and
+forks the real command itself, reporting its own RSS as the floor under the
+reading. It also puts the command in its own process group, so a timeout kills
+the whole tree rather than just the shim.
+
+### CLI options
+
+`--bench` (opt-in; without it `test_benchmarks.py` collects nothing),
+`--bench-baseline`, `--bench-candidate`, `--bench-threshold` (1.15),
+`--bench-min-rounds` (20), `--bench-max-rounds` (60), `--bench-warmup` (5),
+`--bench-budget-seconds` (1500), `--bench-seed` (0), `--bench-out`
+(`test-results/benchmarks`), `--bench-no-gate`.
+
+### Outputs
+
+- `test-results/benchmarks/<sdk>.json` — **every raw per-round sample**
+  alongside the derived statistics, runner metadata, seed, and thresholds.
+  Re-analysing a surprising result offline is the difference between
+  understanding a red build and re-running a 30-minute job to look at the same
+  numbers again.
+- `$GITHUB_STEP_SUMMARY` — one row per (cell, metric): baseline median,
+  candidate median, ratio with CI, adjusted p, verdict, plus the noise floor.
+- Exit status: `pytest_sessionfinish` fails the session on a confirmed
+  regression.
+
+### CI
+
+New `bench` job in `.github/workflows/xtest.yml`: matrix over
+`sdk: [go, java, js]`, one runner each, `timeout-minutes: 45`, platform pinned
+to the `main` SHA, default KAS only, **serial** (no `-n`). Triggers on the
+nightly cron and on `workflow_dispatch`/`workflow_call` with
+`run-benchmarks: true`. Never on pull requests.
+
+## Edge Cases & Constraints
+
+| Threat to validity | Handling |
+|---|---|
+| Runner CPU heterogeneity | Both arms on one runner; ratios, not absolutes |
+| Noisy neighbours, steal time | Paired interleaved rounds; median + Wilcoxon; A/A gate |
+| Thermal and slow drift | Randomized within-round order; pairing differences it out |
+| Page cache, first `go build`, npx resolve | Warm-up rounds discarded |
+| JVM/npx startup dominating small payloads | Reported separately per payload size; 1 KiB *is* the startup cell |
+| Platform/KAS latency in the decrypt path | Shared by both arms in a round; cancels |
+| Arms differing in function, not speed | Container, target mode, and attribute pinned; cell skipped if the arms disagree on `SDK.supports()` for anything in the measured path |
+| Decrypt arms reading different ciphertexts | Both arms decrypt one baseline-produced file |
+| ~14 simultaneous comparisons | BH correction plus an effect threshold |
+| Optional-stopping bias | Stop on precision, never significance; min-round floor |
+| xdist contention | `--bench` under xdist is a hard `UsageError`; CI runs serial |
+| Peak RSS inheriting the measuring process's memory | The command is forked from an empty launcher, not from pytest |
+| Peak RSS pinned at the measurement floor | Both arms clip to the same number, so the ratio is 1.000 with a tight interval — the most convincing PASS the harness can emit, carrying no information. A floored cell is forced `INCONCLUSIVE` and excluded from the BH correction, like the control |
+| A failing operation | Any non-zero exit aborts the cell with captured stderr; a benchmark over the error path is worse than no benchmark |
+
+Payload sizes are 1 KiB / 1 MiB / 32 MiB because they separate two regimes that
+fail independently: at 1 KiB nearly all cost is process startup, so a throughput
+regression is invisible; at 32 MiB crypto and IO dominate, so a startup
+regression is invisible.
+
+Budget: java is the worst case at roughly 1.0 s/op small and 2.5 s at 32 MiB,
+giving ~19 s per round across both operations and both arms; 45 rounds plus the
+control lands around 16 minutes, inside the 30-minute target and the 45-minute
+job timeout. Go and JS are substantially cheaper.
+
+## Out of Scope
+
+Historical trend storage; gh-pages dashboards or `github-action-benchmark`;
+flamegraphs and profiling artifacts; cross-SDK comparison (go vs java is not a
+regression signal); cross-platform-version performance comparison; perf gating
+on pull requests; nano and other container types; in-process microbenchmarks or
+`pytest-benchmark` semantics.
+
+## Acceptance Criteria
+
+- [x] Wall clock, CPU time, and peak RSS are measured per invocation, with
+      descendant processes folded in and RSS units normalized across platforms.
+- [x] Both arms are measured on one runner, paired within randomized
+      interleaved rounds, with warm-up rounds discarded.
+- [x] Decrypt cells compare two arms reading the *same* baseline-produced
+      ciphertext; encrypt cells pin container, target mode, and attribute.
+- [x] A cell is skipped with a stated reason when a build is missing or the two
+      arms disagree on a feature in the measured path.
+- [x] The verdict uses a robust CI, a one-sided signed-rank test, a minimum
+      effect threshold, and BH multiplicity control across the run.
+- [x] An A/A control runs per SDK; if it trips, the run reports but does not
+      fail; if its interval is wider than the threshold, no cell reports PASS.
+- [x] The round loop stops on attained precision, never on significance, and
+      refuses a verdict below the minimum usable round count.
+- [x] Raw per-round samples are written to JSON and a summary table to
+      `$GITHUB_STEP_SUMMARY`.
+- [x] A confirmed wall-clock or peak-RSS regression fails the job; CPU time
+      never does.
+- [x] The offline harness tests demonstrate the gate catching a planted 25%
+      slowdown and *ignoring* a planted 3% one.
+- [ ] A `workflow_dispatch` run with `run-benchmarks: true` produces step
+      summaries and artifacts for all three SDKs inside the 45-minute timeout.

xtest/AGENTS.md

@@ -15,6 +15,7 @@ fixture system.
 | `conftest.py` | `pytest_addoption` + the encrypt/decrypt SDK parametrization. Defines `--sdks`, `--sdks-encrypt`, `--sdks-decrypt`, `--containers`, `--no-audit-logs`. |
 | `fixtures/` | Module-scoped pytest fixtures: `attributes.py`, `keys.py`, `audit.py`, `assertions.py`, `kas.py`, `encryption.py`, `obligations.py`. |
 | `tdfs.py` | SDK abstraction layer — wraps the `cli.sh` shims under `sdk/<lang>/dist/<version>/`. |
+| `perf/` | Paired A/B performance regression benchmarks (opt-in via `--bench`). **Read `perf/README.md` before changing anything in here** — the design decisions fail silently when undone. |
 | `sdk/{go,java,js}/dist/<version>/` | SDK CLI builds. Installed by `otdf-sdk-mgr install` (see `../otdf-sdk-mgr/AGENTS.md`). |
 | `test.env` | Default endpoint and client-credential env vars. Source with `set -a && source test.env && set +a`. |