fix(ci): resolve perf baseline to latest published master release, and let a manual run choose both sides

[raphaelvigee]

Problem

The perf gate's baseline was the previous commit — github.event.before on push, the exact merge-base on PRs — and its release tag was re-derived from that single commit's CI run. Both assumptions fail routinely:

• CI runs once per push, on that push's head commit. Every other commit in a batched push has no run and no release at all. 59585cc6 is on master with zero CI runs — verified against the live API.
• master builds run concurrently (per-sha concurrency groups), so the previous commit's release may not be published yet when this job runs, or ever if its build broke.

Either case failed the job outright with "no baseline found". That is how this gate has spent most of its life: reporting a data-availability problem rather than comparing anything.

Change

Baseline = the latest published master release at/before the baseline point. Walk master-push CI runs newest-first and keep the first ancestor whose re-derived tag exists as a release. PRs still try the exact merge-base first and only walk when it has none — and say so in the report, since a walked baseline widens what the gate attributes to the PR.

Manual runs choose both sides. workflow_dispatch takes candidate and baseline, each a commit SHA or an exact release tag. Both default to what an automatic run would have used, so a no-input dispatch re-runs what push would have, while naming both turns the workflow into "compare these two builds":

gh workflow run Perf --ref master                                    # auto both sides
gh workflow run Perf --ref master -f candidate=85b556b3              # pin the candidate
gh workflow run Perf --ref master \
  -f candidate=v1.0.0-alpha-build.312.1247+gd46c6c4f \
  -f baseline=v1.0.0-alpha-build.301.1219+ge9fa09b8

Nothing is built on a dispatch, so both sides must already be published. The candidate resolves before the baseline, because an explicit candidate also bounds where the baseline walk searches back from — comparing release N against the latest master release would otherwise be wrong whenever N is not the newest build.

Naming a commit works for master commits only: a PR run's tag derives from its ephemeral merge commit, which cannot be re-derived from the branch head the runs API exposes. PRs use the perf-test label, unchanged, or pass a release tag directly.

Guards against quietly-wrong comparisons

• A candidate named by tag whose commit is absent from the checkout fails instead of searching back from master's head. That fallback could pick a baseline newer than the candidate, inverting every delta's sign while reporting an ordinary verdict.
• Candidate and baseline resolving to the same release is refused rather than burning three legs to prove a build equals itself.
• A baseline that is not an ancestor of the candidate is allowed — comparing two release lines is legitimate — but the report says so.

Shared library, and its tests

Both sides now need the same commit → CI run → tag → is-it-published resolution and the same asset download, so it lives in perf-release-lib.sh instead of two copies. It is unit-tested with gh/git stubbed (perf-release-lib.test.sh, wired into CI as perf_lib, ~5s). The paths worth freezing are the failure ones, and none can be provoked on demand against the live API.

Extracting it immediately surfaced four bugs:

• run_for_sha reported an Actions API failure as "this commit has no CI run" — the same misdiagnosis release_exists already guards against, reachable on the most common dispatch of all.
• .workflow_runs | first | @tsv renders a null run as the non-empty string "\t\t-", which read as a successful lookup and produced CI run #- … is still —. Fixed with select(. != null).
• fetch_assets discarded gh release download's exit status, so a transient failure reported as "the release is missing assets"; it also shared stdout with gh, where one future line of chatter would flip every caller's missing-asset test.
• An unguarded version=$(version_for_run …) could carry an empty tag into the error messages — set -e does not reach inside $( ).

Other supporting fixes

• A non-404 from the artifacts repo (expired PAT, outage) no longer reads as "release does not exist" — that turned a dead token into a confident no published master release found aimed at the wrong system.
• The run listing is captured rather than process-substituted, so a failed gh api gets its own message instead of an empty read.
• Assets are verified by name: gh release download exits 0 when only some patterns matched, so a partially published release died on a bare cp: cannot stat. Baseline degrades, candidate fails, both naming what is absent.
• Run lookups use the workflow-scoped endpoint — no .name=="CI" string coupling, and the 50-run window is no longer shared with the other push-triggered workflows (autodoc roughly halved it).
• Tier A is gated on the Tier B assets too; it used to burn its minutes for a report that then hard-failed.

Diagnosability

verdicts/resolution.json records what was actually compared — both versions, SHAs, and how each was chosen — on every trigger, so a dispatcher reads a stable JSON result instead of scraping markdown. The artifact uploads even when no baseline resolved, which is when that answer matters most. The report names the candidate on the failure path too, and PostHog carries candidate_version/baseline_version.

Verification

perf-release-lib.test.sh covers 19 cases (green on bash 5.3 and macOS bash 3.2, and in CI). On top of that, every step's script was extracted from the YAML and run against the live GitHub API:

Case	Result
push, auto	walks past self → v1.0.0-alpha-build.312.1247+gd46c6c4f
dispatch, candidate pins the window	candidate run 1221 → baseline run 1219, not 1247
dispatch candidate by SHA / by tag	both resolve; tag recovers its sha from +g
candidate == baseline	refused, exit 1
candidate by tag, sha unknown, no baseline	refused, exit 1 (would have inverted the verdict)
baseline newer than candidate	ancestor=false + report note
bogus tag / bogus SHA / non-master SHA	exit 1, each with its own message
expired artifacts token	Bad credentials (HTTP 401), exit 1 — not "no release"
asset fetch: real leg / partial / missing	downloads, or names exactly what is absent
baseline missing assets	degrades to ok=false, does not fail the step

Plus YAML parse and bash -n over every run: block.

Note

This PR's own Perf legs only run if the perf-test label is applied — worth adding before merge so the auto-resolution path proves itself end to end.

🤖 Generated with Claude Code

https://claude.ai/code/session_015zSJ4W6qpxdCMJ1XbXnU5X