Skip to content

feat(ci): nightly flake detection — N-run failure rates + serial/parallel divergence - #271

Open
raphaelvigee wants to merge 3 commits into
masterfrom
raphaelvigee/w6-flake-detection
Open

feat(ci): nightly flake detection — N-run failure rates + serial/parallel divergence#271
raphaelvigee wants to merge 3 commits into
masterfrom
raphaelvigee/w6-flake-detection

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

Five distinct flakes surfaced in one day, and the real cost was misattribution: four separate agents each spent significant time proving someone else's flake wasn't theirs, and master itself went red on one. This builds the detection that would have prevented that, in priority order:

  1. Nightly N=20 run, per-test failure rates (heph tool flake-report + .github/workflows/flake-detect.yml). Parses the default cargo test text output (no nightly-only --format json — this workspace pins stable Rust) across N repeated runs and reports, per test, how many runs it failed — a flake is now a number, not folklore. Runs on Linux + macOS nightly (schedule + workflow_dispatch), reporting via step summary + JSON artifact.
  2. Serial-vs-parallel divergence. The same nightly job runs the suite once more with --test-threads=1 and diffs it against the 20-run parallel aggregate: a test that fails under parallel execution but never serially is coupled to process-wide global state — the root cause behind two of the five flakes.
  3. Mutation testing — skipped. cargo-mutants reruns the suite per mutant; this workspace's compile times (18min cold, observed while building this feature) make even PR-scoped mutation testing expensive enough to risk shipping the other two pieces badly. Per the stated preference, shipped (1) and (2) well instead of all three poorly.

Advisory by default — it reports, it doesn't gate master. fail_on_flake: true on a manual dispatch makes it fail instead.

Review

Ran independent code-quality and feature-quality passes against the first commit. Both surfaced real, non-cosmetic bugs, all fixed in the second commit:

  • --fail-on-flake never actually failed the job (set -e without pipefail swallows a piped command's exit code) — reproduced live, fixed.
  • Doctest output was silently misattributed to the last integration-test binary (the exact misattribution class this tool exists to prevent) — fixed.
  • "Consistently failing" didn't distinguish a test seen in every run from one only reached in a few runs before its binary aborted — added a reduced-visibility category instead of overstating confidence.
  • The divergence check compared a single parallel run instead of the 20-run aggregate, missing intermittent-under-parallel tests — fixed to reuse the aggregate.
  • Added missing test coverage the reviews flagged: execute()'s CLI entrypoint (empty dir, --json-out, --fail-on-flake gating, subdirectory skip), and the <binary>::<name> collision key.

Test plan

  • cargo test -p heph --lib flake_report — 14 tests, all green
  • Every new/changed behavior verified red without the fix, then green with it (temporarily reverted each fix, confirmed the corresponding test failed, restored)
  • cargo clippy --all-targets --locked -- -D warnings and --all-features variant, both clean (workspace root, not -p)
  • cargo fmt --check clean
  • cargo test --workspace --exclude bin-e2e --no-run clean
  • Live end-to-end smoke test of the workflow's shell logic (pipefail bug reproduced + fixed; divergence jq diff verified against a synthetic global-state-coupled test)
  • First scheduled/manual run of flake-detect.yml on CI (can't run a GitHub Actions schedule locally)

🤖 Generated with Claude Code

https://claude.ai/code/session_01UbF42bJh9F15S9BmeEfcTY

@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/w6-flake-detection branch 2 times, most recently from 5acd637 to 5f0ebf2 Compare July 30, 2026 08:39
@raphaelvigee
feat(ci): nightly flake detection — N-run failure rates + serial/para…
e716470 
…llel divergence

Four separate agents each burned time proving someone else's flake wasn't
theirs, and master itself went red on one, because flakiness was folklore
with no numbers behind it. `heph tool flake-report` parses repeated `cargo
test` runs and reports per-test failure rates; the nightly workflow runs the
suite 20x on Linux+macOS, aggregates, and also diffs one parallel run against
a serial (`--test-threads=1`) run to flag tests coupled to process-wide
global state — the root cause behind two of the five flakes.

Mutation testing (item 3) is skipped: cargo-mutants reruns the suite per
mutant, and this workspace's compile times (18min cold, ~40s warm, observed
here) make even PR-scoped mutation testing expensive enough to risk shipping
the other two pieces badly rather than well.
@raphaelvigee
fix(ci): address flake-detect review findings — real bugs, not nits
73532c5 
code-quality + feature-quality review of the initial commit surfaced three
real bugs and confirmed the tool's own test coverage had gaps identical in
kind to the problem it exists to solve:

- The nightly workflow's --fail-on-flake never actually failed the job:
  `heph tool ... | tee -a "$GITHUB_STEP_SUMMARY"` under `set -e` (no
  pipefail) reports the pipeline's exit status as tee's, not heph's. Added
  `set -o pipefail`; reproduced the swallow and the fix live before committing.
- Doctest results (`Doc-tests <crate>`, no `Running ` line of its own) were
  silently misattributed to whichever integration-test binary ran last —
  the exact class of misattribution bug this tool was built to prevent.
  parse_run now recognizes the `Doc-tests` header.
- "Consistently failing" only checked failed == seen, not seen == total
  runs — a test only reached in 2 of 20 runs (because an earlier test
  aborted the process) could be labelled "broken every run" on 2 samples.
  Added has_reduced_visibility/is_consistently_failing and a markdown
  section surfacing partial-visibility tests instead of overstating them.
- The serial-vs-parallel divergence check compared a single parallel log
  against the serial run, so an intermittent-under-parallel test had a good
  chance of passing in the one sampled run and never showing up as
  divergent. It now diffs against the already-computed 20-run aggregate.

Also: execute()'s CLI entrypoint had zero test coverage (empty dir, --json-out,
--fail-on-flake gating, subdirectory skip), and the <binary>::<name> collision
key had no test proving two same-named tests in different binaries don't
collide — both added. Plus workflow hygiene: r2-secret for warm sccache,
retention-days on scratch/report artifacts, an honest "Build test binaries"
step that actually builds them, zero-padded run-log filenames, and a stated
(not silent) linux/arm64 exclusion from the matrix.
@raphaelvigee
refactor(ci): move flake-report out of the heph binary into its own c…
bcd4e08 
…rate

heph tool flake-report shipped in every user's install of the heph binary
for something that has nothing to do with building or running targets — it
exists solely for this repo's own nightly CI. Moved the tool to a standalone
crate (crates/flake-report) with its own tiny binary, dropped the
heph-tool-subcommand wiring, and pointed the nightly workflow at the new
binary directly (cargo build -p flake-report; ./flake-report --logs-dir ...
instead of heph tool flake-report --logs-dir ...).

No behavior change: same parsing/aggregation logic, same 14 tests (moved
as-is), same CLI flags. Added the crate to the workspace member list and to
devenv.nix's qualityCrates so lint/fmt still cover it.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/w6-flake-detection branch from 5f0ebf2 to bcd4e08 Compare July 30, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raphaelvigee