fix(tests): e2e.sh discarded all mix test output, making CI undebuggable#642
Merged
Conversation
`tests/e2e.sh` judged the Elixir suite with:
mix test --trace 2>&1 | tail -5 | grep -q "0 failures"
The pipe threw the entire run away. When the suite failed, CI printed exactly
one line:
FAIL: Elixir unit tests
Results: PASS=7 FAIL=1 SKIP=0
No failing test name, no assertion, no stacktrace. The `E2E — Elixir Scanner
Pipeline` job has been red on main with literally nothing to diagnose from.
Now: capture the run to a log, judge it by mix's own exit status (mix already
exits non-zero on failure — the `grep "0 failures"` was a weaker proxy that
also mis-reports a crash that never reaches the summary line), and echo the
tail of the output when it fails.
Verified by stubbing a failing `mix` and confirming the diagnostic surfaces:
FAIL: Elixir unit tests
--- mix test output (tail 120) ---
1) test foo (Hypatia.ReflexiveTest)
** (exit) no process
--- end mix test output ---
bash -n clean; shellcheck reports nothing on the new block.
What this immediately reveals (reproduced locally, Elixir 1.18.3/OTP 27):
`mix test` = 1398 tests, 20 failures, ALL in test/reflexive_test.exs, all
`GenServer.call(Hypatia.Supervisor, {:terminate_child, ...})` -> "no process
... possibly because its application isn't started". Running that file ALONE
gives 16 tests, 0 failures — so it is a test-isolation defect, not a product
bug. Filed separately; this commit only makes it visible.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hyperpolymath
marked this pull request as ready for review
July 21, 2026 12:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
tests/e2e.sh:118judged the whole Elixir suite with:The pipe throws the entire run away. When the suite fails, CI prints exactly this:
No failing test name. No assertion. No stacktrace.
E2E — Elixir Scanner Pipelinehas beenred on
mainwith nothing whatsoever to diagnose from — the job reports that it failedwhile actively destroying why.
Changes
Capture the run to a log, judge it by mix's own exit status, and echo the tail on
failure. Exit status is also strictly better than
grep "0 failures", which mis-reports acrash that never reaches the summary line as a pass-check failure indistinguishable from a
test failure.
Verification
Stubbed a failing
mixand confirmed the diagnostic now surfaces:bash -nclean;shellcheckreports nothing on the new block.What it immediately revealed
Reproduced locally (Elixir 1.18.3 / OTP 27):
mix test(full suite)mix test test/reflexive_test.exs(alone)All 20 failures are in
test/reflexive_test.exs, all the same shape:Since the file passes in isolation,
Hypatia.Supervisoris being torn down by somethingearlier in the full run — a test-isolation defect, not a product bug.
take_supervised/1(
test/reflexive_test.exs:29) doesterminate_child+delete_childand restores viaon_exit; under:one_for_onea child crash-looping pastmax_restartswould also take thewhole supervisor down.
That is a separate fix and is not attempted here — this PR only stops the harness from
hiding it. Filed separately.
Scope
One file. Does not change what is tested, only whether a failure is legible.
🤖 Generated with Claude Code