Skip to content

fix(tests): e2e.sh discarded all mix test output, making CI undebuggable#642

Merged
hyperpolymath merged 2 commits into
mainfrom
fix/e2e-surface-mix-output
Jul 21, 2026
Merged

fix(tests): e2e.sh discarded all mix test output, making CI undebuggable#642
hyperpolymath merged 2 commits into
mainfrom
fix/e2e-surface-mix-output

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

The problem

tests/e2e.sh:118 judged the whole Elixir suite with:

mix test --trace 2>&1 | tail -5 | grep -q "0 failures"

The pipe throws the entire run away. When the suite fails, CI prints exactly this:

  FAIL: Elixir unit tests
  Results: PASS=7  FAIL=1  SKIP=0

No failing test name. No assertion. No stacktrace. E2E — Elixir Scanner Pipeline has been
red on main with nothing whatsoever to diagnose from — the job reports that it failed
while actively destroying why.

Changes

Capture the run to a log, judge it by mix's own exit status, and echo the tail on
failure. Exit status is also strictly better than grep "0 failures", which mis-reports a
crash that never reaches the summary line as a pass-check failure indistinguishable from a
test failure.

Verification

Stubbed a failing mix and confirmed the diagnostic now surfaces:

  FAIL: Elixir unit tests
--- mix test output (tail 120) ---
  1) test foo (Hypatia.ReflexiveTest)
     ** (exit) no process
--- end mix test output ---

bash -n clean; shellcheck reports nothing on the new block.

What it immediately revealed

Reproduced locally (Elixir 1.18.3 / OTP 27):

Run Result
mix test (full suite) 1398 tests, 20 failures
mix test test/reflexive_test.exs (alone) 16 tests, 0 failures

All 20 failures are in test/reflexive_test.exs, all the same shape:

** (exit) exited in: GenServer.call(Hypatia.Supervisor, {:terminate_child, Hypatia.SelfDiagnostics}, :infinity)
    ** (EXIT) no process: the process is not alive ... possibly because its application isn't started
  test/reflexive_test.exs:29: Hypatia.ReflexiveTest.take_supervised/1

Since the file passes in isolation, Hypatia.Supervisor is being torn down by something
earlier in the full run
— a test-isolation defect, not a product bug. take_supervised/1
(test/reflexive_test.exs:29) does terminate_child + delete_child and restores via
on_exit; under :one_for_one a child crash-looping past max_restarts would also take the
whole supervisor down.

That is a separate fix and is not attempted here — this PR only stops the harness from
hiding it. Filed separately.

Scope

One file. Does not change what is tested, only whether a failure is legible.

🤖 Generated with Claude Code

`tests/e2e.sh` judged the Elixir suite with:

    mix test --trace 2>&1 | tail -5 | grep -q "0 failures"

The pipe threw the entire run away. When the suite failed, CI printed exactly
one line:

    FAIL: Elixir unit tests
    Results: PASS=7  FAIL=1  SKIP=0

No failing test name, no assertion, no stacktrace. The `E2E — Elixir Scanner
Pipeline` job has been red on main with literally nothing to diagnose from.

Now: capture the run to a log, judge it by mix's own exit status (mix already
exits non-zero on failure — the `grep "0 failures"` was a weaker proxy that
also mis-reports a crash that never reaches the summary line), and echo the
tail of the output when it fails.

Verified by stubbing a failing `mix` and confirming the diagnostic surfaces:

    FAIL: Elixir unit tests
    --- mix test output (tail 120) ---
      1) test foo (Hypatia.ReflexiveTest)
         ** (exit) no process
    --- end mix test output ---

bash -n clean; shellcheck reports nothing on the new block.

What this immediately reveals (reproduced locally, Elixir 1.18.3/OTP 27):
`mix test` = 1398 tests, 20 failures, ALL in test/reflexive_test.exs, all
`GenServer.call(Hypatia.Supervisor, {:terminate_child, ...})` -> "no process
... possibly because its application isn't started". Running that file ALONE
gives 16 tests, 0 failures — so it is a test-isolation defect, not a product
bug. Filed separately; this commit only makes it visible.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@hyperpolymath
hyperpolymath merged commit 3aeb988 into main Jul 21, 2026
47 of 52 checks passed
@hyperpolymath
hyperpolymath deleted the fix/e2e-surface-mix-output branch July 21, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant