fix(ci): arangodb probe can never pass (401) + stress-test fake/impossible steps

[hyperpolymath]

Context

tests.yml was red on main with 4 failing jobs. Two of them — Integration Tests and
Haskell Integration Tests — fail with:

##[error]Failed to initialize container arangodb:3.11
##[error]One or more containers failed to start.

…immediately after ArangoDB itself logs ArangoDB (version 3.11.14 [linux]) is ready for business.
That paradox is the tell: the server is fine, the health probe is impossible.

Root cause — two independent faults, both measured

The probe was curl -f http://localhost:8529/_api/version || exit 1.

Fault	Evidence (run against the exact arangodb:3.11 image)
1. curl is not in the image	curl MISSING, python3 MISSING, openssl MISSING; wget, nc, arangosh present
2. localhost → IPv6, Arango binds IPv4	/etc/hosts maps both 127.0.0.1 localhost and ::1 localhost; probing localhost → Connection refused, probing 127.0.0.1 → {"server":"arango","license":"community","version":"3.11.14"}

Either fault alone is fatal, so fixing only one would not have worked — and indeed a
wget + localhost variant was tried first and still went unhealthy after 10 retries.

Changes

.github/workflows/tests.yml — all three arangodb service definitions
(integration-tests, haskell-integration-tests, and the third stack job):

-  --health-cmd "curl -f http://localhost:8529/_api/version || exit 1"
+  --health-cmd "wget -q --spider http://127.0.0.1:8529/_api/version || exit 1"

The redis-cli ping probes on the dragonfly services are correct and untouched.

Verification — end-to-end against the real image, not by inspection

Probe	Result
curl + localhost (old)	command absent — can never run
wget + localhost	unhealthy after 10 retries (Connection refused)
wget + 127.0.0.1 (shipped)	healthy in 15s ✅

YAML re-parsed clean: 11 jobs, 3/3 probes replaced, 0 curl+localhost occurrences left.

Scope note

An estate-wide sweep found only 2 repos using curl-on-localhost health checks: this one
(proven broken) and echidna/s4-loop.yml (ghcr.io/hyperpolymath/verisimdb-api:latest,
unverified — worth checking whether that image ships curl). Every other service probe in
the estate correctly uses pg_isready / redis-cli. This is therefore a small class, not
template rot.

Remaining red jobs (not addressed here — deliberately out of scope)

• E2E — Elixir Scanner Pipeline — PASS=7 FAIL=1; the failing check is mix test
  (tests/e2e.sh:118). Separate, real test failure.
• stress-test — not yet diagnosed.

Draft until CI confirms the two integration jobs go green on a real runner.

🤖 Generated with Claude Code

[hyperpolymath]

Correction: a third fault, and my "verified in CI" claim was wrong

Two things in this PR as originally written need retracting.

1. There is a THIRD fault, and it is the one that actually mattered

The body above lists two faults (curl absent, localhost → IPv6). Both are real, but
fixing them was not sufficient, because /_api/version requires authentication
whenever ARANGO_ROOT_PASSWORD is set — which all three service blocks set. And busybox
wget, the only HTTP client in the image, has no --user/--password option at all, so
the probe cannot be authenticated:

$ wget --spider --user=root --password=testpassword http://127.0.0.1:8529/_api/version
wget: unrecognized option: password=testpassword

Measured against the real image, polling once per second for 25 seconds from container start:

endpoint	result
/_api/version	exit 1 at every single timepoint (401 Unauthorized)
/_admin/server/availability	exit 1 for 3s, then exit 0 from t=3s onwards

So the probe never succeeded on any job, ever.

2. My earlier "verified healthy in CI" was a won race, not a working probe

I previously reported this fix as confirmed because a CI run logged
arangodb service is healthy. That was wrong, and the log itself shows why — the lines
appear in this order:

docker create ... --health-cmd "wget -q --spider .../_api/version" ... arangodb:3.11
arangodb service is healthy.          <-- declared healthy...
 Initializing root user...Hang on...  <-- ...before the DB had even initialised

The runner polled .State.Health.Status before Docker had recorded any health state, read
an empty string, and reported healthy. A job that polled a moment later saw starting,
waited, and eventually failed unhealthy — which is exactly the split seen in run
29828464533, where Integration Tests and Haskell Integration Tests had byte-identical
service config and opposite outcomes.

Do not read arangodb service is healthy in these logs as evidence the probe works.

Fix

Switch all three arangodb services to /_admin/server/availability — the unauthenticated
liveness endpoint intended for exactly this — which removes the race rather than winning it.

Also in this push, a separate fault in the same workflow: stress-test consisted of one
real step, two steps that could not fail, and one that could not pass. Details in the
commit message for d6b241c.

Status: pushed, CI not yet green. I am not calling this verified until the run completes.

[hyperpolymath]

Final state of this branch — measured, run 29831653210 (1dd31af)

job	before	after
stress-test	failure (11 of last 12 runs)	✅ success — twice consecutively
Build Test Images	success	✅ success
criterion + baseline gate	success	✅ success
Aspect — Rule Module Coverage	success	✅ success
E2E — Rust CLI Scan	success	✅ success
Haskell Integration Tests	failure (never once green)	removed — orphan, see 1dd31af
E2E — Elixir Scanner Pipeline	failure	❌ still failing — real Elixir test failures
Integration Tests	failure (exit 127, no BEAM)	❌ still failing — but now for an honest reason
Integration Status	failure	❌ roll-up of the above

ArangoDB now transitions properly in both remaining service jobs:
starting → healthy, rather than the instant pre-init "healthy" that a broken probe used
to produce by winning a race. Rust: 16 test suites ok, 0 failed.

The two remaining failures are not plumbing

Integration Tests previously died at exit 127 — the job installed the Rust toolchain
and then ran mix test with no BEAM on the runner at all. With erlef/setup-beam added it
now runs the suite and reports genuine failures. That is the intended outcome of this PR:
the CI was hiding the state of the test suite, and now it shows it.

Those failures are tracked separately and are not in scope here:

• test: 7 DETERMINISTIC Elixir failures (fail in isolation) — separate from the #643 order-dependent cluster #645 — 7 deterministic failures (identical count in isolation and in combination)
• test(isolation): 20 reflexive tests fail in the full suite but pass alone — Hypatia.Supervisor is torn down mid-run #643 — the ~66 order-dependent ones, plus a note that CI does not pin the ExUnit
  seed, which makes the failure count non-comparable between runs (73 @ seed 20912 vs
  79 @ seed 406044, with zero Elixir source changed in between)

What this PR contains

1. arangodb probe → /_admin/server/availability on all three services (the previous
   /_api/version probe 401s under ARANGO_ROOT_PASSWORD and busybox wget cannot
   authenticate, so it could never pass — see the correction comment above)
2. erlef/setup-beam added to integration-tests (fixes exit 127)
3. stress-test rewritten: two steps that could not fail and one that could not pass,
   replaced with assertions verified locally against the real binaries
4. haskell-integration removed — its registry/ project was deleted in f2fe2df
   (2026-03-06, "superseded by Elixir") and the job has failed every run since

Ready for review. I have not marked it non-draft because the workflow still ends red, by
design — the red is now #645/#643, not this file.