Skip to content

fix(ci): arangodb probe can never pass (401) + stress-test fake/impossible steps #640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hyperpolymath merged 6 commits into from
Jul 21, 2026
Merged

fix(ci): arangodb probe can never pass (401) + stress-test fake/impossible steps #640

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7bfa53f
fix(ci): arangodb health check can never pass — curl absent + IPv6 lo…
hyperpolymath Jul 21, 2026
24f3ac0
fix(ci): move comments OUT of the `options: >-` folded scalar
hyperpolymath Jul 21, 2026
311b504
fix(ci): Integration Tests ran `mix test` with no BEAM; harden arango…
hyperpolymath Jul 21, 2026
d6b241c
fix(ci): arango probe could never pass; stress-test was 2 fakes + 1 i…
hyperpolymath Jul 21, 2026
1dd31af
fix(ci): remove orphaned Haskell job — its project was deleted in March
hyperpolymath Jul 21, 2026
b82a59d
Merge branch 'main' into fix/arangodb-healthcheck
hyperpolymath Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
244 changes: 161 additions & 83 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,11 +191,40 @@ jobs:
ARANGO_NO_AUTH: "0"
ports:
- 8529:8529
# THREE separate faults made this probe unable to ever succeed. All
# three were measured against the real arangodb:3.11 image, not inferred:
#
# 1. `curl` is NOT installed in the image -> use wget (present)
# 2. `localhost` resolves to ::1 there, but -> use 127.0.0.1
# ArangoDB binds IPv4 only
# 3. `/_api/version` requires authentication -> use
# whenever ARANGO_ROOT_PASSWORD is set, /_admin/server/availability
# so it returns 401 forever; and busybox
# wget has no --user/--password option at
# all, so the probe CANNOT be authenticated
#
# Measured, polling once a second for 25s from container start:
# /_api/version -> exit 1 at EVERY timepoint (401)
# /_admin/server/availability -> exit 1 for 3s, then exit 0 onwards
#
# Fault 3 is why this looked intermittent. A probe that never passes
# leaves the service permanently `starting`; whether the job survived
# depended on a race in the runner's own poll (if it read
# .State.Health.Status before Docker had recorded any health state it
# saw an empty string and reported "healthy" — which is why one job
# once logged "arangodb service is healthy" BEFORE the container had
# even logged "Initializing root user"). Do not read that log line as
# evidence the probe works.
#
# NB: keep comments OUTSIDE the `options: >-` folded scalar — inside it
# `#` is literal text, gets folded into the docker args, and breaks
# `docker create` with "invalid reference format".
options: >-
--health-cmd "curl -f http://localhost:8529/_api/version || exit 1"
--health-cmd "wget -q --spider http://127.0.0.1:8529/_admin/server/availability || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 10
--health-start-period 40s

dragonfly:
image: docker.dragonflydb.io/dragonflydb/dragonfly:latest
Expand Down Expand Up @@ -285,6 +314,19 @@ jobs:
TEST_GITHUB_TOKEN: ${{ secrets.TEST_GITHUB_TOKEN }}
RUST_LOG: debug

# This job installs the RUST toolchain and then ran `mix test` — with no
# BEAM anywhere on the runner. `mix` was simply not on PATH, so the step
# died with exit 127 (command not found) and took `Integration Tests` red
# with it. Same pins as the E2E job, so the two agree on the toolchain.
- name: Setup Erlang/Elixir
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.18.2
with:
otp-version: '27.0'
elixir-version: '1.17'

- name: Install Elixir dependencies
run: mix deps.get

- name: Run Elixir integration tests
run: mix test --include integration
env:
Expand All @@ -307,70 +349,30 @@ jobs:
DRAGONFLY_URL: redis://localhost:6379

# ============================================================================
# Haskell Integration Tests
# REMOVED — "Haskell Integration Tests" (haskell-integration)
#
# The job ran `cabal build all` / `cabal test all` with
# `defaults.run.working-directory: registry`, and cached on
# `hashFiles('registry/hypatia.cabal')`.
#
# `registry/` does not exist. It was deliberately deleted in f2fe2df
# (2026-03-06), whose own commit message reads:
# "Remove registry/ (30 files - Haskell, superseded by Elixir)"
# The CI job was left behind and has failed EVERY run since: 11 failures
# and 1 cancellation across the last 12 runs, never once green. The repo
# now contains 0 .cabal files, no cabal.project, no stack.yaml, and a
# single .hs file — test/soundness/fixtures/code_safety/unsafe_coerce.hs,
# which is a scanner test fixture, not a Haskell project.
#
# Each run installed GHC 9.6 + cabal 3.10 and stood up an ArangoDB service
# container before failing at the first `run:` step with
# No such file or directory ... /home/runner/work/hypatia/hypatia/registry
# so removing it also stops burning those runner minutes.
#
# If Haskell returns to this repo, reinstate the job together with the
# project, not before.
# ============================================================================

haskell-integration:
name: Haskell Integration Tests
runs-on: ubuntu-latest
timeout-minutes: 60
needs: [build-images]
services:
arangodb:
image: arangodb:3.11
env:
ARANGO_ROOT_PASSWORD: testpassword
ports:
- 8529:8529
options: >-
--health-cmd "curl -f http://localhost:8529/_api/version || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 10

defaults:
run:
working-directory: registry
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

- name: Setup Haskell
uses: haskell-actions/setup@cd0d9bdd65b20557f41bea4dbe43d0b5fbbfe553 # v2.11.0
with:
ghc-version: '9.6'
cabal-version: '3.10'
cabal-update: true

- name: Restore cached dependencies
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
id: cache
with:
path: ~/.cabal/store
key: ${{ runner.os }}-ghc-9.6-cabal-integration-${{ hashFiles('registry/hypatia.cabal') }}
restore-keys: ${{ runner.os }}-ghc-9.6-cabal-

- name: Install dependencies
run: cabal build all --only-dependencies

- name: Wait for ArangoDB
run: |
for i in {1..30}; do
if curl -s http://localhost:8529/_api/version > /dev/null; then
echo "ArangoDB is ready"
break
fi
sleep 2
done

- name: Run Haskell integration tests
run: cabal test all
env:
ARANGODB_URL: http://localhost:8529
ARANGODB_DATABASE: cicd_hyper_a
ARANGODB_USER: root
ARANGODB_PASSWORD: testpassword

# ============================================================================
# End-to-End Tests
# ============================================================================
Expand Down Expand Up @@ -457,11 +459,40 @@ jobs:
ARANGO_ROOT_PASSWORD: testpassword
ports:
- 8529:8529
# THREE separate faults made this probe unable to ever succeed. All
# three were measured against the real arangodb:3.11 image, not inferred:
#
# 1. `curl` is NOT installed in the image -> use wget (present)
# 2. `localhost` resolves to ::1 there, but -> use 127.0.0.1
# ArangoDB binds IPv4 only
# 3. `/_api/version` requires authentication -> use
# whenever ARANGO_ROOT_PASSWORD is set, /_admin/server/availability
# so it returns 401 forever; and busybox
# wget has no --user/--password option at
# all, so the probe CANNOT be authenticated
#
# Measured, polling once a second for 25s from container start:
# /_api/version -> exit 1 at EVERY timepoint (401)
# /_admin/server/availability -> exit 1 for 3s, then exit 0 onwards
#
# Fault 3 is why this looked intermittent. A probe that never passes
# leaves the service permanently `starting`; whether the job survived
# depended on a race in the runner's own poll (if it read
# .State.Health.Status before Docker had recorded any health state it
# saw an empty string and reported "healthy" — which is why one job
# once logged "arangodb service is healthy" BEFORE the container had
# even logged "Initializing root user"). Do not read that log line as
# evidence the probe works.
#
# NB: keep comments OUTSIDE the `options: >-` folded scalar — inside it
# `#` is literal text, gets folded into the docker args, and breaks
# `docker create` with "invalid reference format".
options: >-
--health-cmd "curl -f http://localhost:8529/_api/version || exit 1"
--health-cmd "wget -q --spider http://127.0.0.1:8529/_admin/server/availability || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 10
--health-start-period 40s

dragonfly:
image: docker.dragonflydb.io/dragonflydb/dragonfly:latest
Expand Down Expand Up @@ -539,7 +570,6 @@ jobs:
- integration-tests
- e2e-tests
- coverage-report
- haskell-integration
if: always()
steps:
- name: Check integration status
Expand All @@ -552,7 +582,6 @@ jobs:
["integration-tests"]="${{ needs.integration-tests.result }}"
["e2e-tests"]="${{ needs.e2e-tests.result }}"
["coverage-report"]="${{ needs.coverage-report.result }}"
["haskell-integration"]="${{ needs.haskell-integration.result }}"
)

failed=false
Expand Down Expand Up @@ -591,38 +620,87 @@ jobs:
- name: Install stress testing tools
run: |
sudo apt-get update
sudo apt-get install -y stress-ng valgrind
sudo apt-get install -y stress-ng

- name: Build release
run: cargo build --release --all-features

- name: Concurrent operations stress test
# `./target/release/*` globs BUILD DIRECTORIES (build/, deps/, examples/,
# incremental/) and .d files — not just binaries. The previous steps ran
# `timeout 1s ./target/release/build` 100 times, printed "Permission
# denied" 100 times, and still reported SUCCESS, because a bare `wait`
# returns 0 no matter what the background jobs did. Resolve the real bin
# targets from cargo metadata so this cannot rot when a binary is added
# or renamed, and assert each one exists before relying on it.
- name: Resolve release binaries
run: |
# Run binary with high concurrency
for i in {1..100}; do
timeout 1s ./target/release/* &
set -euo pipefail
BINS="$(cargo metadata --no-deps --format-version 1 \
| jq -r '.packages[].targets[] | select(.kind[] == "bin") | .name' \
| sort -u | tr '\n' ' ')"
[ -n "${BINS// /}" ] || { echo "::error::no bin targets resolved from cargo metadata"; exit 1; }
echo "resolved binaries: $BINS"
for b in $BINS; do
[ -x "target/release/$b" ] \
|| { echo "::error::target/release/$b missing or not executable"; exit 1; }
done
wait
echo "STRESS_BINS=$BINS" >> "$GITHUB_ENV"

- name: Memory pressure test
- name: Concurrent launch stress test
run: |
# Run under memory constraints
ulimit -v 512000 # 500MB virtual memory limit
cargo test --release
set -euo pipefail
pids=(); names=()
for b in $STRESS_BINS; do
for _ in $(seq 1 25); do
"./target/release/$b" --help >/dev/null 2>&1 &
pids+=("$!"); names+=("$b")
done
done
failed=0
for i in "${!pids[@]}"; do
if ! wait "${pids[$i]}"; then
echo "::error::${names[$i]} exited non-zero under concurrent launch"
failed=$((failed + 1))
fi
done
echo "launched ${#pids[@]} concurrent processes; failures: $failed"
[ "$failed" -eq 0 ]

- name: Long-running scenario test
- name: Memory-constrained execution
run: |
# Test for memory leaks over time
timeout 300s valgrind --leak-check=full --error-exitcode=1 \
./target/release/* || true
set -euo pipefail
# The cap goes on the BINARY, never on cargo/rustc. The previous
# `ulimit -v 512000; cargo test --release` failed on 6 of 6 runs:
# 500MB of virtual address space cannot host rustc's LLD linker,
# which aborts with
# std::system_error ... Resource temporarily unavailable
# collect2: fatal error: ld terminated with signal 6 [Aborted]
# The very same build scripts (zerocopy, crossbeam-utils, num-traits)
# compiled fine in the preceding step, which set no ulimit — that
# in-log A/B is what identified the cause.
for b in $STRESS_BINS; do
if ( ulimit -v 1048576; "./target/release/$b" --help >/dev/null 2>&1 ); then
echo " ok: $b runs under a 1GiB virtual-memory cap"
else
echo "::error::$b failed under a 1GiB virtual-memory cap"
exit 1
fi
done

# REMOVED — "Long-running scenario test": it ran
# valgrind --leak-check=full --error-exitcode=1 ./target/release/* || true
# which globbed a directory (so valgrind never profiled a binary) and
# ended in `|| true` (so it could not fail even when it did). It reported
# success without testing anything. Deleted rather than left in place.
# To reinstate as a real gate: name a specific binary and drop `|| true`.

- name: Stress test with stress-ng
- name: Test suite under CPU and I/O stress
run: |
# CPU and I/O stress
set -euo pipefail
stress-ng --cpu 4 --io 2 --timeout 60s &
STRESS_PID=$!
trap 'kill "$STRESS_PID" 2>/dev/null || true' EXIT
cargo test --release
kill $STRESS_PID || true

bench:
name: criterion + baseline gate
Expand Down
Loading