[Tech Debt] Flaky integration-test CI failures survive the existing test reruns

[weiqingy]

Search before asking

• I searched in the issues and found nothing similar.

Description

Two integration-test CI jobs fail intermittently for reasons unrelated to the code under test, and — importantly — these failures survive the test reruns already configured for the live-LLM e2e suite. They cost every contributor a "is this red mine?" investigation on unrelated PRs.

The existing mitigation from #716/#717 retries flaky tests: Python e2e runs with pytest --reruns 2 --reruns-delay 5 and Java e2e with -Dsurefire.rerunFailingTestsCount=2. Recent failing runs show the reruns firing (16 rerun, 56 rerun in the pytest summaries) and the jobs still ending red. So turning the retry count up will not help — a smarter mitigation is needed.

Failure A — local llama-server segfault (the dominant one).
The Ollama model server crashes at inference time while serving qwen3:1.7b:

ollama._types.ResponseError: llama-server process has terminated: signal: segmentation fault (core dumped)
io.github.ollama4j.exceptions.OllamaException: llama-server process has terminated

It is not a version regression: within a single CI run every matrix job installs the same-day Ollama binary, yet one job crashes while its siblings pass (e.g. it-java [flink-2.0] crashed while it-java on 1.20/2.1/2.2/2.3 passed in the same run). Because it varies at a fixed binary, pinning an Ollama version would add reproducibility but would not stop the crash. The model pulls fine (1.4 GB), so it is an inference-time crash, not a download issue.

Why the current reruns miss it: a rerun retries the same test ~5s later against the same ollama serve process whose llama-server subprocess just died, so it hits the dead server again. A rerun only helps if Ollama (or the model) is restarted between attempts.

Failure B — PyFlink gateway death on Flink 1.20 remote-submit e2e.
Only on it-python [python-3.11] [flink-1.20]; when it fails, all six PyFlink remote-submit e2e tests fail together on a dead py4j gateway:

py4j.protocol.Py4JError: org does not exist in the JVM

(The odd KeyError: 'p' seen alongside it is raised inside py4j/protocol.py when the client reads a garbled type-code byte off a broken gateway channel — a symptom of the same dead gateway, not repo code.) This is pre-existing and intermittent (~1 in 10 recent main runs) and also survives the reruns, which suggests the minicluster/gateway stays broken for the whole retry window.

Proposed direction (to investigate, not prescribe)

• A: detect the "process has terminated" signature and restart ollama serve (or re-pull the model) before retrying, rather than retrying into the dead server — e.g. a rerun condition hook, or a shell retry loop around the e2e step in tools/ut.sh. Verify a restart actually clears the crash in the runner before committing to it.
• B: determine whether a fresh minicluster per retry attempt recovers the gateway; if the flake is purely environmental, a tracking note may be the honest outcome rather than a code change.
• Any mitigation must stay scoped to the live-LLM / external-service e2e tests so it does not mask genuine intermittent regressions in unit tests.

This extends the flaky-CI line started in #716/#717/#719. It is independent of #889 (which hardens the Ollama install and bounds job runtime).

Are you willing to submit a PR?

• I'm willing to submit a PR!

[weiqingy]

Additional evidence from re-running the two CI-hardening PRs (#891, #892): the failures roam between matrix jobs from one run to the next, while the root cause stays the same. This is the clearest signature that it is the intermittent llama-server crash rather than any deterministic bug — a real regression would fail the same job every time.

Same two PRs, two consecutive CI runs each:

• Run 1 — [ci] Retry Ollama install and propagate download failures #891 failed it-python [3.11] [flink-1.20] + it-python [3.12] [flink-2.1]; [ci] Bound the runtime of the integration-test jobs #892 failed it-java [java-21] [flink-2.0] + it-python [3.11] [flink-1.20].
• Run 2 (fresh trigger) — [ci] Retry Ollama install and propagate download failures #891 failed cross-language + it-java [flink-1.20] + it-java [java-21] [flink-2.0]; [ci] Bound the runtime of the integration-test jobs #892 failed it-java [flink-1.20] + it-python [3.11] [flink-1.20].

Spot-checking the failing jobs confirms the same signatures: the roaming ones (it-java, it-python, cross-language across various Flink versions) all show the llama-server process has terminated: signal: segmentation fault crash (failure A), and it-python [flink-1.20] shows the py4j gateway death (failure B). Both survive the existing --reruns 2 / rerunFailingTestsCount=2.

Rough flake rate: across the Ollama-dependent jobs (~11 per run), 2–3 crash each run, i.e. an order of ~20% per-job segfault rate. That puts the probability of an all-green run at roughly 0.8^11 ≈ 8% — so re-triggering (even via a fresh commit) is not a reliable way to get green; it just relocates which job crashes, and it burns shared runner capacity doing so.

This reinforces the direction in the description: raising the retry count cannot help (the retry fires ~5s later into the same dead ollama serve), so the mitigation has to restart Ollama on the "process has terminated" signature before retrying — and that restart needs verifying against the runner before it is treated as a fix, since the crash may simply recur on the fresh server.

[weiqingy]

Update — the flake has a dated onset, which points at an external change (most likely the unpinned Ollama version) rather than anything in the repo, and revises the mitigation priority.

main's CI history:

• 07-06: 2 runs, all pass
• 07-07: 2 runs, all pass
• 07-08: 4 runs, all pass
• 07-09: no runs (nothing merged)
• 07-10: 2 runs, both fail — and one of them is a docs-only change that cannot touch the inference path

So main was green for eight straight runs through 07-08 and both runs on 07-10 failed with these same two signatures (one llama-server segfault, one py4j-1.20 gateway death). A regression introduced by a merge would have reddened the first main run after that merge; instead the failures appear on a day (07-09) when nothing merged. That rules out a repo change as the cause.

The most likely trigger is the unpinned Ollama version: tools/start_ollama_server.sh installs the latest Ollama with no version lock, so an upstream release around 07-09 would land in CI automatically with no repo change — which matches the green-through-07-08 / red-from-07-10 boundary exactly. (I could not extract the exact installed version from the job logs to prove a version bump specifically — a runner-image update is a secondary possibility — so this is a strong hypothesis rather than a proven cause. The "external, dated onset, not a PR" conclusion is solid regardless.)

This changes the recommended first step. An earlier read here was that pinning a version would only add determinism, based on same-binary mixed pass/fail within a single run. But a dated onset means something changed, so pinning Ollama back to the last-known-good version (whatever was latest around 07-06/07-08) is now the cheapest thing to try first and could restore green outright — before investing in the more involved restart-Ollama-on-crash approach. Suggested order: (1) identify the last-good Ollama version and pin OLLAMA_VERSION; (2) if segfaults persist at that pin, fall back to restart-on-crash; (3) the py4j-1.20 gateway flake is separate and may only warrant a tracking note.

[weiqingy]

Correction to my previous comment: the release timeline rules out the Ollama-version-bump hypothesis, so a version pin is unlikely to help.

Ollama's release dates around the onset (from the GitHub releases API):

• v0.31.1 — 2026-06-30
• v0.31.2 — 2026-07-06
• (no release after 07-06)

tools/start_ollama_server.sh installs the latest Ollama, which has been v0.31.2 since 07-06. So the green runs on 07-07 and 07-08 already installed v0.31.2, and the red runs on 07-10 installed that same v0.31.2 — the version was constant across the green→red boundary. Pinning OLLAMA_VERSION would only pin back to a version that was already both green and red, so it cannot explain or fix the onset.

The "external, dated onset, not a PR" conclusion still holds — eight straight green runs through 07-08 rule out chance — but the external change is not the Ollama version. The leading explanation is the GitHub-hosted runner image rolling forward within ubuntu-latest (kernel 6.17.0-1018-azure on a current runner), which is not pinnable via OLLAMA_VERSION. Two more data points fit an environmental cause: v0.31.2 runs thousands of concurrent, grammar-constrained, model-churning inferences without a crash in an isolated Ollama-only job, yet within a single CI run the same v0.31.2 both passes and crashes across different matrix legs.

Since the cause is environmental and per-runner, the mitigation that is robust regardless of the exact trigger is to re-run only the failed integration-test jobs on a fresh runner, which resets the whole job environment. Opened as #895.

[weiqingy]

Root-cause update: the segfault is a per-runner-VM lottery, not something we can pin or configure away.

I tried to reproduce and stabilize it, and the evidence converges on a hardware/environment cause tied to the specific GitHub-hosted runner VM:

• Not reproducible in isolation. A stress harness on the same ubuntu-latest image ran thousands of concurrent, grammar-constrained (format=json), model-churning qwen3 inferences — 0 crashes.
• Not reproducible even with the real workload. Looping the actual segfaulting Java integration tests (YamlLoaderIntegrationTest, ChatModelIntegrationTest, ReActAgentTest) ~90 times across 6 runner VMs — 0 crashes — while CI on this repo crashes ~18–20% per job in the same window.
• Constraining Ollama did not change anything. An A/B of OLLAMA_NUM_PARALLEL=1 + OLLAMA_MAX_LOADED_MODELS=1 vs default came back 0 vs 0 — inconclusive, because the baseline would not crash either.
• Not a version change. Ollama v0.31.2 has been the installed "latest" since 07-06, spanning both the green (07-07/08) and red (07-10) days, so a version pin cannot explain or fix the onset.

Putting this together with the earlier signals — the ~19-minute in-job wedge (once a job's VM starts crashing it stays crashing), the failures roaming between matrix jobs, and the version-neutral pass/fail within a single run — the model that fits everything is: a subset (~20%) of GitHub's runner VMs are "cursed" by a hardware/microarchitecture condition that trips the underlying llama.cpp segfault, and a given VM is effectively deterministic (good or bad). The dated onset (~07-09) lines up with a GitHub runner-image rollout that made that subset bad, not with anything in this repo or in Ollama.

Consequence: nothing that stays on the same VM can fix it — not a bigger in-suite retry (retries into the same crashed server), not an in-process Ollama restart, not a version pin, not an Ollama resource flag. The only reliable escape is to re-run the failed job on a fresh runner, which re-rolls the VM (~80% land healthy).

Mitigation opened as #895: a workflow_run companion that, when CI fails, re-runs only the failed integration-test legs on a fresh runner (unit-test/lint/build failures are never re-run, so real regressions stay red). It cannot demonstrate itself on its own PR — workflow_run companions run only from the default branch — but once merged it applies to every subsequent CI run: a flaky integration-test leg auto-heals to green while a genuine failure that reproduces on every fresh runner stays red.