Skip to content

[ci] Retry Ollama install and propagate download failures#891

Open
weiqingy wants to merge 4 commits into
apache:mainfrom
weiqingy:889-ollama-install-retry
Open

[ci] Retry Ollama install and propagate download failures#891
weiqingy wants to merge 4 commits into
apache:mainfrom
weiqingy:889-ollama-install-retry

Conversation

@weiqingy

Copy link
Copy Markdown
Collaborator

Linked issue: #889

Purpose of change

The it-java, it-python, and cross-language CI jobs install a local Ollama server via tools/start_ollama_server.sh, which pipes https://ollama.com/install.sh into sh. That upstream installer probes for the ollama-linux-${ARCH}.tar.zst release asset with a silent curl --fail --silent --head, and on any failure of that probe it falls back to a .tgz asset that current releases no longer publish (.tgz returns 404). A single transient network error on the probe therefore turns into a hard failure, before any test compiles or runs:

>>> Downloading ollama-linux-amd64.tgz
curl: (22) The requested URL returned error: 404

gzip: stdin: unexpected end of file
tar: Child returned status 1
##[error]Process completed with exit code 2.

This retries the install up to three times with a linear backoff, so a transient failure recovers instead of failing the job. The underlying .tar.zst asset does exist, so a retry is a genuine remedy rather than another roll of the dice.

It also fixes a latent bug in the wrapper. The old script read ret=$? after curl -fsSL ... | sh, which captures the exit status of sh, not curl. With no pipefail, a failed download left sh reading empty stdin and exiting 0, so the step "succeeded" with no Ollama installed and the failure surfaced later somewhere less obvious. The installer is now downloaded to a file and curl is checked directly, under set -euo pipefail.

Tests

Verified locally by exercising both paths against stubbed install targets: the script recovers and exits 0 when a transient failure clears before the retries are exhausted, and it exits 1 after three failed attempts against a persistent 404. bash -n passes; the CI lint job exercises the script's shell checks.

The full CI matrix on this PR exercises the change end-to-end — every it-java / it-python / cross-language job runs this installer.

API

No public API changes.

Documentation

  • doc-not-needed

The upstream Ollama installer probes for the .tar.zst release asset with a
silent HEAD request and, on any failure of that probe, falls back to a .tgz
that current releases no longer publish. A single transient network error
therefore produces a hard 404 and fails the integration-test jobs before any
test runs. Retry the install up to three times with a linear backoff.

Reading the exit status after "curl ... | sh" also captured the status of sh
rather than curl, so a failed download left sh reading empty stdin and exiting
zero. The install step then "succeeded" with no Ollama present. Download the
installer to a file and check curl directly, under set -euo pipefail.
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 10, 2026
@weiqingy

weiqingy commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

The 2 failing checks are unrelated to this change — the Ollama install step this PR touches actually succeeds in both failing jobs (>>> Downloading ollama-linux-amd64.tar.zst>>> Install complete), so the retry/propagation fix is working; the failures are downstream of a healthy install.

  • it-python [python-3.12] [flink-2.1]: the local llama-server crashes mid-run (ollama._types.ResponseError: llama-server process has terminated: signal: segmentation fault (core dumped)). This is the model-server crash, not the installer — intermittent on main too (passes ~5/6 runs). Log.
  • it-python [python-3.11] [flink-1.20]: a py4j/Flink-1.20 classpath failure (org does not exist in the JVM, KeyError: 'p'), not Ollama-related. The same job fails identically on current main (main run), so it's pre-existing and flaky. Log.

This PR hardens the install step against transient download 404s; it doesn't address the llama-server segfault, which is tracked as the open item on #889.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant