[ci] Retry Ollama install and propagate download failures#891
Open
weiqingy wants to merge 4 commits into
Open
Conversation
The upstream Ollama installer probes for the .tar.zst release asset with a silent HEAD request and, on any failure of that probe, falls back to a .tgz that current releases no longer publish. A single transient network error therefore produces a hard 404 and fails the integration-test jobs before any test runs. Retry the install up to three times with a linear backoff. Reading the exit status after "curl ... | sh" also captured the status of sh rather than curl, so a failed download left sh reading empty stdin and exiting zero. The install step then "succeeded" with no Ollama present. Download the installer to a file and check curl directly, under set -euo pipefail.
Collaborator
Author
|
The 2 failing checks are unrelated to this change — the Ollama install step this PR touches actually succeeds in both failing jobs (
This PR hardens the install step against transient download 404s; it doesn't address the |
1 task
2 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issue: #889
Purpose of change
The
it-java,it-python, andcross-languageCI jobs install a local Ollama server viatools/start_ollama_server.sh, which pipeshttps://ollama.com/install.shintosh. That upstream installer probes for theollama-linux-${ARCH}.tar.zstrelease asset with a silentcurl --fail --silent --head, and on any failure of that probe it falls back to a.tgzasset that current releases no longer publish (.tgzreturns404). A single transient network error on the probe therefore turns into a hard failure, before any test compiles or runs:This retries the install up to three times with a linear backoff, so a transient failure recovers instead of failing the job. The underlying
.tar.zstasset does exist, so a retry is a genuine remedy rather than another roll of the dice.It also fixes a latent bug in the wrapper. The old script read
ret=$?aftercurl -fsSL ... | sh, which captures the exit status ofsh, notcurl. With nopipefail, a failed download leftshreading empty stdin and exiting0, so the step "succeeded" with no Ollama installed and the failure surfaced later somewhere less obvious. The installer is now downloaded to a file andcurlis checked directly, underset -euo pipefail.Tests
Verified locally by exercising both paths against stubbed install targets: the script recovers and exits
0when a transient failure clears before the retries are exhausted, and it exits1after three failed attempts against a persistent404.bash -npasses; the CI lint job exercises the script's shell checks.The full CI matrix on this PR exercises the change end-to-end — every
it-java/it-python/cross-languagejob runs this installer.API
No public API changes.
Documentation
doc-not-needed