Skip to content

[ci] Bound the runtime of the integration-test jobs#892

Open
weiqingy wants to merge 4 commits into
apache:mainfrom
weiqingy:889-ci-job-timeouts
Open

[ci] Bound the runtime of the integration-test jobs#892
weiqingy wants to merge 4 commits into
apache:mainfrom
weiqingy:889-ci-job-timeouts

Conversation

@weiqingy

Copy link
Copy Markdown
Collaborator

Linked issue: #889

Purpose of change

None of the jobs in ci.yml set timeout-minutes, so a hung test runs until GitHub's six-hour per-job ceiling before it is killed. A recent run consumed 6h0m16s on a stalled it-python job — the local llama-server segfaulted mid-run, pytest hung, and the job burned shared Apache runner capacity for a result that was knowable in minutes.

This bounds the three jobs that install and exercise a local Ollama server (it-python, it-java, cross-language), which are the ones exposed to that hang. Each limit is roughly two to three times the job's slowest observed run over recent green builds on main:

job slowest observed (p100, 6 green main runs) timeout-minutes
it-python 26.2m 60
cross-language 24.8m 60
it-java 14.0m 40

The segfault itself is a separate question tracked in #889; this change only ensures such a hang fails fast instead of running to the six-hour ceiling.

Tests

No test changes. Verified that the workflow YAML still parses and that the three timeout-minutes keys attach at job level to python_it_tests, java_it_tests, and cross_language_tests. The behavior — a job aborting once it exceeds its limit — is a GitHub Actions primitive.

API

No public API changes.

Documentation

  • doc-not-needed

The integration-test jobs set no timeout-minutes, so a hung test runs until
GitHub's six-hour job ceiling before it is killed. A recent run consumed six
hours on a stalled it-python job whose result was knowable in minutes.

Bound the three jobs that install and exercise a local Ollama server. The
limits are roughly two to three times each job's slowest observed run over
recent green builds on main: it-python 26.2m, cross-language 24.8m, and
it-java 14.0m.
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 10, 2026
@weiqingy

weiqingy commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

The 2 failing checks are unrelated to this change — this PR only adds timeout-minutes to three jobs, so it changes how fast a hang surfaces, not whether any test passes. The Ollama install succeeds in both jobs (>>> Install complete).

  • it-java [java-21] [flink-2.0]: the local llama-server crashes (io.github.ollama4j.exceptions.OllamaException: llama-server process has terminated in YamlLoaderIntegrationTest / ChatModelIntegrationTest). This is the model-server crash — intermittent on main too (this job passes 6/6 recent runs). Log.
  • it-python [python-3.11] [flink-1.20]: a py4j/Flink-1.20 classpath failure (org does not exist in the JVM, KeyError: 'p'), not Ollama-related. The failure count is identical to the same job on [ci] Retry Ollama install and propagate download failures #891 (which doesn't touch ci.yml), and the same job fails identically on current main (main run), so it's pre-existing and flaky. Log.

The llama-server segfault is exactly the intermittent hang this PR's timeouts are meant to bound (rather than fix) — the root-cause discussion is the open item on #889.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant