Skip to content

test: add Linux GUI end-to-end coverage#7

Open
cm2435-hcomp wants to merge 16 commits into
feat/linux-runtime-0.1.9from
charlie/linux-gui-e2e
Open

test: add Linux GUI end-to-end coverage#7
cm2435-hcomp wants to merge 16 commits into
feat/linux-runtime-0.1.9from
charlie/linux-gui-e2e

Conversation

@cm2435-hcomp

@cm2435-hcomp cm2435-hcomp commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds first-class Linux GUI end-to-end coverage for HoloDesktop CLI, stacked on #6.

  • provisions a deterministic Ubuntu 22.04 X11 desktop with Xvfb, Openbox, Picom, D-Bus, and AT-SPI
  • maps the existing eight stable task contracts to Mousepad, Thunar, KCalc, and Google Chrome
  • adds independent KCalc accessibility and opened-file window evaluators
  • adds Linux to live smoke and the four-shard full E2E workflow
  • bounds full-suite model concurrency to two sessions and adds a platform-scoped manual qualification input (all remains the default)
  • extends reporting, contract tests, coverage documentation, and local reproduction guidance

Why

#6 publishes and installs the managed Linux x86_64 runtime, but the existing live suite only proves desktop control on macOS and Windows. This PR closes that confidence gap by exercising the shipped runtime inside a real rendered Linux GUI session and checking externally observable final state.

Stack

  • Base: feat/linux-runtime-0.1.9
  • Base SHA when branched: 8955498cec62c34912d757909071fa9a1884e05e
  • Head: 8ef65190e1f941cd9ff6aa9dfc7de3b5a34238b5
  • Runtime: hai-agent-runtime 0.1.9 Linux x86_64

Local validation

  • uv run ruff format --check .
  • uv run ruff check .
  • uv run mypy src/
  • uv run pytest -q — 428 passed, 14 skipped
  • managed Linux runtime CDN artifact/SHA network test
  • workflow/action YAML parsing and duplicate-key validation
  • composite-action Bash syntax validation
  • git diff --check

Hosted validation

Final-head required checks are green:

The hosted work also found and fixed real Linux workflow defects before this head: missing Xauthority initialization, the missing gnome-screenshot backend required by PyAutoGUI, an Openbox readiness race, cross-drive Windows artifact upload, and KCalc's misleading Qt STATE_SHOWING value. The corrected KCalc evaluator has hosted evidence reading 2 + 2 = 4 from the AT-SPI text interface and extracting result 4.

Linux full-suite qualification runs on the final head:

  1. Normal mode — matrix/setup/report/artifact paths worked; three second tasks passed in 8–15 seconds, while unrelated first turns simultaneously produced empty event logs and hit 240-second timeouts.
  2. Fast mode — exposed an upstream runtime packaging defect: runtime 0.1.9 does not contain Hydra config holo-desktop-holo-3-1-35b-visual-multi-fast; tasks fail before CUA starts.
  3. Normal mode — reproduced the model-service no-event timeout pattern; KCalc completed and independently evaluated 4 in 26 seconds, while other turns stalled at the model boundary.

Artifacts are preserved on every pass/failure and aggregate reports fail honestly when a task is red. The Linux-only dispatch created exactly four Ubuntu shards, with two admitted concurrently and no macOS/Windows jobs.

Promotion status

Keep this PR draft. Linux task mappings remain release-candidate coverage until all eight full-suite tasks pass in three independent Ubuntu workflow runs. That 3/3 bar is not met today because of the model-service no-event timeouts above. Do not weaken evaluators or add workflow retries to hide that signal. Fast qualification is additionally blocked on the missing runtime config in 0.1.9.

@cm2435-hcomp

Copy link
Copy Markdown
Collaborator Author

Qualification timeout diagnosis

The no-event timeout is upstream of CUA and is not Linux-specific. A temporary DEBUG run reproduced the same sequence on macOS and Linux:

  • the vLLM adapter starts the holo3-1-35b-a3b chat request;
  • the POST connects and sends successfully, then waits 120 seconds for response headers;
  • httpx.TimeoutException fires;
  • OpenAI Python 2.24.0 handles that internally and retries (OpenAI(...).max_retries == 2);
  • the E2E task deadline stops the runtime while SDK attempt two is again waiting for response headers.

Evidence: debug smoke run. The first request timed out at 11:29:07.202 on macOS and 11:29:37.039 on Linux; each SDK retry began within 0.5 seconds. Windows happened to receive a response and passed.

This explains the previously empty event traces and absent HAI retry warnings: the model call has not returned to VLLMChatProvider._completion_with_retries before the harness deadline. HAI then has a second, outer retry policy (max_retries_on_exception=8), so retry ownership and the total deadline are currently misaligned.

This PR now preserves runtime logs on timeout and across attempts, using graceful shutdown first. The temporary DEBUG override was removed after diagnosis because it records full model payloads/screenshots and creates oversized artifacts.

The production fix belongs in HAI/runtime: make one layer own retries (disable OpenAI SDK retries for the wrapped vLLM client) and enforce a total retry deadline compatible with the agent/task budget. Full-suite 3/3 qualification should resume after that runtime fix is released and pinned.

@cm2435-hcomp
cm2435-hcomp marked this pull request as ready for review July 16, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant