Skip to content

acceptance scenario 39 discards the server's output when healthz times out on CI #1237

Description

@jayminwest

The defect

When acceptance scenario 39 times out waiting for the public-mode warren instance to become healthy on CI, the child process's stdout and stderr have already been discarded, so the failure leaves no evidence of why the server did not come up.

scripts/acceptance/lib/inproc.ts:169-177 spawns the server with stdout and stderr set to "ignore" unless WARREN_ACCEPTANCE_WARREN_STDOUT=1 or WARREN_ACCEPTANCE_WARREN_STDERR=1 is exported. .github/workflows/acceptance-public.yml:42-46 sets neither. waitForHealthz (scripts/acceptance/lib/poll.ts:88-108) throws healthz did not reach a terminal state within 30000ms (last state: ...) and that one line is all the CI log keeps.

The same failure hit three times in the last eight days on main (runs 33132730151, 33561435518, 33799958392), so the flake is real and the 30-second window with backoff introduced in PR #909 did not remove it. Without the child's output nobody can tell whether the process crashed, exited early, or bound a different port.

Fix direction

Capture instead of discarding. scripts/acceptance/lib/self-host-docker.ts:38-43 already spawns with stdout: "pipe" and stderr: "pipe" and reads them with new Response(proc.stdout).text() on exit. Copy that into inproc.ts: pipe both streams into buffers, and when waitForHealthz throws, append the buffered output (the last few hundred lines is enough) to the AcceptanceError message before rethrowing. Keep the existing WARREN_ACCEPTANCE_WARREN_STDOUT and WARREN_ACCEPTANCE_WARREN_STDERR env vars working as the live-streaming option.

Scope

Out of scope: the root cause of the boot failure on CI runners. That needs the diagnostics this issue adds. Also out of scope: changing the timeout again.

Tests: no test pins inproc.ts. Add a small unit test for the capture-and-attach path using a fake child that writes to stderr and exits non-zero. bun run acceptance:public runs the scenario locally against a real boot.

Getting started

AGENTS.md covers setup and conventions, and the "Acceptance harness" section describes scenario 39. Run bun run check:all before pushing. Warnings count as failures.

Tracked internally as warren-f074.

Revised 2026-09-04. The original report asked to raise the 10-second healthz window and add backoff. PR #909 (fe428898) did both: the window is 30 seconds and waitForHealthz backs off from 100ms to 1s. That bullet is retired. The surviving problem is the recurrence with no diagnostics, as described above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/acceptanceEnd-to-end acceptance harness (scripts/acceptance/)effort/small< 1 day of focused workgood first issueGood entry point for first-time contributorshelp wantedMaintainers welcome external contributions herepriority/P3Low: nice-to-have; pick up when capacity allowstype/testTest coverage: add, fix, or strengthen tests

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions