-
Notifications
You must be signed in to change notification settings - Fork 84
acceptance scenario 39 discards the server's output when healthz times out on CI #1237
Copy link
Copy link
Open
Labels
area/acceptanceEnd-to-end acceptance harness (scripts/acceptance/)End-to-end acceptance harness (scripts/acceptance/)effort/small< 1 day of focused work< 1 day of focused workgood first issueGood entry point for first-time contributorsGood entry point for first-time contributorshelp wantedMaintainers welcome external contributions hereMaintainers welcome external contributions herepriority/P3Low: nice-to-have; pick up when capacity allowsLow: nice-to-have; pick up when capacity allowstype/testTest coverage: add, fix, or strengthen testsTest coverage: add, fix, or strengthen tests
Description
Activity
Metadata
Metadata
Assignees
Labels
area/acceptanceEnd-to-end acceptance harness (scripts/acceptance/)End-to-end acceptance harness (scripts/acceptance/)effort/small< 1 day of focused work< 1 day of focused workgood first issueGood entry point for first-time contributorsGood entry point for first-time contributorshelp wantedMaintainers welcome external contributions hereMaintainers welcome external contributions herepriority/P3Low: nice-to-have; pick up when capacity allowsLow: nice-to-have; pick up when capacity allowstype/testTest coverage: add, fix, or strengthen testsTest coverage: add, fix, or strengthen tests
The defect
When acceptance scenario 39 times out waiting for the public-mode warren instance to become healthy on CI, the child process's stdout and stderr have already been discarded, so the failure leaves no evidence of why the server did not come up.
scripts/acceptance/lib/inproc.ts:169-177spawns the server withstdoutandstderrset to"ignore"unlessWARREN_ACCEPTANCE_WARREN_STDOUT=1orWARREN_ACCEPTANCE_WARREN_STDERR=1is exported..github/workflows/acceptance-public.yml:42-46sets neither.waitForHealthz(scripts/acceptance/lib/poll.ts:88-108) throwshealthz did not reach a terminal state within 30000ms (last state: ...)and that one line is all the CI log keeps.The same failure hit three times in the last eight days on
main(runs 33132730151, 33561435518, 33799958392), so the flake is real and the 30-second window with backoff introduced in PR #909 did not remove it. Without the child's output nobody can tell whether the process crashed, exited early, or bound a different port.Fix direction
Capture instead of discarding.
scripts/acceptance/lib/self-host-docker.ts:38-43already spawns withstdout: "pipe"andstderr: "pipe"and reads them withnew Response(proc.stdout).text()on exit. Copy that intoinproc.ts: pipe both streams into buffers, and whenwaitForHealthzthrows, append the buffered output (the last few hundred lines is enough) to theAcceptanceErrormessage before rethrowing. Keep the existingWARREN_ACCEPTANCE_WARREN_STDOUTandWARREN_ACCEPTANCE_WARREN_STDERRenv vars working as the live-streaming option.Scope
Out of scope: the root cause of the boot failure on CI runners. That needs the diagnostics this issue adds. Also out of scope: changing the timeout again.
Tests: no test pins
inproc.ts. Add a small unit test for the capture-and-attach path using a fake child that writes to stderr and exits non-zero.bun run acceptance:publicruns the scenario locally against a real boot.Getting started
AGENTS.mdcovers setup and conventions, and the "Acceptance harness" section describes scenario 39. Runbun run check:allbefore pushing. Warnings count as failures.Tracked internally as
warren-f074.Revised 2026-09-04. The original report asked to raise the 10-second healthz window and add backoff. PR #909 (
fe428898) did both: the window is 30 seconds andwaitForHealthzbacks off from 100ms to 1s. That bullet is retired. The surviving problem is the recurrence with no diagnostics, as described above.