Work type
Bug / regression
Observed behavior
PR #1051 hosted test-race run 30583930460 failed TestMatrix_ProviderAPIKeyCapture at cmd/harnessd/main_test.go:4130: server at 127.0.0.1:43417 never became healthy within 3s. The same log then showed the server listening on 127.0.0.1:43417 roughly one second after the assertion failed. The branch production diff is confined to internal/harness/runner.go and cannot delay harnessd startup. Related timing-flake family: #958.
Expected behavior
The provider API-key capture matrix test must allow a contended race runner enough time to reach health, then still prove the exact captured API key and bounded graceful shutdown. A genuinely broken startup must continue to fail with the address and deadline.
Reproduction
Preconditions: Linux GitHub-hosted runner, full make test-race package fan-out. Exact command: make test-race. Actual result in run 30583930460: TestMatrix_ProviderAPIKeyCapture exhausted its 3-second awaitHealthy budget; harnessd began listening after the test had failed. Reproduction is contention-dependent; the hosted failure is the deterministic red artifact for this slice.
User and operational impact
The false negative blocks review and promotion of otherwise valid PRs. It does not alter production runtime or user data, and no security or persistence behavior is implicated. Local workaround is rerunning CI, which is not an acceptable merge gate.
Suspected seam and search evidence
Owning seam: cmd/harnessd/main_test.go TestMatrix_ProviderAPIKeyCapture -> awaitHealthy. Searches: rg -n "func awaitHealthy|awaitHealthy\(" cmd/harnessd/main_test.go and rg -n "3\time.Second|3 \ time.Second" cmd/harnessd/main_test.go. The shared runMatrixTest helper already uses a 10-second health budget for the same runWithSignals startup lifecycle. The log proves startup eventually succeeded, ruling out provider capture, config, listener, and production retention logic.
Blast-radius impact map
Callers and data flow: test-only provider-config capture and health polling. Config/env/defaults: fixture map only; no production env/default change. API/CLI/wire formats/tools: none. Persistence/schema/cache: none. Concurrency/lifecycle/cancellation/retries: only test startup scheduling tolerance; shutdown assertion unchanged. Security/auth/permissions/privacy: fake API key fixture only, no credential exposure. TUI/web/macOS/other clients: none. Provider/model/tool catalog: provider factory capture assertion unchanged. Deployment/observability/runbooks: hosted race reliability and exact timeout diagnostic only. Compatibility: no runtime artifact change. Existing tests/fixtures: TestMatrix_ProviderAPIKeyCapture and adjacent matrix tests. Documentation: plan, impact map, engineering/long-term logs, plans index.
Regression test first
Test file and test name: cmd/harnessd/main_test.go TestMatrix_ProviderAPIKeyCapture. Red command/artifact: make test-race in hosted run 30583930460. Expected failure: awaitHealthy exhausts 3 seconds before the contended server goroutine listens. This proves the fixture budget measures shared runner scheduling rather than the API-key capture invariant. False-positive controls: keep the health requirement, captured-key equality, and bounded graceful shutdown; change only this startup wait to the established 10-second matrix budget.
Fix boundaries
In scope: raise this fixtures health wait from 3 seconds to the established 10-second matrix-test budget, then stress it normal/race and run the complete regression gate. Out of scope: production startup, provider construction, listener allocation, all other deadlines, and broad #958 cleanup. Existing awaitHealthy abstraction remains; no parallel helper is needed.
Diagnostic and observability evidence
Red log: run 30583930460 job 91010705523 lines show the timeout at 21:35:31 and server listening at 21:35:32. Green evidence must preserve awaitHealthy failure diagnostics and record focused stress plus full-gate results. No secrets are included.
Verification plan
Use the hosted log as red evidence. Green: go test ./cmd/harnessd -run TestMatrix_ProviderAPIKeyCapture -count=100 and go test -race ./cmd/harnessd -run TestMatrix_ProviderAPIKeyCapture -count=100. Adjacent: go test ./cmd/harnessd -run TestMatrix_ -count=1 and race equivalent if practical. Required full: ./scripts/test-regression.sh in the foreground non-TTY environment required for macOS Keychain access. Push and recheck PR #1051 hosted checks.
Rollout and rollback
Test-only change; no deployed behavior, migration, or data repair. Revert if the expanded budget masks a deterministic startup failure or makes a failing focused test exceed the documented bound.
Documentation and handoff
Add issue-specific plan and impact map, update docs/plans/INDEX.md, engineering log, and long-term thinking log. Update PR #1051 to close this issue and report exact hosted/focused/full evidence.
Definition of done
Process acknowledgement
I will preserve the failing regression test, update the issue if the root cause changes, and record any newly discovered bug as its own issue.
Work type
Bug / regression
Observed behavior
PR #1051 hosted test-race run 30583930460 failed TestMatrix_ProviderAPIKeyCapture at cmd/harnessd/main_test.go:4130: server at 127.0.0.1:43417 never became healthy within 3s. The same log then showed the server listening on 127.0.0.1:43417 roughly one second after the assertion failed. The branch production diff is confined to internal/harness/runner.go and cannot delay harnessd startup. Related timing-flake family: #958.
Expected behavior
The provider API-key capture matrix test must allow a contended race runner enough time to reach health, then still prove the exact captured API key and bounded graceful shutdown. A genuinely broken startup must continue to fail with the address and deadline.
Reproduction
Preconditions: Linux GitHub-hosted runner, full make test-race package fan-out. Exact command: make test-race. Actual result in run 30583930460: TestMatrix_ProviderAPIKeyCapture exhausted its 3-second awaitHealthy budget; harnessd began listening after the test had failed. Reproduction is contention-dependent; the hosted failure is the deterministic red artifact for this slice.
User and operational impact
The false negative blocks review and promotion of otherwise valid PRs. It does not alter production runtime or user data, and no security or persistence behavior is implicated. Local workaround is rerunning CI, which is not an acceptable merge gate.
Suspected seam and search evidence
Owning seam: cmd/harnessd/main_test.go TestMatrix_ProviderAPIKeyCapture -> awaitHealthy. Searches: rg -n "func awaitHealthy|awaitHealthy\(" cmd/harnessd/main_test.go and rg -n "3\time.Second|3 \ time.Second" cmd/harnessd/main_test.go. The shared runMatrixTest helper already uses a 10-second health budget for the same runWithSignals startup lifecycle. The log proves startup eventually succeeded, ruling out provider capture, config, listener, and production retention logic.
Blast-radius impact map
Callers and data flow: test-only provider-config capture and health polling. Config/env/defaults: fixture map only; no production env/default change. API/CLI/wire formats/tools: none. Persistence/schema/cache: none. Concurrency/lifecycle/cancellation/retries: only test startup scheduling tolerance; shutdown assertion unchanged. Security/auth/permissions/privacy: fake API key fixture only, no credential exposure. TUI/web/macOS/other clients: none. Provider/model/tool catalog: provider factory capture assertion unchanged. Deployment/observability/runbooks: hosted race reliability and exact timeout diagnostic only. Compatibility: no runtime artifact change. Existing tests/fixtures: TestMatrix_ProviderAPIKeyCapture and adjacent matrix tests. Documentation: plan, impact map, engineering/long-term logs, plans index.
Regression test first
Test file and test name: cmd/harnessd/main_test.go TestMatrix_ProviderAPIKeyCapture. Red command/artifact: make test-race in hosted run 30583930460. Expected failure: awaitHealthy exhausts 3 seconds before the contended server goroutine listens. This proves the fixture budget measures shared runner scheduling rather than the API-key capture invariant. False-positive controls: keep the health requirement, captured-key equality, and bounded graceful shutdown; change only this startup wait to the established 10-second matrix budget.
Fix boundaries
In scope: raise this fixtures health wait from 3 seconds to the established 10-second matrix-test budget, then stress it normal/race and run the complete regression gate. Out of scope: production startup, provider construction, listener allocation, all other deadlines, and broad #958 cleanup. Existing awaitHealthy abstraction remains; no parallel helper is needed.
Diagnostic and observability evidence
Red log: run 30583930460 job 91010705523 lines show the timeout at 21:35:31 and server listening at 21:35:32. Green evidence must preserve awaitHealthy failure diagnostics and record focused stress plus full-gate results. No secrets are included.
Verification plan
Use the hosted log as red evidence. Green: go test ./cmd/harnessd -run TestMatrix_ProviderAPIKeyCapture -count=100 and go test -race ./cmd/harnessd -run TestMatrix_ProviderAPIKeyCapture -count=100. Adjacent: go test ./cmd/harnessd -run TestMatrix_ -count=1 and race equivalent if practical. Required full: ./scripts/test-regression.sh in the foreground non-TTY environment required for macOS Keychain access. Push and recheck PR #1051 hosted checks.
Rollout and rollback
Test-only change; no deployed behavior, migration, or data repair. Revert if the expanded budget masks a deterministic startup failure or makes a failing focused test exceed the documented bound.
Documentation and handoff
Add issue-specific plan and impact map, update docs/plans/INDEX.md, engineering log, and long-term thinking log. Update PR #1051 to close this issue and report exact hosted/focused/full evidence.
Definition of done
Process acknowledgement
I will preserve the failing regression test, update the issue if the root cause changes, and record any newly discovered bug as its own issue.