Fix routine-run status never reaching terminal (CL-6595, CL-6667) - #324
Merged
Conversation
CL-6595: a run whose event log arrives pre-combined in a single push (no per-event blobs) is invisible to the current newly-terminal scan. CL-6667: Mission Control's "this week" total must include today.
…gs (CL-6595) enumerateEventBlobs skips a run whose events already live in one combined events.jsonl instead of per-event <seq>.json blobs, so a run sealed from birth (its whole log, terminal event included, arriving in a single push) never surfaced to the newly-terminal scan and markTerminal never fired -- workflow_run.status stayed "running" forever despite the run having genuinely finished and delivered. validatePush now also walks the combined-run set and reports a newly-sealed run as terminal by reading its log's last event. hub-session-lookups.ts adds a same-push backfill via the new readCommittedWorkflowRunTerminalStatus export, so a run that still somehow slips past the primary detection self-heals on its next pack instead of every future reader hitting the same stale column.
"This week" summed over priorDays, which explicitly excludes today -- so it could read lower than "Runs today" on the same page, which is never legitimate since a week necessarily includes today.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
workflow_run.statusnever reaches a terminalvalue for a run whose entire event log arrives pre-combined in a single
push (
events.jsonlsealed from birth, no per-event blobs). Verifiedagainst the live
wb_verify_0822DB: 94 rows, allrunning/deployed,zero terminal, even for routines that had visibly delivered. Root cause:
enumerateEventBlobsskips a sealed run entirely, sovalidatePush'snewly-terminal scan never saw it and
markTerminalnever fired, eventhough the committed git log (read by
readCommittedWorkflowRunLifecycle)correctly reports the run terminal — which is exactly why the run's own
trace page said "finished" while the Routines list / detail / Mission
Control (all reading the stale DB column) stayed stuck on "Running now."
validatePushnow also scans the combined-run set for newly-sealed runs;hub-session-lookups.tsalso gets a same-push backfill via a newreadCommittedWorkflowRunTerminalStatusexport, so a run that stillsomehow slips past the primary detection self-heals on its next pack.
"This week" card summed over
priorDays, which explicitly excludestoday — so it could read lower than "Runs today" on the same page. Now
sums over all days including today.
live DB's
model_pricingtable is completely empty, so cost/tokens arenull->0 for every model regardless of run status. This is a pricing-data
seeding gap, unrelated to workflow_run/status. Left scoped for a
follow-up.
Known limitation
The 94 already-stuck live rows in
wb_verify_0822won't self-heal fromthis fix alone, since they're genuinely finished and no future pack will
ever arrive for them — the
hub-session-lookups.tsbackfill only fires ona live push. A one-time backfill script (or wiring
readCommittedWorkflowRunTerminalStatusinto the DB-facing resolversbehind Routines list/detail/Insights/Mission Control) is a reasonable
follow-up; flagging rather than stretching scope here.
Live proof
Not performed against the shared
:3000verify stack for this PR: thatstack (
.worktrees/cl-e2e-main) is a shared, already-running devenvironment and restarting it to pick up this branch's vendored changes
risked disrupting other concurrent work. Confidence instead comes from:
red-before/green-after unit tests reproducing the exact sealed-run
scenario against a real (non-mocked)
CommittedReadsfake, plus anindependent re-run of the full gate on this branch.
Test plan
WORKBENCH_CHECK_SINCE=origin/main bun run typecheck— exit 0WORKBENCH_CHECK_SINCE=origin/main bun run test— exit 0, all pass(new
workflow-run-kind.test.ts: sealed-run terminal detection,red without the fix / green with it; new mission-control test for
this-week >= today)
bun run lint— exit 0 (only pre-existing, unrelated warnings)bun run scripts/checks/killdates.ts— ok, hash updatedmerge; see note above