Skip to content

Fix routine-run status never reaching terminal (CL-6595, CL-6667) - #324

Merged
TheGreatAxios merged 4 commits into
mainfrom
cl-6595-run-state
Aug 22, 2026
Merged

Fix routine-run status never reaching terminal (CL-6595, CL-6667)#324
TheGreatAxios merged 4 commits into
mainfrom
cl-6595-run-state

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

  • CL-6595 (root cause): workflow_run.status never reaches a terminal
    value for a run whose entire event log arrives pre-combined in a single
    push (events.jsonl sealed from birth, no per-event blobs). Verified
    against the live wb_verify_0822 DB: 94 rows, all running/deployed,
    zero terminal, even for routines that had visibly delivered. Root cause:
    enumerateEventBlobs skips a sealed run entirely, so validatePush's
    newly-terminal scan never saw it and markTerminal never fired, even
    though the committed git log (read by readCommittedWorkflowRunLifecycle)
    correctly reports the run terminal — which is exactly why the run's own
    trace page said "finished" while the Routines list / detail / Mission
    Control (all reading the stale DB column) stayed stuck on "Running now."
    validatePush now also scans the combined-run set for newly-sealed runs;
    hub-session-lookups.ts also gets a same-push backfill via a new
    readCommittedWorkflowRunTerminalStatus export, so a run that still
    somehow slips past the primary detection self-heals on its next pack.
  • CL-6667: independent bug, not the same root cause. Mission Control's
    "This week" card summed over priorDays, which explicitly excludes
    today — so it could read lower than "Runs today" on the same page. Now
    sums over all days including today.
  • CL-6659 (investigated, confirmed separate, not touched here): the
    live DB's model_pricing table is completely empty, so cost/tokens are
    null->0 for every model regardless of run status. This is a pricing-data
    seeding gap, unrelated to workflow_run/status. Left scoped for a
    follow-up.

Known limitation

The 94 already-stuck live rows in wb_verify_0822 won't self-heal from
this fix alone, since they're genuinely finished and no future pack will
ever arrive for them — the hub-session-lookups.ts backfill only fires on
a live push. A one-time backfill script (or wiring
readCommittedWorkflowRunTerminalStatus into the DB-facing resolvers
behind Routines list/detail/Insights/Mission Control) is a reasonable
follow-up; flagging rather than stretching scope here.

Live proof

Not performed against the shared :3000 verify stack for this PR: that
stack (.worktrees/cl-e2e-main) is a shared, already-running dev
environment and restarting it to pick up this branch's vendored changes
risked disrupting other concurrent work. Confidence instead comes from:
red-before/green-after unit tests reproducing the exact sealed-run
scenario against a real (non-mocked) CommittedReads fake, plus an
independent re-run of the full gate on this branch.

Test plan

  • WORKBENCH_CHECK_SINCE=origin/main bun run typecheck — exit 0
  • WORKBENCH_CHECK_SINCE=origin/main bun run test — exit 0, all pass
    (new workflow-run-kind.test.ts: sealed-run terminal detection,
    red without the fix / green with it; new mission-control test for
    this-week >= today)
  • bun run lint — exit 0 (only pre-existing, unrelated warnings)
  • bun run scripts/checks/killdates.ts — ok, hash updated
  • Live proof against the shared verify stack — recommended before
    merge; see note above

CL-6595: a run whose event log arrives pre-combined in a single push
(no per-event blobs) is invisible to the current newly-terminal scan.
CL-6667: Mission Control's "this week" total must include today.
…gs (CL-6595)

enumerateEventBlobs skips a run whose events already live in one
combined events.jsonl instead of per-event <seq>.json blobs, so a run
sealed from birth (its whole log, terminal event included, arriving in
a single push) never surfaced to the newly-terminal scan and
markTerminal never fired -- workflow_run.status stayed "running"
forever despite the run having genuinely finished and delivered.

validatePush now also walks the combined-run set and reports a
newly-sealed run as terminal by reading its log's last event.
hub-session-lookups.ts adds a same-push backfill via the new
readCommittedWorkflowRunTerminalStatus export, so a run that still
somehow slips past the primary detection self-heals on its next pack
instead of every future reader hitting the same stale column.
"This week" summed over priorDays, which explicitly excludes today --
so it could read lower than "Runs today" on the same page, which is
never legitimate since a week necessarily includes today.
@TheGreatAxios
TheGreatAxios merged commit 4c219a7 into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant