merge-steward should make in-progress reconcile ticks transparent and self-healing

[krasnoperov]

Problem

During the LSR-656 / subtitles PR #1147 merge, merge-steward looked stuck in validating even though GitHub Actions had already completed successfully. The operator view made the queue look wedged and led to a manual merge-steward service restart, after which the PR merged normally.

After investigating the logs and code, this was not a red-CI or PR-content problem. It was a status/transparency problem around an active reconcile tick, with a related reliability gap if a tick actually hangs.

Incident timeline

All UTC unless noted.

• 2026-05-22 07:11:04 — PR #1147 enqueued for krasnoperov/subtitles.
• 2026-05-22 07:11:09 — promoted.
• 2026-05-22 07:11:45 — spec ready, CI SHA 49324934..., entry moved to validating.
• 2026-05-22 07:14:37 — GitHub Actions run for spec SHA 493249340a54a7d4b8d619a46751df83557965e6 completed successfully.
• Around 2026-05-22 07:14:37 — merge-steward pr status --repo subtitles --pr 1147 --json still reported kind: validating, status: validating, checkedAt: 2026-05-22T07:14:37.145Z.
• 2026-05-22 07:14:54 — a reconcile tick started.
• 2026-05-22 07:14:56 — service log emitted Queue: ci_passed PR #1147.
• merge-steward queue reconcile --repo subtitles --json returned started: false because tickInProgress: true, with lastTickStartedAt: 2026-05-22T07:14:54.449Z and lastTickCompletedAt: 2026-05-22T07:14:24.449Z.
• 2026-05-22 07:15:04 — operator restarted the service, believing the queue was wedged.
• 2026-05-22 07:15:39 — restarted service emitted merge_revalidating.
• 2026-05-22 07:15:43 — PR #1147 merged; post-merge checks passed.

What made this confusing

The queue was probably not truly stuck at the time of restart. It was inside a normal reconcile tick shortly after CI passed. But the CLI and status surfaces did not make that obvious:

• pr status still showed the entry as validating until the tick committed the state change.
• queue reconcile returned started: false, which is technically correct when a tick is already running, but it did not say "a tick is already running; it started N seconds ago; last event was ci_passed".
• queue status exposes tickInProgress, lastTickStartedAt, and lastTickCompletedAt, but the meaning is easy to misread during a race between GitHub check completion and the next poll tick.
• There is no visible watchdog / stale-tick warning in merge-steward. If a reconcile tick really hangs, operators currently discover it by noticing an old lastTickStartedAt and restarting the service.

Code pointers

• packages/merge-steward/src/service-runtime.ts
  • runTick() sets tickInProgress = true, records lastTickStartedAt, then awaits beforeTick and reconcile.
  • triggerReconcile() returns started: false whenever tickInProgress is true, but does not include tick age, current/last action, or guidance.
  • There is no bounded reconcile timeout/watchdog here.
• packages/merge-steward/src/reconciler-validate.ts
  • checkValidation() emits ci_passed and transitions the head entry from validating to merging when CI passes.
• packages/merge-steward/src/reconciler.ts
  • After processing active entries, each tick also calls verifyMergedEntriesPostPush(ctx), which can add more work after an entry emits ci_passed.
• Compare PatchRelay's own ServiceRuntime tests:
  • test/service-runtime.test.ts includes service runtime recovers after a background reconciliation timeout, but merge-steward's runtime does not have an equivalent guard.

Requested behavior

Make this impossible to misread, and make true stuck ticks self-healing.

Suggested acceptance criteria:

1. queue status and pr status expose active tick age and current/last reconcile action when tickInProgress is true.
2. queue reconcile --json returns started: false with an explicit reason such as already_running, plus tickAgeMs, lastTickStartedAt, lastTickCompletedAt, and the latest queue event for the head entry.
3. The human CLI output should say something like: reconcile already running for 12s; head PR #1147 last event ci_passed; wait for the next tick before restarting.
4. Add a stale tick threshold. If tickInProgress exceeds the threshold, surface a warning in queue status, service status, and logs.
5. Add a watchdog or timeout strategy for merge-steward reconcile ticks. It should either abort safely via AbortSignal plumbing or mark the runtime failed/stale and allow a new tick/restart path without requiring the operator to infer the problem.
6. Add tests covering:
   • triggerReconcile() while a tick is already running
   • status output during an active tick
   • stale tick warning after threshold
   • recovery/timeout behavior for a hanging reconcile pass

Why this matters

PatchRelay and merge-steward are meant to support many parallel delegated branches. When the queue is healthy but briefly between CI completion and reconciliation, the operator should see that clearly. When it is genuinely stuck, the system should say so directly and recover or give a precise action.