Quarantine permanently unrestorable workflow deployment records

[TheGreatAxios]

Summary

The boot-time restore loop replays every persisted deployment record serially, before hubLink.connect() — so a pile of dead records gates the hub handshake at every boot. On the owner's machine that meant 62-63 records replayed serially at every boot, most of them long dead. Root cause: a record that fails to restore is never marked, so a permanently broken one warns on every boot forever and nothing ever removes it.

This is scoped to fixing that accumulation at its root — not the restore loop's serial-vs-parallel question, which is PR #288's territory and stays untouched here.

What changed

• apps/sidecar/src/workflow-deployment-record.ts: the record now carries an optional restoreFailure marker (kind, attempts, reason, lastAttemptAt), plus WorkflowRestoreFailure (an error class the restore path throws to say which kind a failure is), RESTORE_QUARANTINE_THRESHOLD = 3, and three functions: recordWorkflowDeploymentRestoreFailure, clearWorkflowDeploymentRestoreFailure, isWorkflowDeploymentRestoreQuarantined.
• apps/sidecar/src/workflow-host-wiring/index.ts: the boot loop skips an already-quarantined record entirely (no attempt, no per-record warning — just a count toward one summary line), tracks a new failure's kind/reason on success or failure, and clears the marker the moment a restore succeeds.

The quarantine rule, and why N=3

After 3 consecutive permanent restore failures, a record stops being attempted and folds into one summary line (Skipped N quarantined workflow deployment record(s)...) instead of a per-record warning every boot. Permanent failures are deterministic — derived only from the record's own immutable closure pin and address, never from runtime environment — so they don't actually need repeated confirmation to be believed. 3 is a small buffer against a misclassification, while still bounding "warn forever" to a handful of boots. The record file itself is never deleted by quarantine — an operator still reclaims it by undeploying the address, same as before.

Permanent vs. transient — the part that has to be right

Only two restore-path gates are classified WorkflowRestoreFailure("permanent", ...), because they're the only two that are deterministic given solely the record's own persisted bytes:

1. Address-derivation mismatch — the stored agentAddress derives a different slug than its own directory. Corrupt or misplaced; re-deriving the same address on a later boot yields the identical mismatch every time.
2. Closure-derived definition validation failure — the pinned closure re-evaluates to a projection that fails WorkflowProjectionDefinition arktype validation, or fails validateWorkflowProjection's structural invariants. The closure is an immutable pin, so re-evaluating it later produces the identical broken shape every time.

Everything else is treated as transient — retried every boot for as long as the record exists, on a separate counter that can never trip the quarantine, however high it climbs:

• assertSourceBuildable failures (the explicitly-called-out case) — an unbuildable inference provider is this boot's adapter registry, not the record's content. A later boot with the provider registered can succeed with the exact same record, unchanged. This is now wrapped explicitly at its call site to mark it "transient".
• Closure-materialization failures (asset-store miss) and spawn-core failures fall through as plain, unwrapped Errors. The boot loop's default for anything that isn't an explicit WorkflowRestoreFailure is "transient" — the safe default, since misclassifying a recoverable failure as permanent stops future restore attempts for it, which is exactly the failure mode this PR is careful not to introduce.

The legacy-address prune (an ins_-prefixed address the current parser rejects outright) is untouched: it still deletes the record immediately, same as before — that's today's only GC path, extended rather than replaced.

Boot log, before vs. after

Before, a pile of dead records produced one warn line per record, every single boot, forever:

WRN Failed to restore workflow deployment dep_1: run_x@example.com derives slug ..., not its directory dep_1
WRN Failed to restore workflow deployment dep_2: run_y@example.com derives slug ..., not its directory dep_2
... (60 more, every boot, forever)

After, the first 3 boots look the same (still confirming it's really dead), the 3rd carries an explicit quarantine notice, and every boot after that collapses to one line:

# boots 1-2: same per-record warning as before
WRN Failed to restore workflow deployment dep_1: run_x@example.com derives slug ..., not its directory dep_1

# boot 3: explicit quarantine notice
WRN Workflow deployment dep_1 failed to restore 3 consecutive times and is now quarantined -- it will not be retried again until the address is undeployed. Last failure: ...

# boot 4 onward: one line covers the whole pile
WRN Skipped 62 quarantined workflow deployment record(s) (permanent restore failures, already reported); undeploy an address to clear its record

A transiently-unbuildable-provider record keeps warning every boot exactly as before — no change to that behavior, since it never quarantines.

Test plan

• apps/sidecar/src/workflow-deployment-record.test.ts — unit tests for recordWorkflowDeploymentRestoreFailure (counter start/increment/kind-reset), clearWorkflowDeploymentRestoreFailure, isWorkflowDeploymentRestoreQuarantined
• apps/sidecar/test/workflow-restore-quarantine.test.ts — integration test through the real router: a permanent failure (address/directory mismatch) quarantines after 3 boots and the attempt counter stops moving; a transient failure (unbuildable provider) is retried every boot for 5 boots straight and never quarantines
• apps/sidecar typecheck clean
• All 194 existing apps/sidecar tests still pass (no regressions)
• prettier --check clean on changed files, eslint clean on changed files
• Rebased onto latest origin/main (no overlap — nothing under apps/sidecar/ changed on main since this branch forked)