Skip to content

Evict process-lifetime rate-limit and retry-hold maps in onboarding - #506

Merged
TheGreatAxios merged 9 commits into
mainfrom
cl-7233-evict-onboarding-maps
Aug 31, 2026
Merged

TheGreatAxios merged 9 commits into
mainfrom
cl-7233-evict-onboarding-maps

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

CL-7233: two in-memory maps in packages/onboarding grew for the life of the hub process with no eviction, plus an orphaned-retry-hold defect of the same class folded into this ticket.

  • lastProvisionByUser (routes.ts) — one entry per distinct user who ever attempted named provisioning. Fixed by adding a new @corbits/collections package with createExpiringMap<K,V>, a TTL-eviction map (get() drops an expired entry lazily on read; set() opportunistically sweeps every expired entry once per TTL window, no background timer). The rate-limit window (10s) is the correct TTL here, since an entry has no reason to outlive the window it gates — this also let the manual now - lastAttempt < window comparison collapse to a plain get() !== undefined check.
  • retryAfter/failureCount (bench-provisioning.ts) — consolidated into one holds map carrying userId/tenantId inline, and reclaimed once a bench's pending_seed row is confirmed gone (via store.read, not just absent from a given tick's listDue page) rather than only on convergence. A permanently-failing bench no longer leaves a stale backoff behind for a later, unrelated connect to the same user/tenant to inherit.

Why a new package

@corbits/collections exists so CL-7229 (orchestrator's postedApprovalIds/pendingDelegationThreads) and CL-7223 (folded-runs' crypto-provider cache) — the same bug class, called out explicitly in this ticket — have somewhere to land instead of each writing a third one-off bounded collection. apps/hub/src/launch-caches.ts's BoundedCache is a sibling primitive (size-capped LRU, no TTL, app-private) that predates this package; noting it here rather than touching it, since folding it in is out of scope for this ticket.

Eviction policy reasoning

  • Rate-limit map → TTL. The map's only invariant is "how recently was this key touched"; an entry past the rate-limit window is worthless. TTL = the window itself.
  • Retry-hold map → tie to the row's actual lifetime, not a TTL or LRU cap. An LRU cap risked evicting a hold for a bench that's still actively backed off, letting it get hammered again before its exponential backoff elapses. Confirming via store.read (authoritative regardless of listDue's paging) is slightly more expensive but provably correct rather than heuristic.

A bug found along the way (filed separately, not fixed here)

Greybeard's review of this approach surfaced that pending-seed.ts's listDue is not paginated across calls — a plain unordered LIMIT 50 re-scanned from the start of the table every tick, with no cursor. If outstanding pending_seed rows ever exceed 50, rows past that page are never drained until something ahead of them clears. This is a real, separate liveness bug, not something this ticket's outcome asked for — filed as CL-7252 rather than folded in here or dropped.

The store.read-based reconciliation in this PR is unaffected by that gap either way: it never trusts listDue's page as proof a row is gone, precisely because that page can't be trusted to be complete.

Test plan

  • cd packages/onboarding && bun run typecheck && bun test — 165 pass
  • cd packages/collections && bun run typecheck && bun test — 9 pass
  • bunx prettier --check on all touched files
  • bun run lint (root) — 0 errors
  • Greybeard-reviewed design; Critique-reviewed both commit pairs, clean on the second pass

Stacking

This is PR 1 of 2 for CL-7233/CL-7234. CL-7234 (routing onboarding's caught errors through reportError) branches off this branch's tip — do not merge out of order.

Do not merge — flag if CI is red.

@TheGreatAxios
TheGreatAxios force-pushed the cl-7233-evict-onboarding-maps branch 2 times, most recently from e00dcfc to 67e612c Compare August 30, 2026 21:45
CL-7233's onboarding maps need an eviction policy, and CL-7229/CL-7223
name the same "process-lifetime collection with no eviction" bug class
against the orchestrator and folded-runs' crypto cache. Scaffolds a new
@corbits/collections package (rather than burying this in onboarding)
so those sibling fixes have somewhere to land instead of writing a
third one-off map.
get() drops an expired entry lazily on read; set() opportunistically
sweeps every expired entry once per TTL window rather than checking
every key on every call, and there is no background timer to unref or
leak. A rate limiter or dedupe guard keyed by an ever-growing set of
distinct callers (a user id, say) can now bound its own memory to
recent activity instead of lifetime traffic.
lastProvisionByUser grew by one entry per distinct user who ever
attempted provisioning, for the life of the hub process, with no
eviction (CL-7233). The rate-limit window is the correct TTL for this
map: an entry has no reason to outlive the window it gates, so
createExpiringMap's own get() already implies the rate-limit check.
A bench that fails permanently accumulates a retryAfter/failureCount
hold that is only ever cleared on convergence — if its pending_seed row
disappears some other way (TTL-expiry, an admin action) first, the
hold outlives the row it was guarding and a later, unrelated connect
for the same user/tenant inherits a stale backoff (CL-7233).
Consolidates retryAfter/failureCount into one holds map that also
carries each hold's own userId/tenantId, and reclaims a hold once its
pending_seed row is confirmed gone rather than only on convergence
(CL-7233). A hold still present in the current tick's due batch is left
alone unconditionally; every other hold is checked directly against the
store (store.read is authoritative regardless of listDue's capped,
unordered page) before being evicted, so a row merely absent from this
tick's page is never mistaken for a row that no longer exists.
Notes that a permanently-failing bench's backoff state no longer
outlives its pending_seed row (CL-7233), so the doc doesn't drift from
the eviction behavior the previous two commits added.
routes.ts, bench-provisioning.ts, and plant-env-credentials.ts only
logged a caught failure; provision.ts already routes through
@corbits/error-sink's reportError for the operation/tenant context, a
refId, and secret redaction (CL-7234). Asserts on the reportError call
itself (operation, tenantId, extra) rather than only observable
response shape, and proves the two plant-env-credentials.ts catches
stop leaking a raw cause message once they're fixed.
routes.ts, bench-provisioning.ts, and plant-env-credentials.ts only
logged a caught failure and never reached @corbits/error-sink,
inconsistent with this same package's own provision.ts (CL-7234).

- routes.ts's reportOnboardingError funnel (every route catch) now
  also calls reportError, with a snake_case operation per route action
  (never the failure code itself, which is an unbounded per-error
  taxonomy) plus tenantId wherever a route already resolves one before
  it can fail, and userId in extra everywhere.
- bench-provisioning.ts's drain-level catch (a whole-tick failure, not
  scoped to any one bench — per-seed failures already carry tenant
  context via holdOff) now reports with no tenant/user, since none
  exists at that scope.
- plant-env-credentials.ts's two seedCatalog-throw catches now report
  with tenantId + provider, and — this module handles a live API key —
  the raw cause text logged and returned in PlantEnvProviderCredentialsOutcome
  now goes through the same sanitizeProviderMessage redaction the
  probe-failure path already applied, instead of logging the
  unredacted message next to a safe copy.
pruneOrphanedHolds needs the current tick's due batch to know which
holds are still on the page; drainOnce passed the undefined due
instead of page.seeds, breaking the typecheck.
@TheGreatAxios
TheGreatAxios force-pushed the cl-7233-evict-onboarding-maps branch from 67e612c to 1bda975 Compare August 31, 2026 02:17
@TheGreatAxios
TheGreatAxios merged commit b44e870 into main Aug 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant