Evict process-lifetime rate-limit and retry-hold maps in onboarding - #506
Merged
Merged
Conversation
5 tasks
TheGreatAxios
force-pushed
the
cl-7233-evict-onboarding-maps
branch
2 times, most recently
from
August 30, 2026 21:45
e00dcfc to
67e612c
Compare
CL-7233's onboarding maps need an eviction policy, and CL-7229/CL-7223 name the same "process-lifetime collection with no eviction" bug class against the orchestrator and folded-runs' crypto cache. Scaffolds a new @corbits/collections package (rather than burying this in onboarding) so those sibling fixes have somewhere to land instead of writing a third one-off map.
get() drops an expired entry lazily on read; set() opportunistically sweeps every expired entry once per TTL window rather than checking every key on every call, and there is no background timer to unref or leak. A rate limiter or dedupe guard keyed by an ever-growing set of distinct callers (a user id, say) can now bound its own memory to recent activity instead of lifetime traffic.
lastProvisionByUser grew by one entry per distinct user who ever attempted provisioning, for the life of the hub process, with no eviction (CL-7233). The rate-limit window is the correct TTL for this map: an entry has no reason to outlive the window it gates, so createExpiringMap's own get() already implies the rate-limit check.
A bench that fails permanently accumulates a retryAfter/failureCount hold that is only ever cleared on convergence — if its pending_seed row disappears some other way (TTL-expiry, an admin action) first, the hold outlives the row it was guarding and a later, unrelated connect for the same user/tenant inherits a stale backoff (CL-7233).
Consolidates retryAfter/failureCount into one holds map that also carries each hold's own userId/tenantId, and reclaims a hold once its pending_seed row is confirmed gone rather than only on convergence (CL-7233). A hold still present in the current tick's due batch is left alone unconditionally; every other hold is checked directly against the store (store.read is authoritative regardless of listDue's capped, unordered page) before being evicted, so a row merely absent from this tick's page is never mistaken for a row that no longer exists.
Notes that a permanently-failing bench's backoff state no longer outlives its pending_seed row (CL-7233), so the doc doesn't drift from the eviction behavior the previous two commits added.
routes.ts, bench-provisioning.ts, and plant-env-credentials.ts only logged a caught failure; provision.ts already routes through @corbits/error-sink's reportError for the operation/tenant context, a refId, and secret redaction (CL-7234). Asserts on the reportError call itself (operation, tenantId, extra) rather than only observable response shape, and proves the two plant-env-credentials.ts catches stop leaking a raw cause message once they're fixed.
routes.ts, bench-provisioning.ts, and plant-env-credentials.ts only logged a caught failure and never reached @corbits/error-sink, inconsistent with this same package's own provision.ts (CL-7234). - routes.ts's reportOnboardingError funnel (every route catch) now also calls reportError, with a snake_case operation per route action (never the failure code itself, which is an unbounded per-error taxonomy) plus tenantId wherever a route already resolves one before it can fail, and userId in extra everywhere. - bench-provisioning.ts's drain-level catch (a whole-tick failure, not scoped to any one bench — per-seed failures already carry tenant context via holdOff) now reports with no tenant/user, since none exists at that scope. - plant-env-credentials.ts's two seedCatalog-throw catches now report with tenantId + provider, and — this module handles a live API key — the raw cause text logged and returned in PlantEnvProviderCredentialsOutcome now goes through the same sanitizeProviderMessage redaction the probe-failure path already applied, instead of logging the unredacted message next to a safe copy.
pruneOrphanedHolds needs the current tick's due batch to know which holds are still on the page; drainOnce passed the undefined due instead of page.seeds, breaking the typecheck.
TheGreatAxios
force-pushed
the
cl-7233-evict-onboarding-maps
branch
from
August 31, 2026 02:17
67e612c to
1bda975
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CL-7233: two in-memory maps in
packages/onboardinggrew for the life of the hub process with no eviction, plus an orphaned-retry-hold defect of the same class folded into this ticket.lastProvisionByUser(routes.ts) — one entry per distinct user who ever attempted named provisioning. Fixed by adding a new@corbits/collectionspackage withcreateExpiringMap<K,V>, a TTL-eviction map (get()drops an expired entry lazily on read;set()opportunistically sweeps every expired entry once per TTL window, no background timer). The rate-limit window (10s) is the correct TTL here, since an entry has no reason to outlive the window it gates — this also let the manualnow - lastAttempt < windowcomparison collapse to a plainget() !== undefinedcheck.retryAfter/failureCount(bench-provisioning.ts) — consolidated into oneholdsmap carryinguserId/tenantIdinline, and reclaimed once a bench'spending_seedrow is confirmed gone (viastore.read, not just absent from a given tick'slistDuepage) rather than only on convergence. A permanently-failing bench no longer leaves a stale backoff behind for a later, unrelated connect to the same user/tenant to inherit.Why a new package
@corbits/collectionsexists so CL-7229 (orchestrator'spostedApprovalIds/pendingDelegationThreads) and CL-7223 (folded-runs' crypto-provider cache) — the same bug class, called out explicitly in this ticket — have somewhere to land instead of each writing a third one-off bounded collection.apps/hub/src/launch-caches.ts'sBoundedCacheis a sibling primitive (size-capped LRU, no TTL, app-private) that predates this package; noting it here rather than touching it, since folding it in is out of scope for this ticket.Eviction policy reasoning
store.read(authoritative regardless oflistDue's paging) is slightly more expensive but provably correct rather than heuristic.A bug found along the way (filed separately, not fixed here)
Greybeard's review of this approach surfaced that
pending-seed.ts'slistDueis not paginated across calls — a plain unorderedLIMIT 50re-scanned from the start of the table every tick, with no cursor. If outstandingpending_seedrows ever exceed 50, rows past that page are never drained until something ahead of them clears. This is a real, separate liveness bug, not something this ticket's outcome asked for — filed as CL-7252 rather than folded in here or dropped.The
store.read-based reconciliation in this PR is unaffected by that gap either way: it never trustslistDue's page as proof a row is gone, precisely because that page can't be trusted to be complete.Test plan
cd packages/onboarding && bun run typecheck && bun test— 165 passcd packages/collections && bun run typecheck && bun test— 9 passbunx prettier --checkon all touched filesbun run lint(root) — 0 errorsStacking
This is PR 1 of 2 for CL-7233/CL-7234. CL-7234 (routing onboarding's caught errors through
reportError) branches off this branch's tip — do not merge out of order.Do not merge — flag if CI is red.