Small queries should avoid K8s worker cold-start latency

[bill-ph]

Context

docs/design/worker-ttl-pool.md documents the current remote/K8s worker TTL model and explicitly leaves per-org reserved minimum capacity as future work. Today, small/default queries benefit from hot-idle reuse only when a compatible worker already exists. After an idle gap, the first small query still has to wait for a new worker pod to become query-ready.

Why this matters

In testing, spawning a worker on an existing node still took roughly 3-5 seconds before the query could run. That delay is not only node provisioning; it includes pod creation/startup, worker process startup, DuckDB extension/bootstrap/warmup, control-plane health-check confirmation, and tenant activation.

That latency is acceptable for cold or large batch work, but it is a poor fit for small/interactive queries where users expect the query path to feel already warm. The current TTL model improves repeated queries, but it does not protect the first small query after all compatible hot-idle workers have expired.

Desired outcome

Small/default query traffic should have an operator-controlled way to avoid paying cold K8s worker startup latency after ordinary idle gaps, while preserving the existing remote-worker safety properties: one query session per worker, tenant isolation, version-aware reuse, and predictable per-query resources.

Related

• docs/design/worker-ttl-pool.md
• Remote/K8s spawn-on-demand + hot-idle worker model

[bill-ph]

Implemented design

This ended up as a narrower retention-floor implementation rather than a proactive warm-pool reconciler. See PR #764.

Add default_worker_min_hot_idle as an org-level setting. The default is 0, which disables the behavior. When set, it tells the control plane to retain up to that many compatible default-profile workers in hot_idle instead of retiring them at TTL.

What the floor means

The implemented floor is a retention floor, not a spawn target.

It only preserves workers that were naturally created by prior traffic and later returned to hot_idle. It does not proactively create workers for cold orgs, and it does not asynchronously replenish the floor after a retained worker is claimed.

So the behavior is:

• first request for a cold org still spawns on demand;
• after the session ends, the worker can enter hot_idle;
• once its TTL expires, janitor checks whether retiring it would drop the compatible hot-idle count at or below default_worker_min_hot_idle;
• if yes, janitor skips TTL retirement and keeps the worker available for future same-shape traffic.

This keeps the change small and avoids bringing back the old shared/neutral warm-pool model.

Compatibility

A worker only counts toward the floor when it matches the org's current default worker shape:

• same org;
• same target worker image/version;
• same default worker CPU/memory profile.

Hot-idle acquisition is also exact image/profile compatible. Old-image or wrong-profile hot-idle workers are not reused for new requests and are not protected by the floor.

Arbitrary per-profile reserved capacity remains future work. The implemented setting only covers the org default profile.

Admission and caps

The implementation keeps worker caps simple:

• incompatible hot_idle workers do not count against the org cap for admitting a replacement spawn;
• compatible hot_idle workers do count for org admission, because they are already usable capacity for that request;
• all active worker pods, including hot_idle, still count against the global cap.

There is no inline retire-and-replace path. If global capacity is full because stale/incompatible hot-idle pods still exist, admission returns capacity exhausted and normal cleanup is left to janitor/version reaper/TTL behavior.

This preserves global cap as a hard pod-count cap instead of treating incompatible hot-idle pods as invisible capacity.

Lifecycle

The main lifecycle remains unchanged:

idle -> reserved -> activating -> hot -> hot_idle -> retired

The only new lifecycle behavior is inside janitor's hot-idle TTL cleanup: before retiring an expired hot-idle worker, janitor asks whether that worker is protected by the configured default-profile floor.

Observability

This implementation does not add a separate floor reconciler, so there are no floor-deficit or replenishment metrics. The useful signals are:

• existing acquire path metrics for cold spawn versus hot-idle claim;
• worker lifecycle/state metrics;
• janitor debug logs when TTL retirement is skipped because a worker is protected by the floor.

Rollout

Start disabled by default. Enable per org by setting default_worker_min_hot_idle to a small value, typically 1, for orgs where default-profile cold-start latency matters and the extra idle pod cost is acceptable.