Concurrency is 1 (default) for all controllers: instrument before it becomes a ceiling

[nickmarden]

Context

No controller sets MaxConcurrentReconciles, so every reconciler runs with the controller-runtime default of 1. This was never an explicit decision. Analysis (following the #37 investigation) says the default is right for some controllers and a latent problem for others.

Per-controller assessment

• workloadwatcher (pod + profile): keep at 1. Reconciles are cheap post-fix: serve activeWorkloads recounts from a profile-ref pod index #37 (indexed lookups). Concurrency would widen transient miscount windows (concurrent reconciles of same-profile pods each carry only their own self override; status merge patches are last-writer-wins on the conditions array) in exchange for parallelism these queues don't need. Cross-object serialization also keeps the stamp-then-recount ordering easy to reason about.

• metricscollector: the real scaling ceiling. It is a polling loop in controller clothing: every profile self-requeues at the poll interval (default 60s), so single-worker capacity is poll_interval / avg_collection_time. Past the ceiling nothing errors and nothing is dropped; reconciles queue, effective sampling cadence stretches to N x work_duration, and percentiles are quietly computed from sparser samples than the policy assumes. Silent measurement degradation is the worst failure mode for a measurement product. Groundwork for raising it already exists: kubelet plugin caches are mutex-guarded (nodeCacheMu, podCacheMu, backoffMu) and go-redis is goroutine-safe. Watch for kubelet thundering herd at startup (the per-node summary cache blunts it).

• resourceadjuster: serialization is an accidental safety feature. Concurrency 1 means at most one in-place resize at a time, which is de facto blast-radius pacing. It exists only as an unstated side effect of a default; bumping concurrency later would silently delete the rate limit. Either document that serialization is intentional pacing or make pacing explicit (rate limiter) so the knob becomes safe to turn.

• killswitch: single object, irrelevant.

Proposed actions

1. Instrument first (ties into the SigNoz dashboards task). Controller-runtime already exports workqueue_depth, workqueue_queue_duration_seconds, and workqueue_work_duration_seconds per controller; put them on the dashboards.
2. Define the tripwire alert: collector work-duration sum per poll window approaching the poll interval. That is the signal that the ceiling is near.
3. When the tripwire fires: raise the collector's concurrency (and only the collector's) to a small bounded value (4 to 8), exposed as a flag / Helm value rather than a hardcode.
4. Document the adjuster's pacing as intentional (comment) or replace it with an explicit rate limit.

Non-goal

Do not raise concurrency as a performance fix for expensive reconciles. Concurrency multiplies per-reconcile cost; it does not fix it. With the pre-#37 unindexed lists, concurrency 4 would have pegged four cores instead of one.