l1: default-route unresolvable S3 backend selectors instead of rejecting

[shreyas-blacksmith]

Why

The backend-selector trust interceptor enforced the forwarded (x-blacksmith-s3-endpoint, x-blacksmith-s3-bucket) pair fail-closed: anything not in the node's backends map got an InvalidArgument trust rejection. Two problems, both proven in production on 2026-09-03 (us-west legacy MinIO teardown):

1. Every config race is build-visible. A pin or host env value naming a backend the map no longer contains turned into per-RPC rejections — 450k+ in 24h — degraded to metered misses at best.
2. One stale host env value can knock a host off the ring entirely. The FA boot probe presents Doppler's MINIO_ENDPOINT as its routing identity; when that value left the map, probes failed and 134 hosts went boot-sticky s3_fallback, dialing the same dead endpoint directly — no working cache path at all.

The check never protected tenant isolation — that is the storage-prefix interceptor's job (unchanged, still fail-closed). The selector only ever picked which MinIO shard serves the request.

What

Resolution now never rejects, mirroring the actions cache's shard router (clientFor): placement values are hints, anything unresolvable falls back to a default.

• Selector missing / duplicate / not in the map → the map's designated default backend.
• Bucket missing / duplicate / not in the resolved entry's allowed set → that entry's default bucket.
• Resolved pairs behave exactly as before.

The L1 owns which backends exist (its config is the map, and the fork's per-backend circuit breakers already own backend health); the upstream needs no endpoint knowledge of its own to be served. This is what lets the FA agent stop presenting MINIO_ENDPOINT as probe identity (companion FA PR).

Observability

• New counter bazel_remote_s3_backend_selector_defaulted_total{reason} (reasons: missing/duplicate/unknown, bucket_missing/bucket_duplicate/bucket_unknown) plus a rate-limited journald line. Nonzero during a map rollout = FA/L1 drift; sustained nonzero = a namespace pin serving from the default shard it wasn't allocated on (cold reads until the pin and map are reconciled) — worth an alert.
• bazel_remote_s3_backend_selector_rejected_total is gone (nothing rejects anymore). Dashboards/alerts referencing it need to move to the new counter.

Compatibility

• RejectionReasonS3BackendSelector stays defined: L1s predating this change still mint it, and upstream grpcproxy clients must keep degrading it to a metered miss. Client-side degradation tests unchanged and passing.
• Single-backend deployments: no interceptor installed, no change.
• Config invariant (exactly one default: true entry) now also enforced at interceptor wiring via RoutingBackends(), same as s3proxy.NewMulti.

Testing

go build ./... and full go test ./... green, including rewritten server/grpc_s3_backend_test.go (every former rejection case now asserts which backend/bucket the request routes to) and the updated config routing-table test.

Made with Cursor

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)}

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}