GCP us-central1-b incident: entrypoint served global 429 "Rate exceeded." from 16:16 UTC 2026-09-01; no monitoring caught it

[MaxGhenis]

What happened

On 2026-09-01 ~16:16–16:31 UTC, the production Cloud Run entrypoint (policyengine-simulation-entry, project policyengine-simulation-entry) returned an instant 429 Rate exceeded. (Google Frontend, ~0.1s) for every request on every path (/health, /docs, /versions/*). App-v2 UK society-wide reports failed with a 500 wrapping the upstream 429 from GET /versions/policyengine (v1 API SimulationEntrypointClient.resolve_app_name).

Evidence gathered during the incident:

• https://simulation.api.policyengine.org/health → 429 Rate exceeded. (via the LB)
• https://policyengine-simulation-entry-4xy55an4yq-uc.a.run.app/health → 429 (direct stable URL, bypasses the LB/Cloud Armor) — so this was Cloud Run itself refusing admission, not an edge policy
• Tagged revision URL prod-10---… → 429 as well
• policyengine-simulation-entry-staging (same project/region) → 200 (5.5s cold start) — rules out project-wide quota/billing
• Modal gateway …-web-app.modal.run/health → 200 — backend healthy
• No deploy in the window (last: run 33270913487, 2026-08-29, same SHA as main)

An instant service-scoped 429 with that body is Cloud Run's admission-control refusal: all instances busy at max capacity with a full request queue (prod: --concurrency 80 --max-instances 20), i.e. either a large request burst or wedged/slow instances collapsing effective capacity. The service recovered on its own at ~16:31 UTC before any intervention reached prod (a re-deploy of the same SHA was dispatched as an instance refresh — run 33531984855 — and the recovery preceded its prod phase). A UK society-wide calculation (/uk/economy/82209/over/1?region=uk&time_period=2026) completed end-to-end right after.

Follow-ups this issue asks for

1. Forensics (needs GCP access): Cloud Run metrics/logs for policyengine-simulation-entry (us-central1) around 16:00–16:35 UTC — request rate, container instance count, request latencies, 429 count, and the request log by client identity/path just before onset. Was this a burst (from which caller?) or instances wedged behind a slow Modal upstream (25s proxy timeout × 80 concurrency can hold a lot of slots)?
2. Uptime monitoring: there is no Better Stack (or equivalent) monitor on https://simulation.api.policyengine.org/health — the v1 API host has one, this host does not; nobody was paged and the outage was found by a user (report page failure). Add a monitor.
3. Capacity review: whether --max-instances 20 × --concurrency 80 and the 25s upstream timeout compose safely when the Modal gateway is cold/slow; consider a lower per-request admission ceiling or faster shed for /versions/* (cacheable, tiny) so version resolution doesn't die with job traffic.

Related context: the July bot-wave incidents on the v1 API LB (docs/migration in policyengine-api, lb-cloud-armor-runbook.md) — but note Cloud Armor could not have caused this one (direct URL reproduced it), and the sim LB policy state should be checked separately if forensics point to a flood.

[MaxGhenis]

Root cause confirmed: Google Cloud incident, not anything on our side

GCP incident J5ia5t9p3g9Q5Wi7r8Ev — "Multiple products in us-central1-b are experiencing network service degradation", began 2026-09-01 14:44 UTC (07:41 PT), ongoing as of 16:37 UTC. Cloud Run is on the affected-products list. Google's 16:19 UTC update: network infrastructure recovered, product teams validating individual services.

Forensics from Cloud Logging (policyengine-simulation-entry project):

• Total request volume 14:00–16:40 UTC: 87 requests. There was no flood, no bot wave, no retry storm. Traffic was the normal trickle.
• First 429: 16:16:49 UTC (the failed UK report in app-v2 at 16:16 was among the first casualties). 429s continue through 16:36+, with one recovery window ~16:31–16:33 UTC during which a full UK society-wide calculation completed end-to-end.
• 10-minute buckets: 16:10 {429: 8} → 16:20 {200: 3, 429: 5} → 16:30 {200: 8, 202: 14, 429: 47} (the 16:30 bucket includes this run's verification traffic and probes).
• Both policyengine-simulation-entry and policyengine-simulation-entry-staging refuse with instant Rate exceeded. — Cloud Run cannot schedule instances for either service (min-instance unreachable/gone, new instance creation failing), consistent with the zonal network degradation. The v1 API project (policyengine-api, warm long-lived instances) is unaffected.
• Deploy run 33531984855's beta phase deployed successfully but failed verification at 16:33 UTC on the same instant 429s — expected during the incident window; no rollback needed (prod phase never ran).

No action needed on our side for recovery — this resolves when Google finishes validating Cloud Run in us-central1-b. Once probes go green, re-dispatch Deploy Simulation API to leave the pipeline qualified-green (the same-SHA refresh is harmless).

The durable takeaways stand:

1. Uptime monitor on https://simulation.api.policyengine.org/health — this outage ran 20+ minutes before a user found it via a failed report.
2. Retry/backoff + friendly error mapping for simulation entrypoint 429s; cache the versions map policyengine-api#3811 — bounded retry/backoff would have ridden out the flapping windows, and the raw httpx error string should not reach the app's report card.

[MaxGhenis]

Recovery confirmed: the entrypoint went green at ~16:58 UTC and has held (probed every 30s). A fresh, uncached UK society-wide calculation (/uk/economy/82209/over/1?region=uk&time_period=2027) completed end-to-end at 17:03 UTC. Outage window for the entrypoint: ~16:16–16:58 UTC (with a brief good window 16:31–16:33), inside GCP incident J5ia5t9p3g9Q5Wi7r8Ev. Re-dispatched the deploy pipeline for a green qualification run: https://github.com/PolicyEngine/policyengine-sim-api/actions/runs/33535236643 — will confirm promotion here. Remaining asks in this issue: the uptime monitor (item 2) and the capacity-composition review (item 3, now demoted to nice-to-have given root cause was zonal, not capacity).

[MaxGhenis]

Deploy run 33535236643 completed green end-to-end (beta + prod qualified, promoted) at ~17:40 UTC — the pipeline is back to a clean state and the entrypoint has stayed healthy since 16:58 UTC. Incident fully closed on our side; leaving this issue open for the uptime-monitor ask.