Skip to content

fix(gateway-session): retry a transient mint failure instead of ending the run - #1261

Draft
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixwizard-retry-a-transient-503-from-d1e157
Draft

posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixwizard-retry-a-transient-503-from-d1e157

Conversation

@posthog

@posthog posthog Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Problem

  • A wizard run dies before it starts any work when the gateway token mint answers a transient HTTP 503. The only workaround is to run the whole command again.
  • mintGatewayToken classified only 400, 401, 403, 404 and 429 as a refusal it can explain. Every other status fell through to a terminal GatewayMintFailed — no retry, no body read, no Retry-After.
  • A 503 there is routine, not an outage: the app view answers one for any gateway transport failure, and the gateway itself answers one while a pod drains during a normal deploy or while its model catalog is briefly unreadable. So a deploy window can strand any run that mints inside it.
  • The thrown message carried the bare status, so nobody could tell draining apart from a real outage.

Changes

  • 5xx and 408 on the mint are now transient. The mint re-asks up to three times with a doubling backoff (500ms, 1s), honours Retry-After in both its seconds and HTTP-date forms, and stops as soon as the remaining budget cannot hold another attempt.
  • The 20s mint budget is now shared across attempts, not spent per attempt, so the worst-case wall clock does not grow.
  • A refusal still ends the attempts. A 429 is the daily run limit; re-asking would only spend the mint it was counting.
  • The failure message carries the server's reason (outcome and detail) and the attempt count, so the next 5xx is diagnosable from the message alone.
  • A retry captures gateway mint retried, next to the existing gateway mint refused. Without it a mint that recovers leaves no trace, so there is no way to measure the fix.
Mint answer Before After
400 / 401 / 403 / 404 / 429 refusal, run ends unchanged
5xx, 408 run ends on the first one up to 3 attempts inside the 20s budget
transport error run ends unchanged

Test plan

  • src/lib/__tests__/gateway-session.test.ts covers: recovery on the second attempt, the bounded give-up at three attempts for 500/502/503/408, a refusal never re-asked, Retry-After in both forms, the budget refusing an absurd Retry-After, the server reason in the message, and the gateway mint retried capture.
  • All 10 new or changed assertions fail with the production file stashed.
  • Full suite green: 2831 tests, 175 files. tsc --noEmit reports nothing new in the changed files.
Agent context
  • The inbox report named posthog/ai-gateway as the repository, but src/lib/gateway-session.ts lives here. The gateway repo only holds the two sources of the 503 the CLI reacts to.
  • Considered and rejected: reusing fetchWithRetry from src/lib/fetch-retry.ts. It takes no request init (so no POST body or auth header), treats 429 as retryable where the mint treats it as a terminal refusal, has no Retry-After awareness, and times out per attempt rather than against one shared deadline.
  • Also rejected: extracting a shared Retry-After parser and backoff primitive. This is the third place in the repo to want one (fetch-retry.ts and src/lib/task-stream/destinations/posthog.ts are the others), but the extraction would change two modules outside this fix, one of them on the skill-download critical path. Worth its own change.

Created with PostHog Desktop from this inbox report.

The mint treated every status outside its refusal set as terminal, so one HTTP
503 ended the run before any work started. A 503 there is routine: the app
answers one for any gateway transport failure, and the gateway answers one while
a pod drains during a normal deploy.

5xx and 408 are now transient. The mint re-asks up to three times with a
doubling backoff, honours Retry-After, and shares the one 20s mint budget, so
the worst-case wall clock does not grow. A refusal still ends the attempts.

The thrown message now carries the server's own reason, and a retry captures
`gateway mint retried`, so a recovered mint is measurable against the failures
that still end a run.

Generated-By: PostHog Desktop
Task-Id: 8968884d-aaba-4285-8ccd-ccb074764953
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant