Skip to content

Harden Cost Management export retries for HTTP 429 #2257

Description

@MSBrett

🐛 Problem

Start-FinOpsCostExport can fail instead of retrying when the Cost Management Run API returns HTTP 429. Invoke-Rest emits Write-Error for the response before the caller can reliably inspect Throttled, and the loop condition does not repeat a single-export run even when throttling is returned non-terminatingly.

The current unit test mocks Invoke-Rest returning a throttled response directly, so it does not cover the terminating error behavior observed in CI.

👣 Repro steps

  1. Run New-FinOpsCostExport -Execute or Start-FinOpsCostExport in a context where PowerShell treats Write-Error as terminating.
  2. Have the Cost Management export Run API return HTTP 429 with a Retry-After value.
  3. Observe that the command exits with an error instead of waiting and retrying.

🤔 Expected

  • HTTP 429 responses are returned to Start-FinOpsCostExport in a form it can handle.
  • The command honors the server-provided Retry-After value, with a safe fallback when absent or invalid.
  • Both single-export and multi-month runs retry the same request after throttling.
  • Retries are bounded to prevent an infinite loop.
  • Unit tests reproduce the real Invoke-Rest/Write-Error behavior and cover single-export, multi-month, retry-delay, retry-limit, and eventual-success cases.

📷 Screenshots

N/A

🔧 Environment

ℹ️ Additional context

Observed in the Hubs (manual) check for PR #2187: deployment completed, then the initial export run failed with Too many requests. Please retry after 60 seconds. (Code: 429).

Failed run: https://github.com/microsoft/finops-toolkit/actions/runs/31678558903

The two immediately preceding PR deployment runs succeeded, indicating the service-side 429 was transient while the retry-handling failure is deterministic.

🙋‍♀️ Ask for the community

We could use your help:

  1. Please vote this issue up (👍) to prioritize it.
  2. Share other Cost Management throttling responses, especially variations in status codes or retry headers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Tool: PowerShellPowerShell scripts and automation

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions