Symptoms
- Requests succeed at low concurrency and fail in bursts.
- Retrying immediately creates more 429 responses.
Likely causes
- Client concurrency exceeds a TokenReply, route, or upstream window.
- A free daily allowance or shared upstream capacity is exhausted.
How to diagnose
- Measure request rate and concurrency by model and key.
- Check response details and status before treating every 429 as the same quota.
- Run one request after the relevant window instead of a retry storm.
Fixes
- Use exponential backoff with jitter and a maximum attempt count.
- Reduce concurrency or queue requests.
- Switch to another eligible route only when fallback semantics are acceptable.
How to verify
A controlled low-rate request succeeds and the client stops retrying after its configured cap.