Symptoms
- Long requests fail near a repeatable duration.
- A shorter prompt or non-streaming smoke test completes.
Likely causes
- The client timeout is shorter than model latency.
- A reverse proxy or upstream closed an idle or long-running connection.
How to diagnose
- Measure which layer closes the connection and at what elapsed time.
- Compare connect, first-byte, and total response timing.
- Test a small request with one deliberate timeout value.
Fixes
- Set bounded timeouts appropriate to the operation and route.
- Use streaming when supported so intermediaries see response activity.
- Retry only idempotent or safely deduplicated operations.
How to verify
The request completes within the chosen deadline or fails once with a controlled timeout.