Timeouts and Retries: Stop Turning One Failure Into an Outage

When a dependency slows down, the instinctive fix is often to retry. But if every incoming request creates several extra attempts, a small slowdown can become a traffic surge. The service that needs less pressure receives more of it. Timeouts and retries should be designed together. A timeout limits how long an attempt can consume resources. A retry policy decides whether another attempt is useful within the user's remaining time budget. ## Start with an end-to-end deadline Suppose a page should complete within a few seconds. Giving every downstream request its own full-length timeout can make the overall request last much longer. Allocate time within a shared deadline and propagate cancellation when the caller gives up. Distinguish connection establishment, response headers, and response body timeouts where the client library supports them. A server can accept a connection quickly and then stop sending useful data. Choose values from observed latency and the user experience, not a universal constant. An interactive search and an overnight export have different tolerances. Also reserve time for a useful error response instead of using the entire budget on the last attempt. ## Retry only when another attempt makes sense Some network interruptions and temporary service errors are reasonable retry candidates. Invalid input, authentication failures, and deterministic application errors usually are not. Respect rate-limit responses and documented retry guidance. An ambiguous timeout after a write is particularly important. The operation may already have succeeded. Use idempotency protection before retrying actions such as creating orders or charging accounts. Keep retries in one deliberate layer where possible. If a browser, API server, SDK, and worker all retry independently, their attempt counts can multiply. Document which layer owns the policy. ## Bound the extra traffic Exponential backoff increases the wait between attempts. Jitter spreads clients across time rather than having them retry at the same instant. Cap both attempts and total elapsed time; a mathematical backoff formula alone is not a resource limit. Consider a retry budget that limits additional attempts relative to normal traffic. When a dependency remains unhealthy, a circuit breaker can temporarily stop calls, but its state transitions and recovery probes also need testing. Measure original requests separately from attempts. Track timeout rate, retry rate, final outcomes, and dependency latency. Otherwise a dashboard may show stable request volume while the underlying service sees several times as much work. A good retry policy helps with short-lived failures without hiding permanent ones. Its job is controlled recovery, not endless persistence. ## Further reading - [AWS Builders' Library: Timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) - [Google SRE: Handling overload](https://sre.google/sre-book/handling-overload/)