Retries without the retry storm

In one sentence

A retry is a decision to send more traffic to a service that just told you it was struggling, so the interesting part is not whether to retry but how hard to try not to.

Why it matters

Retries are the most common self-inflicted outage in distributed systems. The mechanism is always the same: a dependency slows down, every caller retries, the dependency now receives three times its normal load, it slows down further, and the system converges on a stable state where nothing succeeds and everything is retrying. This is a metastable failure — removing the original trigger does not fix it, because the retries are now the load.

Worse, retries multiply through a call chain. If four services each retry three times, a single user request can become 81 calls to the service at the bottom. The service that is already unhealthy gets the amplified version.

Retry only what is retryable

Two conditions must both hold:

1. The error is transient. A 503, a connection reset, or a timeout might succeed next time. A 400 or a 422 will not — retrying a malformed request is pure waste, and retrying a 401 can lock the account.
2. The operation is idempotent. Either naturally (a GET, a PUT of a full resource) or made so with an idempotency key. Retrying a non-idempotent charge is how customers get billed twice.

A timeout is the dangerous case, because you do not know whether the work happened. Treat a timed-out write as "unknown", not "failed", and only retry it if it carries an idempotency key.

Exponential backoff with jitter

Fixed-interval retries synchronise: every client that failed at the same moment retries at the same moment, producing a thundering herd at each interval. Exponential backoff spreads the retries out in time but does nothing about the synchronisation — clients still land together, just at exponentially spaced moments.

Jitter is what breaks the synchronisation, and full jitter is the version to reach for:

async function withRetry<T>(
  operation: () => Promise<T>,
  { attempts = 3, baseMs = 100, capMs = 20_000 } = {},
): Promise<T> {
  let lastError: unknown

  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      return await operation()
    } catch (error) {
      if (!isRetryable(error)) throw error
      lastError = error

      // Full jitter: uniform in [0, min(cap, base * 2^attempt)].
      const ceiling = Math.min(capMs, baseMs * 2 ** attempt)
      await sleep(Math.random() * ceiling)
    }
  }

  throw lastError
}

The uniform draw over the whole window — rather than the window's endpoint — is what makes the arrival pattern flat instead of spiky. It is a one-line change from the naive version and it is the entire point of the exercise.

If the server sends Retry-After, obey it. It knows something you do not.

Deadlines, not attempt counts

An attempt count is the wrong budget. Three retries of a call with a 30-second timeout is a two-minute wait for a user who left after five seconds.

Give the request a deadline at the edge, propagate it through every hop, and check it before each attempt:

const deadline = Date.now() + 2_000

while (Date.now() < deadline) {
  const remaining = deadline - Date.now()
  try {
    return await operation({ timeoutMs: remaining })
  } catch (error) {
    if (!isRetryable(error)) throw error
    await sleep(Math.min(jitteredDelay(), deadline - Date.now()))
  }
}

throw new DeadlineExceeded()

The deadline also gives downstream services useful information. A service that receives a request with 40ms of budget left can reject it immediately rather than doing work whose result nobody will wait for.

Retry at one layer

If your HTTP client retries, your service wrapper retries, and your job runner retries, you have 27 attempts, not 3. Pick one layer — usually the outermost one that understands whether the operation is idempotent — and make every other layer fail fast.

The two things that make retries safe at scale

A retry budget. Cap retries as a fraction of total requests — say 10%. Under normal conditions nothing is ever refused, because failures are rare. Under a partial outage the budget is exhausted immediately and retries stop, which is exactly when you want them to. This bounds amplification no matter how many clients you have.

A circuit breaker. After a threshold of consecutive failures, stop calling the dependency entirely for a cooldown, then let a single probe through. This turns a slow cascading failure into a fast, local, obvious one — and gives the dependency the idle time it needs to recover.

Common pitfalls

• Retrying on a 429. The server is telling you to slow down; back off hard and honour Retry-After rather than treating it as a normal transient error.
• Retrying inside a database transaction, holding locks for the whole backoff.
• Logging one line per attempt, so a retry storm also becomes a logging incident.
• Testing retries only against a dependency that is down. The hard case is a dependency that is slow, because that is where timeouts and deadlines interact.

Further reading

• Timeouts, retries, and backoff with jitter — the Amazon Builders Library article, with the jitter measurements.
• Google SRE: Handling Overload — where retry budgets and their numbers come from.