The retry you added to make the system more reliable is often what takes it down.
Here's the pattern we see in code reviews more than any other. A call to a payment provider, an email service, or an internal API occasionally times out. Someone wraps it in a retry loop. It works in staging. Everyone moves on.
Then one day the downstream service slows down instead of failing outright. Every caller times out and retries. Those retries pile onto an already-struggling service. It gets slower, so more calls time out, so more retries fire. A brief latency blip becomes a full outage — and the retries are the thing keeping it down.
Two things separate a retry that heals from a retry that amplifies:
Idempotency. If the operation can run twice safely — an idempotency key on writes, a dedup check on the consumer — a retry is free insurance. Without it, a retry that half-succeeded now double-charges or double-sends.
Backoff with jitter and a budget. Retry immediately and you synchronize every client into a thundering herd. Exponential backoff with randomness spreads the load; a cap on total retries stops the loop from feeding itself.
Our rule in review is simple: no retry gets merged until we can answer two questions. Is this operation safe to run twice, and what stops it from retrying forever? If either answer is unclear, the retry is a liability, not a safeguard.
When a retry has bitten you in production, was it the missing idempotency key or the missing backoff that did the damage?