A card got declined. Our retry wrapper tried again, and then again. One checkout attempt turned into three decline notifications for the customer, and support spent the afternoon explaining what had happened.
The bug was not in the retry loop. It sat in what we thought a retry was for.
We had built retries for flakiness: a connection dropping mid-call, a timeout, a gateway hiccup. Those deserve another attempt. Our wrapper, though, treated every non-success the same way. A decline is not a failure to deliver the request, it is the answer, and resending it only asks the same question louder.
So we split provider errors into two groups, and that split now sits explicitly in the code instead of being implied.
Network errors are retryable: we never learned the outcome, so asking again is legitimate.
Declines come back untouched. So does an invalid state transition, because capturing something already cancelled will not become valid in eight hundred milliseconds. So does an idempotency conflict, which is the gateway telling us the key has already been used for a different payload. Retrying that one does active harm.
The second change mattered more.
Every retry now resends the identical arguments, including the idempotency key the caller chose. We do not generate a fresh key per attempt. Written down it sounds obvious, but a new key on attempt two is how you turn one hold into two holds. The provider is meant to recognise the repeat and replay its original answer instead of performing the operation again. A rotating key removes the gateway's only means of recognising it.
Two smaller rules we settled on while cleaning this up.
When the attempt budget runs out, we return the last response as it is, failure included. The wrapper does not raise on exhaustion, because the caller needs to see what the provider actually said.
And a raised exception is not an answer, so we do not retry those. If the transport blew up in a way we did not classify, the honest move is to surface it rather than guess that another round trip will help.
Backoff is exponential with a ceiling, and the sleep function is injectable, which is the only reason the retry tests run in milliseconds instead of minutes. The constructor refuses fewer than one attempt and negative backoff outright, because a misconfigured retry policy should probably fail at startup and not at 2am against real money.
This lives in our open-source payment-gateways project, as a wrapper that implements the same provider interface and forwards hold, capture, refund, partial refund, cancel and status through one shared helper. One place to change the policy, every operation covered.
If you run retries against a payment provider, it is worth checking where your idempotency key gets created: at the call site, or inside the retry loop.