Shipmind Labs

Two days went into chasing a bug that lived only in production, and the reason was that our fake acquirer was too agreeable.

The setup was ordinary. A hold goes out, the acquirer answers, the order moves on. In the tests we had, the hold came back authorized immediately. In the real world some holds come back later: the acquirer accepts the request, decides nothing, and tells you about it on its own schedule, through a callback you did not ask for.

So our code had a state it had not been exercised against: a transaction that exists, is addressable, reports a status, and can do nothing. Capture was wrong. Cancel was wrong. Refund was wrong. It all looked right in staging, because staging did not produce that transaction.

The fix did not start in the payment code. It started in the double.

In our open-source project payment-gateways, the fake provider now has a counter you arm deliberately: you ask for one pending outcome, and the next hold (only the next one) comes back waiting instead of authorized. Everything after it behaves normally. That matters, because the bug we were reproducing was not 'the acquirer is slow', it was 'one hold out of many is slow while the rest settle around it'.

And a waiting hold is inert on purpose. Capture and cancel refuse it, because they require an authorized transaction. Refunds refuse it, because they require something captured. Each one answers with an invalid-state error and changes nothing: the transaction is still readable, still reports what it is, still reports the full amount as outstanding, because nothing has been captured against it.

The only legal move is the callback. It takes a transaction id and an outcome, and we model it as its own operation with no idempotency key, because the caller did not initiate this exchange, the acquirer did, and pretending otherwise puts a key on a message you didn't send.

The part we would argue for in any gateway integration is the order of the refusals. Unknown transaction first. Then any transaction that is not actually waiting. Then any outcome that is neither authorized nor failed. Only after all three does the status change. That last guard is the one teams skip, and it is the one that stops an acquirer (or a replayed webhook) from walking a transaction into a state your own code can never produce.

A failed hold then reports zero outstanding, same as a cancelled one. A second callback for the same transaction finds nothing pending and declines. The answer goes to an audit sink either way, because 'we refused this callback' is a thing you will need to prove later.

What we do differently now: when a provider has a state we cannot trigger on demand, the test double gets the knob before the integration gets the feature.

Probably the useful thing to compare is how you reproduce the slow-callback path in your own payment tests, with a scriptable fake or a recorded sandbox.

Was this useful?

Building something similar?

or email hello@shipmindlabs.com