A deploy went out, the service restarted cleanly, and one notification never arrived.
No error. Nothing in the logs about a failed send. The send had been enqueued a few milliseconds before shutdown, and then it was gone.
We traced it to the worker that pulls from our outbox in notify-dispatch, our open-source dispatcher. The loop was simple: check a stop flag, wait on the outbox for a short interval, drain whatever showed up. Shutdown was just as simple, and that was the problem. We set the flag and joined the thread.
Two things went wrong in that gap.
First, a worker parked in wait had no idea anything had changed. It sat out the rest of its interval while the process was trying to exit, so shutdown felt slower than it should, and under pressure someone eventually shortens the join timeout, which is how you turn a slow shutdown into a lost message.
Second, and worse: a ticket enqueued after the flag was set but before the join completed did not get looked at again. The loop had already decided to leave. The queue still had work in it. Nothing drained it.
The fix needed no new machinery. It was ordering.
Stop now does three things in a fixed sequence. It sets the stopping flag. Then it wakes the outbox condition, so a worker parked in wait returns immediately instead of sleeping out its interval. Only then does it join the thread.
And the loop drains one final time after the while loop exits. That last drain carries the guarantee: anything that arrived in the window between the flag and the join still gets sent and settled, not dropped.
The small guards matter too. The interval has to be positive, and the constructor rejects anything else instead of quietly spinning a hot loop. A second start raises rather than leaving two threads fighting over one queue. Stopping a worker that was not started is a no-op, because there is no thread to join. And the worker is a context manager, so entering starts it and exiting stops it through the same safe path. You cannot forget the ordering if you do not write it by hand.
What changed in how we work: we stopped treating shutdown as cleanup and started treating it as a delivery path. If a queue can accept work right up to the moment the process exits, then the exit sequence is part of your delivery guarantee, and it deserves the same tests as the happy path.
Idle cost stayed where it was, by the way. The worker only drains when something actually arrived, so waiting is still waiting. It parks on a condition rather than polling.
The bug cost us one notification and an afternoon. The lesson was cheap at that price.
If you run background workers, it is worth checking where your shutdown sequence actually sits in your test suite: next to the feature tests, or nowhere.