Moving work to Celery makes your API fast. It also quietly turns every user action into a promise nobody is tracking.
The pattern is familiar. A signup triggers a welcome email, a KYC submission triggers a verification call, a payment triggers a ledger update. All of it goes to a queue so the HTTP response stays under 200ms. The endpoint is fast. Everyone is happy.
Then a verification provider times out at 3am, the task raises, the worker moves on, and a user sits in "pending review" for two days. No error page. No alert. The request succeeded — the promise didn't.
What we've learned running payment and compliance systems on Python queues: the moment a task represents something a user is waiting for, it stops being infrastructure and becomes product state. It needs a row in your database, not just a job in Redis. Enqueued, started, failed, succeeded, retried — visible to the people who answer support tickets, not only to the engineer with terminal access.
The practical version is boring. Every user-visible async action gets a status record written before the task is enqueued. The task updates it. A reconciliation job scans for records stuck in a non-terminal state longer than they should be. That job is the actual monitoring — it catches the failures your exception tracker never saw because the worker died mid-task.
Celery is excellent at running code later. It was never designed to tell your business what didn't happen.
For teams running background work in production: how do you find the tasks that failed silently — and how long does it usually take?