The most dangerous code in most Python systems is the code nobody watches: the background task.
An HTTP endpoint fails loudly. Someone gets a 500, support gets a ticket, you get a trace. A Celery task fails into a log line nobody reads, and the system keeps looking healthy.
We have shipped queue-backed services in payments, e-commerce, and notification fan-out for years, and the failures repeat with the same shape. A payout webhook retried three times because the ack happened after the work. A catalog import that half-finished and left prices from two different runs in the same table. A notification service that quietly stopped delivering because one poisoned message kept the worker busy forever.
None of those were queue bugs. Celery and RabbitMQ did exactly what they were told.
What actually holds up in production is boring:
Every task takes an idempotency key, not just arguments, so a redelivery is a no-op instead of a second charge.
Every task has a hard time limit and a dead-letter path, because a task with no ceiling will eventually consume the whole pool.
Every task reports failures where humans already look, not into a log file. If your API errors page someone and your worker errors do not, you have two reliability standards in one system.
And tasks stay small. A task that does five things fails in the middle of the third one, and you get to reason about partial state at 3am.
Our rule when we take over an existing codebase: read the task modules before the views. That is where the unowned complexity lives.
What finally forced your team to take background jobs as seriously as your API — a duplicate charge, a silent data drift, or something worse?