Queue Topology: One Broker, Several Queues, No Shared Pool
A single task queue served by a single worker pool works until the day a bulk job — a catalog scrape, a translation batch, a re-notification sweep — holds every process for minutes at a time. Payment provider callbacks, refund confirmations and verification emails wait in the same line behind it, and the incident is reported as a payments outage even though no payment code failed. The fix is queue topology, and topology is mostly a decision about which work is allowed to wait.
We run this shape on payment services, marketplaces and lending systems, and it survives contact with production because it is small: three classes of work, three pools, one rule enforced by the build.
Classify by what breaks when the task is late#
The useful axis is not the domain of the task. It is the blast radius of delay.
Money path. Provider callbacks, ledger writes, payout and settlement transitions — anything a reconciliation job will later compare against a bank statement. Late here means a customer sees a paid order as unpaid, or the provider re-sends a webhook we have not processed yet and we now have to tell a retry apart from a second payment. Delay does not just annoy; it manufactures ambiguity in the ledger.
Interactive. Password resets, verification codes, push and chat notifications, on-demand document rendering. Somebody is looking at a screen waiting for the effect.
Bulk. Parsers, reindexing, catalog translation through an LLM, report generation, collections sweeps. Nobody is waiting. The only requirements are that it finishes and that it does not eat the machine.
Three classes are enough for most systems. The temptation is to create a queue per feature, which produces thirty queues, no owner for any of them, and a worker fleet nobody can reason about during an incident. A new queue is justified when the work has a genuinely different latency contract or a different failure mode — not when it has a different product manager.
Route explicitly, and make the default queue a build failure#
The dangerous task is the one nobody routed. It lands on the default queue, which is usually the same queue the money path uses, and it sits there quietly until it is slow.
# tasks/routing.py
from kombu import Exchange, Queue
QUEUES = (
Queue(
"money",
Exchange("money"),
routing_key="money",
queue_arguments={
"x-dead-letter-exchange": "dlx",
"x-dead-letter-routing-key": "money.dead",
},
),
Queue(
"interactive",
Exchange("interactive"),
routing_key="interactive",
queue_arguments={
"x-dead-letter-exchange": "dlx",
"x-dead-letter-routing-key": "interactive.dead",
},
),
Queue(
"bulk",
Exchange("bulk"),
routing_key="bulk",
queue_arguments={
"x-dead-letter-exchange": "dlx",
"x-dead-letter-routing-key": "bulk.dead",
"x-overflow": "reject-publish",
},
),
)
ROUTES = {
"payments.*": {"queue": "money"},
"ledger.*": {"queue": "money"},
"payouts.*": {"queue": "money"},
"notify.*": {"queue": "interactive"},
"documents.render_*": {"queue": "interactive"},
"catalog.*": {"queue": "bulk"},
"search.reindex_*": {"queue": "bulk"},
"reports.*": {"queue": "bulk"},
}The default queue is named unrouted and no worker consumes it, so an unrouted task is visible as a growing queue instead of invisible as added latency somewhere else. Better still, it never reaches production, because the routing table is a test:
from celery import current_app
def test_no_task_falls_through_to_the_default_queue():
router = current_app.amqp.router
unrouted = sorted(
name
for name in current_app.tasks
if not name.startswith("celery.")
and router.route({}, name, [], {})["queue"].name == "unrouted"
)
assert unrouted == [], f"tasks without an explicit queue: {unrouted}"This is the whole governance mechanism. Adding a task forces a five-second decision about what breaks when it is late, and the reviewer sees that decision in the diff. Our review gate rejects a new task whose route was added to bulk purely to make the test pass when the task clearly writes to the ledger.
One pool per class, with its own concurrency and prefetch#
Separate queues consumed by one worker fleet buy nothing. The pools have to be separate processes with separate limits, because the classes fail differently.
services:
worker-money:
image: platform:${TAG}
command: >
celery -A platform worker -Q money -n money@%h
--concurrency=8 --prefetch-multiplier=1 --max-tasks-per-child=500
worker-interactive:
image: platform:${TAG}
command: >
celery -A platform worker -Q interactive -n interactive@%h
--pool=gevent --concurrency=64
worker-bulk:
image: platform:${TAG}
command: >
celery -A platform worker -Q bulk -n bulk@%h
--concurrency=4 --prefetch-multiplier=1 --max-memory-per-child=400000The money pool runs with acks_late and a prefetch multiplier of one: a task is acknowledged after it completes, and a worker holds exactly one message, so killing a pod during a deploy re-delivers the callback instead of losing it. That choice makes delivery at-least-once, which is only safe because every money task is idempotent on a deterministic key derived from the provider event — the same discipline the money path already needs for retried webhooks.
The interactive pool is IO-bound and gets high concurrency on a green-thread pool; it spends its life waiting on SMTP, push gateways and chat APIs. The bulk pool is memory-hungry and gets recycled by max-memory-per-child, because a parser that leaks over a long run should die quietly on its own queue rather than take a machine that also runs settlements.
Retries that do not become the second outage#
When a provider goes down, naive retries turn a dependency failure into a self-inflicted one: thousands of tasks re-queued at a fixed delay, all waking at the same moment, saturating the pool that the recovery also needs.
@app.task(
bind=True,
autoretry_for=(ProviderUnavailable,),
retry_backoff=2,
retry_backoff_max=900,
retry_jitter=True,
max_retries=12,
acks_late=True,
)
def confirm_payment(self, event_id: str) -> None:
with claim(event_id) as first_time:
if not first_time:
return
apply_provider_event(event_id)Exponential backoff with jitter and a bounded retry count; after the last attempt the message dead-letters instead of disappearing. The dead-letter queues exist for all three classes, but only one of them is an alert: depth above zero on money.dead pages someone, because a payment event we permanently failed to apply is a discrepancy that a human will otherwise find in a reconciliation run days later. The consumer on the dead-letter queues records and classifies; it never replays automatically, because a poison message replayed on a schedule is just a slower infinite loop.
What it costs to run#
Three deployment units instead of one, three sets of dashboards, and three alert thresholds that must be different or they are pointless: seconds of queue latency on money, tens of seconds on interactive, depth-and-trend only on bulk. Capacity planning stops being a single concurrency number and becomes three, tuned independently — which is more work up front and considerably less work at three in the morning.
There is also a discipline cost. The topology decays the moment somebody adds a queue without an owner or routes a ledger write to bulk because it happened to be slow. The routing test catches the first mistake; only review catches the second.
Closing#
The insight is unglamorous: most "payments were down" incidents in systems we have inherited contained no payment defect at all. They were scheduling accidents, where work that could have waited an hour was standing in front of work that could not wait a second. Separating the two costs one routing table, one test, and three worker definitions.