Notification Fan-Out Needs a Stop Condition, Not a Retry Loop
A service that turns one event into push, chat and email messages is easy to build and hard to keep honest. The failure that reaches support is almost never a dropped message; it is the second and third message, arriving after the user has already handled the thing they were being nagged about. We have built multi-channel notification layers more than once — fan-out to WhatsApp, Telegram, mobile push and web push — and the part that decides whether the system feels competent is not the sending. It is knowing when to stop.
The event is not the notification#
An event happens: a payment needs confirmation, a document is waiting for a signature, a verification case has been returned to the customer. The naive service treats that as one message per channel, enqueued at once. Every channel then races the user, and at least one of them loses.
The model that survives production has three levels rather than one.
A notification is a decision that this recipient should be told this thing, once. It carries a dedup key, so that the same underlying state producing two events does not produce two ladders.
A delivery step is one attempt, on one channel, at one time. Steps are ordered.
A resolution is the fact that the notification no longer needs delivering. It is written by whoever observes the user acting: the API handler that accepts the confirmation, the endpoint that marks an in-app item as read, the webhook that reports a reply.
Only the first two belong to the notification service. The third is written from outside it, and that asymmetry is the whole design.
create table notification (
id bigserial primary key,
recipient_id bigint not null,
kind text not null,
dedup_key text not null,
payload jsonb not null,
created_at timestamptz not null default now(),
resolved_at timestamptz,
resolution text,
unique (recipient_id, dedup_key)
);
create table delivery_step (
id bigserial primary key,
notification_id bigint not null references notification (id),
step_no smallint not null,
channel text not null,
send_after timestamptz not null,
state text not null default 'pending',
attempts smallint not null default 0,
provider_ref text,
unique (notification_id, step_no)
);
create index delivery_step_due
on delivery_step (send_after)
where state = 'pending';Everything below is the machinery that keeps resolved_at authoritative.
The ladder is data, not code#
Escalation is configuration, one ladder per notification kind, because the delays are a product decision and they change far more often than the dispatcher does.
@dataclass(frozen=True)
class Step:
channel: str
after: timedelta
LADDERS: dict[str, list[Step]] = {
'payment_action_required': [
Step('push', timedelta(0)),
Step('chat', timedelta(minutes=10)),
Step('email', timedelta(hours=2)),
],
'document_awaiting_signature': [
Step('push', timedelta(0)),
Step('email', timedelta(hours=4)),
],
}
def schedule(conn, notification_id: int, kind: str, now: datetime) -> None:
rows = [
(notification_id, i, s.channel, quiet_hours_shift(now + s.after))
for i, s in enumerate(LADDERS[kind])
]
conn.executemany(
'insert into delivery_step '
'(notification_id, step_no, channel, send_after) '
'values (%s, %s, %s, %s) on conflict do nothing',
rows,
)Two details earn their place. All steps are written at scheduling time rather than lazily after the previous one completes, so the ladder is inspectable: when someone asks why a customer received three messages, the answer is a single row-set, not a reconstruction from logs. And quiet hours move a step forward instead of dropping it — a suppressed step is a message the user never gets, while a delayed step is one they get in the morning.
Cancellation happens at claim time#
The obvious way to stop a ladder is a sweep: when the notification resolves, set its pending steps to cancelled. Do that, but do not depend on it. It is an optimisation, and it races with a dispatcher that has already picked the row up.
Correctness comes from re-checking the stop condition inside the same statement that claims the work.
with due as (
select s.id
from delivery_step s
join notification n on n.id = s.notification_id
where s.state = 'pending'
and s.send_after <= now()
and n.resolved_at is null
order by s.send_after
limit 50
for no key update of s skip locked
)
update delivery_step s
set state = 'sending',
attempts = attempts + 1
from due
where s.id = due.id
returning s.id, s.notification_id, s.channel, s.attempts;The join is the point. A step is never sent because it was due; it is sent because it was due and the notification was still open at the instant of claiming. With that in place the cancellation sweep can be late, can be lost, can run in another transaction entirely, and the worst outcome is a row that sits in pending until a dispatcher discards it.
One race remains and it cannot be closed: the user acts while the provider call is already in flight. That window is the duration of one HTTP request rather than the ten minutes between ladder steps, and it is the right place to stop engineering. Write it down in the runbook and move on.
The adapter contract, and the key that must not change#
Every channel is an adapter with the same three outcomes: the provider accepted it, the provider permanently refused it, or the call failed in a way worth retrying.
def dispatch(conn, step, notification, adapter) -> None:
key = f'{notification.id}:{step.step_no}'
result = adapter.send(
recipient_id=notification.recipient_id,
payload=notification.payload,
idempotency_key=key,
)
match result:
case Sent(provider_ref):
mark(conn, step, 'sent', provider_ref=provider_ref)
case Rejected(reason):
mark(conn, step, 'rejected')
disable_contact(conn, notification.recipient_id, step.channel, reason)
promote_next_step(conn, notification.id, step.step_no)
case Failed():
mark(conn, step, 'pending', send_after=backoff(step.attempts))The idempotency key is built from the notification id and the step number, and deliberately not from the attempt counter. A retry after a timeout must present the same key as the call that timed out, because the ordinary explanation for a timeout is a provider that accepted the message and failed to say so. Put the attempt number in the key and every ambiguous timeout becomes a duplicate on someone's phone.
The Rejected branch is the one that gets skipped. A revoked push token, a chat account that blocked the sender, an address that hard-bounced — these are permanent facts about the contact, not transient send failures. Retrying them burns quota, and worse, it leaves the ladder waiting on a channel that will never work. So the branch does two things at once: it marks the contact dead so no future ladder schedules it, and it pulls the next step forward instead of leaving the user in silence for two hours because step one was addressed to a phone that no longer exists.
What it costs to run#
The partial index on pending steps keeps the poll to one small index scan per tick regardless of table size, and the table does grow — delivery_step is where the volume lives, so it needs a retention job from day one, not after the first slow query.
Three signals are worth a dashboard.
Due lag: now() minus the send_after of the oldest pending due step. Non-zero and rising means the dispatcher is behind. For a payment ladder that is a product outage even though nothing has thrown an error.
Escalation depth: the distribution of the step number at which notifications resolve. This is the most useful number the service produces and it is not an infrastructure metric. If a kind of notification routinely resolves at step three, either channel one is not arriving or the text in it does not tell people what to do.
Cancellation ratio: the share of steps cancelled by the sweep or discarded at claim time. It should be substantial. A ladder that almost never cancels is a ladder whose stop condition is not being written — somebody shipped a new action path and forgot to resolve the notification, and the system is now sending messages nobody needs.
Close#
The fan-out is the easy half; anything can put a message on three queues. What makes a notification service feel like it was built by people who have watched real users is that it treats delivery as a sequence whose stop condition is owned elsewhere, re-reads that condition at the last possible moment, and records a dead contact as a fact rather than an error. The rest is adapters.