Shipmind Labs

A failing hook must not hand back the moved order

· 8 min read

A transition that writes the new state and then fails in one of its callbacks leaves the order in the new state with the work half done. The record says shipped. The stock was never reserved, or the customer was never told. The next handler (a nightly job, a mobile endpoint, a support screen) reads a shipped order that nobody shipped, and there is nothing in the record to say so. This is a shape problem rather than a bug in any one callback: as long as the state change and the side effects are separate writes inside the same function, the window between them belongs to whichever one crashes first.

The handler that writes the state too early#

This is the version that gets written first, and it is the one we quote in the README of our own order-lifecycle library (https://github.com/shipmindlabs/order-lifecycle) as the thing the library exists to replace:

python
def ship(order, user):
    if order.status != "paid":
        raise ValueError(f"cannot ship an order in {order.status}")
    if user.role != "warehouse":
        raise PermissionError("only the warehouse may ship")
    if not order.payment_confirmed:
        raise ValueError("payment is not confirmed")
    order.status = "shipped"
    reserve_stock(order)
    mailer.send(order, "shipped")
    log.info("order %s shipped by %s", order.id, user.id)

The guards are fine. The defect is the ordering of the last four lines. order.status = "shipped" mutates the object the caller is going to persist, and whether that mutation reaches the database when reserve_stock raises depends on the session, the transaction boundary and how far up the stack the exception is caught. Which is to say it depends on things you cannot see in this function.

Moving the assignment to the bottom only changes which failure you prefer. A raise inside mailer.send then leaves stock reserved for an order that is still paid, and the warehouse holds goods against a move that never happened. No ordering of those four lines makes all of them atomic, because they do not all live in the same system.

What you can decide is narrower, and still worth having. Of those four effects, exactly one lives in our own record. That is the only one we can hold back, so it is the one that must never be the first thing to land.

The move is a value, not a mutation#

In order-lifecycle a Machine is a frozen dataclass of a transition table, the current state and a history, and applying a trigger returns a new machine rather than editing the one you have:

python
def apply(
    self,
    trigger: Trigger,
    *,
    role: Role | str | None = None,
    context: Any = None,
    reason: str = "",
    at: datetime | None = None,
) -> Machine:
    transition = self.resolve(trigger, role=role, context=context)
    self._fire(transition.before, "before", transition, role, context)
    moved = replace(
        self,
        state=transition.target,
        history=self.history.log(transition, role=role, reason=reason, at=at),
    )
    self._fire(transition.after, "after", transition, role, context)
    return moved

The statement order is the whole argument. moved is constructed before the after hooks run, and it has to be, since the hooks describe work that happens around a settled target state, but it is only returned once the last of them has come back. If an after hook raises, _fire turns it into a HookFailed that propagates out of apply, moved goes out of scope, and the caller is still holding the machine it called apply on: source state, history without an entry for the abandoned move.

There is no rollback branch to forget to write, because nothing was written. The moved order existed as a local variable for the duration of the after hooks and was then dropped. That turns the caller's shape into the entire discipline:

python
try:
    moved = machine.apply(SHIP, role=user.role, context=order)
except HookFailed as failure:
    report(failure)
else:
    save(moved)

One persistence write, reached only after every hook has returned. The state that gets written is a state whose hooks all completed.

Failure has to say how far it got#

Holding back the state does not undo what the hooks did, and pretending otherwise would be the more dangerous lie. An after hook that raises was necessarily preceded by before hooks that returned, and in a multi-hook phase by siblings of its own that returned. Those effects stand. So the error names them: HookFailed carries the hook, its TransitionContext, the cause, the phase, the transition, and completed, the hooks of that phase that already ran.

python
except HookFailed as failure:
    log.error(
        "order stayed in %r: %s hook %r raised %s; already done: %s",
        machine.state.name,
        failure.phase,
        failure.hook.name,
        type(failure.cause).__name__,
        ", ".join(hook.name for hook in failure.completed) or "nothing",
    )

The message the exception builds for itself ends with the same promise, in the words of the table: the order stays in 'paid'. A caller that needs compensation, releasing a stock reservation or cancelling a scheduled notification, has the facts to do it at the point where the decision belongs, instead of inferring them from a status column that has already moved.

This is also the practical argument for which side of the state settlement a piece of work goes on. In a declared row the two phases are named separately:

python
Transition(
    PAID,
    SHIPPED,
    SHIP,
    roles=WAREHOUSE,
    conditions=(PAYMENT_CONFIRMED, IN_STOCK),
    before=reserve_stock,
    after=notify_customer,
)

Work that must have succeeded for the move to be legitimate goes in before, where its failure stops the move with the clearest possible story. Work that merely announces a settled move goes in after. Both phases leave the source state untouched on failure. The difference is how much you have to clean up when they do, and the row is where that gets argued once rather than per endpoint.

Nothing fires until the refusal is ruled out#

The other half of "the order stays put" is that a move which was never allowed must not run a hook at all. apply calls resolve first, and resolve refuses in a fixed order: IllegalTransition naming the triggers that would be allowed here, then RoleNotPermitted naming the required_roles, then ConditionNotMet carrying every condition that failed. Every one, because the condition evaluation deliberately does not short-circuit, so you get the full list of what is missing instead of one item at a time.

All three happen before the first before hook is constructed, which is why the refusal half of our matrix test can assert the absence of effects directly:

python
with pytest.raises(error) as refusal:
    machine.apply(trigger, role=role, context=context)

assert message in str(refusal.value)
assert machine.state == source
assert not machine.history

The same two assertions hold for a hook failure, for a different reason. Not "we never got that far" but "we got all the way and did not hand it back".

What it costs to run#

Three real costs, worth naming before you adopt the pattern.

Callers must use the return value. machine.apply(SHIP, role=role) on a line by itself is a silent no-op, and no type checker will complain. That is a review item, and it is the trade for never having a half-moved record: the mutable version cannot have this bug, and has the other one instead.

Hooks must not write the order's state themselves. They receive a TransitionContext with the transition, the phase, the acting role and the subject, enough to do their work, and no business reaching around to set a status. A hook that persists the move defeats the whole arrangement.

Hooks run inside the caller's latency, between resolve and return. A slow after hook is a slow handler, so outbound work belongs behind a queue and the hook is the enqueue, not the delivery. For the record itself, time stays controllable: the history is a frozen value whose record and log return new objects, an entry's timestamp defaults to the current UTC moment, and apply(at=...) hands the clock to the caller, so tests and backfills are deterministic rather than dependent on wall time.

The guarantee you buy is small and exact, and it is probably the one that matters most on a money path: every state in the record is a state that something finished producing. Side effects still need compensation, and the error tells you which ones. What never happens is the quiet case, a shipped order nobody shipped, discovered by the next handler, with no entry anywhere saying why.

Was this useful?

Building something similar?

or email hello@shipmindlabs.com