Catching money bugs with ledger invariants, not error logs

30 July 2026 · 8 min read

A payout that credits the wrong sub-account returns HTTP 200. Nothing throws, the worker acknowledges the message, the error dashboards stay green, and the discrepancy surfaces days later when someone reconciles a bank statement by hand.

That gap — between "the code ran without errors" and "the money is where it should be" — is where the hardest incidents in payment systems live. Retries, partial failures, and provider callbacks arriving out of order all produce states that are individually plausible and collectively wrong. Exception tracking cannot see them, because there is no exception.

We work on systems that move money: payment services on card rails and account-to-account flows, virtual bank-account ledgering with per-user attribution, custodial wallets, marketplace payout flows. Across all of them the same practice earns its keep. Write down the properties that must hold over the ledger, check them continuously, and treat a violation with the same weight as a spike of 500s.

The constraint that makes this hard#

A ledger backed by external rails is legitimately inconsistent for a while. A transfer is submitted, the provider confirms asynchronously, the bank statement line arrives on the next value date. A check that ignores that timing produces false positives, and a team that gets paged for false positives stops reading pages. So an invariant is not just a boolean over the tables — it needs a settlement window and a severity.

The second constraint is operational: these are the tables you cannot lock or slow down. Checks must be read-only, run against a replica, and be cheap enough to repeat every few minutes.

Write the invariants as SQL, not as prose#

We keep them in the repository next to the migrations, one file per invariant, under version control and code review like everything else. The contract is deliberately narrow: each query returns zero rows when the system is healthy, and one row per offending entity when it is not. Every file takes a single :settlement parameter cast to an interval.

Double entry, first. Every posted transaction must sum to zero:

-- ledger/invariants/sql/0001_transaction_balances_to_zero.sql
SELECT t.id AS entity_id,
       SUM(p.amount_minor) AS residual_minor
FROM ledger_transaction t
JOIN ledger_posting p ON p.transaction_id = t.id
WHERE t.posted_at < now() - CAST(:settlement AS interval)
GROUP BY t.id
HAVING SUM(p.amount_minor) <> 0;

Next, the cached balance every read path uses must match the postings that produced it. This is the invariant that catches a balance updated outside a transaction, or updated twice by a retry:

-- ledger/invariants/sql/0002_account_balance_matches_postings.sql
SELECT a.id AS entity_id,
       a.balance_minor AS cached_minor,
       COALESCE(SUM(p.amount_minor), 0) AS derived_minor
FROM ledger_account a
LEFT JOIN ledger_posting p ON p.account_id = a.id
WHERE a.updated_at < now() - CAST(:settlement AS interval)
GROUP BY a.id, a.balance_minor
HAVING a.balance_minor <> COALESCE(SUM(p.amount_minor), 0);

Then attribution. In a virtual-account design, where each user pays into their own requisites and the bank reports movements on the parent account, incoming money has to be matched to exactly one user. Zero matches means funds are sitting unattributed; two matches means one payment was credited twice:

-- ledger/invariants/sql/0003_inbound_lines_attributed_once.sql
SELECT l.id AS entity_id,
       count(p.id) AS attribution_count
FROM bank_statement_line l
LEFT JOIN ledger_posting p ON p.statement_line_id = l.id
WHERE l.direction = 'inbound'
  AND l.value_date < current_date - CAST(:settlement AS interval)
GROUP BY l.id
HAVING count(p.id) <> 1;

The rest of the set follows the same shape: no custody or escrow account below zero at any moment, no posting written to a closed account, the sum of user balances equal to the omnibus balance reported by the provider, no transaction in a non-terminal state older than its expected lifetime.

Two conventions do a lot of work here. Amounts are integers in minor units, never floats, so equality comparisons are meaningful. And every invariant returns an entity_id, so an alert points at rows an engineer can open rather than at a number that dropped.

Run them like tests, on a schedule#

The runner is intentionally dull. It loads the SQL, executes it read-only, counts rows, samples a few for the alert payload, and reports how long it took.

# ledger/invariants/runner.py
import time
from dataclasses import dataclass
from pathlib import Path

from sqlalchemy import text

SQL_DIR = Path(__file__).parent / "sql"


@dataclass(frozen=True)
class Invariant:
    name: str
    severity: str        # "page" or "ticket"
    settlement: str      # interval literal, e.g. "5 minutes"
    sample_size: int = 20


REGISTRY = (
    Invariant("0001_transaction_balances_to_zero", "page", "1 minute"),
    Invariant("0002_account_balance_matches_postings", "page", "5 minutes"),
    Invariant("0003_inbound_lines_attributed_once", "ticket", "2 days"),
    Invariant("0004_custody_never_negative", "page", "0 seconds"),
)


def run(invariant, session):
    sql = (SQL_DIR / f"{invariant.name}.sql").read_text()
    started = time.monotonic()
    rows = session.execute(
        text(sql), {"settlement": invariant.settlement}
    ).mappings().all()
    return {
        "name": invariant.name,
        "severity": invariant.severity,
        "violations": len(rows),
        "sample": [dict(r) for r in rows[: invariant.sample_size]],
        "duration_s": time.monotonic() - started,
    }

The scheduled task wraps it, pins the session to a read-only transaction on the replica, and publishes one gauge per invariant:

# ledger/invariants/tasks.py
from sqlalchemy import text

from app.celery import app
from app.db import replica_session
from app.metrics import gauge
from app.alerts import emit
from ledger.invariants.runner import REGISTRY, run


@app.task(name="ledger.run_invariants")
def run_invariants():
    with replica_session() as session:
        session.execute(text("SET TRANSACTION READ ONLY"))
        for invariant in REGISTRY:
            result = run(invariant, session)
            tags = [f"invariant:{invariant.name}"]
            gauge("ledger.invariant.violations", result["violations"], tags)
            gauge("ledger.invariant.duration_s", result["duration_s"], tags)
            gauge("ledger.invariant.last_run_ts", time.time(), tags)
            if result["violations"]:
                emit(result)

The third gauge matters more than it looks. A checker that stopped running produces exactly the same picture as a perfectly healthy ledger: no violations, no alerts. So the monitoring has two rules per invariant — one on the violation count, and one on the age of last_run_ts. Without the staleness rule, the whole mechanism can fail silently, which is the failure mode it exists to prevent.

The same invariants in the test suite#

Because the checks are plain SQL against the schema, the test suite can run them too — with a zero settlement window, since a test controls time and has no in-flight provider callbacks. Our money-path tests end with the ledger being asserted clean, not just with the response body being asserted correct:

# tests/conftest.py
import pytest

from ledger.invariants.runner import REGISTRY, run


@pytest.fixture
def assert_ledger_clean(db_session):
    def _assert(exclude=()):
        failures = []
        for invariant in REGISTRY:
            if invariant.name in exclude:
                continue
            result = run(invariant, db_session)
            if result["violations"]:
                failures.append((invariant.name, result["sample"]))
        assert not failures, failures

    return _assert

# tests/payouts/test_reversal.py
def test_failed_payout_reverses_without_residual(
    api_client, payout_factory, provider_stub, assert_ledger_clean
):
    payout = payout_factory(state="submitted")
    provider_stub.reject(payout.reference, reason="account_closed")

    api_client.post("/internal/payouts/poll")
    api_client.post("/internal/payouts/poll")  # duplicate poll, must be idempotent

    assert_ledger_clean()

This is also what our review gate looks for. A pull request that introduces a new money path — a new transfer type, a new reversal branch, a new provider — is expected either to be covered by an existing invariant or to bring a new one with it. Reviewers ask the same question every time: which property of the ledger would be violated if this code is wrong, and what checks it? Answering that in review costs minutes. Answering it during a reconciliation incident costs a week and a lot of trust.

What this costs to run#

The frequent invariants are aggregates over the postings table, so they live or die on indexes: ledger_posting(transaction_id), ledger_posting(account_id), and a timestamp index supporting the settlement filter. Past a certain table size a full sweep every few minutes stops being free, and the fix is to split the cadence:

Incremental mode adds one predicate to the same file, driven by a stored watermark:

AND t.posted_at > CAST(:since AS timestamptz)

The full sweep still matters, because incremental checks only see rows the application touched. Corruption introduced by a manual fix, a bad backfill, or a restore lands outside the watermark, and only the sweep finds it.

Operationally it is a replica, one scheduled task, a handful of gauges, and two alert rules per invariant. No new infrastructure, no vendor.

Close#

Error rates tell you whether the code ran. Invariants tell you whether the money is right, and those are different questions. Since the ledger is the source of truth for every balance a user sees and every figure that ends up in a reconciliation report, it should also be the thing under continuous assertion — in production and in the test suite, with the same SQL doing both jobs.