Shipmind Labs

An audit trail should record why a body is missing, not an empty field

· 8 min read

An audit trail that captures request bodies has to decide what to do with the bodies it cannot keep: a file upload, a 2 MB JSON document, a stream a view already consumed, bytes that do not decode. Drop them and you get a record whose empty payload looks exactly like a request that sent nothing, so whoever reads the trail later cannot tell a quiet endpoint from a quiet logger. In model-audit we fill the slot either way: with the payload, with None, or with a short note naming the reason the payload is not there.

The constraint: the record is the only artifact#

We build the systems where this matters (payment services, KYC and AML moderation tooling, e-signature workflows), and in those systems the trail gets read by people who cannot reproduce the event. Someone reconstructing why a payout changed state, or what a merchant submitted before a decision was made, has the record and nothing else. If the body is blank, they have to pick between three unrelated stories: the client posted nothing, the client posted something we declined to keep, or the endpoint was never captured in the first place.

Those stories support opposite conclusions. "Nothing was submitted" is evidence about the client. "We did not keep what was submitted" is a fact about our own configuration. A logger that collapses them has quietly turned its own limits into claims about user behaviour, and the only way back to the difference is to read the middleware source and reason about what it would have done, which is the work the trail exists to save you.

Three states, kept apart on purpose#

Capture is declarative. A route map says which endpoints are audited and which parts of the exchange each one keeps, so the keys you find in a record are a property of the declaration, not of the traffic:

python
# billing/apps.py
from model_audit import AuditRoute, Capture, audit_routes, subscribe_requests

audit_routes(
    {
        "POST,PUT,PATCH,DELETE /api/invoices/**": AuditRoute(
            Capture.QUERY | Capture.BODY | Capture.HEADERS,
            label="invoice-writes",
        ),
        "POST,DELETE /api/sessions": AuditRoute(Capture.HEADERS, label="sign-in"),
    }
)

subscribe_requests(write)

The middleware builds the payload only for the flags the matched route carries, and stores it in the payload field of RequestRecord under the keys query, body, headers and response. You get three distinguishable states instead of one ambiguous one:

  • No body key. This route does not capture bodies. The sign-in route above keeps headers only; its records are silent about bodies by declaration, and that silence is visible in the shape of the record.
  • body is None. The route captures bodies, the request carried one slot for it, and it was empty. This is the honest record of a request that sent nothing.
  • body is a note. Something was there and we did not keep it. The note says why.

Most loggers skip the third state, and it is the one that makes the other two readable.

The gate: content type, then size, then parse#

python
def _body(request: HttpRequest, redactor: Redactor) -> Any:
    content_type = (request.content_type or "").lower()
    if content_type.startswith(("multipart/form-data", "application/x-www-form-urlencoded")):
        return redactor.data(_pairs(request.POST))
    if "json" not in content_type:
        return {"omitted": "unsupported content type", "content_type": content_type}
    try:
        raw = request.body
    except (RawPostDataException, RequestDataTooBig, UnreadablePostError):
        return {"omitted": "body unavailable"}
    if not raw:
        return None
    if len(raw) > MAX_BODY_BYTES:
        return {"omitted": "body too large", "bytes": len(raw)}
    try:
        payload = loads(raw.decode(request.encoding or "utf-8"))
    except (JSONDecodeError, UnicodeDecodeError):
        return {"omitted": "unparsable body"}
    return redactor.data(payload)

The ordering is the design. Form payloads come first, through request.POST and _pairs, because Django has already parsed them and reading them again costs nothing, and because request.POST carries the form fields, so uploaded file content never enters the payload on the way past.

Anything else that is not JSON returns a note before request.body is touched. That matters twice over. Reading a body is not free and not always safe: it pulls the bytes into memory, and on a request a view intends to stream it changes what the view can do. An audit logger that interferes with the handler it is observing is a worse problem than a missing field, so we decline on the cheapest available signal. The note carries the content_type it declined, which turns a dead end into information. If you see application/octet-stream, you know a binary upload went through that endpoint, which is a different fact from an empty body and a different fact again from a JSON document we lost.

Then the exceptions. RawPostDataException means something ahead of us already consumed the stream; RequestDataTooBig means Django's own upload limits refused it before we got there; UnreadablePostError means the client went away mid-upload. All three are states of the world, not bugs in the caller, and all three get recorded as body unavailable rather than raised. The same principle runs through the library's actor resolution, where an unauthenticated request resolves to ANONYMOUS and an absent one to UNKNOWN: a missing fact is recorded, not thrown. An audit logger that can break a request is an audit logger teams eventually switch off.

Only after all of that does an empty body become None, and only there, where emptiness is the truth.

The byte count is the part that earns its place#

MAX_BODY_BYTES is 64 KB, and a body over it is recorded as {"omitted": "body too large", "bytes": len(raw)}.

The cap is a policy decision, not a technical ceiling. Audit records go to a log sink, and a trail that occasionally carries multi-megabyte documents stops being an index of what happened and starts being an unindexed copy of production data, with the retention and redaction obligations that implies. The byte count is what keeps the decision reviewable. A note that says only "too large" tells you your trail has a hole. A note that says the body was a few hundred kilobytes tells you the endpoint takes bulk payloads and the capture declaration for that route is wrong: it should keep query and headers and leave the body to the service that owns the document. Notes with fields are notes you can act on, and they aggregate, so a sink can count them per route instead of relying on anyone noticing.

The same reasoning explains unparsable body. We do not record bytes we could not decode, because the recursive redaction in Redactor.data only works on parsed structures. It walks mappings and sequences and replaces sensitive entries with REDACTED. Unparsed bytes cannot be walked, so keeping them would mean writing unredacted material into a log under the impression that redaction was applied. Refusing, and saying so, is the only version of that branch we would ship.

Responses take the same ladder, one branch earlier#

_response_body begins with the case the request side does not have:

python
if getattr(response, "streaming", False):
    return {"omitted": "streaming response"}

A streaming response has no content to read without consuming it on the client's behalf, so there is no version of capturing it that leaves the response intact. After that the ladder repeats: non-JSON content types, a size check against the same cap reported as response too large with its byte count, and parse failures as unparsable response. One reason vocabulary across both directions means you learn it once.

What it costs to run#

For a matched route the cost is a route-map lookup, up to 64 KB decoded and parsed, and one recursive redaction walk over the result. For a path the map does not match it is the lookup and nothing else, and the resolution ladder behind that lookup (most literal segments first, then an explicit method list, then patterns without a trailing wildcard) is something we have written about separately. Notes themselves are a dict with two keys, and the branch that produces them is strictly cheaper than the branch that keeps a payload.

The library records and never stores: subscribe_requests hands each RequestRecord to receivers that write wherever the project already keeps its logs. That division is what makes the notes worth the trouble. We do not know what sink a record lands in or who queries it, so the record has to be self-describing for a reader who has the file and not the code.

The source is at https://github.com/shipmindlabs/model-audit.

An audit trail is probably never complete, and pretending otherwise is how it loses its authority. What we can guarantee is that its gaps are labelled, that every slot a body would have filled either holds the body, states that there was none, or names the reason it is not there. The gaps are part of the trail, and they should read like it.

Was this useful?

Building something similar?

or email hello@shipmindlabs.com