Field Notes
- · 9 min readNormalize a one-time code inside its digest, not in the handler
A one-time code leaves the server as six characters and comes back as whatever the person retyped: the hyphen a message template inserted, a trailing space from a paste, lowercase letters from a…
- · permalink
We took over an older company site, and during the handover we put test data into its contact form. The page thanked us. Nothing turned up in the inbox.
The form was posting fine. The thank-you message was written straight into the page, so it showed up whether or not the message was ever delivered. Outbound mail had broken somewhere behind it, and nobody could say when. There were no errors to find because nothing was erroring: the site was quietly taking enquiries and dropping them.
The cost was the part you can't get back. We had no way to tell the owner how many people had tried to reach them.
Two things are standard for us now. The success message only appears after the backend confirms the message was actually handed off. And we alert on silence, so if a form that normally gets enquiries gets none for several days, someone gets pinged. Most monitoring watches for errors, and the worst failure we've seen on a website produced none.
If you own a site, the five-minute version is this: send yourself a message through your own contact form today, and check that it lands in an inbox a human actually reads.
Probably worth knowing when you last tested yours.
- · permalink
We spent a full day on a catalog parser that broke "randomly": half the runs came out clean, half garbage. The retailer was A/B testing a new layout, so we'd been scraping two different sites without knowing it.
Now every parse failure saves the raw page before it alerts. Cheapest debugging habit we have. You probably have one log line of your own that you'd never ship without.
- · permalink
The open-hours filter used to run after the ranking. A closed hub would win on priority, get discarded, and the caller read an empty result as nobody serving the address.
Now the filter goes first, so the ranking only ever sees open zones.
- · permalink
Founders hire for product work. Most of the backlog turns out to be integration work.
A feature takes a week. Then the provider's sandbox doesn't match prod, webhooks arrive twice, their statuses don't map to yours, and that's where the months go. So we stopped asking candidates what they've built, and started asking what someone else's API did to them. We want to hear which integration cost you the most.
- · permalink
A recurring payment notification failed its signature check, and nothing in the flat fields was wrong. The difference was one element that contained other elements.
This came out of bluemedia, our open-source client for a payment gateway's ITN notifications. Most of that notification is flat: a tag, some text, sign it in the declared order. recurringData is not flat. It holds a recurring action and an optional client hash.
Our first instinct was to treat it as a special case, one branch in the parser and another in the signer. That is where the bug lived. Two branches drift.
What we do now is model the nested element as a document of its own. The recurring data is a full model with its own field order, and the notification just declares a nested field that remembers the specification name and the model class and defaults to None.
Parsing became a single walk over the children. An undeclared tag is an error. A repeated tag is an error. A child that itself holds elements is an error unless its declaration names a model, and when it does, the value is that model parsed from the child. Empty text counts as absent. A required element that never arrives raises before anything downstream can read a half-built object.
Signing follows the same idea instead of a parallel one. The nested model contributes its ordered values as a list under its specification name, and flattening expands that list exactly where recurringData sits in the notification's field order, at the end. So the recurring action and the client hash sign one after the other, in the only place they could.
The payoff is the quiet case. A notification with no recurring data has None there, nothing expands, and the string being hashed is identical to what it was before the feature existed. Writing back works the same way: the nested element gets appended as an element instead of having text set on it.
The general lesson we carry into other integrations: when a field's shape is different, that is probably a signal to make a smaller model, not a bigger branch.
Where you draw that line is the interesting part, the point at which a nested structure earns its own model instead of a special case in the code that reads it.
- · permalink
A deletion should not be a log line saying something is gone. In our audit recorder it is the full inventory of what was lost.
The Django adapter treats a delete as the mirror of a create. A creation is a diff from nothing to a snapshot. A deletion is a diff from a snapshot to nothing. We pass the stored snapshot and an empty mapping into the same diff function, so every audited field arrives with its old value on one side and MISSING on the other, and each change reports itself as a removal. No special-case branch, and no second code path to keep in sync with the first.
Three details mattered more than the diff itself.
The primary key has to be read while the signal still has it. The delete collector clears it once the delete completes, so the record captures the pk at post_delete time. A deletion entry you cannot join back to the row it describes is just noise.
Snapshotting must not issue queries. We walk the model's concrete fields, skip the deferred ones, and read relations by their attribute name instead of touching the related object. An audit layer that lazily loads a foreign key during teardown quietly turns one delete into N queries.
Unregistered models emit nothing. The handler looks up the sender in the registry first and returns silently if the model is not in it. Audit coverage is opt-in, not ambient.
After that, the ordering is the boring part done deliberately: redaction withholds sensitive values before anything leaves the recorder, and the actor is resolved exactly as it is on a save, so a delete row is attributable on the same terms as an update. All three signals attach and detach under one dispatch id, which is what makes unregister symmetric with register rather than a partial cleanup.
This came out of our open-source project model-audit, where the rule is that a destructive action should produce the most detailed record in the system, not the least.
If you keep audit logs, the check worth running is whether a deleted record still tells you what it contained, or only that it once existed.
- · 9 min readAn unreadable store is not an empty queue
A queue that persists every change before enqueue resolves is at its most dangerous on the first write after a read that failed. Take a failed read for "the device holds nothing" and the next write…
- · permalink
We once shipped a marketplace for high-ticket asset deals, and in review it looked genuinely good: wide hero photos, clean spec blocks, everything aligned.
Then real listings started arriving. Many came in with no photos yet, specs half-filled, documents still sitting in compliance checks. We had designed the page around a complete, flattering record, so anything partial turned into empty space and ragged cards. Sellers looked at their own listing, decided it looked half-dead, and went back to sending details by email, which is exactly the behaviour the platform existed to replace.
Nothing was technically broken. The design was quietly telling people their listing wasn't ready. Fixing that had little to do with visuals. We had to decide what the page does when data is incomplete: show what's verified, show what's still in progress, and let a gap read as a step instead of a hole.
So now, before anything gets built, we pull the ugliest real record we have and look at the design with that one loaded. If it survives that, it will probably survive the showcase version.
You can run the same check on your own product, on its emptiest and least complete record.
- · permalink
We no longer ask candidates about the worst bug they ever fixed. We ask how they noticed it was happening at all.
The fix story is rehearsed, the detection story isn't. People who have actually run a system talk about thresholds, false positives, and who got paged. Everyone else says a user reported it. You probably know already which of those two answers you'd give.
- · permalink
Two writers, one table model: the column shape is settled before a single row exists.
finmodel is our open-source project, and its render module exposes exactly one table model, which every writer reads and nothing else. Markdown and CSV are not two formatters with their own opinions about amounts. They are two printers for the same frames.
The order is the whole design.
tabulate takes a statement plus either a results object or a mapping, and the columns get decided first.
Results becomes one frame per period, and each column is named by the period id.
A mapping gets inspected value by value. If every value is itself a mapping, its keys become the columns in the order given, where a period key turns into its id and anything else into a string. If no value is a mapping, the whole thing is one frame under a single column named Amount.
If only some values are mappings, we refuse it. Columns must be either all periods or all amounts. That mixture has no honest shape, so it raises rather than guessing. Anything that is neither results nor mapping gets refused by its type name.
Only then does the statement walk its nodes in declaration order. A block becomes a row whose amounts are None in every column. An input line carries its unit. Every other line is computed, with unit left None. Each cell refuses a frame that is missing that line id, and otherwise hands the value off to be converted.
Then comes the part that pays for all of it. The row turns None into empty text and a non-answer into its reason. So Markdown and CSV emit the same strings, and one pads and escapes them while the other prefixes id, label, depth and unit. There is no second definition of "empty" and no second explanation of why a cell has no number.
The business version of this: in a reporting product, every new export format is usually a new place for the numbers to disagree. Adding tests to each exporter treats the symptom. The structural fix, we think, is making exporters incapable of holding an opinion in the first place.
If your product ships more than one export, something in it already decides what a blank cell means, and that component is worth finding.
- · permalink
A support agent sent us a screenshot of an order with two buttons that did nothing.
Clicking either gave back an error saying the transition was not allowed. The buttons came from one list of legal moves, and the error came from the code that actually performs them. Two lists, same intent, drifting apart slowly.
So we rebuilt the listing in our open-source project order-lifecycle so it asks the runtime's question instead of its own. A row survives the listing when its source matches the current state and it allows the given role and context, and allowing is exactly two things: the role is permitted, and the conditions hold. That is the same pair the machine consults before it moves anything. One question, two callers.
Then the uglier half. An admin screen asked "what is possible from here" before any order was loaded, and a condition predicate went looking for a field on nothing. A stack trace, out of a read-only listing.
The rule we landed on: None means absent, never unfiltered.
A missing role fails any row that declares roles. A missing context fails a condition before the predicate ever runs, and the failure carries the explanation of every condition we checked, so we can report all of them instead of just the first. The conditional row simply drops out of the listing. A row guarded only by "always" still lists without a context. An unguarded row still lists without a role.
Refusal then follows the same order the filter does. Unknown transition first, and that error carries the allowed moves computed for the exact role and context the caller passed, so the suggestion cannot contradict the question. Then role. Then conditions.
The payoff is small and daily: a predicate can index straight into the order it is handed. No defensive None checks smeared through business rules.
You probably already know where your permission checks and your "what can we do here" screens get their answers from: one source, or two that agree most of the time.
- · permalink
A founder asked us for one strong backend engineer, available yesterday. Three calls in, nobody on his side could say which of his two services owned a user's balance. He did not need hands. He needed someone allowed to decide that and write it down.
So now we ask it early: who settles it when two services disagree.
- · 7 min readAn outbox ticket is a handle, not a promise of delivery
A notification triggered by a web request should not make that request sit waiting on a provider. Deferring the send does not make it safe, though. The template can still be missing a value, the…
- · permalink
Validate money where you construct it, not at the limit check. Money refuses non-Decimal and non-finite amounts, so NaN never reaches a comparison. Non-positive holds get refused before a transaction exists: nothing recorded, and the idempotency key stays free for the retry.
- · permalink
Most take-home tasks hire for the wrong half of the job. They test building something new, while the actual work is operating a system someone else wrote, live. Reading unfamiliar code, finding the one path that breaks under retries, deciding what not to touch: none of that shows up in a clean repo. We would rather watch someone debug than scaffold. Probably worth checking what your own interview measures.
- · permalink
For years we asked candidates to walk us through the system they were proudest of. The answers were good every time. Then one hire shipped something elegant, moved on, and we learned what it actually did the hard way.
Now we ask what broke in that system after they handed it off. The people who can answer stayed long enough to find out. We're not sure that's a fair thing to ask.
- · permalink
An overlap is not a separate problem from insufficient rest. It is rest equal to zero, measured negative.
We hit this while building shift-planner, our open-source rostering library. Two checks looked independent: shifts must not overlap, and consecutive shifts for one worker must leave a minimum gap. Two rules, two error types, two code paths. That framing is wrong, and it costs you in the places you do not look.
A roster carries its shifts plus a rest value, defaulting to zero. Construction refuses a negative rest outright, sorts the shifts by start, then end, then worker, and walks them once, keeping the latest shift per worker. For two consecutive shifts of one worker the order of checks is fixed: overlap gets tested first, and only if the two do not share time does the gap get compared against rest.
The part worth copying is the type relationship. Our overlap error subclasses the insufficient-rest error. A clash is the zero-rest case of a single rule, and it still keeps its own message built from the overlapping span. Both errors carry the same payload (first shift, second shift, required rest). Code that handles rest violations handles clashes for free. Code that cares about the difference still can. Nobody has to remember to catch both.
The second part is where the rule lives. Equality passes: exactly the required rest between one shift ending and the next starting is legal, and touching shifts stay legal only while rest is zero. And when you grow a roster, the add and extend operations hand the existing rest value to the new roster. A grown roster cannot shed its own rule, and the object you were holding keeps its own shifts unchanged.
The same field exists one level up, on a plan, with the same default and the same negative check. A plan builds its roster from its own shifts and its own rest. When a plan is applied to an existing roster, the result takes the maximum of the two rest values. Drawing a plan onto a roster can tighten the constraint. It cannot loosen it.
That asymmetry was a deliberate choice, and it is the one we would defend hardest. Monotonic constraints are probably the only kind you can reason about in a system that composes. If combining two valid objects could produce something weaker than either input, every guarantee becomes local, and you are back to reading the whole call graph to know what is enforced.
Most validation bugs we have chased in production were not wrong predicates. They were correct predicates that some path forgot to run, or two near-identical predicates that drifted apart over a year of edits. Collapsing a special case into its general rule removes that drift by construction.
Somewhere in your own domain there are two validations that turn out to be one rule and its boundary.
- · permalink
A website rarely looks cheap because of the design. It looks cheap because of what it does in the two seconds after you touch it.
The things customers register as cheap are almost all state and timing problems:
- a button with no loading state, so people tap it twice
- a form that wipes every field when one of them fails validation
- a delivery cost or total that changes between the cart and the card page
- images that arrive at different sizes and shift the page under your thumb
- error text that only says something went wrong
- a search box that returns nothing for a product that exists under a different spelling
- a confirmation message that lands an hour after the payment, or never
None of that is a taste decision. Each one decides who absorbs the failure: the browser, the backend, or the customer. When that call goes unmade, the customer absorbs it, and that feeling is the thing they call cheap.
It is also why a redesign often changes less than you expected. The new layout sits on top of the same empty states, the same silent timeouts, the same unlabelled errors.
If you want an honest read on your own site, stop checking it on a fast laptop. Open it on a weak mobile connection and try to fail on purpose: wrong card, wrong postcode, double tap, back button mid-payment. The gaps show up in about a minute.
We think most teams have a part of their checkout they have not deliberately tried to break.
- · permalink
A duplicate webhook should not reach your business rules at all. If one does, your usage limit will reject your own retry, and it will be correct to do that.
The failure is a quiet one. A promo code carries a per-customer limit of one. The payment succeeds, the redemption gets recorded, and then the payment provider redelivers the webhook, as providers do. The second call runs validation, validation sees one prior use by this customer, and the code is refused with customer_limit_reached. Nothing is broken. Every component did its job. Your customer still gets an error on a discount they already earned.
Same story with a validity window. The payment landed while the code was live, the redelivery lands after it closed, and the rule answers expired, which is a correct answer to the wrong question.
What fixes this is ordering, not more rules. In our open-source promo code library, redeem does the type checks first: a non-string idempotency key is a TypeError, an empty string is a ValueError, and the timestamp is settled from the caller or from current UTC. Then, before anything else, it asks the store whether this key has been seen. A stored reservation short-circuits the whole retry loop. The discount gets recomputed from the code, and a result comes back carrying the stored usage and a replayed flag. Validation does not run, and neither does the write. The rules never learn that a second delivery happened, because as far as the business is concerned, it didn't.
An unknown key returns nothing and the ordinary path proceeds: read usage, validate, then a compare-and-set write retried a bounded number of times. And the write itself still answers a known key with the replayed flag, which is the backstop for two deliveries that both miss the lookup and race into the same insert.
The principle generalises well beyond promo codes. Any rule that counts, expires, or rate-limits is a rule that can refuse a retry, and a retry is not a second request. Deduplication has to sit upstream of policy rather than inside it. Put it downstream and every correctness rule you add becomes a new way for at-least-once delivery to fail your customers.
Somewhere in your stack, a retry is probably still being treated as a fresh attempt.
- · 9 min readRoute maps for audit middleware: a broad False is a default, not a veto
Audit middleware tends to start as one if and end as a page of conditions: this prefix but not that one, write verbs only, skip the health check, keep the body here but not there. Every one of those…
- · permalink
When a batch job hits a file it cannot read, it has two options: end the run, or make the failure part of the output. We picked the second one, and that changed where the work went.
The problem is familiar. You point a tool at a directory, it processes two hundred files, and one of them is a half-written export with a broken byte sequence in the middle. If reading throws, you lose the other one hundred ninety-nine results. If you wrap the read in a try and move on, you get all your results and quietly lose the knowledge that anything was skipped. The second failure is worse, because nobody notices it.
In proseprobe, our open-source text-scoring project, the batch entry points walk a tree, keep what the filesystem confirms is a file, sort the result, and hand the list to one shared reader. The reader works out the document name before it tries to read anything: relative to the base directory when there is one, bare filename when the caller named individual paths. Then the read happens inside a try, and two exception types are caught together, the OS refusing the file and the decoder refusing its contents. Different causes, identical consequence: there is no text to score.
The caught failure gets appended as a record carrying the name, the path, and the text of the error. The loop continues. Only a successful read reaches the scorer.
The useful part is that the report then holds two lists side by side, scored documents and unread files. The headline states how many files could not be read. The closing notes name them as not scored. The machine-readable summary carries them as structured records, so a downstream job can see the gap instead of inferring it from a count that looks low. And when reports are written to disk, unreadable files get no report file of their own, but the summary file is still written. One more honest detail: the sibling function that scores prose already held in memory passes an empty unread list, because nothing was read and nothing could fail.
The trade-off is real. Partial success means callers have to check two fields instead of trusting one exit code, and a run full of unreadable files looks like a success until you read the headline. We took that over a pipeline that stops on file one hundred and forty-three.
If you run batch jobs in production, it is worth checking whether your failed items end up in the output artifact or only in the logs.
- · permalink
We burned most of a call once arguing whether the field should be status or state. We shipped status. Another service added state later anyway, so now both exist and both are load-bearing.
Naming fights get ten minutes and a coin flip now. The coin has a better track record than that call did. We'd be curious about your worst two-word debate.
- · permalink
A mixin that exists to narrow a queryset must not have a path where it hands back everything.
That sounds obvious until you sit down and write it. The natural shape is to look for the thing that defines the slice, and if it is not there, fall back to whatever the view already had. That fallback is the bug. It turns a forgotten declaration into a working endpoint that serves the full table.
In the DRF adapter of our open-source project role-scopes, the scoped queryset mixin deals with this by failing loudly at the wiring boundary.
The sequence is small. It calls the parent get_queryset to get the view's own queryset. Then it walks the view's permission instances in declared order and stops at the first one that is a scoped permission, handing that queryset to the permission along with the request and the view. If the walk finishes and no scoped permission was found, it raises ImproperlyConfigured and names the view class in the message: this view mixes in the scoped queryset mixin but declares no scoped permission, so there is nothing to narrow by.
Two details matter more than the raise itself.
First, non-scoped permissions are passed over rather than rejected. A view can list a throttle-style or authentication-style permission class next to a scoped one and still get narrowed. Being strict about a missing declaration is useful. Being strict about the presence of unrelated ones would probably just push teams into copy-pasted wrappers.
Second, the refusal lives only at the wiring boundary. Inside the scoping call, the ordinary answers are preserved. If there is no actor, or no resolvable permission for that actor, you get an empty queryset. Otherwise the declared slice is applied. Those outcomes depend on data, and empty is the correct response to them. A developer who forgot a line of configuration is a different category of event, and it deserves a different category of response.
The general rule we keep coming back to in authorization code: separate "this request legitimately sees nothing" from "nobody declared what this request is allowed to see." The first one is an answer. The second is a question that was never asked, and defaulting it to an open queryset is how a permission bug ships as a silent data leak (tests pass, the endpoint returns rows, and the review notices nothing because the missing line is missing).
Failing at import-adjacent wiring time means the forgotten declaration shows up the first time anyone touches the endpoint, including in CI, instead of in an access log six months later.
That is where we draw the line in our own permission layers: a missing scope declaration fails the request rather than falling back to a default. Where you draw it in your own code is a separate call.
- · permalink
A founder asked us for one backend developer for a couple of months. We said yes, staffed it that way, and then watched what the work actually turned into: the first weeks were almost entirely infrastructure and deployment setup, the middle stretch was getting a mobile release through store review, and only the tail end was the backend work he had originally described.
One person covered all of it. He could, technically. But he spent most of those weeks outside his strongest lane, so everything took longer and he carried a quiet kind of stress the budget had not accounted for. The founder paid for a backend engineer and got a slower version of three different specialists.
That one is on us for taking the request literally. A project does not need a developer, it needs whatever the current phase needs, and that changes every few weeks. So now we scope the way we would staff our own work: which roles does this stage actually need, and for how long. Infra early, QA before a release, backend when the feature work lands. Roles rotate in and out, the project keeps its context, and people are not pretending to be specialists in something they touch twice a year.
The side effect we did not expect is that it got easier to explain. "Two weeks of DevOps, then a mobile release cycle" is something a founder can sanity-check. "One senior developer" tells him very little about what he is getting.
If you have brought in outside help before, it is worth looking back at whether you scoped it by person or by the work. The second one probably aged better.
- · permalink
A candidate once spent most of the interview asking us questions: who gets paged when a payment retry loops, who owns a migration if they're out that week. We had just passed on people who answered everything cleanly and asked nothing back.
Now we read the questions as closely as the answers. Asking about handover is probably the cheapest signal that someone has actually run something in production, and a candidate's questions tell you plenty.
- · 10 min readThe payment means code decides which account rules fire
An invoice can pass every arithmetic rule in EN 16931 and still be a document nobody can pay. The totals reconcile, the VAT breakdown gets computed rather than summarised, the validator says yes,…
- · permalink
We spent two days chasing a bug we couldn't reproduce on staging. Staging was clean: tidy names, valid requisites, nothing half-finished. Production carried a decade of human improvisation in it. The bug lived in the data we never bothered to imitate, not in the code. So seeding ugly data is now part of setting up an environment, not a nice-to-have. You have probably met a worse row than ours in a production table.
- · permalink
We scoped a fiat-to-crypto onramp as a payments job. Rails, wallets, integrations. Then the KYC queue went live and the moderation screen turned out to be the product: the ops crew lived in it all day, and we had budgeted it as a throwaway admin panel.
Now we ask who sits in the thing for eight hours before we ask what it connects to. Worth knowing who the real daily user is on your side.
- · permalink
We emitted the payment means name as its own element, and the validator refused it. BT-82 is an attribute of PaymentMeansCode, not a child. In invoicerules now the name rides on the code, the due date is written once, and PayeeParty sits next to the seller, never instead of it.
- · permalink
The hosted payment page went out looking almost identical to the merchant's site: same fonts, same spacing, same quiet palette. In design review someone pointed at the little row of card and security marks near the bottom and called it visual noise. We agreed. On a screen that clean it was the one ugly thing, so out it went.
Then people started dropping at the card step. No failed payments, no errors, just customers who reached the form and walked away from it.
Clicking through the flow on a plain phone made it obvious. The page sat on the merchant's domain, wore the merchant's brand, and said nothing about who was actually going to receive the card number. We had deleted the only element that answered the question every buyer asks silently right there: is it safe to type this here.
The fix was small. The cost was the weeks before it, with the client reasonably convinced their payment provider was the problem.
Since then we treat the trust row on a payment screen the way we treat the amount: content, not decoration. If a review wants it gone for being unattractive, fine, but it has to come back as something that answers the same question, not as emptier space.
You probably have your own version of this, something you pulled off a page for looking cluttered and quietly missed afterwards.
- · permalink
A reviewer sent us a screenshot of two numbers that could not both be true.
The score said the text leaned one way. The explanation underneath listed the contributing features, and the percentages next to them added up to a different story: one feature was credited with more influence than the whole measurement had moved.
Nobody had typed a wrong number. The explanation layer had recomputed influence on its own. It read the same features, applied its own weighting, and produced a second opinion that happened to live in the same response as the first one. Two code paths, two answers, one UI.
That cost us a day of bisecting the wrong file. We kept reading the scale, because that is where measurement lives, and the scale was fine.
Here is what we do differently now in our open-source project proseprobe: the explanation layer is deliberately arithmetic-free. It is allowed to copy and to divide, and nothing else.
The headline is the score's own summary. The notes are the score's own notes. The features arrive in the order the scale put them, largest shift first, and every field travels through untouched: name, value, unit, description, the two means, the shift, the direction.
The only numbers the explainer forms are derived. Total movement is the sum of the absolute shifts. A feature's share is its shift over that total, and zero when the total is zero. Nothing is weighted twice.
Even the wording of magnitude is a lookup rather than a judgement. Four named bands (negligible, slight, moderate, strong) whose floors and ceilings tile the whole range up to infinity, and the first band the size fits under wins. No thresholds scattered across call sites.
Quoting is the part that surprised us. The explainer walks the evidence, collapses whitespace, takes the first non-empty quote and shortens it on a word boundary past the halfway mark. When there is no span, it does not reach for nearby prose to look complete. It says there is no excerpt. An empty quote is information, while a plausible wrong quote is a bug report six weeks later.
The general rule we took out of it: an explanation is a projection of a decision, not a second decision. The moment it can compute an opinion, it can disagree, and when it does, your users have to guess which half of your own output to trust.
So if you ship model or scoring output with reasons attached, it is probably worth tracing where those reasons get their numbers from: the decision itself, or a recalculation sitting next to it.
- · permalink
A guard that can raise a score is not really a guard. It is a second scorer wearing a disguise.
In proseprobe, our open-source project, we have a non-native English guard. Its job is narrow: it reports how far the score should be trusted, not whether the text is good. We enforce that distinction in the wiring, not in a comment.
The guard splits the text into words and sentences, then walks a fixed tuple of signal rows. Every row carries a threshold, a saturation point, a weight, and minimum word and sentence counts.
That minimum is the part most teams skip. A signal whose text is too short does not get measured and scored as zero: it gets deferred by name, with a note saying the text is shorter than the signal needs. An unmeasurable signal should leave a trace instead of a number.
The signals that do run become readings, and their strength is a clamped ramp from threshold to saturation. Quotes attach as evidence only when strength sits above zero, so you never see a highlighted phrase that contributed nothing. The weighted sum over total weight lands on a caution band: none, slight, substantial, severe.
Then calibration consumes that one number in exactly one direction. Damping is one minus the guard's strength. It multiplies the mean evidence, scales every contribution's logit shift, and widens the interval. The point value gets pulled toward the 0.5 prior and the interval grows. Nothing in that path can push a score up.
The default matters too. Score a set of features without a guard and it falls back to an explicit unexamined value: strength zero, with a note stating the guard did not run. Absent and clean are not the same thing.
The general rule we keep returning to is that confidence modifiers should probably be monotonic. If a trust input can both tighten and loosen a verdict, you can no longer say which component produced the answer.
We are still working out where the line sits between a signal that scores and a signal that only moderates.
- · 9 min readWhy a quarterly margin cannot be a sum of monthly margins
A financial model that rolls up correctly for revenue rolls up wrongly for everything derived from it. The quarter column in a spreadsheet holds whatever a person typed into it, usually a sum. A sum…
- · permalink
Someone pasted two sentences into an early build of our text checker and got a verdict back. They acted on it.
The verdict was not wrong because the math was wrong. It was wrong because there was nothing to measure. Most of our features need a minimum number of words or sentences before their statistics mean anything, and on two sentences almost none of them qualified. The build reduced that to a label anyway.
So we deleted the label. In proseprobe, our open-source project, the calibrated scale now has exactly one exit, and it carries no boolean field anywhere: a point value, a low and a high bound, the confidence that produced them, the band the value falls in, every band the interval touches, the per-feature contributions, and notes.
The gates are explicit. Any feature whose word or sentence minimum the text does not reach is skipped, and the notes name each skipped feature. If nothing survives the gates, the function returns the prior: value 0.5 across an interval of 0.0 to 1.0, with every band marked as spanned and no contributions at all. You cannot round that shape into anything. Nobody looks at it and sees an answer.
Two more things we would not ship without now.
The interval is not decoration. It comes from the requested confidence, defaults to 0.95, folds in an irreducible spread we refuse to go below, and widens further under 250 words. When the interval touches more than two bands, the result says in writing that the point value is indicative only.
And every surviving feature reports its own signed shift, sorted largest first, with the shifts summing to the log-odds behind the value. Each feature's evidence is clamped to a ceiling, so one strange outlier cannot carry the whole score. The practical effect is that an argument about a result becomes an argument about one feature, and that is a conversation you can actually finish.
The product lesson came cheaper than the incident: if the data cannot support a yes or no, the API should not offer a field where a yes or no could live. Developers will consume whatever is cheapest to consume.
We would like to hear where you have taken a yes/no out of an API and found the honest version was the more useful one.
- · permalink
One field in the model. Two separate elements in the output. And a warning for the case where there is nowhere to put it at all.
That is the shape of a due date in e-invoicing, and it is a good example of a mapping problem that looks trivial right up until you write it down.
The business term is a single date: when payment is expected. In our open-source project invoicerules it is one optional model field on the invoice. The output format, though, does not agree with itself about where that date lives. On an invoice it gets its own top-level element. On a credit note that element does not exist, so the date can only ride inside a payment instruction block.
So the same field has two destinations depending on the document type, and one of those destinations depends on other data being there.
The naive version is a branch at write time: if credit note, write here, else write there. That branch then gets copied into every place that touches the date: serialisation, validation, error paths.
We pushed it down instead. The syntax description for each document type carries an optional element name for the due date. The invoice syntax fills it in. The credit note syntax leaves it blank on purpose, and the path builder turns that absence into the nested payment-instruction path. The result is that the path lookup returns somewhere to write for both document types, and nothing upstream has to know which case it is in.
The second half is where we had to be honest. On a credit note the nesting target may not exist, and with no payment instructions the date has no container. That is not invalid. The document still passes. But the date gets silently dropped on the way out.
So that case warns rather than fails. Validation reports three different things with three different weights. A due date that is not a date fails and stops further checks. A due date before the issue date fails. A due date with nowhere to land warns, and says why.
The principle we keep coming back to is that a validator should distinguish between wrong, impossible and lossy. Wrong is rejected. Impossible is rejected. Lossy is reported, because the sender is the only one who can decide whether losing that value matters, and they can only decide that if someone tells them.
One more detail that costs nothing and saves support time: every rule carries an identifier prefixed with the library's own name. A receiving system should not be able to mistake our check for a standards body's rule and quote it back at the sender as if it were official.
If you build validators, the line between a hard failure and a warning is worth drawing on purpose rather than by accident. Silent data loss is the hardest category to place, we think, and the one people most often collapse into one of the other two.
- · permalink
Code review is our best onboarding tool, and only incidentally a quality gate. A new engineer reads far more of the system in week one than they write. Review is where they pick up our boundaries, naming and failure modes without risking anything, so we treat what you hand a new hire first as a real decision.
- · permalink
A quarterly margin is not three monthly margins added together. You roll the inputs first, then re-evaluate: profit over the span, divided by revenue over the span. Balances do not sum either, you take the first opening and the last closing.
Your quarter column probably does not know the difference. https://shipmindlabs.com/c/5e9f14a2
- · permalink
A client once asked us a simple question about their platform: a user says they were charged twice, can you show what happened? We couldn't. The system knew the balance today, but not how it got there, because every payment had been written as an update over the previous number. The past was gone.
Reconstructing that history took far longer than building it properly would have, and parts of it were simply unrecoverable, so the dispute got settled on trust instead of records.
Now, on anything that touches money, we write every movement down as its own line (what happened, when, and why) and we never edit it afterwards. The balance is just the sum of those lines. It costs almost nothing to do on day one, and it is the one part you can't add back in later.
If your product moves money, you probably want to know whether you could answer that question for a payment from last March.
- · 8 min readNo Patch Is Coming for PHP 7.4, 8.0 or 8.1
On 21 October 2026 CISA published an alert on two remote code execution flaws in PHP core, CVE-2026-109618 and CVE-2026-109619, both rated CVSS 9.2, which let an unauthenticated attacker run code on…
- · 11 min readA hash-chained audit log should verify without its own library
A compliance record usually gets read by someone who has the file and not the code. An auditor, a lawyer, a counterparty's engineer, or one of us two years later gets an export and no running…
- · permalink
Someone forwarded us a link that "did nothing". We had logging around every link we routed, so we went looking for it, and found nothing at all. No acceptance, no refusal. Silence.
The link had been sent twice. The second copy landed inside our dedupe window and returned early, before any call-site logging ran. The one decision we most needed to see was the only one we never wrote down.
A week later, the opposite failure. Our reporting hook threw on a payload it did not expect, inside a Linking handler. A link that anyone on the internet can send turned into a crash in the app.
Both of those shaped how auditing works in deeplinkmap, our open-source deep link router.
One hook, configured on the router, not at each call site. Call sites forget. Constructors do not.
The decision comes first, the event second. Resolution is reached internally, stamped with the router's own clock, and only then handed to the hook. Routing does not depend on the logger.
The report call is wrapped and swallows anything the hook throws. If reporting fails, the event is lost and the link still opens. That trade is deliberate (we would rather lose a log line than crash on untrusted input).
Drops are events too. The intake layer reads the router's hook once when it is built, so a link dropped as a repeat is reported with the source it arrived through, cold start or warm resume, and lands in the same log as the accepted and refused ones.
And that source is recorded, not routed on. The moment delivery channel starts changing behaviour, you have two products to support instead of one.
With no hook configured, nothing is reported at all. That part should probably be a choice, not a default someone discovers during an incident.
Worth knowing where your own dropped requests end up, in the same log as the handled ones, or nowhere.
- · permalink
PHP 7.4, 8.0 and 8.1 get no patch for this week's unauthenticated RCE flaws. Not now, not later. There is nothing to put in a backlog here, the stack is dead.
Staying on PHP? Then move to a patched 8.2+, set session.upload_progress.enabled=0, and ask your host what they actually patched. Want off it? Write to us today with the service, and within a day you get the cost and the week count. We have done legacy rewrites before, and small services take days, not months.
- · permalink
No patch is coming for the PHP2Shell RCEs on PHP 7.4, 8.0 and 8.1. Not a bug, a dead stack.
Today: move to 8.2+, set session.upload_progress.enabled=0, ask your host what they actually patched.
Write to us today, in one day you get the cost and the week count to get off PHP.
- · permalink
If you are running PHP 7.4, 8.0 or 8.1 right now, no patch is coming. Not late, not waiting behind other work. Never.
On 21-23 October 2026, two critical remote code execution flaws landed in PHP core: CVE-2026-109618 and CVE-2026-109619, CVSS 9.2 each. Unauthenticated. Any web-facing PHP runtime counts, so mod_php, PHP-FPM, php-cgi, FrankenPHP, LiteSpeed. CISA and CERT-EU both say patch immediately, and exploit attempts have already been reported.
Emergency fixes exist for 8.5.12, 8.4.27, 8.3.36 and 8.2.35. That is the whole list. A third flaw, CVE-2026-115560, was disclosed on 23 October with no patch at all, and the only mitigation today is session.upload_progress.enabled=0 in php.ini.
Teams keep misreading this part. An unpatched version is a maintenance problem. An unpatchable version is an architecture decision that someone already made for you, two or three years ago, by not moving.
Three things to do today if you stay on PHP:
- Move to a patched 8.2+ build.
- Set session.upload_progress.enabled=0.
- Ask your hosting vendor what they have actually patched, not what their status page says, what the running binary reports.
Then the harder one. If the only way forward is a jump anyway, you get to pick whether it is a version jump or a stack jump.
We have done legacy rewrites before: monolith-to-microservices on inherited systems, runtime migrations, systems with money and compliance on the line. Send us the service today and within one day we tell you what moving it off PHP onto a supported, modern stack costs and how many weeks it takes. Small services come out in days, not months. Large ones we scope honestly, including the parts we would probably leave alone.
Message us with the service and we will come back with numbers.
If you are on 7.4, 8.0 or 8.1 today, tell us what is actually blocking the move. Our experience says it is rarely the code.
- · permalink
Continuity is a review habit more than a headcount number. Cross code review means two people have read a service before it breaks. Rotating who answers alerts surfaces the undocumented step while its author is still around to ask. Add a second engineer and your bus factor is still one if only one of them has ever deployed.
We keep asking what the real bus factor is on the service that touches money.
- · permalink
An audit log you can only verify with our library is a dependency, not a record. Months later the reader has the file, not the code. So the export is the same canonical bytes we hashed, and one entry checks out with a JSON parser and SHA-256. Nothing stops an edit to that file. The design only makes the edit show. https://shipmindlabs.com/c/8d437c2d
- · permalink
A while back we brought on an engineer who gave the cleanest technical interview we'd seen that season: fast reasoning, knew payment flows cold, no bluffing. Three months in they took a week off, and the rest of us couldn't move their work forward. Not because it was bad. Because the only place it was explained was in their head.
We spent days reverse-engineering decisions that would have taken ten minutes to explain, and we had an awkward conversation with a client who only wanted a date.
So we added a last step to our interviews, and it's probably the one we'd suggest you copy. After the technical part, we ask the candidate to hand their solution over: five minutes, explain it to the person who'll own it, as if you're going away tomorrow. We don't re-score the code. We listen for three things: what they chose not to do, where it will break first, and what we should check if it misbehaves on a quiet weekend.
It changed who we hire a little. It changed what we learned about people we were already going to hire a lot. Someone who can hand work over makes the whole team interruptible, and that's the real thing a client buys when they hire a team instead of one person.
It's worth naming the one job on your team you'd want someone else to be able to pick up next week.
- · permalink
A number your system can emit without declaring it first is a number nobody can audit later.
We run into this a lot in analytics and reporting work. A metric gets added in a hurry, the name is invented at the call site, the unit lives in someone's head, and six months later a dashboard shows a value nobody can explain. The code is correct. The output is unverifiable.
In our open-source text analysis project proseprobe, we made the registry the only door.
There is one table of declarations. Each entry is a frozen record holding the name, the unit, and a description of what the number means. Nothing else may declare a measurement.
When the extractor emits a reading, it goes through a single helper, and that helper looks the name up in the declaration index before it is allowed to construct anything. A name that is not declared does not produce an undocumented row. It fails. Every emitted value inherits its unit and description from the declaration, not from whoever wrote the line that computed it.
We care about that inversion more than anything else in the design. Metadata belongs to the schema, not to the caller.
Two decisions around it that we think generalise:
Every value carries evidence, a small tuple of collapsed excerpts from the source text, so a number points back at the prose it was read from. One ratio also carries a detail map with per-item rates per 1000 words. A metric you can click back into is a metric people trust.
Thresholds are named constants, not inline literals. The short-sentence and long-sentence boundaries have names. The expression that strips list and heading markers before punctuation is counted has a name. When someone argues about a result, the argument lands on a named decision instead of a magic number buried in a loop.
The last choice was the one we debated most: division is guarded, not refused. The per-100-words, per-word, per-sentence and per-line helpers return zero on an empty denominator. So empty input still yields the complete set of declared names, all at zero.
That is deliberate. A consumer should not have to handle a shrinking result shape. The schema is fixed by the declaration table, not by whether the input happened to contain data, so downstream code can assume the keys exist and stop writing defensive lookups.
Building this way costs almost nothing at the start, and retrofitting it hurts once dashboards, exports and client reports already depend on the loose version.
If you run a metrics pipeline, it is worth checking where your units actually live: in a schema, in a column name, or in a wiki page someone stopped updating.
- · 9 min readA vendor on two purposes does not run on a partial grant
A consent decision records a category, but the party receiving the request is a vendor, and one vendor often processes for several purposes. When a visitor grants one of those purposes and refuses…
- · permalink
We were reading a consent record that should not have existed anymore.
The decision in it was more than a year old. Our code knew it was stale — the caller got nothing back, the banner came up again, the user re-consented. Functionally correct. But the old record was still sitting in localStorage, untouched, because we had filtered it on the way out instead of removing it.
That gap cost us a conversation we did not want to have. When someone asks what you store and for how long, "we ignore it after thirteen months" is a weaker answer than "it is gone." The first one is a code path. The second one is a fact about the device.
So in consentcore, our open-source consent library, the expiry rule lives in the storage layer and both adapters route every stored record through it before a caller sees anything.
What we do differently now:
Expiry is computed from the decision's own timestamp, not from when we happened to look. Thirteen months is the default, overridable per store.
Month arithmetic takes the shorter window on purpose. A decision made on the 31st, landing in a month with 30 days, rolls back to the last day of that month rather than forward into the next one. When there is ambiguity about a boundary, we expire earlier, never later.
An unparseable timestamp evaluates to the epoch, so it counts as already over. A record we cannot read is not a record we trust.
Equality counts as past. The instant it expires, it is expired.
A hand-edited entry is refused whole, not half-read. Either the shape is right and the timestamp is still inside the window, or the entry is dropped entirely. We never reconstruct intent from a partially valid object.
And the part that actually closed the gap: read filters the log, writes the shortened log back, and removes the key outright when nothing survives. The cookie adapter does the same thing through its write path, and derives Max-Age from the expiry of the last surviving decision — so the browser drops the cookie on the same schedule our code does. Two clocks, one deadline.
The reason that last detail matters is that storage you only filter is storage you still hold. A reader that cleans up as it goes is the only version of this that stays true when nobody is looking at the code.
We have run compliance tooling in production for years — KYC flows, audit exports, retention rules — and this is the pattern that keeps coming back. Retention is not a check you perform. It is a state your storage is in.
If you have a retention window in your product: is it enforced where you read, where you write, or only in the policy document? I am curious which of the three is actually load-bearing for you.
- · permalink
We shipped once with the stub OTP sender still wired in. No errors, no codes, just messages piling into a list in memory. Now every stub send logs a WARNING, and require_real_sender refuses to boot it at all. That is why otpguard ships the guard with the stub.
- · permalink
We gave a new backend engineer a small fix on day one. It took far longer than the fix deserved, not because of the code, but because half the deploy steps lived in someone's head.
Our onboarding was at fault there, not the hire. Now every new engineer ships something tiny in their first week, and whatever blocks them goes straight into our backlog. Something probably broke the last time you onboarded someone too.
- · permalink
One candidate did badly on the coding exercise. Then we showed him a payment flow that was quietly charging some customers twice, and the first thing he asked was whether the retry came from our side or the provider's.
He was the only one that week who asked that. The others went straight to guessing at fixes.
An hour earlier we had almost passed on him, and the lesson stuck: we were testing how fast someone writes code, when the expensive skill is knowing which question to ask before touching anything. Now every interview includes a broken system and no explanation of why it is broken.
We would be glad to hear what you look for in an interview that a resume leaves out.
- · permalink
A vendor with two purposes, one of them granted, is not half-permitted. Send the request anyway and you are trusting that vendor to honor the half it never got, which hands the visitor's decision to the party it was made about. So it stays off for us, even for the purpose that was granted. We pay that cost willingly.
https://shipmindlabs.com/c/91156e90
- · permalink
A ticket came in saying our minus sign sat in the wrong place. Not missing. Wrong place.
Finance exported a report, and the negative amounts showed up wrapped in parentheses on one view and with a leading minus on another. Same number, same currency, two screens. Assuming a bug was reasonable.
It was not a bug. There were two different formatting paths, both of them ours, and each one made its own small decision about what a negative amount looks like.
That is when we stopped treating display as a cosmetic layer.
In our open-source amountfield project the display module is now the showing side of a value, and it has no reading side on purpose. You give it a money value and formatting options, and it formats. That is all it does.
What we pushed into it:
The locale decides the symbol, we do not. If you pass none, it falls back to a single default. Accounting style negatives happen only when the caller asks for accounting explicitly, and otherwise you get a standard minus. One switch, one owner.
The amount's own exponent wins. We pin minimum and maximum fraction digits to the exponent the value carries, instead of letting the formatter use whatever it keeps for that currency code. The value knows how precise it is, while the currency table knows only the general case.
Grouping separators are on by default, because a displayed amount has stopped moving. Typing a number and reading a number are different jobs.
Where the currency mark goes is not a table in our code. We build a formatter, format the number one, read the resulting parts, see where the currency sits relative to the integer, and report before or after, plus whatever literal sits between them, usually a no-break space rather than a plain one. We would rather ask the platform than keep a list we would probably get wrong for some locale we have never shipped to.
Formatters are memoized by the whole combination that produced them: locale, currency, display style, sign style, grouping, exponent. Building a formatter is not free, and this code runs in lists.
The rule that actually saved us is the boring one: display reads a decimal string rather than a float, and nothing takes its output back in. A formatted string offered to the field gets refused outright instead of half-parsed. Parentheses, no-break spaces and locale separators are not input. They are the end of the road.
The real cost of getting this wrong shows up as a support thread where two people look at the same balance and disagree about the sign, which is a good deal worse than a rendering glitch.
If you run a multi-currency product, the same choice lands on you: let the locale decide negatives, or force one house style everywhere.
- · 10 min readWhy a thousands separator is only stripped where the digits are grouped
Two people write the same amount as 1,234.56 and 1.234,56. A money input that assumes either convention reads the other one off by a factor of a hundred, and what lands in the request body looks…
- · permalink
If four workers can touch your queue at once, the write that lands last is not necessarily the newest one.
That is how an offline write queue quietly undoes work it had already recorded.
The setup comes from our open-source offlinequeue library. Pending operations get grouped into lanes keyed by entity, and up to four lanes run concurrently by default. Each lane persists twice per attempt: once when it marks a row in-flight with a start timestamp before the request leaves, and once when it records the outcome.
So at any moment several lanes may be holding a snapshot of the whole operation list and asking storage to save it.
Two overlapping saves is all it takes. Lane A reads the list, lane B reads it, B writes, then A writes, and A's older snapshot lands last, erasing an outcome that was already committed. Nothing errors. The queue just forgets.
The fix is not a lock in the adapter. It is a single serialized chain in the queue itself: a private writing promise, initialized to a resolved one. Persisting no longer awaits the save directly. It chains the save onto that promise, assigns the result back, and attaches a catch that swallows the rejection on the chain only, while the caller still gets the unswallowed promise.
Two consequences fall out of that, and both are deliberate.
Each save serializes the queue as it stands when its turn comes, not as it looked when the lane started. The snapshot is taken late, so there is no stale write left to land.
And a failed write still rejects for whoever asked for it, because enqueue and flush should not lose a change silently, but the next save is not held hostage behind a rejection it cannot act on. One failure is one caller's problem, not the end of the write path.
The part worth stealing is what this does to the extension point. Because the chain exists, the storage contract only ever sees one save in flight. An adapter author writing against AsyncStorage, IndexedDB, SQLite or a plain file does not have to reason about reentrancy or add a mutex, and does not have to wonder whether their driver is safe under concurrent calls.
We pushed the hard guarantee up into the library and handed the adapter a boring job. Boring is the feature: every adapter that follows is correct by default instead of correct if its author happened to think about it.
With concurrency, the real design question is usually about placement: where to draw the line so that everything downstream of it can stop caring, rather than how to make the parallel part safe.
If you have had a concurrency guarantee sitting in the wrong layer, you probably found out the way we did, through corrupted state or a contributor's bug report.
- · permalink
Developer life is mostly queue time, not typing time.
A 12-minute pipeline plus a half-day review wait turns a one-line fix into a two-day ticket, and the work itself is not hard. You start something else while you wait, then pay to reload both contexts. So we cut feedback latency before we cut anything else. Worth knowing which queue costs your team the most.
- · permalink
A deep link should have exactly one door.
If a cold start and a running app go through different handlers, you are keeping two sets of checks in sync, and they drift. One of them will eventually accept something the other refuses, and the bug report will say "only happens when the app was already open".
In our open-source project deeplinkmap, the intake class is that single door. Both paths end in the same method.
The cold path takes whatever the platform's initial-URL lookup resolved. If that value is null or undefined, intake returns null, because no link is not a delivery and it must not be reported as one. Anything else is forwarded with the source marked cold. The warm path calls the same method and the source defaults to warm. Same checks, same order, one difference: a label.
The order inside is fixed, and the order is the design.
Read the clock once. Expire remembered links first: anything whose elapsed time exceeds the dedupe window is forgotten, and so is anything with a negative elapsed time, because a clock that moved backwards must not pin a link in memory forever. Only then consult what has been seen.
If the link is still remembered, it comes back as a delivery with status duplicate, carrying both the url and the source, and it goes through the audit hook before it is dropped. That part matters to us more than the dedupe itself. A dropped repeat is recorded exactly like an acceptance or a refusal. Silent deduplication is indistinguishable from a broken router at 2am.
If it is new, the link is written down at the current time. The window is anchored to the first sighting, not to the latest repeat, so a stream of repeats cannot hold the door shut indefinitely. Then the router resolves it: matched with a match, or refused with a refusal.
Two deliberate refusals to be clever: links are compared as received, so two different URLs stay two links and no normalization quietly merges them. And the window is validated at construction, where not finite or below one millisecond throws immediately rather than degrading into dedupe that never triggers.
The default window is three seconds. Long enough to absorb the double-fire you get when the OS hands the same link to a resuming app, short enough that a user tapping the same notification again on purpose still gets through.
If you dedupe deep links, the anchor you picked is worth a second look (first sighting, or the latest repeat), and so is whether your logs can prove that a specific drop actually happened.
MobileDevelopment #SoftwareArchitecture#
- · permalink
A client messaged us on a Monday: sales were down. Nothing was broken. The site loaded, the catalog worked, the checkout button clicked through.
What had happened: a credential for one of their delivery integrations had quietly expired over the weekend. The page did not crash, it just returned no shipping rates, so the final step showed a blank box, and customers assumed the store could not ship to them and left. No error log, no alert, no angry email. Just a few days of people walking away.
It cost them a weekend of orders, and it cost us an uncomfortable conversation about why this went unnoticed. Fair question. We had been watching whether the server was up. We were not watching whether a customer could actually finish buying something.
So if you own a site, these are the five things we would check today, and most of them fail silently.
One, complete a real purchase or submit a real form yourself, end to end, today. Not the homepage, the last step.
Two, find out when your integration keys and certificates expire, and write the dates somewhere a human will see them.
Three, open your own site on a phone, on mobile data, not office wifi.
Four, check where form submissions actually land. We have seen contact forms mailing to an inbox that had sat unopened for a year.
Five, confirm someone gets told when something fails. Not a dashboard someone could look at, a message that arrives.
What we do differently now: each project we run gets a check that behaves like a customer, not like a server. If the purchase path stops working, we hear about it before the client does.
Probably worth asking yourself when you last bought something from your own site.
- · permalink
1,234.56 and 1.234,56 are the same amount. A parser guessing which one it holds hands you the other, off by a factor of a hundred.
No locale table, just an order of operations: split on the decimal point first, then treat a group separator as decoration only where digits are grouped. A dot before two digits in a German field is a decimal typed with the wrong key, so we refuse it rather than multiply by 100. One refused pattern, no silent hundredfold.
https://shipmindlabs.com/c/58a55254
- · 9 min readOne table, one cancel: why each actor needs its own trigger
A lifecycle table indexed by (state, trigger) answers "what happens now" with a dictionary lookup instead of a walk through handlers. What you pay for that index is a vocabulary problem: two actors…
- · permalink
The field refused an amount a tester had copied out of a finance spreadsheet into EUR. On screen the value looked perfect.
The minus was U+2212, the real typographic minus, and not the ASCII hyphen our parser expected. A spreadsheet writes it. So does a Japanese keyboard.
We fixed that one character and then got a second report, this time from a right-to-left locale: a value copied straight back out of our own formatted output failed too. Intl.NumberFormat writes bidi marks around a number in RTL text. They are invisible. They were in the clipboard.
Two reports, same shape. The paste boundary was doing nothing, so the parser kept getting handed text no human could see was dirty.
What we do now, in our open-source amountfield: there is exactly one place where a raw string becomes a candidate number, and it removes exactly four things.
Bidi marks go, because our own renderer put them there.
Unicode minus and dash characters are folded into an ASCII hyphen, including the case where the locale's own minus is not a hyphen either.
The field's own currency marks go. We ask Intl for symbol, narrowSymbol and code, keep every part the formatter labels as currency, and sort them longest first, so stripping CFA cannot leave a stray F standing in the number. That list is memoized per locale and currency code, because it is the same answer on every keystroke.
Leading whitespace goes, since a space in front of digits reads as a group separator with nothing before it.
Everything else survives. A space inside the number survives, because that is grouping and not noise. The locale's own digits survive, because parsing already reads them.
What took us longest to agree on was the part we refuse. We do not read the currency out of the text. Paste a dollar amount into a EUR field and the dollar sign stays in the string and comes back as a not-a-number failure. The other option is a field that quietly decides 1,200 dollars were 1,200 euros, and nobody finds out until reconciliation.
The caret has to survive all of this too, so cleaning also returns a mapping of the old caret position onto the cleaned text. Otherwise a paste that fixes your number moves your cursor. And a parse failure mid-typing is marked incomplete rather than invalid, because a trailing separator is unfinished and not wrong.
Clean the characters you authored. Refuse the ambiguity you did not. That same line sits somewhere in your own input handling, either repairing the pasted value or handing it back, and we think it pays to know which one you picked.
- · permalink
The design review for that compliance screen went well. Clean layout, generous spacing, document preview in a modal, applicant details in tidy collapsible sections. We were proud of it.
Then we sat down next to the people who actually use it. A moderator opens case after case for a whole shift, the same decision over and over again. Our beautiful screen made each one cost a click to open the document, a scroll to compare the name on the ID with the name in the form, then another click to close the modal again. Multiply that by a shift and the queue stops moving. Within days the team had a second window open for the parts that were simply faster elsewhere.
What it cost us: rebuilding the screen we liked most, and learning that nobody had ever told us how many times a day it would be opened.
So now we ask that question before we call any internal screen finished. How many times will one person do this today? Then we design for that number, with everything needed for one decision visible at once, a keyboard path for the common case, nothing sitting between a reviewer and the thing they are verifying. And we run a batch of real cases ourselves, with a timer, before we hand it over.
Beauty is aimed at the visitor who comes once. The person who lives in a screen all day is paying for something different.
You probably have a screen in your business that someone opens a hundred times a day.
- · permalink
A refund is a type code, not a minus sign. One UBL mapping in invoicerules emits both Invoice and CreditNote, and the UNCL1001 code picks root, namespace and line elements. Parties, taxes and totals stay identical. A credit note has no dueDate, so BT-9 moves to PaymentMeans.
- · permalink
A rate limiter that refreshes its TTL on every attempt limits nothing. It just punishes whoever stops knocking.
That is the quiet failure mode hiding inside "5 attempts per 15 minutes". If each increment re-applies the lifetime, the window slides forward with the attacker. Keep knocking once every 14 minutes and the counter never expires, while knocking fast enough locks out a well-behaved user who typo'd their code twice, long after the real window should have closed. Same key, same code path, opposite outcomes depending on traffic shape.
In otpguard, our open-source OTP library, we pinned the window to its first increment, and we pushed that rule down into the storage layer instead of the policy layer. The storage contract is a small runtime-checkable protocol: get, set, incr, delete, each accepting an optional ttl. The invariant fits in one sentence: incr attaches a lifetime only when it creates the counter.
The mechanics differ per adapter, which is why the invariant has to be stated rather than assumed.
The in-memory adapter computes the deadline first, then reads the key. Absent, the counter starts at 1 with that deadline. Present, the counter becomes current plus one and the expiry comes from the existing entry, so the new deadline is discarded. Later increments cannot push the window out, because the only value they can write is the one already sitting there.
The Redis adapter cannot compute first. It calls incr, then calls pexpire only when the returned counter equals 1. Redis tells you whether you created the key by what it hands back, and that return value is the entire branch condition.
Two details we would not skip again.
A non-positive lifetime is rejected, not normalized. A ttl of zero or negative is a bug in the calling policy, and silently treating it as "no expiry" turns a rate limit into a permanent lockout. It raises. Milliseconds round up rather than down, so a sub-millisecond window never collapses to zero.
And a counter that does not parse as an integer raises instead of resetting to zero. That one is tempting to get wrong, since resetting looks like graceful degradation. In practice it hands a free attempt-counter reset to anyone who can get a bad value into that key. Corrupt state should stop the request, not forgive it.
So "expire after 15 minutes" works as a behavioural guarantee rather than a configuration value, and it lives or dies in whichever line of the adapter decides when to set the TTL. Policy code reads the same either way.
If you run attempt limits in production, it is worth knowing whether your window is pinned to the first attempt or the last one, and whether that was a decision or just a default you inherited.
- · permalink
A founder came to us needing one senior backend engineer, urgently. Two weeks in, the code was not the holdup: nobody on their side could approve a schema change, so the branch just sat there.
We had added hands to a queue that was waiting on a decision, not on capacity. Now we ask who signs off before we ask what to build. You can probably name the spot where work actually stalls on your team.
- · 9 min readReverse charge needs a validated VAT identifier, nothing less
An invoice that shifts VAT to the buyer rests on one fact the seller cannot get from its own records: that the buyer is a taxable person in another member state. What the seller can see is what…
- · permalink
We wanted one test that proved one thing. When a status field goes from pending to settled, the changeset says exactly that.
Writing it took a Django test database, a migration run, two saved instances, and a fixture nobody wanted to own.
Then a payments integration asked for the same diff over something that was never a model, a webhook payload compared against the state we had stored for that transaction. Same question, same output shape. Our diff logic could not answer it, since it only knew how to walk a model instance.
So we cut it apart. The diff core in our open-source project model-audit now takes two plain mappings and returns a typed changeset: added, removed, changed, with old and new per field. There is no Django import anywhere in it. The framework layer's whole job is to produce those two mappings from model state and hand them over.
What that cost us, honestly: the boundary got harder. A deferred field is not the same as a field that is absent. A foreign key has to arrive as an id, not a lazy object. A None value and a missing key mean different things, and now something has to decide which one it is. All of that used to sit implicit inside the ORM walk. It is now an explicit decision at the edge, written down once, where someone can argue with it.
What we got back is the part we would defend in review: the diff tests are dictionaries in, assertions out. No database, no migrations, no app registry. And when another service needs the same changeset over an API response or a parsed file, it imports the core and skips the framework entirely.
The rule we kept: the framework layer feeds data in, it stays out of the comparison.
You probably draw that line somewhere else in your own code, and the thing worth naming is the last piece you would still let the framework reach into.
- · permalink
We stopped scoring candidates on how they would build a thing, and started scoring how they would hand it over. Build skill shows up within a week. The cost of someone who cannot explain state (why this retry is safe, what this flag protects) arrives months later, and whoever is on call pays it. Handover is what we are actually buying.
So we weigh the faster builder against the one whose work survives them being out.
- · permalink
A VAT number that doesn't validate is a no for us, not a softer yes.
Absent, unvalidated, or missing a member-state prefix: our rule treats all three the same, private buyer, seller charges VAT. Reverse charge applies only on a validated ID with a prefix. Get it wrong in that direction and you pay money you can reclaim, not a penalty.
https://shipmindlabs.com/c/a87e5f9a
- · permalink
A site rarely looks cheap because of the design. It looks cheap because of the states nobody designed.
A mockup shows one version of a page: content loaded, images in place, nothing failing. Production shows you the other versions. Text that arrives before the image and shoves the layout down. An empty list with no explanation of why it's empty. A form that rejects your input without saying which field. A button you press twice because nothing acknowledged the first press. Each of those is a small moment where the visitor stops trusting the thing, and trust is what people are reading when they say "this looks cheap."
The mechanics are simple. Every screen that loads data has at least four states: loading, empty, error, and loaded. Most projects only build the fourth one, because it's the only one sitting in the design file. The rest get improvised by whoever is implementing the page that afternoon. That's also why cheapness shows up unevenly. One page feels solid, the next one feels like a different company made it.
So we review those states explicitly: what does this page look like with no data, with slow data, with failed data, with text twice as long as the design assumed. It costs an hour per screen, and it's the difference between a site that feels built and a site that feels assembled.
If you're looking at your own site right now, try it on a bad connection and submit a form wrong on purpose. That's probably the version your visitors see more often than you'd think.
- · permalink
A custom POST action should not inherit the permission that guards create. In our open-source role-scopes it doesn't, and we built that refusal in on purpose; it is not a gap.
The permission class decides which Permission gates a request before it asks whether the actor holds it. That lookup runs in a fixed order.
First the viewset action name, checked against the class's own actions mapping. Then the HTTP method. An explicit entry under either one wins.
Only when neither is mapped does a fallback apply, and what the view claims to be decides which fallback you get. A viewset action consults VIEWSET_ACTIONS, which maps list, retrieve and create, and nothing else. A plain APIView consults METHOD_ACTIONS, where GET, HEAD and OPTIONS are view and POST is create.
update, destroy and custom actions are missing from the viewset table on purpose. So a custom POST route has no verb to fall back on. It stays unmapped, and unmapped means refused.
The denial is loud rather than silent. A name the matrix lacks resolves to None, and the gate turns None into a failure under a rule that says no permission is mapped for this action, naming the actor or ANONYMOUS. Only after that does it reject a request carrying no actor at all. The ordering matters: you learn the route was never wired before you learn you were not logged in. That message travels into the 403 body, so you read the real reason instead of a generic forbidden.
One more line of defence sits earlier than any request. Building the permission class refuses a resource outside the declared set with ValueError, and it resolves every mapped value through Permission() as the class is constructed. An undeclared capability fails at import, not on the first production call.
The trade-off is real and we chose it knowingly. Every custom action now costs one line in a mapping, and a forgotten line shows up as a 403 in review. The alternative, inheriting create from the verb, costs a quiet grant that nobody reviews at all.
When an endpoint in your system is mapped to nothing, it either denies or falls back to the method. We would rather you hit the deny.
- · 8 min readTest a delivery catalogue for the places no zone reaches
A delivery catalogue built from polygons answers one question well: which zones reach this address. That shape makes overlaps announce themselves and leaves gaps silent. Ask about a street covered…
- · permalink
Screening for stack knowledge catches the cheap gap. The expensive gap shows up when someone gets an underspecified ticket: they ask two questions first, or they confidently build the wrong thing for three days. So we hand over a deliberately vague task and read the questions instead of the code.
We'd like to hear which question you hope a candidate asks.
- · permalink
Our admin screen said the notifications had gone out. All that had really happened was that enqueue returned a ticket.
In notify-dispatch, our open-source notification library, the outbox is thin on purpose. Enqueue takes the recipient, the template and a copy of the context, stamps the time from the outbox clock, appends the ticket to an in-memory deque, and hands it back. Nothing is rendered. No dedup key is claimed. No provider is touched.
Everything that decides the outcome runs later, in drain, on whichever thread calls it, because drain just performs the ordinary send. The retry policy, the dead-letter queue, the dedup claim and the quiet-hours check all evaluate at that moment, not at enqueue.
We learned what that means the hard way, with a worker thread that wasn't running in one deployment. Tickets piled up, the screen kept reporting success, and when the process restarted the deque went with it. The queue is memory-only by design. We had been reading it as durable.
The second surprise came when we drained a backlog by hand. Quiet hours were read at the drain moment, so messages queued before the window were judged by the clock after it. Dedup keys were claimed in drain order, which is not necessarily the order anyone intended.
Three things changed.
A ticket is a handle. It is not a promise. Our interfaces now show ticket state (pending until the settle event fires, then sent or failed) instead of treating a returned ticket as delivery.
When the enqueue moment is the policy-relevant moment, we queue that timestamp explicitly, so drain doesn't re-decide it against a later clock.
Anything that must survive a restart is written to durable storage before it is enqueued. The receipt, or the dead letter, is the audit record. The enqueue call is not.
If you run something similar, it is worth knowing where your system draws the line between accepted and sent, and whether the caller is ever told which one it got.
- · permalink
If you don't want to hire a full-time team, you should be shopping for role coverage rather than for developers.
One contractor writing good code still leaves you holding QA, deployment, and the decisions nobody wrote down. Those don't disappear because you skipped the payroll, they land on you. What makes an outside team work is that backend, frontend, mobile, QA and DevOps are all somebody's job by default, and the work keeps moving when one person is out.
So when you compare options, ask who tests it, who deploys it, and who picks it up next month. If all three answers are the same single name, you haven't avoided hiring. You've postponed it.
- · permalink
Every lookup shows you where your coverage overlaps. Not one of them shows you where it doesn't.
The strip between two zones that almost meet stays invisible until an order lands on it. We sample a grid over the catalogue and ask the index about every point, and the misses that touch each other become one named gap. Spacing is the resolution of what you can see at all.
We'd want to hear your test for the places nobody queries. https://shipmindlabs.com/c/e1731228
- · permalink
An off switch that loses to the most specific rule is a default, not a veto. model-audit scores each route by the literal segments it matched, so /api/** False only sets the baseline. A deeper declaration overrides it. An unmatched path costs you one lookup, nothing else.
- · permalink
A read that threw is not an empty queue.
That one distinction keeps an offline-first app from erasing work it never saw. A store that answered with something unusable and a store that could not answer at all look identical at the call site, and treating them the same is how a user's pending writes disappear on the next enqueue.
In our open-source offlinequeue library the contract splits them deliberately.
The decode side is forgiving. The key-value and MMKV adapters read one key ("offlinequeue" unless a second argument names another) and hand the raw value straight to decode. An absent value, whether null, undefined or an empty string, decodes to an empty queue. A string JSON.parse cannot read decodes to an empty queue. A value that parses but is not an array decodes to an empty queue. A half-written record from a process killed mid-save does not brick the app, it starts clean.
The read side is not forgiving at all. A getItem or getString that throws is caught nowhere in the storage layer. It propagates. And load marks the instance loaded only after the awaited storage read has returned, so a failed read leaves the queue unloaded with its in-memory operations untouched.
That ordering is the whole guarantee. Every mutating entry point (enqueue, flush, retry, discard, prune, recover) begins by awaiting load, which retries the read while the instance is still unloaded and rethrows if it fails again. Persistence is reached only after that call returns. So a one-element queue can never be stringified over everything that survived the last run. No write happens until a read has succeeded.
The write side mirrors it. Save is a stringify of the whole array, and persistence chains each save behind the previous one, returning that save's own promise. The caller hears the rejection while the chain itself stays alive through a catch, so one failed write does not poison every write after it. A flush triggered by a connectivity event has no caller to tell, so its error routes to the error handler passed at start.
Our take: durability bugs in local persistence almost never come from the write path. They come from an error on the read path being flattened into a default value. "No data" is a legitimate state. "We don't know" is not, and if your storage layer cannot say the second one, every recovery path downstream is guessing.
The in-memory adapter copies the array on both load and save, for the same reason: a store that hands out a live reference is a store whose contract you cannot reason about.
If you maintain a local queue, cache or draft store, it is worth knowing what your code does today when the read itself fails, rather than when it simply returns nothing.
- · 8 min readAccessibility audit scope is an input, not an inference from checks
A conformance claim can report a page as unchecked only if it knows the page exists. If the list of pages under assessment gets inferred from the checks you recorded, then a page nobody ever opened…
- · permalink
We once shipped a catalog page we were genuinely proud of: clean grid, tight product cards, everything lining up. Then we ran it through our translation pipeline for a second market. It came apart. Names that fit one line in English wrapped onto three, cards grew to different heights, and on phones the price and the buy button slid below the first screen.
Nothing was broken in any way a test would catch. The page looked fine in our language and quietly stopped doing its job in the other one, until someone noticed and told us.
What it cost was mostly trust. We had told the client the page was finished, and it was finished in exactly one language.
So now we design to the longest text we can realistically expect, not the shortest, and we look at every layout in the target language before anyone calls it done. If you sell in more than one language, it's probably worth opening your own storefront in the other one today.
- · permalink
Staging is not a smaller production. It's a cleaner one, and that cleanliness is what trips you up.
Prod's bugs live in the data: rows written three schema versions ago, encodings nobody picked, volumes that turn a fine query into a timeout. Staging carries none of that, so it passes. We rarely see anyone test against prod-shaped data.
- · permalink
If your audit builds its scope list from the checks it ran, a page that went unopened doesn't come back as a gap. It comes back as nothing.
Unknown is not passing, and it is not failing either. We think most coverage reports cannot tell you the difference.
https://shipmindlabs.com/c/7e62f952
- · permalink
A validator that stays quiet about its blind spots is lying by omission.
We ran into this while building invoicerules, our open-source EN 16931 rule engine. The payment checks show it most clearly.
For each payment means entry, the rules do not all fire at once. They are gated on what the instruction itself declares.
BR-49 fails when the type code is blank, with no conditions attached.
BR-50 runs only when a credit transfer block is present, and it fails when the account identifier there is blank. No block, no finding.
BR-61 runs only when the trimmed code is one of three credit transfer codes (30, 31, 58), and then it demands an account identifier. So code 30 with no credit transfer block at all fails BR-61 on its own. BR-50 stays quiet, correctly, because there is nothing for it to inspect.
That gating matters. A global "strict mode" would have produced two findings for one defect, and taught people to ignore both.
The IBAN check follows the same discipline. When an account is present and the compacted value matches IBAN shape, we verify the mod-97 checksum: move the first four characters to the end, map letters to digits, reduce digit by digit, refuse anything outside 15 to 34 characters. A mismatch gets a specific code. An identifier that is not in IBAN shape is left alone, not approved, just not judged. Domestic account formats are not our business.
Then the part we think is underrated: we export UNCHECKED_PAYMENT_RULES. Five entries, each with the rule, what it covers, and a note on why we skip it: BR-CO-25, BR-51, PEPPOL-EN16931-R061, UNCL4461, ISO 13616.
Not a paragraph in the README. A data structure the caller can read, print in a compliance report, or diff against the next release.
Green output from a validator is not "this invoice is valid." It is "none of the rules we implement found a problem." The gap between those two sentences is where audits go wrong, and the honest fix is to publish the gap.
If you ship validation to other teams, the thing worth checking is whether your list of known gaps lives in the docs or in the output.
- · permalink
A quarterly margin is not the sum of three monthly margins. If your model rolls up every line the same way, it is already wrong, and the error sits in the lines nobody checks.
We ran into this while building finmodel, our open-source financial modelling library. The rule we landed on is narrow: only input lines roll up. Everything else is either carried across an edge or evaluated again.
Three kinds of line, and each one behaves differently.
An input line sums by default. Revenue, headcount cost, ad spend: a quarter is January plus February plus March. You can override that, since a rollups mapping can declare FIRST or LAST instead of SUM, which matters for anything that is a level rather than a flow.
A carry is a balance, and it takes an edge rather than a total. A carry names an opening, a closing, and optionally a movement. Cash at the start of Q1 is cash at the start of January, so construction claims FIRST for the opening and LAST for a closing that is an input. Nothing gets summed. Summing opening balances is how you end up reporting four times your cash.
A computed line does not roll up at all. It gets evaluated a second time, over the rolled-up inputs. So a quarter's ratio is the quarter's numerator over the quarter's denominator, not an average or a sum of monthly ratios.
What actually saved us was refusing to be clever about disagreements.
The claim step raises a spec error instead of resolving a conflict: if the mapping disagrees with what a carry requires, if a line is carried twice, if opening and closing are the same id, if an opening is not an input line, or if someone hands a rollup to a computed line. Those are all modelling mistakes, and each of them produces plausible-looking numbers if you let it through. A run also requires values for exactly the twelve month ids, not a superset and not a subset.
An opening is not read from its own month's values. That is a spec error too. It comes from the previous month's closing, and only the first month may take an opening from the mapping, or zero. One source of truth for continuity, and no month can quietly restate where it started.
Then the model proves itself. For every carry that declares a movement, each period gets a proof: opening plus movement must equal closing. If it doesn't, or if the chain touches a not-applicable value, the run fails and carries the proof with it. Not a warning in a log, but a failed run with the arithmetic attached.
The broader point generalises past finance. When you aggregate a derived value, you have a choice: aggregate the derived numbers, or re-derive from aggregated inputs. The first is faster and almost always wrong. The second is the one we think survives an audit.
So if you maintain a model that aggregates across time, it is worth checking how your ratios get computed at the quarter level, re-derived or rolled up.
- · 9 min readA dispatch policy should sort candidates, not filter them
Every dispatcher grows an extension point sooner or later. One operation wants the next job to go to the least loaded courier, another to whoever has been on duty longest, and a third wants a rule…
- · permalink
We quoted a payment integration in weeks. The code took exactly that. Then two months went to the client's compliance sign-off, a merchant domain nobody owned, and a bank contact who answered on Fridays.
So the estimate was right and useless. We now price the dependencies we don't control as explicit line items, with an owner's name next to each one. Half of our estimate meetings run on their side now, not ours.
You probably know who owns the slowest approval in your own delivery chain.
- · permalink
The most expensive mistake we see is not building the wrong thing. It's building the right thing so that only one person can change it.
It looks like speed at first. One engineer holds the whole picture in their head, skips the write-up, skips the tests, and ships fast. On the surface nothing is wrong: the product works, users are happy. The cost shows up later, when that person is unavailable, or busy, or gone, and a small change to how payments are attributed suddenly needs a week of reading code before you can even start. That's when founders pay twice, once for the original build and once for the rewrite.
What prevents it is boring and cheap. More than one person has read every important part of the system, there are tests that explain what the thing is supposed to do, and the setup steps live somewhere other than someone's memory. We cover roles rather than individuals for exactly this reason: if one engineer is out, the work keeps moving.
The test is simple. Someone new should be able to make a small change safely next week, without the original author in the room.
- · permalink
If you let a dispatch policy filter, it can push a courier past their capacity. So ours doesn't get to: the constraints sit in one fixed filter, and the plug-in gets nothing but a sort key over whatever survived. Worst case it orders badly.
The key ends with the worker name, so exact ties get decided by the candidates, and the same roster gives you the same assignment every time. https://shipmindlabs.com/c/6ddb2ace
- · permalink
An auditor sent our evidence export back with a single line: none of these hashes check out.
They were right. Nothing had been tampered with. We had pretty-printed the file on the way out.
The log is an append-only chain. Each record carries its sequence number, the hash of the record before it, and its own hash. Checking it is supposed to be something you can do with a hashing tool and no access to us.
Except what we hashed was a canonical form (sorted keys, no undefined values, no whitespace), and what we shipped was the same data re-serialized for human eyes. Same objects, different bytes. Hashes are over bytes.
That cost us a round trip with the auditor and a day of proving that a formatting change was a formatting change.
Now the exporter and the hasher share one function. The file is JSON Lines, one canonical record per line. A line with its hash field removed is exactly the byte string that was hashed. You can check a record where it sits, without knowing anything about our code.
The second change was less obvious.
Our verifier used to skip records whose format it did not recognize (newer writer, older reader) and return ok. "Verified" quietly meant "nothing in here was understood."
Now an unknown format or an unknown hash algorithm is a failure rather than a skip. The verifier checks in order: the sequence number equals its position, the previous hash links to the record before it, the format is one this version reads, the algorithm is one this version computes, and the recomputed digest matches the stored hash. First failure wins, and it comes back with the index it broke at and a reason, not a boolean.
A verifier that cannot read a record has no business vouching for it.
What this still does not do is stop anyone editing the file. Nothing does. The chain makes the edit show and names the entry where it happened, and that is the whole promise, worth being precise about, because what people hear is "tamper-proof" and what they are getting is "tamper-evident".
This lives in our open-source project disclosurekit, where the same log records each assessment and each disclosure actually shown to a user.
If you keep an audit log, hand it to someone outside your team and watch them verify it without you in the room.
- · permalink
A vendor declares two purposes and gets granted one of them. It does not run. Not partially. At all.
That rule is why we stopped treating a vendor list as configuration and started treating it as data that gets checked.
In consentcore, our open-source project, the declaration is validated before anything is allowed to read it. A purpose with no id, an id declared twice, a category outside the known set, a purpose missing its name or description: rejected. A vendor with no id or name, declared twice, declaring no purpose at all, or pointing at a purpose id that was never declared: rejected. What you get back is plain data in fresh arrays, and lookups by id throw on an unknown id instead of handing you undefined.
Where that validation sits matters more than the checks themselves. The failure lands where the declaration is written, not three layers down in a call site where someone wraps it in a null check and moves on.
Permission then reads through that structure rather than beside it. A purpose gets answered through its category. A vendor gets answered by mapping its purposes to categories and requiring every one of them to be allowed. All, not any. The permitted vendors come back in declaration order, so the list a reviewer reads top to bottom is the list that actually runs.
The hash took us longest to get right. The declaration hash sorts purposes and vendors by id, sorts each vendor's purposes and cookies, and writes an absent policy or transfers link as an empty string before the JSON is hashed. The same value serves as the notice hash. Move a line in the file and nobody gets re-asked. Add a vendor or a purpose and everybody does.
Without that normalisation a formatting diff turns into a consent prompt for every user, and a banner that reappears for no visible reason probably trains people to click accept faster than anything else.
The tag layer reads the same declaration. Registering a vendor tag resolves the vendor first, so an undeclared vendor is refused at registration instead of sitting in memory forever, waiting for a grant that can never arrive. A tag with no declaration fails loudly rather than defaulting to blocked and looking like a consent problem.
The design cost is real. All-or-nothing means a vendor that bundles analytics with personalisation is dark for anyone who wanted only one of them. We think that is the correct cost, because the alternative is a vendor deciding at runtime which half of itself to be.
If you have built this layer, you have had to pick one of two things: let a vendor run on a partial grant, or make it split into separate vendors.
- · 8 min readA footer saying "generated by AI" is not a machine-readable mark
Article 50 of the EU AI Act asks two different things about synthetic content, and only one of them is a wording problem. A sentence under an image telling the reader it was generated by AI…
- · permalink
Years ago we hired a backend engineer who sailed through our technical interview, then spent the first weeks arguing with QA about what "done" meant. The code was fine. They had just never had to prove to anyone else that it worked. So we changed the format: a QA role now sits in on every engineering interview and asks one question, how would you convince someone else this works. The answers sort people quickly, and not by years of experience. If we could keep only one interview question, that would probably be ours.
- · permalink
"Verified" is a timestamp, not a boolean. Store the provider's decision along with its expiry, or you end up trusting a check that nobody has re-run in two years.
- · permalink
Onboarding time is a code-quality metric, not an HR one. When a new engineer needs two weeks of tribal knowledge before touching a payment flow, that's not their ramp-up speed: it's missing tests, undocumented contracts, and setup steps living in someone's head. So we watch where the first useful PR stalls, and we fix the repo instead of the person. Your own ramp-up probably stalls somewhere specific too.
- · permalink
"Generated by AI" in the footer tells your reader something. It doesn't mark the file.
Those are two separate duties, and an assessment that scores the sentence as compliance quietly buries the one you still haven't met. We model each measure by whether text can satisfy it at all, then write the gaps into the evidence. https://shipmindlabs.com/c/44e4f2c4
- · permalink
Every KYC integration we have shipped came with the same surprise: the provider API was the easy part.
The vendor returns pass, fail, or something in between. Teams plan for pass and fail.
Then production arrives and a large share of the volume sits in between: glare on a document photo, a name transliterated differently than on the card, an address that is correct but unreadable. The verdict says "fail." The truth is "ask again."
Without a human loop for that middle, two things happen. Good users churn at signup. And the compliance team starts running the business out of a spreadsheet and a shared inbox.
So we treat the moderation queue as a product surface rather than an admin page. What a reviewer actually needs:
the case with every submission attempt in order, not only the latest one;
the provider's raw response kept verbatim, because you will be asked to explain a decision months later;
decisions stored append-only, so the audit trail is the data and not a log file;
and a hard distinction between "rejected" and "needs a new photo" that the signup flow can act on by itself.
Tiered onboarding carries weight here as well. Let a user in at a limited level, and require the heavier checks at the threshold where the risk actually appears.
Our rule of thumb when scoping a KYC epic: the review tooling and the re-submission path are first-class features, and not phase two.
If you have run onboarding at volume, we would be glad to hear about your most common reject that turned out to be fixable by simply asking once more.
- · permalink
Double-entry is not the hard part of a virtual-account ledger.
What breaks things is money that lands with no clear owner.
We have built per-user virtual accounts on account-to-account flows, including products where one user holds two positions at the same time, one for borrowing and one for investing. The accounting model is the easy half. The incoming transfer is the difficult half, because attribution usually rests on payment requisites and a free-text reference, and people mistype it, pay from a relative's account, or send one transfer meant to cover two different things.
What worked for us was refusing to guess.
Arriving funds get booked straight into an unattributed position that belongs to nobody. Attribution after that is a separate decision, and we record it: either an automatic rule match on requisites, or an operator action in moderation tooling. It is reversible, and you can see it in reconciliation against the bank side instead of in someone's spreadsheet.
The tempting shortcut is fuzzy matching on sender name similarity. In our experience that belongs on the operator's screen as a suggestion, and not in the credit path. In a lending product a wrong credit does not stay a bookkeeping error for long: it becomes somebody's funded deal, and unwinding it touches the counterparty too.
If you run virtual accounts, someone has to own the unattributed bucket, whether that is engineering or finance. We have seen it go badly both ways when nobody owns it at all.
- · 8 min readPer-line VAT rounding and totals that cannot be written
An invoice can fail a tax platform while every number printed on it looks correct. Two defects get you there. VAT rounded once on the invoice total comes out a cent away from VAT rounded on each…
- · permalink
What tells us most about a candidate is how they behave in code they did not write. We hand over a real piece of an inherited system and watch. Do they start suggesting rewrites, or do they first ask what depends on this and what breaks if it changes? Almost all production work is joining something already running, so a careful reader who asks that question is worth more to a client than a fast rewriter. Worth deciding what the first question should be for someone on day one in your product.
- · permalink
Support came to us asking who cancelled an order. We had the event. It said CANCEL, and that was the whole of it.
The actor was in there somewhere, on a field of the payload, written by whichever code path got there first: sometimes a role, sometimes a service name, sometimes empty. Four different things could cancel an order in that system. The customer changed their mind, the merchant rejected it, the payment path failed, finance wrote it off. All four ended up in the same place, so all four used the same trigger. The state was right. The story was gone.
We rebuilt it the other way round in our open-source project order-lifecycle. The transition table is keyed by the pair (source state, trigger), and the constructor indexes every row by that pair and raises on a duplicate, so two rows can never claim the same key. The cancellation policy applies the same rule to its own paths, and the error message says the reason out loud: give each actor its own trigger. Customer gets CANCEL. Merchant gets REJECT. The payment path gets FAIL. Finance gets WRITE_OFF. Each path becomes a row in the table, its follow-up work becomes after hooks, and resolution walks the paths in declaration order and returns the first one that permits the calling role.
The actor is legible in the history now, as the name of the trigger. Nothing to join, nothing to trust.
Two things that cost us.
The history lookup compares trigger equality. Ask it for CANCEL and you get nothing that was recorded under REJECT, FAIL or WRITE_OFF. Every "cancellation rate" query has to name the whole set now. We stopped filing that as a bug, because the set is the honest answer, and the old single number was four different business events wearing one label.
The second one we shipped, and we still live with it. The cancel call checks the policy, not the machine. It resolves the path, accepts the role, accepts the reason, then applies the trigger, and if the machine was built on a table that was never extended with the policy's rows, that apply raises an illegal-transition error. Permission granted, transition refused, one layer too late. Build the machine on the extended table and it does not happen. But the failure surfaces at apply time rather than at construction, which is probably the wrong place for a wiring mistake, and we know it.
If you keep an audit trail of state changes, you encode the actor either in the event name or as a field on the event. The field is easier to aggregate. The name is the only version that cannot drift.
- · 9 min readHalf-open delivery zones: the shared street belongs to exactly one
Draw two delivery zones against the same street and you get a line of addresses sitting exactly on the boundary, and a containment test that is inclusive on both sides hands those doorsteps to two…
- · permalink
We shipped an accessibility report that said satisfied. Then someone asked about the checkout page: no tester had ever recorded it, so it never entered the denominator. Now we hand a11ytrail the scope list up front, and anything unchecked prints as not checked, not as passing.
- · permalink
Two zones meet on a street, and the addresses sitting on that line get assigned twice or not at all, depending on how each side rounds its edge.
Make the boundary half-open so it belongs to exactly one zone. Save the tie-break for real overlaps, not the seam. Worth knowing which side owns your borders. https://shipmindlabs.com/c/04070c37
- · permalink
Our coverage scan reported zero uncovered points. A customer on the eastern edge of the service area still could not place an order.
The scan was ours: walk a lattice over the area, test each position against the zone index, report the misses. The bug was one line of arithmetic. The east-to-west span happened to be an exact multiple of our spacing, but in binary floating point the division came back a hair under the whole number, truncation dropped the last column, and the only uncovered strip sat right on the edge we never sampled.
So the step count now rounds span over spacing to the nearest whole number when the two differ by less than 1e-9, then adds one, so both ends of every axis actually get sampled and not just the start.
The second thing we broke was the opposite mistake. Someone ran the same scan at a ten-metre spacing over the envelope of every zone we had. No output, no progress, a job we eventually killed. Now the scan multiplies columns by rows first and refuses anything above its sample ceiling (a million positions by default) before it touches the index, and the error tells you to widen the spacing or shrink the area. A spacing that is not a finite positive number is refused the same way, as is an inverted box, and an index with no zones to bound: no zones is an error, not an empty success.
The last change was about the report, not the scan. A flat list of every missed position reads as "everything is broken". So we flood-fill the misses by four-neighbour adjacency into gaps, sort them largest first, and give each gap a representative, the member nearest that gap's bounding-box centre, so you get one coordinate per hole that you can paste straight into a map. Completeness is just "no gaps". The one-line description names the uncovered count, the number of gaps, the size of the largest and its representative. And when you already have real addresses, you check those instead, in the order you gave them.
This all lives in our open-source delivery-zones project.
The part we keep relearning: a geometric check is only as trustworthy as its sampling, and a sampling bug fails quietly on the boundary, which is exactly where the customers are. That leaves you with the choice of how to verify coverage of a service area: a lattice scan, your real order history, or both.
- · permalink
If your business has a website, the five things worth checking today are all about failure. Appearance does not come into it. Can someone finish a purchase or a form submission twice by double clicking, and does your system record that as one order or two? When a payment or a signup fails, does the visitor see what went wrong, or a blank screen that sends them to your inbox instead? Is there a person or a screen watching the submissions that need review, or do they pile up unseen? Does anyone get told when the site goes down, or do you hear about it from a customer? And is there a copy of your data somewhere other than the server running the site? A site that looks good and a site that recovers are two different products, and only one of them keeps earning while you sleep. We think you had to stop and consider at least one of those five just now.
- · permalink
A candidate stopped our interview to ask how we handle a bad deploy on a payment service. Not the stack, not the salary. We had never been on that side of the table before. We hired them, and now every interview reserves ten minutes for their questions about our process, because the answers tell us what they have actually survived. If a candidate ever asked you a better question than that, we want to hear it.
- · 8 min readA shift pattern is wall-clock time, a shift is an instant
A weekly shift pattern and a rostered shift look like one fact in two shapes: nine on weekdays, and nine on the ninth. They are different types. A pattern is a wall-clock time that means nothing…
- · permalink
We re-ran our dispatcher on a past day's roster to reproduce a complaint, and it assigned a different worker than production had. Same jobs, same shifts, same input.
Two reasons, and both of them were ours.
The first: the cooldown check asked whether a worker was locked right now, rather than at the job's own moment. In production that distinction never surfaced, because "now" and "the job's time" sat seconds apart. In a replay they sit days apart. A worker who was resting during the original run looked perfectly available during the rerun.
The second one was uglier. Two shifts tied on the leading field of the ordering (same current load, same start time), and the winner was whichever candidate the iteration happened to hand us first. Nothing in the data decided that. The run order did.
So we moved the line between what is fixed and what you can configure.
Eligibility became code that no policy can reach: the shift must be at the job's location, it must cover the job's time, and the worker must not be locked at that time. Load comes from a mapping the caller passes in, so an unknown worker counts as zero, and the library never increments it, because counting assignments is the caller's job and not hidden state we keep. Spare capacity is load subtracted from capacity, floored at zero, and a candidate with none gets dropped before anything is sorted.
A policy now has exactly one method: it returns a sort key for a candidate that already survived. It can prefer the least loaded worker, or the earliest start. It cannot admit a shift the filter rejected, and it cannot remove one the filter passed.
And every key we ship ends in the worker name. The name does not mean anything by itself, it is there because a tie has to be settled by something the key carries, rather than by the order the list arrived in.
That is the change that made a replay count as evidence instead of a second opinion. It lives in our open-source shift-planner.
If you run a scheduler, something already settles your exact ties today: either a field in the key, or luck.
- · permalink
We once shipped a hosted payment page that matched a client's brand so closely that it stopped looking like a payment step. People landed there, saw nothing that plainly said "this is the charge, this is the amount, this is what happens next", and either backed out or paid and then called support to ask whether it had actually gone through. The styling was flawless. The page just wasn't telling anyone what was about to happen to their money. Now those boring lines are fixed content on that screen, and the brand gets to style them instead of removing them. Probably worth looking at which part of your own checkout customers ask support about most.
- · permalink
We deleted an endpoint nobody called. Two weeks of logs, zero traffic, clean removal, good mood.
Turned out it was a monthly reconciliation job, and we found that out on the 1st. Nothing counts as dead now until we have watched it longer than the longest schedule in the repo. Worth deciding how far back you look before you delete.
- · permalink
"Every Monday 09:00" is wall-clock time in a zone. "Monday's shift" is an instant. Store one as the other and you get it wrong twice a year.
We keep the pattern naive, put the zone on the plan, and materialise instants per day, which lets the shift crossing the gap really be an hour shorter. Worth checking how your roster handles that.
https://shipmindlabs.com/c/483916ff
- · 8 min readA failing hook must not hand back the moved order
A transition that writes the new state and then fails in one of its callbacks leaves the order in the new state with the work half done. The record says shipped. The stock was never reserved, or the…
- · permalink
The net totals matched. The VAT line did not, off by one minor unit, on a handful of documents in a batch.
We had computed it the way arithmetic suggests: sum the line nets, apply the rate once, round once. One rounding step, smallest error available. The reviewer on the other side computed it the way the document actually gets read, VAT on each line, rounded at each line, then summed.
Both are defensible. Only one of them is what the recipient's system reproduces, and we had never decided which one we were. The choice sat scattered across a serializer, a PDF template and a totals helper, each quietly rounding at a different moment.
The cent was not the cost. The cost was a day of three people arguing about which number was correct, with no single place in the code that owned the answer.
So in our open-source invoicing library, euinvoice, the decision has exactly one home.
A line's net is unit price times quantity, rounded. A line's VAT is that net times the rate, rounded again. Rounding happens at the currency's minor unit (zero decimals for JPY-type currencies, three for BHD and TND, two otherwise), half-up, every time, with no per-call override.
Then the part that does the real work: an invoice has no total fields. Net, VAT and gross are properties that sum the lines. The VAT breakdown is a property too, bucketed by category and rate in first-appearance order, each row carrying its own gross. The breakdown equals the totals by construction, because both are the same sum read twice. There is no setter to disagree with.
The same instinct runs through the money type. Floats and bools get rejected at the door, not coerced. Every add, subtract and compare checks the currency first and raises on mismatch. No conversion function exists, so nothing silently crosses currencies. An invoice refuses to exist with mixed currencies, with a due date before its issue date, or with reverse charge sitting next to taxable lines.
The lesson we keep reapplying: when a number appears in two places, it is a question of policy rather than arithmetic, and policy belongs in one function instead of three.
If you have shipped invoicing across currencies, you have probably had this argument already, and we would like to hear whether you rounded per line or on the total, and whether anyone downstream read it differently.
- · permalink
Most of what makes a website look cheap has little to do with the design, it's the half-second after a click. A form that gives no sign it was received, a submit button that can be pressed twice and sends twice, an error that says "something went wrong" without saying what: each one quietly tells the visitor that nobody is home behind the page. We build those states with the feature instead of adding them as polish at the end. You probably remember the last site that left you unsure whether your form actually went through.
- · permalink
A shared border has to belong to exactly one zone. Our delivery-zones ring test uses strict inequalities, so edges come out half-open: on a zone's southern or western edge you're in, on its northern or eastern edge you're out. Two zones along one street never both claim it.
- · permalink
A hook after the status change failed and the order stayed shipped, except nobody shipped it. Now a transition is not done until every hook returns, and on failure the record sits where it was. Worth checking which of your post-save hooks are actually part of the transition. https://shipmindlabs.com/c/4c66d83d
- · permalink
A client came to us wanting two mobile engineers. We asked what the app was doing badly: screens felt slow, they said. The traces showed one screen making nine calls, three of them for the same user object. They didn't need mobile hands, they needed someone to own the API contract. So now we ask to see one real screen's network trace before we scope a role. It costs an hour, and it has changed the ask twice.
Worth knowing what the last role you hired turned out to actually be.
- · 8 min readHidden is not denied: one row declaration, two checks
A delivery backend looks like one product, and it answers to half a dozen audiences, most of them reading the same order table. The list endpoint filters it correctly (a courier sees the shipments…
- · permalink
We used to write the take-home spec until nothing was left open, every edge case listed, and the candidates who finished it clean got offers. Then one stalled on their first real ticket: a three-line message about refunds that turned out to be a reconciliation problem. They could build anything we specified and nothing we didn't.
Now the task ships half-defined on purpose and we grade the questions, not the code. Probably worth looking at what you screen for that the job never asks for.
- · permalink
Our courier app scoped rows in two places: a filter in the list view, an ownership check in the detail view. Same rule, written twice. One edit later they disagreed, and a courier could open by id what the list never showed.
Declare the slice once, derive both from it, and you want a missing key to be an error, not a wider result. https://shipmindlabs.com/c/06d8af13
- · permalink
If you don't want to hire a full-time team, the gap is usually role coverage rather than hours. Product work is bursty by role: QA matters most in the days around a release, DevOps when infrastructure or deployment changes, mobile for a few intense weeks and then not for months. Hiring for that shape means paying for idle time, or hiring one generalist who ends up as the only person who knows how your deploy works, and then your roadmap stops when they take a vacation. We work as a team with cross code review and shared context, so the knowledge lives in the process rather than in one head, and the role you need is available the week you need it. Probably worth checking which role in your product is currently held by exactly one person.
- · permalink
A shift pattern is wall-clock time. A shift is an instant. If your model lets those two share a type, DST will eventually hand you a scheduling bug you cannot reproduce.
That is the rule we ended up encoding in our open-source scheduling library, shift-planner, and it reshaped the whole API.
The pattern says "nurses start at 22:00". That is an intent. It is not a moment in time until you pick a day and a zone. So the pattern object refuses a start time carrying tzinfo at construction: the plan already owns the timezone, and a pattern that smuggles in its own gives you two sources of truth that disagree twice a year. The instant gets built once, in the one place where both halves are present: combine the day, the wall-clock time, and the plan's zone.
The other direction matters just as much. When a plan decides which existing shifts belong to it, it draws day boundaries by converting each shift's start back into the plan's zone and taking that date. Not UTC. A shift starting at 22:00 local belongs to that local day, even though in UTC it has already rolled over.
Then the consequence that surprises people. Shift end is start plus length in wall-clock arithmetic, while coverage and overlap checks compare instants. Across a spring-forward gap, a two-hour shift is one hour of real duty. That is not a bug to paper over. It is what happens to the person working it, and payroll and coverage rules need to agree on which of the two numbers they mean. We keep a test file whose only job is to pin that behaviour down, because it is exactly the kind of thing a well-meaning refactor quietly normalises away.
Three smaller decisions fell out of the same principle:
Shifts, jobs and cooldown locks all reject a naive datetime at construction. A timezone-less timestamp is not a moment; it is a note to yourself.
Every interval is half-open. Back-to-back shifts must not count as overlapping, and that is a one-line decision you only get to make once.
No source file calls datetime.now(). Every check takes the moment as an argument. That is what makes DST testable at all, and it means an operator can ask "is this roster legal as of Sunday 02:30" without touching the system clock.
One last property worth copying: applying a plan keeps every shift the plan does not claim, then appends the shifts it generates. Redrawing a plan is idempotent. Reruns are boring, and boring is what we were after.
Most scheduling bugs we have been called in to look at are not scheduling bugs. They are one timestamp that lost its zone, or a duration that was assumed to equal elapsed time.
If you run shift-based operations, it is probably worth working out where your wall-clock intent and your instants currently meet, and whether that is one place or several.
- · 9 min readFour things a formula evaluator must do that eval() cannot
A spreadsheet formula is a program somebody else wrote, and sooner or later it has to run outside the spreadsheet, either in a service that recomputes a model every month or in a test that proves…
- · permalink
We hired someone strong and they looked slow for a week. We blamed ramp-up. Then we read their first ticket: it crossed four services, and only one had a working local setup. The interview measured them. Onboarding measured us, and we had been failing it quietly for months.
Now the first week of a new hire is our documentation audit, and their questions go straight into the backlog. Worth asking what your last hire accidentally revealed about your setup.
- · permalink
Everyone argues about eval() and security. What gets discussed less: a formula has to fail when you write it, not on the first number, and you need the dependency order before any value exists.
A tokenizer and a recursive-descent parser give you that. eval() gives you none of it. https://shipmindlabs.com/c/a8242fe4
- · permalink
A state machine that hands back a half-moved object is worse than one that refuses to move at all.
The usual shape of a transition API goes: validate, fire the hooks, mutate the entity, save. The problem is that the mutation and the hooks live in the same mutable object, so when an after hook raises, you are holding something that is neither the old state nor a committed new one. The exception propagates, the entity in memory has already changed, and whether that leaks to the database depends entirely on who remembered to wrap the call in a transaction.
We took a different route in our open-source project order-lifecycle. The transition resolves the row, runs the before hooks, builds the new machine with a dataclass replace, runs the after hooks, and only then returns it. The new machine is a separate value that lives inside the call and never escapes unless every phase completed. So a failure in an after hook leaves you in the source state with no history entry at all, not a rolled-back one, just none.
The error has to carry more than a message. When a hook raises, it surfaces as a single failure type carrying the phase it happened in, the original cause, and the list of hooks of that phase that already ran before the raise. That last field is probably the one people forget. If three after hooks fire and the second one dies, you need to know the first one already sent its side effect, because that is what you compensate for.
Two supporting decisions fall out of this.
History is a frozen value. Recording an entry or a log line returns a new object instead of appending in place, so the half-written history of a failed transition cannot exist by construction. Entries default their timestamp to UTC now, but the transition call accepts an explicit clock, because replaying a backlog of events with wall-clock time stamps gives you a history that is ordered wrongly and silently.
Refusals are ordered and complete. Resolution rejects in a fixed sequence: an illegal transition first, carrying the triggers it would have allowed; then a role that is not permitted, carrying the required roles; then conditions that were not met, carrying every failed condition, because condition evaluation deliberately does not short-circuit. One round trip tells you everything wrong with the attempt instead of one problem at a time.
The pattern underneath all three is that the failure path needs to be designed with the same care as the success path, and it needs to be as informative. Most production incidents we have cleaned up in payment and compliance flows were not caused by a transition that failed. They were caused by a transition that failed halfway and told nobody which half.
If you run a lifecycle with side effects attached to transitions (notifications, ledger writes, document generation), the hook that succeeded before the one that died is the part you are left holding, and the error object is where we would rather see it named.
- · permalink
A site we worked on launched looking great: fast, clean, and the signup flow tested well. Then the verification queue started filling up. Every new signup had to be checked by a compliance person, and that part of the product was a plain list. No filters, no way to see who had been waiting longest, no record of who had already opened an application. Within a week the people doing the checking were keeping a side spreadsheet to stay sane, and new customers were waiting days for an approval the marketing page had implied would take minutes. Nothing was technically broken. The front door was beautiful and the room behind it had no furniture. Now, when a site creates work for a human, we design that human's screen in the same sprint as the landing page: queue order, assignment, and a visible history of who touched what. Worth knowing where the work goes after someone signs up on your site, and who they are waiting on.
- · 9 min readYour audit trail is a second copy of every sensitive value that changed
An audit trail records which fields changed on a row, and that means it also records the values those fields held. If one of them is a password hash, a card number or a passport number, the trail…
- · permalink
The most expensive mistake we see in early products isn't bad code. It's data recorded in the wrong shape.
Code is cheap to replace, and a messy service can be rewritten in weeks. The records your product has been quietly piling up cannot. If a system stored that a balance changed but not why, or took a payment without keeping what it was for, that gets very hard to reconstruct later: not by your accountant, not by your support team, not by the engineer who eventually rewrites it. A function you can fix. A field you left unrecorded, you cannot go back and capture.
That is why "we'll clean it up after launch" costs so much more than it sounds like. The cleanup runs while real money and real users are already moving through the system, and every correction has to be made twice, once in the new code, once across the history.
So it's probably worth naming the one thing your product doesn't record today that you'd want to look up a year from now.
- · permalink
Syntax is the manageable part of a Python 2 to 3 migration. The week when both versions are running at once is the part that costs you.
Codemods and a test suite cover the code. Data in flight is what they leave alone.
During a rolling migration, old workers and new workers share infrastructure: the same queues, the same cache, the same session store. A task serialized by one interpreter gets picked up by the other, and a cached blob written yesterday gets read today by a different runtime. The failures land there, not in the files you changed but in the bytes crossing a process boundary.
The fix is boring, and it has to happen before the migration starts. Move every cross-process payload to an explicit, interpreter-neutral format, JSON over pickle for queue messages. Put a format version into the cache key namespace, so the two runtimes do not read each other's blobs. Once that holds, the cutover goes per worker instead of per release night, and a rollback is just scaling one deployment back down.
We plan for the overlap window rather than pretending it won't exist. Both paths run, both are monitored, and the switch moves service by service.
If you have done a runtime or framework migration on a live system, we would like to hear what actually broke: the rewritten code, or something that travelled between two processes. Probably the second one.
- · permalink
On a Python 2→3 migration we deleted a file that nothing imported and nothing referenced. Just a comment at the top: do not delete.
Two hours later half the handlers weren't registered, because the import side effects were the architecture. Now anything "unused" gets grepped, then sits one release before it dies. Most codebases probably have one of those load-bearing junk files.
- · permalink
Manual KYC review is a queue sitting in your signup path. Staff it, measure its wait, alert on it. Treat compliance as back-office and onboarding stalls silently for days. Someone needs to be watching that queue depth.
- · 9 min readA promo usage limit needs compare-and-set, not a read then a write
A promotion capped at one use per customer and a thousand overall is two counters, and the obvious way to enforce them is a read, a check against the limits, and a write of the incremented values.…
- · permalink
We once quoted a KYC integration straight out of the provider's docs. Writing the code was the easy part. Then their onboarding showed up: sandbox credentials, a compliance review, a go-live checklist nobody had mentioned. We had committed to a date that depended on a queue we didn't own.
So now every third-party dependency gets its own line in the estimate, flagged as a date outside our control. You probably want to know who owns that date on your projects.
- · permalink
The most useful exercise in our engineering interview is reading code rather than writing it. We hand over a small, real module and ask two things: what would you change first, and what would you deliberately leave alone. The first answer shows judgment. The second shows restraint, and restraint is the rarer skill, because most of the work on a live product is changing something another person wrote without breaking the parts that already work. Candidates who want to rewrite everything and candidates who won't touch anything fail in the same way. If you hire engineers, it's probably worth knowing which one your process actually tests, writing or reading.
- · permalink
A shipping rate shown at checkout is not a price.
It is a quote with an expiry, and the carrier is under no obligation to honour it when you buy the label two hours later.
This is one of the quietest money leaks we see in cross-border e-commerce. Rates come from one API call at checkout. Labels come from a different call later, after payment, often after the warehouse has already packed and weighed the parcel.
Between those two moments things move. Rate tables get updated. Surcharges change. The real weight and dimensions are not the ones the customer's basket estimated. The address gets normalized into a different zone than the one you quoted.
If you treat the checkout number as a fact, the difference lands quietly on your margin, and nobody notices until someone reconciles a month of shipments.
What works better for us: store the quote as a record, not a number. Carrier, service level, the exact inputs you quoted on, the carrier's quote id when it gives one, and a validity window. At label purchase, re-quote and compare against that record.
Now the delta is visible, and it becomes a product decision instead of an accident: absorb it below a threshold, re-price above it, refuse and escalate when the service is no longer available at all.
The shift that matters is treating that gap as expected rather than exceptional. Most multi-carrier bugs we have debugged were never API failures. They were one number from checkout being reused as truth three steps downstream.
If you run multi-carrier shipping, you probably already have a line drawn somewhere between absorb and re-price, and someone who owns that number, ops or engineering.
- · permalink
A promo code capped at 100 uses is not capped if you enforce it by reading the counter and writing it back. Two webhooks land at once, both read 99, both write 100, and both customers get the discount.
The cap needs a compare-and-set against the counts you actually validated, re-read on conflict. And a repeated idempotency key should hand back the first reservation instead of minting a second one. https://shipmindlabs.com/c/343a64c3
- · 8 min readA blocked tag left on the page is one line of code away from running
The usual way to hold a third-party script until the visitor consents is to leave it in the markup with type="text/plain" and swap the type back on accept. Nothing has executed, no request has gone…
- · permalink
We used to drop refused vendor tags onto the page as text/plain and call it consent. It isn't: one line flips the type back, and if you withdraw mid-session, whatever the tag already started keeps running.
Treat tags as a registry. Hold until there's an answer, release per purpose, and on withdrawal remove the element and clean up after it.
https://shipmindlabs.com/c/063e5250
- · permalink
Most permission bugs we find in audits are not missing checks. They are two checks that were meant to say the same thing and quietly stopped saying it.
The shape repeats. A list endpoint filters the queryset down to "the rows that belong to you." A detail endpoint fetches one row and asks whether it belongs to you. Two developers, two tickets, two implementations of one sentence. Six months later someone adds a relation, updates the list filter, and forgets the object check. The list stays correct. The detail view hands out someone else's record by ID.
So in our open-source project role-scopes we stopped treating those as two features. Reach gets declared once, and both paths are derived from it.
There are two read-only tables. One answers whether an actor may act at all. The other, keyed by the resource the permission refers to, answers which rows are theirs. If a resource was never declared, the answer is Scope.NOTHING, not "everything," and not something inherited from a parent role. Reach is granted rather than assumed. A forgotten line in config fails closed.
The list path checks the permission first and then applies the slice: no permission or NOTHING returns an empty queryset, EVERYTHING returns the queryset untouched, OWN returns it filtered by the scope's keys.
The object path reuses that same slice instead of reimplementing it. It follows "__" lookups across relations the way the ORM would join them (order.shipment.courier_id resolves the same on one object as it does in a WHERE clause), and it denies on a mismatch or on a NOTHING slice.
The decision we would defend hardest is the error behaviour. When the principal has no value for the scope key, or the object has no such attribute, it raises. It does not widen the result to be helpful, and it does not quietly return nothing.
Both of those failure modes are worse than a 500. Widening leaks data. Emptying looks exactly like "you have no orders," so support closes the ticket, the customer shrugs, and a broken scope lives in production for a quarter. A loud exception in staging costs an afternoon. A silent empty list costs trust, and you probably hear about it from the wrong person.
That is why we treat authorization as data with one source of truth instead of logic sprinkled at call sites. Worth asking of any codebase: if someone adds a relation tomorrow, how many places have to change for "yours" to stay correct? If the answer is more than one, the two answers will drift, and the only thing left open is which endpoint tells you first.
We keep list filters and object checks from drifting apart by giving them one declaration to read from, and we would rather hear how you do it than assume ours is the only way.
- · permalink
A new engineer closed their first payments ticket in a day. Then it sat for three, because only one of us could actually review that module. The delay came from the review bench, not from onboarding. So now we count how many people can review each service, instead of who owns it. One reviewer is the same single point of failure as one owner.
Worth checking how many of your services have exactly one person who can approve a change.
- · permalink
A formula DSL is safe when there's nothing to escape into. In finmodel the tokeniser allows numbers, names, arithmetic, parens. No calls, attributes or subscripts. Cycles and unknown refs fail at construction. Division by zero returns NA with a reason, not a mid-model exception.
- · permalink
A candidate was halfway through our take-home review when they stopped to ask which number was the real one: the balance shown on a user's account, or the sum of the transactions behind it.
Nobody had asked that before. We were treating the exercise as a coding test, and they read it as a reconciliation problem, which is what it actually is in production. Two places show money, and if you have no stated rule for which one wins when they disagree, someone will eventually pick the wrong one in front of a customer.
They were right, and it cost us an afternoon of embarrassment plus a rewrite of our own task description. The ambiguity was baked into it, which means every earlier candidate had quietly guessed instead of asking.
So we left the gap in on purpose. The only thing we really score now is whether someone asks about it before they start building.
We'd like to hear the question you wish more candidates asked you.
- · 9 min readA money input that regroups on every keystroke eats a digit
A money field that reformats its own text while someone is still typing will swallow a character sooner or later. The user adds a digit, the field rewrites the string to move a group separator, the…
- · permalink
Redaction and exclusion are not two settings of the same feature. Redaction keeps the fact that a field changed and withholds the value. Exclusion leaves no trace the field was ever touched. An audit trail that conflates the two is lying by omission.
We ran into this while building model-audit, our open-source change-tracking library. The pressure is real: someone asks you to keep a password hash out of the log, and the easy fix is to drop the field. Now the trail says the record was not modified. Six months later, in an incident review, that silence reads as evidence.
So we split them into two different calls. Register a field with redact and the change gets recorded by field name, with the before and after values withheld. Register it with exclude and the field drops out entirely, so the trail never claims it was involved.
The mechanics that make redaction safe are small, and worth stealing.
The placeholder is a str subclass that reads "[redacted]", so every consumer that serializes records to JSON, logs, or a diff view keeps working untouched. Nothing downstream needs special-casing. But because it is a singleton, an identity check still tells you a value was deliberately withheld rather than genuinely empty.
Redaction also preserves the shape of the change. If a field had no prior value, the sentinel for absence survives redaction, so an addition stays an addition instead of a mysterious edit from one hidden value to another. Lose that distinction and you cannot tell account creation from account modification.
Matching is a substring test on normalized field names against a sensitive-field set. Deliberately blunt, deliberately over-eager. We left email out of the defaults, because for a lot of products the email address is the business record and not a secret. And we added an allow list specifically to release false positives, because a blunt matcher without an escape hatch gets disabled wholesale by the first team it annoys. Noisy fields (the ones that change on every save and tell you nothing) are excluded by default, with a flag to keep them when you actually want the churn.
One decision we deliberately did not make: where the trail lives. The library records changes and hands each record to subscribed receivers. Database, log pipeline, append-only store, retention window, all of that is the project's call, shaped by its own compliance obligations. A library that picks your storage has picked your retention policy too, and it has no business doing that.
The general rule we keep coming back to: an audit trail's value is in what it still proves after you remove the sensitive parts. Removing the value is a tool. Removing the evidence is a bug.
For those of you running audit logs under a retention policy, the choice between redacting at write time and redacting at read time is probably the next thing to settle.
- · permalink
Live formatting on an amount field is how you lose a digit. Regroup the thousands separators on every keystroke and the caret quietly slides out from under the person typing. We leave the text alone during entry and group on blur. If a rewrite is unavoidable, put the caret back after the same digits it followed before. https://shipmindlabs.com/c/269160b9
- · permalink
In a P2P lending ledger the hard part was never the interest math, it was incoming transfers we couldn't attribute. Requisites get mistyped. Real money sits in limbo while the loan still reads as unfunded. We fixed it by giving unattributed payments their own holding state and a review queue, instead of letting the system guess an owner. Your platform puts money it can't explain yet somewhere, and you probably want to know where.
- · permalink
If your business has a website, the five things worth checking today are not on the page. They are the five messages that are supposed to leave it: the order confirmation, the password reset, the contact form, the payment notification from your provider, and the error alert nobody has opened since launch. All five fail the same quiet way. A mailbox fills, a forwarding rule changes, a key expires, and nothing on the site looks different, because the broken part sits downstream of everything a visitor can see. Trigger one of each yourself this afternoon. The one that does not arrive is probably the one that has been costing you orders. We would guess you cannot remember the last time a message reached you from your own contact form.
- · 10 min readWhere a discount belongs: the invoice or the line, never both
An invoice can agree with itself in every direction and still be the wrong document. The case that survives validation is a discount granted once and recorded twice, once as a line level allowance…
- · permalink
Your validation does not enforce a promo code limit. The write does.
Read the usage counts, check the rules against them, then increment, and you have a window between the check and the write. Two concurrent checkouts both read "9 | 10 used", both pass validation, both increment. Eleven redemptions on a ten-use code. The rule was correct every single time it ran.
So in our open-source promocodes library the store's record call is a compare-and-set. It takes the usage counts the decision was made against, and it returns nothing at all if those counts moved underneath it. No exception, no partial write, just a signal that the decision is stale.
The redeem path then loops, up to three attempts by default. Every pass re-reads usage and re-runs validation from scratch. It is not a retry of the write but a retry of the whole decision, because the right answer on the second pass may be "rejected" rather than "try again". After the attempts are exhausted it raises a conflict error, deliberately a runtime error and not a rejection. Contention is an infrastructure outcome. A customer being out of uses is a business outcome. Collapse them into one error type and you end up telling a user their valid code is invalid because two people clicked at once.
Two more pieces of the same design.
A known idempotency key returns a reservation marked as replayed and increments nothing. The retry of a request is not a second redemption.
And the read and the write are split on purpose. Quoting validates and prices without spending anything, while capture re-runs every rule through redeem at payment time, optionally against a total that changed since the quote. If a rule now refuses, the rejection is re-raised as a capture failure carrying the stale quote, and nothing is recorded. A refused capture must leave the counters exactly where they were.
The honest limitation: the in-memory store holds the rule with a thread lock. That is correct for one process and no substitute for a database. The compare-and-set contract exists so that a real backing store can honour it with a conditional update, and the calling code does not change.
If you handle promo codes, discounts, or any budget shared across concurrent requests, it is worth knowing which layer does the enforcing, the validator or the write.
- · permalink
If a role has no backup, hiring for it buys you a single point of failure instead of capacity. Before we open a position we write down which existing role absorbs that work for two weeks, and what that costs us. If the answer is nobody, then the first task for the hire is the handover doc and the backlog waits.
Worth checking who covers your newest role tomorrow.
- · permalink
An invoice can add up perfectly and still be wrong. Put a line discount at document level as well and you have counted it twice, leave it out of the line net and it vanishes from the totals and the VAT breakdown. We keep one home for it, then validate both paths.
https://shipmindlabs.com/c/fd5d0afc
- · permalink
A user withdrew consent and we just flipped the tag to type=text/plain. The script was dead, its cookie wasn't. Now we remove the node and run cleanup for whatever it set. consentcore's registry logs both directions: held, released, removed.
- · permalink
A beautiful site can still be a bad site. The polish usually covers the states the designer controls, and the money sits in the states the network controls.
The mechanic that catches people is simple. A submit button that animates the instant you click it feels fast, but that animation is reacting to your click, not to the server's answer. On a weak connection you get a spinner, then nothing, so you tap again, and one patient customer turns into two orders, or two charges.
The fix is not visual. The form has to carry a stable identifier for that single attempt, so a repeated request lands on the same record instead of creating a new one. Your layout can stay exactly as it is. It is the submit that has to be safe to repeat.
If someone double-tapped the pay button on your site today, you would end up with either one order or two, and we think knowing which one is worth more than the polish.
- · 8 min readOrder per entity, not per queue, in an offline write queue
An offline write queue has to honour two rules that pull against each other: two changes to the same record have to arrive in the order the user made them, and one stuck change should not hold…
- · permalink
We had a test we just re-ran until it went green. Everyone knew the one; it even had a nickname. Then the same race condition showed up in a payment flow in production, and we realised the test had been telling us about it for months.
Now a flake that only passes on retry opens a ticket instead of a green build. You probably have a nicknamed test of your own.
- · permalink
Our QA lead filed a bug that we nearly closed as "works as designed."
Someone entering an amount on a payment form typed 12.50 and submitted 1250. The field was reformatting on every keystroke. As soon as a group separator appeared, the caret jumped to the end of the text, and the next digit landed where the person never aimed it. They did not notice, because the field looked tidy the whole time.
Then a second report arrived. A user pasted an amount copied out of a bank statement, and it came with a non-breaking space between the thousands and a minus sign that was not the ASCII one. The field rejected it. They retyped it by hand, got it wrong, and opened a support ticket instead.
Both reports came down to the same mistake: the field rewrote text while the user's hands were still in it.
So in amountfield, our open-source input, we hold to a few rules now.
We do not reformat while you type. Grouping is applied on blur, once the person is done.
Half-typed input is a state rather than an error. "12." is on its way to "12.50". A group of two digits has simply not reached three yet. Neither of them deserves a red border.
When we do rewrite the text (grouping on blur, or removing a pasted symbol), the caret goes back after the same digits it was after before. Not to the end of the field. After the same digits.
A paste gets cleaned rather than translated. Our own currency symbol or code and non-breaking spaces are stripped, a non-ASCII minus is normalised, and a foreign currency symbol is refused outright.
That last rule is the part we argued about. Quietly reading a pasted euro symbol as our currency would have made the field feel smarter and the number wrong. With money, asking the person to fix the input is probably the cheaper failure.
It is worth checking where your own input layer still rewrites text under the user's hands.
- · permalink
Our offline queue ran as one line. A stuck photo upload held back a note edit behind it, and an edit and a delete on the same record could land out of order.
The fix: one lane per entity, lanes running side by side, and a lane stops at the first change that needs a retry instead of stepping over it. Order where it matters, parallel where it doesn't. Where you draw the entity boundary is yours to decide.
https://shipmindlabs.com/c/463d56c8
- · permalink
A few years back, a checkout we worked on sent people to a payment page with someone else's logo on it and a different address in the bar. Nothing was broken. Customers stopped there anyway, and support started getting messages asking whether the site was a scam. The cheap feeling usually comes from one moment where the site stops looking like itself, not from the design. We keep payment pages on the client's own branding and domain now, because that's the screen where trust actually gets spent. Worth checking where your own site hands a customer off to something that doesn't look like you.
- · 9 min readA deep link that arrives before login: park it, replay it once, expire it
Sometimes a link arrives before the app can do anything with it. A cold start resolves the URL that opened the process before the navigator exists, and a link into a screen behind a login arrives…
- · permalink
You probably don't need a full-time team. You need role coverage, and those are two different purchases. One hire can't be your backend, your mobile build, your testing, and the person who gets up when the servers do something strange at 2am, so one role on paper quietly turns into three before anything ships. Renting the roles means they show up when the work calls for them and step back when it doesn't: a build phase needs all of them at once, and the quiet months of upkeep afterwards need a fraction of that. So it's worth asking what drives your hiring plan right now, the work actually in front of you, or the busiest month you're bracing for.
- · permalink
An offline queue that keeps global order is a queue that stops working the first time one change fails.
That trade-off is the one most offline-first implementations get backwards. The instinct is to hold pending changes in one list and drain it in the order the user made them. It feels safe. It is also why a single rejected edit on one record can hold back thirty unrelated changes sitting behind it.
Order does matter, but it rarely matters globally. It matters per entity. If a courier renames a stop and then marks it delivered, those two have to land in that sequence. If some other stop was edited in between, it has no stake in either of them.
So in offlinequeue, our open-source library, order is scoped by entity. Each operation names its entity through a function we call entityOf, which defaults to the operation kind, and changes that share an entity form a lane. A lane is attempted oldest first and stops at the first change that has to be tried again, because anything after it may depend on it. Lanes do not wait for each other. A flush runs up to a concurrency limit of lanes at once, four by default.
That gives you two guarantees that usually fight each other: causality per entity, and the rest of the app carrying on syncing while one record is stuck. A change can also be parked, and its lane continues past it instead of treating one poison message as a full outage.
The second half is idempotency, and that is the part teams skip. Keys are generated on the device at enqueue time, not by the server. Enqueueing a key that is already in the queue returns the row that is already there instead of creating a second one. And the key does not change across retries, so the same change carries the same key on its first attempt and on its fifth.
That last detail is what makes the retry story honest. Without a stable device-side key, a timeout is unresolvable: the client cannot tell a request that never arrived from one that succeeded and lost its response, so it either risks a duplicate or risks dropping the change. With a stable key, a repeat on the wire is something the server can recognise and collapse. Retry safety comes from the key, not from the network code.
We built this after enough mobile and field-operations work to know how it fails in production. The bug reports are rarely "sync is slow", they are "the change disappeared" or "it saved twice", and both trace back to ordering and key decisions made early and then left alone.
If you run offline writes, it is worth being precise about what your unit of ordering actually is, and about what happens to the queue behind a change the server keeps refusing. We think those two answers describe your sync layer better than any benchmark does.
- · permalink
A founder asked us for one strong backend engineer so things would move faster. We staffed it. Two weeks in the work had stalled: every ticket needed a product decision only they could make, and they surfaced once a day.
What they needed was someone allowed to decide, not another pair of hands. Our intake now includes a question about exactly that: who can approve a scope change while you are unreachable for three days.
- · permalink
A deep link can show up before a session exists, or before the navigator is there on a cold start. So we park it. The part teams forget: it has to be consumed exactly once, and it has to expire.
Replay that link on the next launch, or honour it twenty minutes later, and you yank the user out of whatever they were actually doing. We treat that as a stale write rather than navigation.
Worth deciding how long your parked link stays valid.
https://shipmindlabs.com/c/d99ae83e
- · 8 min readMake a staging OTP shortcut fail at startup, not at verification
A fixed code that opens every account is the fastest way to test a login flow, and a breach on the day it reaches production. The awkward part is that the verification function cannot catch this: by…
- · permalink
The totals on an invoice we generated added up perfectly. The VAT breakdown was off by exactly one discount.
We had taken a discount that belonged to one line and also added it into the document-level allowance total. Each number looked fine on its own. Together they pulled the same money out twice in one place and not at all in the other.
That is the part that bites you. An allowance at document level and an allowance on a line are not two sizes of the same thing. They reach the totals by different paths.
A document-level allowance carries its own VAT category and rate. It gets summed into BT-107, the document allowance total, and it moves the taxable amount of the specific VAT breakdown group it names. It is a first-class tax object.
A line-level allowance carries no category at all. It reaches the totals only through the line net amount, BT-131 (quantity times price, less that line's allowances, plus that line's charges). By the time it gets there, it is invisible as a discount. It is just a smaller line.
So in invoicerules, our open-source rule set for invoice documents, we made both directions of that mistake refusals instead of warnings. A line discount that also shows up in the document allowance total is rejected. A line allowance missing from the line net amount is rejected. And a percentage allowance is checked against the base amount it claims to be a percentage of, because a percentage with no base it matches is a number someone typed, not a calculation.
What changed in how we work: we stopped treating totals as something to verify at the end. Anything derivable gets derived, and the derivation is the check. If a value can be computed two ways, the library refuses to take it as input.
If you have built invoicing against a national e-invoicing standard, you probably know where your mismatches came from: the tax rules, or the places where the format lets you say the same thing twice.
- · permalink
That universal OTP code someone added for staging is the one credential in your system that never expires, doesn't rotate, and stays out of your audit log.
If it has to exist, it should expire on its own, it should make noise in the logs on every use, and the app should refuse to start in prod while it's set. Worth knowing who on your team owns that check. https://shipmindlabs.com/c/dce0b383
- · permalink
A parked deep link works once. take() releases the hold before it returns, so a double-fired login or a remounted screen gets null. Newest link wins, holds expire in ten minutes, sign-out clears. In deeplinkmap, cold and warm share one door, and accepts and refusals are audited.
- · permalink
A candidate went through three rounds with us, and all three sets of notes said some version of "solid, communicates well." None of the notes mentioned what we were unsure about, so each round quietly re-confirmed the last one's good feeling instead of testing anything. Now every round ends with one written doubt the next interviewer has to probe. Half of them resolve fine. The other half we'd have hired on momentum.
Worth checking what your own process does with an unwritten doubt.
- · permalink
A beautiful website can still be a bad website, because beauty gets judged in the one state the site will rarely be in again: empty. Designs get approved on ideal-length product names, photos cropped to the exact ratio, one currency, no loading, no errors. Then the real content arrives, with 90-character titles, a supplier image in the wrong shape, a price that needs a second currency, and a form that has to say something sensible when the payment provider times out. Those states rarely get designed, so they get written by whoever happens to be in the code that day, and those are exactly the states a paying customer is most likely to hit. We now review designs against the ugliest real record we can find rather than the sample one. Probably worth finding the messiest piece of real content your site has to display before it finds you.
- · 8 min readAn unanswered question is not a no
A compliance checklist made of tick boxes has exactly two states, and an empty box reads as "no". That is a quiet way to ship a chatbot without its disclosure: nobody ever decided whether the system…
- · permalink
We hired someone strong and watched them stall for two weeks on our payments service. That gap was ours, not theirs. The things they kept asking about were the things nobody had written down, and we only noticed because a newcomer said it out loud.
Now week-one questions go into a list we fix, instead of a ramp-up checklist we forget about. You can read a new hire's confusion as their problem, or as a hole in your documentation.
- · permalink
Startups worry about building the wrong feature, and that is the cheaper mistake. Writing the wrong record is the expensive one. A feature you regret costs you the week it took to build, and then you delete it. A data shape you regret costs you everything already saved in it, because you cannot go back and re-derive history you never stored in the first place. So we move fast on screens and slow on anything that records money, identity, or who agreed to what. Those are the parts you pay for twice. Probably worth working out which record in your product would be the scariest to change six months from now.
- · permalink
If your checklist only takes yes or no, an unanswered question lands as a no. That's how a chatbot ships without its disclosure: nothing on file said it was needed, and nothing said it wasn't.
Unknown is a third state. It goes to review, and it gets recorded as not stated, not as false. https://shipmindlabs.com/c/08c26941
- · permalink
An unanswered question is not a no.
In disclosurekit, our open-source project, assess() derives the transparency duties of Article 50 of the EU AI Act from a single table of five rows: 50(1) for providers of systems that interact with people, 50(2) for machine-readable marking of synthetic content, 50(3) for emotion recognition and biometric categorisation, and 50(4) for deep fakes and for AI-generated text on matters of public interest.
The table is the easy bit. What matters is what happens when the caller cannot answer a row.
An input given as unknown does not decide anything. The duty goes to needsReview, neither applied nor cleared, and it sits on a list that a human has to close.
An input left out decides like false, but gets recorded as not stated. Same outcome, different provenance. Six months later you can still tell "we assessed this and it does not apply" apart from "nobody ever filled this field in."
Scope is a separate thing from obligation. A duty addressed to a role you never claimed shows up in neither list, not applied and not pending. If you did not claim the deployer role, the deployer duties are not your open questions.
Exemptions are mostly not exemptions either. An artistic deep fake comes back as an applied duty with a note about the manner: the disclosure still exists, it just has to be made in a way that does not ruin the work.
Our take: any engine that maps regulation to a boolean will quietly turn missing data into compliance. The honest output has three states, applied, not applied, and we do not know yet, and the third one is the only one that produces work.
If you have built this kind of checker, we would like to hear where your unknowns end up and who owns closing them.
- · 10 min readWithdrawing consent should be the same call as granting it
The GDPR asks that consent be as easy to withdraw as it is to give. That reads as a requirement about buttons, and it is one, but it lands on the API underneath first: if refusing takes a different…
- · permalink
Years back one of our engineers went quiet on a payments service for a few days. They had hit a reconciliation mismatch, assumed it was their own bug, and kept digging alone. It was not their bug. It was upstream, and the people handling customer money needed to know that morning, not later.
So the last part of every interview now is one question: tell us about something that broke in production. We barely listen to the fix. We listen for when they told someone, and who they told.
Good engineers debug quietly. The ones you can hand a live service to will say it out loud, early: they do not understand what they are seeing yet, but here is what they are seeing. That instinct is almost impossible to train in, and very easy to hear in ten minutes.
That is the one question we would never skip when hiring.
- · permalink
We shipped a universal OTP code for QA. It sat in prod for two weeks before anyone noticed. Now otpguard test mode wants a code and a TTL, caps at a day, logs CRITICAL when it switches on, ERROR on every use, and won't load if the env says production.
- · permalink
We kept finding that granting consent is one clean call, while withdrawing it is a second path with an extra flag and a confirm step. That gap lives in the API, not the banner. Same call, same log record, opposite value, or the symmetry is already gone.
Probably worth checking where yours breaks. https://shipmindlabs.com/c/f8abb8a0
- · permalink
Refusing consent is the same call as granting it.
Most consent code treats refusal as an absence: the user didn't tick the box, so nothing gets written. Then the audit question arrives. Did this person decline analytics, or were they never asked? An empty record answers both, which means it answers neither.
So in consentcore, our open-source consent library, grant() and refuse() are deliberately the same function twice. Same parameters, same order, same return value, same entry in the log. A refusal is not a missing grant, it is a decision with a timestamp, and it costs the caller exactly as much code to record as a yes.
The asymmetry lives in the effect, not the signature.
A grant is additive. It adds to what was already granted and leaves the rest of the state alone.
A refusal is subtractive and narrow. It takes away what it names, and nothing else. It does not reset the record, because a later "no" to marketing is not a withdrawal of everything decided before it.
One exception, and it is not negotiable: necessary survives being named. If a caller refuses the necessary category, the call still succeeds, still writes a log entry, and the category stays on. There is no state in which a service refuses its own ability to function, so the library will not construct one.
The second decision is that purposes and vendors are declared as data, not as identifiers scattered across call sites. Every purpose belongs to a category, and a purpose is answered through its category. Two purposes under one category are one answer. You cannot end up with analytics granted for one vendor and refused for another under the same declared category, because there is only one place that answer can be stored.
That constraint annoys people for about a day, and then it starts paying. It is what makes the state finite, the log replayable, and the question "what is this user's current answer for X" a lookup rather than a reconstruction.
The general rule we keep relearning in compliance tooling: if your negative path is cheaper to write than your positive path, it will probably be under-recorded, and the gap only becomes visible when someone external asks you to prove a decision existed.
So it matters whether you store a refusal today as a row, or as the absence of one.
- · permalink
A client called it a five-minute change: one extra field on the product card. Three days later we were still finding out who read that field, a parser, a cache, a mobile build nobody had touched in months.
Now we count the readers before we quote. Your smallest ticket is probably bigger than it looks.
- · 9 min readSplit five cents three ways: allocation that sums back exactly
Totals drift at the point where an amount is divided. Five cents split three ways cannot be three equal amounts, and a payout split pro rata across investors almost never lands on whole minor units,…
- · permalink
We were hiring for a payments team a while back, and two candidates looked more or less identical on paper. During the walkthrough we asked each of them how a particular provider behaves when a charge gets retried, a detail nobody could know from the outside. One answered confidently and was wrong. The other said they didn't know, then described exactly how they'd find out: read the provider's retry rules, reproduce it on a test account, and leave a test behind that fails if the behaviour ever changes. We hired the second one.
Since then every interview we run includes at least one question the candidate can't possibly know the answer to. In systems that move money the expensive mistake is rarely ignorance. It's someone quietly guessing instead of checking, and you probably want to know which of the two you are hiring.
- · permalink
We once quoted a payment integration by counting endpoints. Eleven of them, two weeks, easy. Then production arrived with partial captures, late refunds, and a chargeback that landed after the order was already archived, states nobody had written down anywhere. The overrun came from states rather than code.
Now we size an integration by how many states money can sit in, not by how many endpoints the docs list. Curious whether anyone estimates these on a better unit.
- · permalink
Splitting money is not division. It is handing out every minor unit and being able to say who got the last one.
The failure is familiar. You split 100 across three parts, round each one, and suddenly the parts no longer sum to what you received. Someone patches it by adding a cent to the first row. Then a negative amount arrives (a refund, a reversal) and the patch allocates in the wrong direction.
In our open-source project amountfield we made that arithmetic explicit. Everything is bigint minor units, so there is no float to lose. Allocation floors each share, then hands the leftover units to the largest remainders. The parts sum back to the original exactly: for equal parts, for weights like 3 and 7, and for negative amounts. No two equal shares end up more than one unit apart.
What the API refuses is the more useful part.
Multiplying by a ratio accepts text, a bigint, a whole number, or a numerator and a denominator, but not the JavaScript number 0.1, because that value is not 0.1. It also throws if you did not name a rounding mode, since there is no safe default when the result lands between two units. Addition refuses to mix currencies outright.
Our take: in money code, a thrown error at the boundary is cheaper than a silent cent. Rounding policy is a product decision, so the library should probably make you state it rather than guess on your behalf.
Somewhere your codebase already decides rounding, either down in the domain layer or wherever the formatting happens.
- · permalink
Split five cents three ways and one party gets the extra cent. It reads like a rounding bug. It is a decision nobody wrote down.
We work in minor units, floor the shares, then give leftovers to the largest remainders so the parts sum back exactly. And multiplication has to ask for a rounding mode with us, instead of quietly picking one. Probably few people on your side can say who gets that extra cent. https://shipmindlabs.com/c/bd7c04f1
- · 8 min readA credit note is a type code, not a minus sign
A refund issued as an invoice with negative amounts puts the direction of the money in the sign of every figure on the document, instead of in the one field a receiver is required to read. EN 16931…
- · permalink
We keep seeing refunds booked as invoices with negative lines. Any receiver that reads the amount instead of the document type will book that money twice.
A refund is a credit note: positive amounts, direction carried by the type. Reject negative totals at your own edge. Better you do it than someone downstream.
https://shipmindlabs.com/c/77cc04c4
- · permalink
Most of what breaks a business website happens somewhere other than the website. Check who actually controls the domain, whether certificate renewal is automated or sitting in someone's calendar reminder, whether anyone receives the form submissions today, whether your backups have ever been restored, and whether one person is the only one who can deploy a change.
Four of those five live in accounts and processes, not in pages, which is why a site can look perfect the day it quietly stops working. We'd guess one of the five would take you much longer to answer with certainty than the other four.
- · permalink
If you split a monolith while both services still write the same tables, you get a distributed monolith with extra latency. We draw the boundary at data ownership rather than deployment units, so the question we start with is who owns the write path for each table.
- · permalink
A teammate went on leave and we found out their service had a deploy step that lived only in their head. Nothing broke, we just waited.
Now, before any planned absence, someone else runs that service for a day while the owner stays quiet in the channel. Handover docs always pass review. That day doesn't.
You probably have one thing on your team that only one person can actually do.
- · permalink
A credit note is not an invoice with a minus sign. It is a different document type, and every amount inside it stays positive.
Easy to agree with, awkward to implement, because most billing code models a refund as a sign flip on an existing invoice. The receiving system then rejects the file, and nobody can tell whether the data or the format is at fault.
Here is how we settled it in invoicerules, our open-source validator.
The document type code, BT-3, with 380 for an invoice, 381 for a credit note, and the rest of the UNCL1001 list, is the single switch. It decides the root element, the line element and the quantity element. Nothing downstream re-decides it. One rule set checks both documents, so a rule you write once applies to a refund and to a sale alike.
The sign rules follow from that. A negative line amount, a negative taxable amount or a negative document total is refused, and in our codes that is INVOICERULES-CN-01. A negative item net price is refused under BR-27. Direction of money is carried by the document type, not by the sign of the numbers.
One exception is real. The payable amount, BT-115, may be negative, which is what you get when more was prepaid than was owed, and refusing it would break legitimate settlements.
A credit note also carries no DueDate element. There is nothing to be due.
The last part is not cosmetic. Failures are reported on paths inside the CreditNote tree, the same paths the receiver will quote back when they reject the file. A validator that reports Invoice paths for a credit note sends your finance team hunting for an element that does not exist.
So it comes down to how you model refunds today, as their own document type, or as negative lines corrected at export time.
- · permalink
We ran a product catalog through an LLM translation pipeline and shipped it. Search broke the next day. The prose was fine, but model numbers and pack sizes got translated too, so nothing matched anymore. We had handed the model a product title and called the whole thing text.
Now non-translatable spans get marked before the model sees them, and a batch fails if a digit changed. Some part of your data is probably not language at all.
- · permalink
Splitting one repayment across many investors is not division. It is allocation, and somebody has to own the last cent.
On a P2P lending platform a borrower repays a single amount, and that amount has to become a credit for every investor in the deal, pro-rata to their share. The arithmetic almost never divides cleanly. Two decimals and a rounding mode look harmless until the parts you wrote to the ledger stop adding up to the money you received. The gap is tiny per payment and permanent per platform.
What worked for us:
Do all of it in minor units, as integers. Floats have no business in a ledger.
Floor every share, then hand out the remaining units by an explicit rule (largest fractional part first, ties broken by a stable key), so the same input always produces the same split.
Assert that the sum of the parts equals the amount received before a single row is written. That is an invariant in the code path, not a test you run nightly.
Store the rounding rule with the allocation, not just the result.
The effort pays off for a reason that has little to do with accounting tidiness. Support can answer why one investor got 3.33 and another got 3.34, and a replay produces the same answer. Rounding is a product decision, and it needs an owner who is not the developer who happened to write the loop.
If you move money between multiple parties, you probably face the same choice: spread the remainder across participants, or designate one bearer for it.
- · permalink
A client brought us a redesign they were genuinely proud of. It looked great. Then they asked us to change one price on the pricing page, and we found out why they were asking us at all: every number, every line of copy, every quote was baked straight into the layout. Nothing on that site could change without a developer.
So nothing changed. Prices drifted away from reality, an old promotion sat live for months, and a product they had stopped selling still had its own page. The site was beautiful and quietly wrong.
We left the visuals alone. The first stretch went into pulling the parts that change (prices, copy blocks, service descriptions, case summaries) out of the code and into a small admin panel their operations lead could use without opening a ticket.
Now, before we sign off on any design, we ask one plain question about every screen: who changes this, and how often? Anything that changes monthly gets an editor before it gets polish. Anything nobody will ever touch can be as handcrafted as you like.
Something on your site is probably out of date right now, purely because updating it means asking a developer.
- · permalink
What someone knows matters less to us than what they do when the requirement is missing.
So our task is deliberately underspecified: no stated behaviour for a failed callback. Juniors pick one and ship. Seniors ask who owns the retry, or they write the assumption down in the PR. Same code either way, and the difference is whether the gap got named.
That gap is probably the thing a CV cannot show you.
- · permalink
A courier taps "Delivered" in a basement with no signal, and what gets recorded is an intent, not a fact.
We ran into this building role-specific apps for couriers, warehouse staff and store operations. Field work happens exactly where connectivity dies: freight elevators, loading docks, cold rooms. The app has to accept the tap anyway, and the truth gets reconciled later.
Two things make that safe.
First, every action carries a client-generated operation id and the device timestamp of the tap. The id makes replays harmless when the queue retries. The timestamp preserves what actually happened, because sync order is not event order.
Second, the server applies a state machine instead of last write wins. A "picked up" that arrives after a "delivered" is not the newest truth. It is a late message, so it gets rejected or parked for a human, and it does not get written over a final state.
The part most teams skip is the UI. If a queued action looks identical to a confirmed one, the courier assumes the job is closed, and support inherits the gap between what the app shows and what the system knows. Pending has to look pending.
Offline-first is probably closer to a second source of events with its own ordering rules than to a caching feature. It belongs in the domain model rather than the network layer.
If you run field or warehouse apps, you end up choosing between auto-resolving the actions that sync hours late and parking them for a human.
- · permalink
A client once told us their checkout looked cheap and asked us to make it prettier. We opened it, and honestly the design was fine. The problem was the seam: clicking Pay moved the customer to a page on a different domain, the logo arrived a beat after everything else, and the font wasn't the one used on the rest of the site. For about two seconds it felt like you had been handed off to a stranger to type your card number into.
We changed almost nothing visual. The payment page moved under their own domain, the branding got served with the page instead of loading after it, and the error messages got rewritten in the shop's voice rather than the payment provider's: "that card was declined by your bank, try another one" instead of a code and a shrug.
Now we walk the handoff points before anyone discusses fonts or spacing. Where does the customer leave your site, even briefly? What loads last, and who wrote the words that appear when something goes wrong? Those three moments do more damage to how expensive a site feels than any design choice, because they're the moments when money is on the line and the polish drops.
Somewhere on your site a customer gets handed over to something that isn't yours, and that spot is probably worth walking through yourself.
- · permalink
We closed the same reconciliation ticket three times. Different reporter each time, different month, and every fix looked right locally. On the fourth one we finally read all three closures together and saw it: nobody had written down which timezone the daily cutoff belonged to. Each of us had assumed a different one.
Now a third reopen stops being a bug report and becomes an architecture question. We are curious what a repeatedly reopened ticket usually means on your team.
- · permalink
Every pro-rata payout leaves a remainder cent. If nobody decided who gets it, your rounding code made that business decision on its own, and reconciliation is where you find out. We think it's worth writing down where the remainder lands in your system.
- · permalink
A scraper that returns nothing is not an empty catalog. It is an outage that reports success.
We have kept retail catalog parsers running for a cross-border e-commerce platform for years, and the failure mode is almost never a crash. The site ships a layout change, the price selector stops matching, and the request still comes back 200. The job finishes. The dashboard is green. Downstream, products quietly lose their price, or drop out of the catalog entirely, and the first person to notice is a customer or a support agent.
What fixed this for us was refusing to let a run write straight into the live catalog.
A run goes into a staging batch. The batch carries its own statistics: rows parsed, share of rows with a missing price, share with no image, count against the previous successful run for the same source. Promotion into production data is a separate step with rules attached, so a source that suddenly yields a fraction of its usual rows does not promote, it raises an alert and keeps yesterday's data.
The mental shift is small but it changes the whole design: parsing is not a write path, it is a proposal that has to pass review. Stale data is a known, bounded problem. Silently corrupted data is not.
If you run scrapers or third-party imports in production, you probably know the version of this that bit you: what signal actually told you a source had broken, and how long it took.
- · permalink
A founder asked us for one backend developer, part-time, for a couple of months. Two weeks in, most of the work turned out not to be backend at all: a payment flow needed reconciling, the release process lived in one person's head, and nobody had checked what happens when a card payment retries.
The developer they asked for could have handled any one of those. Doing all three in sequence, alone, would have eaten the whole engagement. So we put the hours where the month actually needed them: backend for a stretch, then a QA pass, then someone to make deploys boring. Same budget, different shape.
Now we open these conversations by asking which roles a month needs rather than how many people, because part-time headcount is easy to price and usually the wrong question. If you had one developer for a month, what you would hand them first is the thing worth naming out loud.
- · permalink
You can't recompute a credit score. It is a decision, and a decision has to be reproducible months later.
We hit this while building in-house scoring and rating models for a lending marketplace with borrower tiers and moderation.
The pattern repeats. A borrower disputes a rejection, or a reviewer asks why a deal was issued at the tier it was. Someone reruns the scorer against the current database, and it comes back with a different answer.
Nothing is broken. The borrower's data moved on, the model was retrained, a threshold was tuned. But now you have no way to show what was actually decided, or why.
What works for us: treat every scoring run as a write-once artifact. The feature values exactly as they were read, the model version, the thresholds in force, the resulting tier, and the reason codes. The live model stays free to change, the record does not. Re-screening writes a new decision next to the old one instead of overwriting it.
The storage cost is trivial next to the cost of arguing without evidence.
There is a second payoff that we think most teams miss: this is also the only honest way to evaluate the model. Comparing predictions against real repayment behaviour needs the inputs that were actually used, not today's version of them.
So if you run scoring or risk models in production, it is worth checking whether you persist the feature snapshot with the decision or recompute it when someone asks.
- · permalink
Founders ask for "a senior Python dev." What they usually need is someone to own the three services nobody wants: payments, KYC, notifications. Stack is the cheapest thing to match, since a good engineer picks up your framework in a week.
Ownership of an unloved subsystem takes months to transfer, and it's what stalls your roadmap when someone leaves. We'd hire against the map of what's unowned, not the job title. Most products have a subsystem that only one person understands.
- · permalink
A beautiful website can still be a bad website, because beauty gets judged on the one screen where everything works.
Design reviews happen in the happy path. The form is filled in correctly, the payment goes through, the search returns results, the image is exactly the right size. Nobody signs off on the screen where the card is declined, the upload times out, or someone taps Pay a second time because the button gave no sign it had heard them. Those screens still exist. They get invented late, probably by whoever happens to be writing that part of the code, and you meet them as a raw error message or a page that just sits there.
That is where the cost lands. A payment button with no visible submitting state is how someone gets charged twice. A slow response with no feedback is how people refresh and order again. A form that loses what you typed is how a qualified lead quietly leaves. None of that shows up in a design mockup, and all of it shows up in the support inbox.
More polish does not fix this. What we do instead is decide, before anything gets built, what each screen does when the thing it depends on fails or takes too long. That work is unglamorous, and it separates a site that looks finished from a site that behaves well on a bad day.
Most failure states on your own site are ones you have never actually sat down and looked at.
- · permalink
We used to run panel interviews with everyone in the room. Whoever spoke first set the tone: one "weak on architecture" in minute two, and the rest of the hour was the rest of us agreeing with it.
Now each of us writes a verdict before anyone speaks. They disagree far more often than we expected, and that disagreement is the part worth discussing. Your own process probably rewards the loudest interviewer somewhere, quietly.
- · permalink
We once took over a payment service that had run in production for two years with no staging environment. Everything worked. The team had simply not needed one, until they did, and by then every change had to be tested on live traffic with real money moving through it.
The cost showed up in speed, not in a single outage. Small improvements sat untouched for months because no one wanted to be the person who broke checkout on a Friday. The founders thought they had a code quality problem. What they had was a missing environment problem.
Setting up a proper staging copy took us a couple of weeks. Reconstructing two years of undocumented integration behaviour, so that staging actually resembled production, took much longer, and that is the part few people budget for. The groundwork you skip early does not stay skipped. It waits and charges interest.
Now we treat the boring setup (staging, deploy path, a way to roll back) as part of the first delivery, not as something to do once the product proves itself. It is much cheaper to build a second environment when there is almost nothing in it.
You probably have a piece of groundwork you postponed and later wished you hadn't.
- · permalink
Two orders, one balance. That is the bug class an exchange has to design out before it has a single real user.
The naive flow reads like it works: check the user's free balance, accept the order, put it on the book, settle on match. It survives testing because a test client places orders one at a time.
In production two requests land in the same millisecond, read the same free balance, and both pass the check.
What fixed it for us was moving the truth out of the order row. Placing an order is a transfer from available to reserved, in one transaction, against a locked balance row, not a write to the book. The book only receives the order if the reservation succeeded. Cancels and partial fills release the exact remainder that was held, never a recomputed number.
The same rule applies to custodial withdrawals: a pending withdrawal is reserved funds, not a row with a status. If it only exists as a status, a second withdrawal request will be authorised against money that is already leaving.
So the test that matters is not whether a user can place an order. You want to know whether two concurrent requests can both succeed when only one should.
If a balance check and the action it authorises still live in two separate statements somewhere in your system, that is probably where this shows up.
- · permalink
A webhook endpoint is a public API you never designed. Payloads arrive out of order, twice, or a day late. We treat the payload as a hint and re-read state from the provider. Somebody owns ordering in your integrations, and it is worth knowing who.
- · 8 min readA deep link routing table is an authorisation boundary
A deep link is a URL that anyone can send you. It arrives from an email, a text message, another app, or a QR code on a poster, and it picks a screen in an app that may already be signed in. The…
- · permalink
We hired an engineer who aced every round. Two weeks in, on a payments service we'd inherited, his retry fix double-charged test accounts in staging. Not really his fault, we had tested writing code and never once tested reading someone else's.
Now the trial task is a repo we didn't write: tell us what breaks and why. Harder to grade, and different people pass it. You are hiring either for writing or for reading, and those turn out to be two different things.
- · permalink
The most useful signal from a candidate shows up before a single line of code: the questions they ask about a vague task.
We hand people something small and deliberately under-specified, like adding refunds to an existing payment flow. The good ones stop to ask who is allowed to issue one, whether partial refunds count, and what happens if the original payment settled days ago. The ones who start building right away usually build something perfectly correct for a problem nobody actually asked about.
Real client work arrives exactly this under-specified, so we hire for the instinct to clarify first and build second. You probably have a question you wish someone had asked you before they started.
- · permalink
A deep link is a URL anyone can send, and it picks a screen. So the routing table is an authorisation boundary, not a switch statement.
Placeholders declare a kind and get validated before a screen sees them. Host and scheme are allow-listed per route, query params too, and a link that matches nothing goes nowhere. Undeclared placeholder, and the route fails at construction.
Worth checking where deep-link validation lives in your app: router or screen. https://shipmindlabs.com/c/0a5f4a4a
- · permalink
A blockchain deposit is not a fact when you first see it. It is a proposal the chain can still retract.
Most integrations treat the node callback as an event: transaction seen, balance credited, user notified. That holds until the chain reorganises, until the provider replays the same transaction after a reconnect, or until a pending transaction gets replaced with a higher fee.
We keep the deposit as its own record with its own state machine, separate from the balance: seen, confirmed to a depth, credited, and, because this state has to exist, reversed. Crediting becomes a transition rather than a side effect of receiving a webhook.
Two details carry most of the weight.
Confirmation depth is a risk policy, not a constant in a config file. Small amounts can move on shallow confirmation, and amounts that change your exposure wait longer. Same chain, different thresholds, decided by the people who own the risk rather than by whoever wrote the listener.
The reversal path has to be designed on day one. A deposit that disappears after being credited is not a rare exception you handle later, it is what happens when a probabilistic ledger meets a deterministic one.
We have built custodial wallets and on-chain payment flows through three market cycles, and the pattern holds: the hard part was never watching the chain. It is deciding at which point what you saw becomes money.
That line between seen and spendable is a number, and someone in your organisation owns it.
- · 9 min readConsent is a log bound to the notice it answered
A consent banner that flips a boolean can tell you what a visitor chose. It cannot tell you when they chose it, which text they were answering, or what had already run before they answered at all.…
- · permalink
We inherited a service with a cron job named temp_fix. No docs, no owner, older than half the stack. We left it alone for a month and then finally traced it: it was quietly patching a broken reconciliation join. Nothing about it was temp.
Now every legacy handover we do starts with listing what we are scared to delete. You probably have a list like that sitting somewhere too.
- · permalink
Verified is not something a user is. It is a decision about one document, from one provider, at one moment in time.
Store it as a boolean on the user record and you have thrown away everything that made it defensible.
We have built KYC and KYB flows with real-time moderation queues behind them, and the failures all have the same shape. An ID expires. A provider revises a verdict. A screening list updates overnight. Nothing can be recomputed, because the only thing left in the database is true.
We keep the decision instead: which provider answered, which document version it saw, what it returned, who on the compliance side overrode it and on what grounds, and when the decision stops being valid. The status you show the user becomes a derived value, computed from decisions still in force rather than from whatever was written last.
Two things change once that is in place. Re-screening stops being a data migration and becomes a query: find every decision past its validity window. And a compliance officer asking why an account is open gets an answer from the record, instead of an engineer reading code and guessing.
Screening is continuous, and modelling it as a one-time event is the design error, which tends to surface during an audit rather than in testing.
If you run KYC in production, the move from storing the status to storing the decision probably came out of an expiry, a provider changing its verdict, or an audit you could not answer.
- · permalink
A while back we were hiring for a backend seat on a payments team, and one candidate looked like the strongest on paper. Near the end we asked the question we now put to everyone: tell us about something you broke in production. They said nothing came to mind, then offered a story where a release went wrong and QA had missed it.
We passed. Not over the incident itself (most people who ship code that moves money have one), but because the version they told had no version of themselves in it.
The person we did hire answered the same question with a duplicate-charge bug: what caused it, what the fix was, and what they added to the deploy checklist afterwards. That is the person you want reachable at 2am, because they already think in terms of what they can prevent next time.
We kept the question. It tells us more than the take-home task did.
What's the one interview question that tells you the most about someone?
- · permalink
A migration that runs is not finished. It is finished when the old read path is deleted. Every day both paths stay live is a day the data can drift apart unnoticed, and we suspect your dual-read window has been open longer than you planned.
- · permalink
A consent flag says yes. It can't say yes to what.
Bind the decision to a hash of the notice and an edited banner re-asks on its own, no version bump needed, and the log exports as a chain you can verify instead of trust. In an audit, what your consent record can actually prove is the part that counts.
https://shipmindlabs.com/c/83c16653
- · 7 min readDeduplicate notifications by the caller's key, not the message text
A retried dispatch and a double-fired event look the same from inside a notification service: same recipient, same template, same rendered body. Deduplicating on that body suppresses the wrong…
- · permalink
The client says the upload finished. That is a claim about the browser, not a fact about your storage.
With a presigned URL the bytes go straight from the browser to object storage, so the backend never sees them. Most implementations still mark the document attached the moment the client posts "done".
Then a connection drops mid-transfer, and a deal ends up with a signed contract that is forty kilobytes of nothing.
What we do instead: the presign call is the reservation. It creates the record in a pending state, and the upload policy pins what is allowed to arrive, the key prefix, the content type, a maximum size. A client cannot rewrite the key or push a huge file into an avatar slot.
The record becomes attached only when we confirm the object from the storage side, either a storage event or a HEAD on the key, checking size and type against what was reserved. Anything still pending after a timeout gets swept, object and row together, so orphans do not accumulate.
You get extra validation out of this, but the change that matters is that "uploaded" stops being something the client can assert. In document and compliance workflows, where the file is the evidence, that distinction is the whole feature.
So it is probably worth looking at where the flag gets flipped in your system, on a storage confirmation or on a client callback.
- · permalink
Most website problems stay quiet. They fail without a sound, and you hear about it from a customer weeks later. Five things worth checking today, about ten minutes of work.
-
Send a message through your own contact form and check that it lands in the inbox. Forms break silently whenever a mail provider, a domain setting or a plugin changes, and the page still shows the visitor a cheerful thank-you.
-
Check when your SSL certificate and your domain renew. Both expire on a calendar date, not on a warning, and an expired certificate greets your customers with a browser security screen instead of your homepage.
-
Do a real signup or a small order and watch for the confirmation email. Receipts and password resets are the messages people need most, and they are the first to quietly start landing in spam after any change to how mail is sent.
-
Open the site on a phone using mobile data, not the office wifi. That is how most people arrive, and heavy images that feel instant on a fast connection are what loses visitors on a slow one.
-
Ask who finds out first when the site goes down. If the honest answer is that a customer tells you, that is the real gap, and the first four items are just symptoms of it.
None of this needs a developer. It needs someone whose job it is to know the answer.
You probably already suspect which of the five is out of date on your site.
-
- · permalink
Endpoint count tells you very little about what an integration will cost. How the provider models failure tells you most of it.
One with clear error codes, idempotency keys and a sandbox that reproduces edge cases takes a week. One that returns 200 on a rejected payment, with no replay path, takes a month of building the reconciliation the vendor should have shipped.
We read the error docs before the feature docs. That is our first check on a new provider.
- · permalink
Deduplicating notifications by message text fails in both directions. A retry and a double-fired event look identical to the sender, so one gets swallowed. Two genuinely different events can word themselves the same way, and one gets dropped for no reason.
The key has to come from the caller: order id plus event. Text is a rendering detail, not an identity.
We are curious what you dedupe on today.
https://shipmindlabs.com/c/bf2cbcda
- · 8 min readAccessibility evidence belongs to the build it was taken on
Automated accessibility tools settle roughly a quarter to a third of WCAG. The rest is judgement: whether alt text describes the picture, whether the focus order makes sense, whether an error…
- · permalink
We once took over a shop that looked genuinely good. Clean, well-photographed, the kind of site people compliment. Then we watched the team try to run it: changing a price meant writing to us, fixing a typo meant writing to us, adding a seasonal banner meant waiting for a deploy. So they stopped asking.
Within a few months the storefront was beautiful and wrong: old promotions, products that no longer existed, a contact page nobody could update. Nothing was broken in the way a developer would notice. The site just slowly drifted away from the business. What it cost was trust, mostly. Customers assumed the shop was abandoned, and support spent its days apologising for prices that hadn't been true for weeks.
Now we treat the back-office as part of the build, not a later phase. Before we talk about the homepage, we ask who will change this content on a Tuesday afternoon, and what they should be able to do without us. If the answer is "open a ticket", the design isn't finished yet. You probably already know who updates your site day to day, and whether they can do it without calling a developer.
- · permalink
The ticket described tapping the link in the email and landing on someone else's order screen.
It took us most of a day to reproduce, because nothing had crashed and nothing had thrown. The app did exactly what we had told it to do.
Our deep-link handler was a chain of prefix checks: one long function, maybe forty branches, grown over two years by different people on the team. Each branch matched a path fragment, pulled whatever segments it needed, and pushed a screen. The last branch was a catch-all that tried to be helpful, so if nothing matched cleanly it sent the user somewhere reasonable.
That catch-all was the bug. A malformed link fell through several near-matches, hit the fallback, and the fallback reused an identifier it had parsed two branches earlier. The screen opened with a stale value. No error, no log line, just a wrong screen for a real user.
Once we went looking, the routing code had three more problems we had never named. It did not care what host the link came from, so a link from a domain we do not own opened the same internal screens. It did not care about scheme either, so an insecure link and a custom-scheme link were treated identically. And it forwarded every query parameter it found straight into the destination, including ones we had never designed for.
So we stopped treating routing as control flow and started treating it as a boundary that either accepts input or refuses it.
What we run now is a declared route table. Each route states its pattern, the type of every placeholder, and a constraint per placeholder. Each route also states which hosts it accepts and which schemes. Query parameters are allow-listed per route, and anything not listed is dropped rather than passed through. Validation happens before dispatch, so a route either produces a fully typed destination or produces a refusal.
The refusals are explicit and each one has a name: foreign host, insecure scheme, unknown scheme, form not accepted. We log the name. That single change turned "the link did something odd" into a line in a dashboard that tells you which policy rejected it and why.
And there is no fall-through. If nothing matches, the answer is no match, not "whatever matched last". A user landing on a safe default is a product decision we make deliberately in one place, not an accident of branch ordering.
We pulled the pattern out into an open-source library we call deeplinkmap, mostly so we would stop rewriting the same forty-branch function on each new mobile project.
The part that surprised us: once the table was declared as data, we could review it. A route table you can read in one sitting is an artifact a QA engineer can audit. A chain of prefix checks is not.
For those of you running deep links in production, the place where host and scheme policy actually gets enforced, in the app or at the link-generation layer, is worth naming out loud.
- · permalink
Continuity has little to do with headcount. What matters is whether someone other than the author can cut a release this week.
The mechanics are boring: we rotate who ships, who watches reconciliation, who takes the first alert. Anything that only works when one person is online is just a habit with good luck attached. Docs you never execute are fiction.
Worth asking yourself when you last let someone else run your deploy.
- · permalink
A scanner settles maybe a quarter of WCAG. The rest is judgement, and what decides an audit is what was checked, by whom, on which build.
Evidence taken on a build you no longer ship has quietly turned into a memory. So check your accessibility report and see whether it says which build each check came from. https://shipmindlabs.com/c/54e13c6a
- · 9 min readA money field should refuse an ambiguous amount, not guess it
Three things go wrong in every money input, and only the first one is well known. Floats: 0.1 + 0.2 is 0.30000000000000004, and a total ends up a cent off for reasons the user cannot see. Minor…
- · permalink
We shipped a fiat-to-crypto onramp and watched the queue grow. Not the payment queue, the moderation queue. Real-time KYC meant a human had to look at every borderline case, and we had designed for request volume, not for how fast a compliance team can actually decide.
Now we size review tooling before we size throughput: batching, clear reject reasons, no ambiguous states. Somewhere your system probably assumes a person keeps up.
- · permalink
Dedupe notifications by the caller's key, not the message text. Order id plus event name, checked before any provider call.
Two identical strings can be two real events. The second one is the alert a user actually needed. Text equality is not identity, so pick your dedup key.
- · permalink
An auditor asked us a question we could not answer: show me what this user agreed to.
We had the record. User id, timestamp, analytics true, marketing false, policy version 3.2. It looked complete. Then came the follow-up — what did version 3.2 say on that date? Nobody knew. The notice text lived in a CMS that had been edited in place. The version label had not changed, but two sentences about a processor had. The label was a name someone typed, not a fact about the document.
So we had a consent log that proved a user clicked something. It could not prove what.
What that cost was not a fine. It was weeks — reconstructing page history, comparing exports, ending up with a written statement that said we believe. On a compliance question, believe is an expensive word.
We rebuilt it around one rule: a decision is only meaningful next to the thing it answered. Now the notice text, the purpose list and the processor list get hashed at render time, and that hash is stored in the decision record. Not a version label. The content itself. If a sentence changes, the hash changes, and every later decision is visibly bound to different text. Old records stay valid for the old text, which is the honest outcome — consent does not silently migrate to terms the user never saw.
The second thing we found was uglier. Our analytics snippet loaded before the banner did. For a fraction of a second on every first visit the pixel was already firing, and we were asking for permission after taking it. The record said the user decided. The network tab said the decision arrived late. Auditors read network tabs.
So the gate moved. Nothing tracking-related initialises at all until a decision exists for the current notice hash — no decision, no load, not queued and flushed later. Absent is treated as no, not as not yet.
The last piece is the part I would not skip again. Each decision record includes the hash of the previous one, so the log is a chain. If a row is edited or removed afterwards, verification breaks at that point. It does not make us more honest. It makes our honesty checkable by someone who does not trust us, which is the only kind that counts in an audit.
We ended up extracting this into an open-source project we call consentcore, mostly so we would stop rebuilding it per client.
The habit it left me with is broader than cookies. Any record of a decision — terms acceptance, a risk disclosure, a signed quote — is weak evidence unless the exact wording is bound to it cryptographically.
For those of you who have been through a data protection audit: what did they ask for that your logs turned out not to contain?
- · permalink
A money input breaks in three ways: floats (0.1 + 0.2), the assumption that every currency has two decimals (JPY has none, KWD has three), and separators that mean the opposite thing one locale over.
We store amounts as integer counts of minor units, take the exponent from a bundled ISO 4217 table with an explicit override, and parse partial input using the locale's own separators. Worth checking what your own field does with "1.234".
https://shipmindlabs.com/c/135a55be
- · permalink
A client once asked us to fix the "cheap look" of their shop, and pointed straight at the fonts and the product photos. We spent the first day in the checkout instead: paying handed the customer off to a generic page on someone else's domain, different logo, different everything. We rebuilt that step as a hosted payment page under their own brand and domain, and the visual design of the site never changed at all. What we do differently now is walk the money path before we touch a single pixel, because the moment a site stops looking like itself is probably the moment it asks to be trusted with a card. Your own site hands the customer over somewhere quietly, and that spot is worth finding before a client points at your fonts.
- · 9 min readThe offline write path is a storage problem, not a network problem
Reading offline is a cache: keep the last response, show it again, refresh when the signal returns. Writing offline is a different problem, and it is where mobile apps break. A change made in a…
- · permalink
Reading offline is a cache. Writing offline is a queue you have to design.
A change held in memory dies with the app. Retries without backoff flatten the battery and hand the outage back to your own server. A request in flight when the OS kills the app stays stuck, and one rejected change blocks every good one behind it. Four failures, four different fixes, and "try again later" is none of them. One of them probably bit you first. https://shipmindlabs.com/c/a7dd682a
- · permalink
An accessibility check is only true about the build it ran on. Date it instead, and your statement is a guess.
This is where teams slip when they move from ad-hoc audits to a real compliance obligation. The audit arrives as a document with a timestamp, findings get fixed, the next audit happens months later, and by then nobody can say which of those findings still describe what is in production. The evidence stopped tracking the artifact it was about.
We hit this while building a11ytrail, our open-source accessibility evidence tracker, and the fix turned out to be structural rather than procedural.
Every piece of evidence (an automated run, a manual keyboard pass, a screen reader walkthrough) is stored against a build identifier. Not a date, not a release name someone typed in. The build.
Three rules fall out of that.
A claim that mixes evidence from several builds gets flagged at assessment time. It is not silently averaged into a pass, because a criterion proven on build A and a criterion proven on build C do not add up to a statement about build C.
When two checks disagree, the one taken on the assessed build wins, even if the other one is newer. Recency is the intuitive tiebreaker and probably the wrong one. A check run last week against a different build is a fact about a different product.
And publishing is refused outright until every criterion has conclusive evidence on the assessed build, or the statement itself says the assessment is under way. Partial confidence is allowed. Partial confidence presented as a finished claim is not.
The expensive version of this mistake is not a failed audit. It is a support ticket saying a feature is unusable, next to an internal record that cheerfully reports the criterion as passing, because the passing check belonged to a build that shipped two quarters ago.
Accessibility conformance is a property of an artifact, the same way a test result is. Once you treat it that way, the reporting problem turns into a data modelling problem, and data modelling problems have answers.
So the question worth asking on your own team is what your accessibility evidence is actually pinned to today: a release, a date, or nothing in particular.
- · permalink
A trial task came back with a third of the code we expected. The candidate had found an internal service that already did most of it and wired into that instead. Our old scorecard read that as incomplete, and we nearly passed on them. On inherited systems the expensive habit is rebuilding what already runs, and we've paid for that in payment code before. Now we ask what someone looked for before they wrote anything.
You probably have something your rubric quietly punishes too.
- · permalink
A client brought us in for what they called a few hours a week on a payment integration. The first month was quiet. Then a provider certificate expired over a weekend, a reconciliation mismatch showed up days later, and a mobile release needed a change to the same flow. Three different roles, and none of them fit inside "a few hours of a backend developer."
None of that was expensive to do, and each item was small. The expensive part was waiting: everything sat until the one person who knew the system had a free evening.
So we stopped selling part-time engagements in hours and started scoping them by role coverage. The written context lives with the whole team, so whoever is free picks it up (backend, QA, DevOps) and nothing queues behind a single calendar. Some months that really is a few hours. The difference is those few hours happen on the day you need them.
If you work with an outside developer today, your project probably just waits out the week they are unavailable.
- · 9 min readA discount has a VAT rate: EN 16931 totals from the bottom up
An invoice carrying a discount is the easiest way to produce a document that agrees with itself and is wrong anyway. The lines add up, the totals agree with the lines, and every arithmetic rule on…
- · permalink
We once deleted a cron job nobody could name. No docs, no owner, no recent commits. The next reconciliation run came back wrong — that job had been quietly re-queuing failed payouts for years. Now anything unowned gets a log line and one full cycle of watching before it's removed. Dead code is easy to spot; dead-looking code isn't. What's the thing in your repo nobody will claim?
- · permalink
Our courier app kept offline edits in memory. A phone rebooted mid-shift and the queue went with it. Now the change hits disk before it is enqueued, each attempt is leased so stuck ones recover on start, and exhausted ones get parked with a reason instead of blocking the rest.
- · permalink
From 2026 a rejected invoice is no longer an unanswered email. It's a document the platform refuses to accept as issued: Peppol in Belgium, KSeF in Poland, mandatory reception in France.
Under clearance, failed validation means the invoice never existed. So validate at your own desk, and fail with the rule id and the element path, not a generic 500. You probably want to know where your invoice pipeline currently finds out it was wrong. https://shipmindlabs.com/c/e2a2f004
- · permalink
A money input should not hold a float, and it should not assume two decimals.
Both assumptions look harmless right up until they reach production.
The float is the obvious one. 0.1 + 0.2 in binary floating point is not 0.3, and once a value has passed through a JavaScript number you cannot reliably get the original back. Rounding at the end does not fix that. What fixes it is that the amount never becomes a float at all: it is an integer in minor units from the moment you parse it, it stays an integer through validation and transport, and it becomes a human-readable string only at the last step, for display. Formatting is an output concern. Arithmetic does not touch the formatted form.
The two-decimals assumption is the quieter one. ISO 4217 assigns each currency an exponent, and it is not always 2. JPY has 0 minor units. Several currencies use 3. So your field cannot hardcode a scale, it needs the exponent per currency, and it needs an explicit override path for the cases the table does not cover: internal accounting precision, crypto amounts that need more digits than any fiat table will ever list, a pricing unit your finance team defined for one specific product. An override the caller sets deliberately is safe. A silent fallback to 2 is not.
Then comes the part most teams underestimate: partial input. A user typing an amount produces a broken string on every keystroke, and the separators differ by locale. "1,234.56", "1.234,56" and "1 234,56" are the same amount written by three different people. Guessing which one is the decimal separator with a regex works until someone from another market opens your form. Intl already exposes the locale's group and decimal separators, so your parser can read them from the locale instead of inventing its own rules, and it has to stay stable mid-typing, when the input is still "1." or "1,2" and not yet a valid number.
We pulled this into a small open-source component we maintain, amountfield, because we kept rebuilding the same logic across payment forms, invoicing screens and admin panels. Same three problems every time.
The general shape holds well beyond the UI: integers in minor units, currency exponent as data rather than a constant, formatting strictly at the edge.
If you work across multiple currencies, the two-decimals assumption probably surfaced somewhere already, in the input, in the totals, or in a downstream report.
- · permalink
A beautiful website can still be a bad website. Most design work shows one version of every screen: the one where the data loaded, the form was filled in correctly, and the payment went through. Real users land on the other versions, the empty catalogue, the slow response, the declined card, the half-finished order, and those screens usually get invented at the last minute by whoever is building them. That's where a polished site starts to feel cheap, not in the typography but in the moment something goes wrong and the page has nothing honest to say. When we review a design, we ask for the unglamorous states before we ask about the beautiful ones. What does a page look like when it has no data yet, when it's waiting, when it fails? There's probably a state on your site that users hit often and nobody ever designed.
- · permalink
Under clearance e-invoicing, an invoice that fails validation was never issued. Not delayed, not rejected: legally nonexistent.
That moves where correctness has to live. If the tax platform is the thing that grants existence, our own validation has to reach the same verdict before we submit, otherwise we are shipping documents whose legal status we cannot predict.
Three parts of that in practice.
We evaluate EN 16931 business rules locally, and every report entry carries the rule id and the element path that failed. "Invalid invoice" is not a result. BR-CO-13 on a named element is, because that maps to a field in the interface and to an answer for whoever has to fix it.
We compute the VAT breakdown per category and rate, not per line. Lines get grouped by tax category, rate and exemption reason, the taxable amount is summed for the group, the tax is computed on that sum, and rounding happens at group and total level in the order the standard defines. Round every line and add the results up, and a one-cent drift turns into a rejected document.
We model line-level allowances and charges as their own structures, the way BR-41 through BR-44 require: amount, reason, and a base amount plus percentage that reconcile with each other. Fold a discount into a unit price and the totals stay arithmetically correct while the structure does not.
Our take: in a clearance regime, validation is the issuing step rather than a pre-send check. The document stays a draft until the platform says otherwise, and the data model should say so too, with the clearance response stored as the artifact that proves the invoice exists.
If you are already live on a clearance platform, you probably know which rule class generates most of your rejections: totals and rounding, or party and identifier data.
- · permalink
A founder asked us for a fast senior Python dev. Two weeks in we hadn't shipped a single feature: we were reading an undocumented payments ledger, trying to work out why balances drifted. Speed wasn't what he needed. He needed someone to own reconciliation.
So now we ask what's currently broken before we ask what stack you're on. Worth thinking about what your last hire actually spent their first month doing.
- · permalink
The most useful thing we ask an engineer in an interview isn't to build something. It's to read a piece of code they've never seen and tell us what they would delete.
Writing from scratch is the easy half of the job. Most of the work on a live system is arriving in the middle of something someone else built, under deadline, and figuring out what is safe to touch. Deletion is the sharpest version of that question: to remove something confidently you have to know what depends on it, what happens to the data when it's gone, and what nobody wrote down. Strong candidates slow down, ask what runs in the background, and name the one thing they wouldn't touch without checking first. Weaker ones tidy up whatever looks ugly.
We like it because it can't be rehearsed, it takes about twenty minutes, and it predicts the thing clients actually feel: whether a new person can join a running project without breaking it.
What's the last thing you removed from a working system that turned out to be holding something up?
- · permalink
An LLM translation is not content, it is a cached derivation of a source field, and if you don't record what produced it, you can't tell stale from approved.
We run LLM pipelines that translate and adapt product catalogs at scale. The failure mode is rarely bad language quality. It is silence.
Someone edits a product description or a spec line in the source locale. The translated rows stay exactly as they were. Nothing throws. No queue backs up, no alert fires. The catalog quietly serves last quarter's specification to everyone reading it in another language, and you hear about it from a customer.
What fixes it is treating every generated field as derived data with a provenance record: the hash of the source text it came from, the model, the prompt version, and a review state. Regeneration then runs off hash mismatch instead of a schedule or a human hunch. Anything a reviewer approved stays pinned to the hash it approved, so a pipeline re-run cannot silently overwrite editorial work.
Our take: tokens are the cheap part here. The cost sits in re-translating text nobody touched, and re-reviewing text a human already signed off on. Provenance turns "what is stale" from a guess into a query.
If you run generation over content that keeps changing, you probably need both: a rule that decides what to regenerate, and a human approval that survives the next run.
- · permalink
A success response from a payment provider is a claim. The truth shows up the next day, in reconciliation against their ledger. Every system we've built with money in it eventually grows that job. Yours probably ran on trust for a while before you wrote it.
- · permalink
A candidate kept stopping mid-task to ask what the client actually wanted. We scored that as dodging and nearly passed on them. Hired anyway. Two weeks in, on a payment integration, they flagged that the spec double-charged on retry, before writing a line of code. We'd have found that in production, with money moving. Now "asks before building" is a scored signal for us, not a red flag.
Probably worth a look at which habits you screen out that you shouldn't.
- · permalink
We once gave a trial task on an inherited lending service. One candidate wrote almost nothing for two days: he read migrations, then asked why one balance column existed twice. That was the old attribution bug nobody had documented. We had been screening for how fast people write code on a clean repo, when most of our real work is reading someone else's decisions first. The task now ships with legacy baggage on purpose.
The choice is what you hand a candidate: a blank file or a mess.
- · permalink
The shipping price your customer paid and the price the carrier eventually charges you are two different numbers. Most checkouts are built as if they were the same one.
A rate call at checkout is a quote. It is priced from the dimensions and weight you declared, for one service level, at one moment in time. Weeks later the carrier re-measures the parcel, applies a surcharge, or reprices the lane, and the adjustment turns up in an invoice with nothing linking it back to the order.
What we settled on across multi-carrier delivery work: treat the quote as a stored artifact, not a number rendered on a page. Persist the full input set next to the order (declared dimensions, weight, destination, service, carrier, timestamp, validity window). Then re-quote at label purchase, compare against the stored quote, and apply an explicit tolerance rule that decides in advance who absorbs a difference: the customer, the margin, or a manual review queue.
The payoff comes later, not at checkout. It shows up when the monthly carrier invoice lands and every adjustment can be matched to an order, a declared measurement and a quote, so you can dispute the wrong ones instead of accepting all of them as the cost of doing business.
Unmatched carrier adjustments are probably one of the quietest margin leaks in e-commerce, because they never look like a bug.
For teams running multi-carrier shipping it tends to come down to one of two things: you reconcile carrier invoices line by line against stored quotes, or the delta gets written off as shipping cost variance.
- · permalink
In an interview, the thing we actually watch for is what a candidate does when the task we hand them is missing a detail on purpose. That tells us more than what they can build.
One recent decision came down to exactly that. Two people wrote comparable solutions: one asked what should happen when the same payment is retried, the other quietly picked an answer and moved on. We hired the one who asked. In payment work a silent guess about retry behaviour doesn't fail in code review, it fails on a customer's statement, weeks later.
If someone joined your team and turned out to be a great hire, we'd like to hear what you noticed about them in the very first conversation.
- · permalink
Staging is a shared hallucination we all agree to have before a release.
It has clean data, one user, and sandbox endpoints that answer on time. Production has ten years of weird rows and a partner API that returns 200 with an error inside. The bug was not in your code, it sat in the gap. We think your staging has yet to reproduce that gap even once.
- · permalink
Retries are not a safety feature. They are a duplication feature, unless the receiver can tell a repeat from a new request.
Every team adds retries eventually. The network times out, the client resends, and now the same payout exists twice. The usual fix is an idempotency key. The usual mistake is generating that key from the HTTP request: a hash of the body, a header the client sets fresh on every attempt, a UUID created inside the retry loop. All of those change when the caller changes, so the second attempt looks brand new to your system.
The key has to belong to the business operation, not the transport. It gets allocated once, when the intent is formed (this payout, for this order, for this period), and it survives the client restarting, the queue redelivering, and the operator clicking twice.
The second half gets skipped more often than the first. Recognising a duplicate is not enough. You have to store what the original attempt produced, then return that same stored result, byte for byte. If a repeat re-runs the handler and merely suppresses the side effect, the caller gets a different response for the same operation, and reconciliation later cannot tell which answer was true. The key maps to the outcome, not to a boolean "already seen".
We build this into payment services and ledger flows by default now, because every provider on the other side assumes we handle replays, and most of them are right to assume it.
So it is worth knowing where your idempotency key actually gets created: inside the client's retry loop, or at the moment the business intent is recorded.
- · permalink
A few years back we ran white-label hosted payment pages: each merchant's own branding, their own domain, our checkout behind it. One morning one of those pages stopped taking cards. Nothing had been deployed, no code had changed. The certificate on that domain had quietly expired overnight, and browsers were refusing to load the page at all.
Nobody was watching that date. The page had worked the day before, so everyone assumed it would work again. We heard about it from a confused customer's support message instead of a monitor, and by then a good part of a business day of card payments on someone else's storefront was simply gone.
These days certificate and domain expiry get monitored the same way we monitor databases and queues: an alert with days of warning, going to the same place as everything else that can take a product down. It's unglamorous, and it sits above design, copy and page speed on our list, because all three of those are useless on a page that won't open.
So if you're checking five things on your website today, start with this one: what gets told before your certificate and your domain renewal lapse. A person's memory is probably not the thing you want there, you want something that will wake someone up.
- · 8 min readA notification template is a contract, not a string
A notification is one of the few outputs in a production system with no undo. Once a provider accepts the payload, the SMS is on a phone, the email is in an inbox, and the push has already lit a…
- · permalink
Integration estimates go wrong because teams price only their own work. The expensive part sits on the other side: sandbox parity, credential turnaround, a spec that only matches production half the time. Your velocity is capped by someone else's release cycle, and adding an engineer does not move that line. We plan integrations around waiting rather than coding. Somewhere in your timeline there is probably an assumption that the counterparty behaves.
- · permalink
A queue that keeps draining can still be hours behind. Depth tells you nothing, the age of the oldest message does. That's probably the thing you want to alert on.
- · permalink
A notification template is a function signature we pretend is a string. The arguments go untyped, so a missing order_id fails at the provider, not in CI.
Declare the variables, validate before dispatch, and the failure moves back to your test suite where it costs nothing. That part is cheap. Who owns that contract, backend or whoever wrote the copy, is probably less settled on your team than it looks. https://shipmindlabs.com/c/cc5346c3
- · 7 min readWhy a six-digit code is safe: entropy, attempts and the pepper
A six-digit numeric code carries 19.93 bits of entropy, a number no password policy would accept. It is still the correct primitive for passwordless login and step-up confirmation, but only because…
- · permalink
The OTP bugs we inherit are rarely in the code generation itself. They sit in the state around it: cooldowns per channel, attempt counters that reset on resend, lockout that quietly never fires.
Swap SMS for email and the invariants should hold. If they move, your state machine probably lives in the wrong place.
Worth checking where yours lives. https://shipmindlabs.com/c/c8269d48
- · permalink
Ten years in, the part of the job nobody warns you about: most of your work gets read by people who weren't there. Not tests, not docs. Naming, boundaries, the commit that explains why. We've inherited enough systems to know the fastest engineer on a team is often the one who made the last person's code easy to change. Someone taught you that at some point, probably later than you needed it.
- · permalink
Most sites that look cheap don't have a design problem. They have a "what happens after you click submit" problem: the button stays clickable, nothing changes for two seconds, so you click again, and now there are two of you in the database.
Polished sites feel expensive because someone decided in advance what the slow moment looks like. You have probably left a checkout without knowing whether the order actually went through.
- · permalink
Divide one payment across many recipients and rounding stops being cosmetic.
Round each share on its own and the parts no longer sum to the whole. You end up a cent short, or a cent over. Nothing errors. It surfaces later, when the payout account drifts from the ledger and someone has to explain the difference in a meeting.
We hit this in lending flows, where a single borrower repayment is distributed pro rata across every investor in a deal.
What works for us: treat the split as one allocation function, not as N independent calculations. It takes the total and the shares, returns parts that provably sum back to the total, and applies a deterministic rule for the remainder (largest fractional part first, ties broken by a stable key, not by whatever order the database happened to return).
Then you write the remainder as a visible ledger entry instead of absorbing it quietly. If someone asks why they received an extra cent, the answer sits in the data, not in the source code.
The same goes for anything that gets divided: fees, refunds, revenue share, tax. Each of those needs one owner in the code and an explicit rule for the leftover. Rounding scattered across a codebase is a leak with no alarm on it.
Your system already decides who gets the leftover cent, probably without anyone having written that rule down.
- · 9 min readAn accessibility statement your audit trail can support
An accessibility statement is a public compliance claim, and in most organisations it is produced the way a summary gets produced: someone reads the last audit report, forms an impression, and picks…
- · permalink
Most people asking "should we hire or outsource?" are really asking who will still understand this system a year from now.
Outsourcing fails when it's one contractor who leaves with everything in their head. It works when the arrangement covers roles, backend, QA, delivery, someone who knows how the deploy works, so the answer to any question is never "we'd have to ask him."
That part is worth checking before you sign anything: if the person you talk to went quiet for two weeks, does the work stop? If you've outsourced before, the thing that broke was probably the handover and not the code.
- · permalink
Founders write the hire spec by stack: Python, React, done.
In payment and KYC work, most of the hard hours go to someone else's system misbehaving: a provider timing out mid-flow, a webhook arriving twice, a sandbox that lies. Framework depth is cheap to hire. Judgement about other people's failures is not, and we're not sure what you screen that on.
- · permalink
A default timeout is someone else's guess about your money. Payment calls, catalog scrapes, and KYC checks each deserve their own number. Most of your timeouts were probably never chosen, they just came with the client library.
- · 8 min readOverlapping delivery zones resolve by explicit priority
Two zones covering the same street is normal: a city hub reaches it, and the store's own ring reaches it too. The overlap is fine. What goes wrong is the resolution, because most implementations…
- · permalink
The expensive mistakes we see are rarely about a bad technology choice. They come from building a product that can't answer a simple question: what happened to this one customer's payment last Tuesday?
Teams spend months on features and nothing on the boring record of what the system actually did, and then every support question turns into an engineer digging through logs for an hour.
If a customer asked you today about a charge from last month, explaining it would probably take someone on your side a while.
- · permalink
One hiring habit we changed: we stopped asking what someone built and started asking what they deleted.
Adding a service is easy. Removing one means you knew who called it, what depended on the shape of its data, and how to prove nothing quietly broke. That is system understanding, not resume surface.
So we want to hear about the last thing you deleted in production, and how you convinced yourself it was safe.
- · permalink
A payment webhook is not an event. It is a retry with an opinion about the past.
We have built payment services on card rails and account-to-account flows for years, and the same failure turns up in almost every codebase we inherit: the handler reads the payload, trusts it, and updates the balance.
That holds until the provider does what every provider does. It retries the same notification because your 200 came back slow. It delivers "captured" before "authorized" because two workers fired in parallel. After an outage it resends a three-hour-old status and overwrites a refund that already happened.
None of that is a bug on their side. Delivery is at-least-once and unordered by design, and the documentation usually says so in one line nobody reads.
What we do instead: the endpoint stores the raw payload keyed by the provider's event id and returns immediately. Processing happens after, once, guarded by that id. State changes are applied as transitions rather than assignments, so a transaction that is already refunded rejects a late "captured" instead of accepting it. And the provider's timestamp, not our arrival time, decides what is stale.
The reframing that helps: the webhook is a hint that something changed, not the description of what it changed to. When money is involved, that hint triggers a reconciliation against the provider's own API, and that answer wins.
If you run payment integrations in production, you have probably already picked a side here: process webhooks inline, or store first and reconcile after.
- · 8 min readCourier dispatch: refusal cooldown instead of a penalty score
Dispatch systems that let a courier decline a job usually record the refusal as a penalty: a reliability score goes down, and the next assignment round reads that score. It works until someone asks…
- · permalink
When we're hiring, we skip the clean task description and hand over a bug report written the way real users write them: something is wrong, no details, no logs.
The fix matters less than what the candidate asks first: who noticed, when it started, whether money moved. That's the same instinct that keeps a live system from getting worse while you debug it. You probably have your own question you wish people asked before they started fixing things.
- · permalink
A retry limit without a dead-letter queue is just a delayed delete. After the last attempt the job either lands somewhere a human looks, or it never happened. You probably can't say where your fifth failure ends up.
- · 7 min readInvoice numbers that cannot skip or repeat
Two invoices carrying the same number, or a year whose numbering jumps from 0041 straight to 0043, are not cosmetic defects. Someone reading the books finds them years later, long after everyone has…
- · permalink
The day a payment provider renames a field, you find out how many files know that field name.
Most integrations start honestly. You get a provider SDK, the response comes back already parsed, and copying it into your own structure feels like wasted work. So the provider's shape leaks: into your ORM columns, into your serializers, into the frontend, into analytics. Six months later that vocabulary is your product's vocabulary.
Then the business wants a second provider in a new market, or the first one deprecates an endpoint, and the estimate comes back as a rewrite.
What we do instead is boring and cheap at the start: one translation layer per provider, and its only job is to turn their response into our own type, and their failures into our own errors. Business code never sees a provider status string. It sees our states.
Two details make it actually work.
We keep the raw response stored next to the translated record. When the provider disputes something, the argument ends with data, not with logs.
And we map error taxonomies deliberately (retryable, declined, needs-manual-review), because every provider names those differently, and the retry logic is the part that costs real money when it's wrong.
We've built payment services, KYC integrations, and multi-carrier delivery on top of this. The layer never looked impressive in code review. It's probably what made adding the next provider a week instead of a quarter.
When you added a second provider, something leaked further into your codebase than you expected.
- · permalink
On a long project, most of the work is not writing new code, it's changing code someone else wrote years ago. So we hand candidates an unfamiliar piece of a system and ask what they'd be nervous to touch, and why. Caution in the right places is what keeps a client's system running while we're changing it.
Every product has a part nobody wants to touch, and you probably already know yours.
- · permalink
Most people treat invoice numbers as labels. They are not. In most tax regimes the sequence itself has to hold up: no gaps, no duplicates, per series and per period. That gets uncomfortable the moment two orders are paid in the same second and both ask for the next number.
We wrote up how we keep that counter honest under real traffic, and why the accounting rules decide what goes on the invoice while the sequence decides whether it counts at all: https://shipmindlabs.com/c/0ed96d4b
Explaining a missing invoice number to an accountant is probably not a conversation you want to have.
- · permalink
Invoice numbers look cosmetic until two requests race and you get 1043 twice, or nothing at all between 1042 and 1044. Gaps and duplicates are both audit problems.
So the counter belongs in the store, scoped per series and per period. VAT rules shape the lines, and the sequence is what makes the document legal.
Worth checking where yours increments. https://shipmindlabs.com/c/10a95223
- · permalink
Founders tell us they need one senior engineer. Then the scope turns out to be backend, infra, release process and QA, three roles wearing one title.
Those hires rarely fail on skill. They fail on an undeclared boundary: nobody said which part gets dropped when the week is short, so everything gets dropped a little. Probably worth rereading the last senior role you wrote, and counting what it actually contained.
- · permalink
A missing invoice number is not a cosmetic bug. It is a gap an auditor will ask you to explain, and "our sequence skipped" does not count as an explanation.
Most teams hit this the same way. The number gets produced inside the request that renders the document, a count plus one, or a max plus one. Two checkouts land in the same second, both read the same value, and you ship either a duplicate number or, once one transaction rolls back, a hole in the series.
The reflex fix is a database auto-increment sequence. It solves the duplicates and keeps the hole: sequences are deliberately non-transactional, so every rolled back attempt burns a number.
What holds up is a counter that lives in the store as a row, scoped per series and per period, locked for the moment of allocation. And allocated when the document becomes final, not while a draft is still being edited. You serialize issuance inside one series, which is affordable, because nobody issues invoices at request-per-millisecond rates.
The split we keep coming back to: the VAT regime decides what lines appear on the document. The sequence decides whether the document is legal at all. Two different concerns, and mixing them is how numbering ends up in the template layer.
So it is worth checking where your invoice number gets assigned, at draft creation or at the moment the invoice is issued.
We wrote the full breakdown here: https://shipmindlabs.com/c/4a83d145
- · 8 min readA permission check should return a denial, not False
Most Django backends answer authorization questions with a boolean, and everything the check knew gets thrown away on the way out. A courier's app shows a refund button that returns 403, support…
- · permalink
Your support team should not have to read source code to answer "why can't this be edited?"
That is exactly what a boolean permission check costs you. It returns False and discards the only three things anyone downstream needed: which actor, which permission, which ownership rule failed.
The user gets a generic 403. Support escalates. An engineer opens the repository to reconstruct a decision the code already made and then threw away.
We build the check to return an explainable denial instead. Same call site, richer answer: the actor, the permission requested, and the specific rule that rejected it. The interface can then say something true and specific, and a support agent can close the ticket without a developer.
The part teams underestimate comes next. Once those rules are declared as data rather than scattered across view conditionals, the same rule set can drive the Django admin: which roles see which models, which fields are read-only, who may act on an object they do not own. One definition, two consumers. No second, informal permission model quietly drifting out of sync with the first.
That drift is the real failure mode we keep meeting in inherited codebases. The API says no, the admin says yes, and nobody can tell you which one is correct.
So it is worth looking at how access questions get answered on your product today. If the denial does not carry enough context to resolve the ticket, it probably lands on an engineer's desk.
https://shipmindlabs.com/c/056a6fe2
- · permalink
When someone on your team says "the system won't let me do this," a permission check that only answers no leaves everyone guessing — was it their role, the record they touched, or something nobody remembers configuring. We write our permission checks to answer with a reason instead: which person, which permission, which ownership rule failed. The same rule set then drives what each role sees in the internal admin panel, so access is described once rather than re-invented in two places. How often does an access question in your company end up as a message to a developer?
https://shipmindlabs.com/c/43f0cfa3
- · permalink
We once passed on a strong candidate over a single answer. They walked us through a production incident they had fixed, cleanly and honestly, but when we asked how they found out it was broken, there was nothing. No alert, no dashboard, no memory of who noticed first.
That gap is the part clients actually feel. Nobody experiences the fix, they experience the hours before anyone knew. So we hire for both halves now, the repair and the noticing, because a team that only knows how to fix things is still waiting on a customer to tell them something is wrong.
It is worth knowing how your own team usually finds out first: a monitor, or a message from a user.
- · permalink
Most accessibility statements get written by someone who was not in the room when the checks were run.
That is the whole problem. The statement is a public claim about the product, and in many markets it is a legal one. But it usually gets composed at the end, from memory, from a half-remembered audit, from a screenshot someone pasted into a ticket four months ago. Nobody is lying. The evidence just never existed in a form anyone could check.
We hit this while building a11ytrail, our open-source accessibility tooling, and the design decision that mattered was not the checking. It was inverting the direction of authorship.
Every check gets recorded as evidence: who ran it, when, what was checked, what the result was. Automated scan and manual review are both first class. A human confirming keyboard navigation on a component is evidence in exactly the same shape as a rule engine flagging a contrast failure.
Then the statement is generated from that record. A sentence claiming conformance for a criterion can only appear if evidence supports it. If nobody checked it, the statement cannot say it was checked. If the last check was a year ago against a page that has since been rebuilt, that shows.
The generator is not allowed to be more confident than the audit trail.
What this changes in practice is unglamorous and useful. The gap between what you claim and what you verified stops being invisible. It becomes a diff, and someone can look at it before a regulator, a customer, or a user with a screen reader does.
The general shape probably applies well outside accessibility. Compliance reports, security questionnaires, SLA claims: anywhere a document asserts something about a system, the document should be downstream of recorded evidence, not upstream of it. Prose is easy to write and impossible to audit.
If you publish an accessibility statement, it is worth knowing what it is actually generated from today, and whether you could reconstruct the checks behind any single sentence in it.
- · permalink
"Denied" is not an answer. "Not the owner of this record" is.
Most permission layers we inherit return a bare False, so nobody can tell a missing role from an ownership rule, and the admin quietly grows a second copy of the same logic. One rule set, one explanation, in both places.
When a user pings support about a 403, your system should be able to name the rule that denied them. https://shipmindlabs.com/c/824eabe2
- · permalink
A hiring habit we dropped: reading long tenure as loyalty.
Now we ask what state the system was in when they left, what was written down, who could run it on Monday. After a few multi-year engagements it became clear that leaving well is a skill, and it shows up in how someone builds long before they go. You can ask candidates about their exits.
- · 9 min readThe order lifecycle belongs in a table, not in your endpoints
In most order systems, the rules that govern an order live nowhere in particular. The rule that the warehouse may ship only a paid order with stock on hand gets written in the ship endpoint, then…
- · permalink
When an order breaks, the hard question is usually not what went wrong. It is who allowed this, and when. If the rules for moving an order from paid to shipped to refunded sit as conditions spread across half a dozen services, probably nobody can answer that without reading code.
We now declare the states and the allowed transitions as plain data, with rules about who may trigger what. The system either applies the change or refuses it with a stated reason, and every applied change lands in a history nobody can edit afterwards.
We are curious how your team answers the "who allowed this" question today. https://shipmindlabs.com/c/f732b545
- · permalink
Two delivery zones overlap. Which rate applies?
If you cannot answer that from the data alone, then your system is answering it from insertion order.
We ran into this while building courier and warehouse tooling for delivery marketplaces. Zones start clean: one polygon per area, no ambiguity anywhere. Then operations adds a surcharge ring around a bridge, a promo zone for a new district, a restricted area that overrides everything. Now a single address falls inside three polygons, and the lookup returns the first row the query happened to find. That result looks stable in staging, and it shifts the day someone reimports a shapefile.
The fix is small and boring: priority is a field on the zone, not a rule in the code. The point-in-zone lookup returns all matches and picks the highest priority one. The overlap is still there, it is just a decision somebody made and can review, instead of an accident.
We kept this pattern in our open-source delivery-zones project. The geometry is fairly ordinary. The part we care about is that the tie-break lives in the data, where operations people can see it, instead of buried in a query builder.
Our general take, and we think it holds pretty widely: any lookup that can return more than one row needs a declared ordering, or you have shipped a coin flip.
Somewhere in your system a silent tie-break probably decides something a customer actually pays for.
- · permalink
The clearest hiring signal we get comes from how a candidate talks about the worst code they inherited, not from their best project.
Contempt means they will rewrite it. Curiosity ("someone shipped this under a constraint we can't see yet") means they will read the history first. We have modernised enough legacy to know which one we want, and you probably know which one you would hire.
- · permalink
Part of our hiring process isn't technical at all: we ask a candidate to explain something they built to someone with no engineering background. If the tradeoff can't be made plain, the client probably ends up approving decisions they don't really understand, and that's usually where a project starts drifting.
We're always curious about the clearest explanation you've ever gotten from a technical person.
- · permalink
In our shift-planner, declining a shift isn't a penalty — it lowers your priority for the next few assignments, then decays. Punishment makes people accept shifts they'll drop later; a cooldown just moves work to whoever's actually available. Where else are you punishing a signal you need?
- · permalink
Every order bug we have inherited traces back to the same thing: transitions living as if-chains across three services, so nobody can answer "who could move it here, and when."
We declare states, guards and hooks as data, so the machine applies the trigger or refuses with a reason, and every applied transition lands in append-only history. You probably already know where your order logic sits. https://shipmindlabs.com/c/6ed94e2d
- · permalink
Ask an engineer which order transitions are legal and you probably get a shrug. Ask the codebase and you get eleven if-chains in six services.
That is the actual problem with order status. The status field is easy. The rules around it are the product: who may cancel after payment, what happens when a refund lands on a partially shipped order, which state a compliance hold drops you into. Those rules end up as conditionals spread across the services that happened to need them first.
Nobody owns that map. So adding one state means auditing every service that reads status, and hoping the review catches the branch you forgot.
We have been declaring it as data instead. States and transitions live in a table: trigger, source, target, the role allowed to fire it, the hook that runs on success. The machine applies a trigger or refuses with a reason. Every applied transition lands in append-only history.
What changes is not elegance. It is that the transition map becomes reviewable by someone who is not reading source code, and "how did this order get here" has an answer instead of a log grep. In deal flows where money and signed documents accumulate against the state, that history is the audit trail.
We think it is worth checking where your order rules actually live right now: in one place, or wherever the last feature needed them.
https://shipmindlabs.com/c/bff07710
- · permalink
We once made an offer mostly because of a question the candidate asked us. We gave them a small task around recording a payment, and before writing a single line they asked what happens if it runs twice.
That instinct is a big part of the job. Real systems retry — a network hiccup, an impatient user, a queue that redelivers — and code written as if that never happens is the code that charges a customer twice. We can teach a framework in a week; the habit of asking what the world does to your work is much harder to install.
What's the best question a candidate has ever asked you in an interview?
- · permalink
Two invoices issued in the same second, one number. That is not a race condition you fix later — that is an accounting document you now have to explain to an auditor.
Most teams treat the invoice number as a formatting concern. Take a counter, add a prefix, pad it to five digits, render it on the PDF. It works in development, where requests arrive one at a time.
Then two workers hit the same series concurrently. You get a duplicate, or you get a gap — and a gap is not neutral either. In several VAT regimes the sequence itself is the thing being audited: numbers within a series and period must not repeat and must not skip. A missing number reads as a deleted invoice until you prove otherwise.
The fix is boring and structural. The number is allocated by the store, per series and per period, under a lock the database owns — not computed in application code and hoped to be unique. Application code decides which lines appear: domestic sale, private buyer, reverse charge. The store decides which number is legal.
We pulled this apart while building euinvoice, our open-source invoicing work. The split that made it survive: VAT logic is a pure function of the transaction, numbering is a transactional guarantee of the storage layer. Mixing them is how teams end up retro-editing issued documents.
One consequence worth planning for early: a failed invoice after allocation still consumes a number. You cancel it, you do not reuse it.
How does your system handle an invoice that fails validation after the number is already allocated?
- · permalink
Founders almost always ask for speed. What they usually need is someone who knows which 5% of the system must be slow: the ledger write, the migration, the payout path. The rest is recoverable, so it can be messy. We've never once regretted moving slowly there. Which part of your product would you never let anyone ship fast?
- · 9 min readPromo code rejections belong in the type system
A promo code that fails at checkout has to serve two audiences at once: the buyer, who needs to know whether to fix something or give up, and the business, which needs to know which rule fired and…
- · permalink
Your promo endpoint returns false. Support now has three questions and no answer: expired, already used, or does the cart just not qualify?
A boolean flattens the reason into nothing. So teams patch it with strings — "PROMO_EXPIRED", "not eligible" — and those rot the first time someone rewords a message for the frontend and quietly breaks the analytics query that counts rejections by reason.
What we keep coming back to: the rejection reason belongs in the public API as a closed set of typed outcomes the caller has to handle. Expired. Usage limit reached. Minimum order not met. Wrong customer segment. The frontend maps types to copy in one place. Analytics counts types, not sentences. Adding a new rule becomes a compile-time conversation with every caller instead of a surprise in checkout.
The other half is arithmetic. A percentage discount that rounds the wrong way, stacks with a second code, or runs past the order total is a refund your billing logic never agreed to issue. The clamp belongs inside the money type — one place that cannot produce a negative total — not in call-site ifs spread across the checkout flow.
Where do your promo rejection reasons live today: in the type system, or in the copy?
Full write-up: https://shipmindlabs.com/c/332e0017
- · permalink
When a promo code fails at checkout, most systems just say no. That single word costs you twice: the customer doesn't know whether the code expired, didn't apply to their cart, or was already used, and you can't tell later which of those was happening most. We write the rejection reason into the type the checkout returns, so the page can explain itself and the numbers stay countable. Same story with the discount arithmetic — it belongs inside the money type with hard limits, not scattered across the places that happen to call it. Our new article walks through it: https://shipmindlabs.com/c/42616974
What does your checkout tell a customer when their code doesn't work?
- · permalink
Most promo validators return a boolean. The user reads "invalid code", support gets a ticket, and analytics gets nothing: expired, wrong region, basket below minimum — all collapse into the same false. Rejection reasons are public API surface, so give them a type. Strings rot the first time someone renames one. And the discount arithmetic belongs in the money type with hard clamps, not in ifs at the call site. Where does your checkout actually decide a code is invalid? https://shipmindlabs.com/c/005dff7c
- · permalink
Most interviews ask what a person can build. We also ask them to describe handing over a feature they'd only half-finished: what they'd write down, what they'd warn the next person about, what they know is fragile. The ones who answer that well are the ones whose work keeps moving when they're out for a week. What's the one question you'd never skip when hiring?
- · permalink
Every order table has a state machine in it. Most are undeclared: a status column and if-checks scattered across the codebase. Declare the transitions as data, guard them by role, append every applied one to immutable history. Then "who changed this and when" stops being a forensics project. Where does your state logic actually live?
- · permalink
A permission check that returns False tells you nothing. It tells the user nothing, it tells your support engineer nothing, and it tells the next developer nothing.
That is the part teams underestimate.
The access rule itself is usually simple: this actor, this permission, does this object belong to them. The expensive part shows up months later, when a customer says "I should be able to edit this" and nobody can answer without opening the code and reconstructing the decision by hand.
So we changed what a check returns.
Not a boolean. A denial that explains itself: which actor was evaluated, which permission was required, which ownership rule failed. Support reads the answer instead of escalating it. The engineer debugging a report gets the reason, not a stack trace hunt.
Declaring actors and permissions explicitly buys a second thing we did not expect at first. Once the rules are data rather than scattered conditionals, the same rules can drive the admin interface. In our own Django work that means an admin adapter separated by role — a compliance reviewer and an operations user see different objects and different actions, from the identical rule set that guards the API.
We extracted this into an open-source package we call role-scopes, because we kept rebuilding it. Compliance tooling made the need obvious: when a regulator or an auditor asks why someone was allowed to approve something, "the code said no" is not an answer.
One rule set, two consumers, and every denial able to state its own reason.
How does your system answer the question "why was this user denied?" — does it come from the code, or from someone reading it?
- · permalink
A hiring habit we changed: we stopped asking what someone is proudest of. Now we ask what part of a system they owned they still don't fully understand.
The proud answer is rehearsed. The second one is only answerable if you actually ran the thing after launch. Vagueness there usually means they built it and left before it got interesting. What's your version of that question?
- · permalink
Our most useful interview question is one we leave underspecified on purpose.
Some candidates start coding. Some ask what happens when the payment provider returns a duplicate. The second group is the one that survives contact with production — most real work arrives half-specified, and the assumptions you make silently become bugs someone else inherits. What underspecified detail would you have asked us about first?
- · permalink
If your order service knows the difference between an SMS and a push, the notification layer has already leaked.
It starts small. One if-statement for the channel. Then a second one for the provider that needs a different payload shape. Then retry logic in the checkout handler, because that provider times out. A year later, adding a fourth channel means a diff across every feature that ever notified a user, and testing any of it means mocking three vendor SDKs.
We kept hitting this while fanning out notifications to messengers, mobile and web push, so we settled on one rule: the calling code says who, what event, and what data. Nothing else.
Behind that single dispatch call sit provider adapters, one per channel, each owning its own payload translation, its own failure modes, its own retries. Delivery runs in the background, so a slow provider never shows up as a slow API response. We packaged the pattern as an open-source project we call notify-dispatch.
The part that actually decides whether this holds: the dispatch signature has to stay narrow. The moment it grows optional channel-specific fields, you have rebuilt the branch, just further from the code that needed it.
The honest test is adding a channel. If that is a new adapter plus config and nothing else, the abstraction is real.
Where did yours break first — the payload shape, or the per-provider failure semantics?
- · permalink
Halfway through one interview we stopped asking about code and asked the candidate to explain a feature they had built as if we were the client paying for it. They did it in two plain sentences, no acronyms, and that is what got them hired: an engineer who cannot explain their own work leaves the client guessing for months, and guessing is where budgets quietly go. We check for it now in every interview, because most of the friction we have seen on projects was never a technical failure, it was a translation failure. What is the last technical explanation you actually understood the first time you heard it?
- · permalink
The worst notification bug we've shipped wasn't a delivery failure. It was a message that arrived perfectly — reading "Your order {order_id} has shipped."
Nothing errored. The queue was green. The provider returned 200. The customer got a literal placeholder.
This is what happens when a template is treated as a string instead of a contract. Most notification code renders whatever it's handed, and a missing key becomes an empty string or a leftover brace. The pipeline has no opinion about it, because the pipeline never knew what the template needed in the first place.
So in our open-source notify-dispatch we made the template declare its own variables and their types up front. Rendering isn't a string operation anymore — it's a call against a signature. Values get validated at dispatch time, before any provider sees them. A forgotten order_id raises where it belongs: in the test that builds the payload, not in someone's inbox.
The expert take: in multi-channel systems this matters more than it looks. The same event fans out to email, push and messenger templates, each with a slightly different variable set, each maintained by a different person on a different day. Without declared variables you have four silent contracts and no way to test them together. With them, adding a channel is a schema change you can review.
A notification is the one part of your system that a customer reads word for word. It deserves the same type discipline as an API boundary.
How do you catch template drift today — schema validation, snapshot tests on rendered output, or does it only surface when support forwards the screenshot?
- · permalink
SMS, email, messenger — the channel is transport. Rate limits, attempt counters, single-use expiry: those live above it, once. We learned this building otpguard. If your OTP logic changes when the channel does, what exactly did you make pluggable?
- · permalink
Most of our interviews are spent reading code, not writing it. We hand a candidate a working piece of someone else's service and ask what they'd want to understand before changing it — because on real projects, the first months of any engagement are spent inside systems somebody else built, and the people who ask good questions there are the ones who don't break things. Writing new code from a blank page is the easiest part of the job and the least of what we hire for. What's the last thing you looked at in your own product and thought: I'd want to know why this was built this way before touching it?
- · permalink
Founders ask us for a stack match. What they describe two questions later is someone who'll still understand the system in year three.
Stack is the easiest thing to teach. Staying is the hard part — and most job posts don't price it at all. What are you actually hiring for?
- · permalink
We once interviewed someone who spent the first ten minutes asking what happens when things break: who gets woken up, how we find out a payment silently failed, what we do when a customer says money left their account and nothing arrived. Not one question about frameworks. We hired them, because anyone can learn our stack in a month, but the instinct to ask "what does this look like at 3am when it goes wrong" is what actually keeps a client's system running. What's the one question you wish more people asked before joining your team?
- · permalink
One interview habit we dropped: asking what someone built from scratch. We ask how they read code they inherited instead. Most of our work is somebody else's system — legacy migrations, monoliths where accidental behaviour turned load-bearing. A greenfield story tells us almost nothing about that. What's the interview question you stopped asking and never missed?
- · permalink
Most OTP bugs we see are not in the code generation. They are in what the caller forgot to check before verifying it.
The usual shape: a library gives you generate() and verify(). Everything protecting that pair — how soon a user can request a new code, how many wrong guesses before the code dies, how long the account stays locked — is left to the endpoint. So it gets written once in the login handler, differently in password reset, and not at all in the phone-change flow someone shipped later.
Each of those is a separate brute-force window, and none of them look like bugs during review.
Our take, after years on flows where a wrong code means money or identity: a one-time code is not a string. It is a small state machine that knows when it was last sent, how many attempts it has absorbed, and whether it is currently locked. If that state lives inside the library, every caller inherits the protection whether or not the developer was thinking about it that day. If it lives in the caller, protection is a convention — and conventions decay per endpoint.
The practical test: add a new OTP-protected action to your product. If you have to remember to re-implement cooldown and attempt limits, they are in the wrong place.
We put ours in a small Python package, otpguard, open on GitHub under shipmindlabs — resend cooldown, attempt counting and lockout are part of the verification result, not a separate thing to wire up.
Where does the attempt counter live in your auth stack right now — the code, the session, or the endpoint?
- · 7 min readNotification Fan-Out Needs a Stop Condition, Not a Retry Loop
A service that turns one event into push, chat and email messages is easy to build and hard to keep honest. The failure that reaches support is almost never a dropped message; it is the second and…
- · 9 min readAn audit trail is a typed changeset, not a log line
Most Django projects discover their audit requirement late. A compliance reviewer asks who moved an application to the next issuance tier and when, and the answer has to be assembled from…
- · permalink
When we take over an inherited system, the hard part is rarely the old code — it's that nobody left in the building can explain why a particular rule exists. So before rewriting anything, we spend time working out what each odd behaviour was protecting against, because half of those quirks turn out to be someone's fix for a real problem. What's the strangest rule in your business software that nobody can explain anymore?
- · permalink
When a customer asks why a number changed six months ago, the honest answer depends on one design choice made much earlier: did the system record who made the change at the moment it happened, or does someone now have to reconstruct it from server logs? We build audit so the actor is resolved explicitly and saved alongside the field-level diff — the record itself says who changed what, so nobody is stitching timestamps together under pressure. We keep our approach to this open at github.com/shipmindlabs/model-audit. If someone asked you today who changed a key setting last spring, how long would it take you to find out?
- · permalink
We wrote our field-diff engine with zero Django imports. It takes two mappings, returns a typed changeset. The framework layer just feeds it model state.
Not purity — testability. No ORM to spin up, and it survives whatever we migrate to next. It's open: github.com/shipmindlabs/model-audit
Where's the line in your codebase between the framework and the thing it's calling?
- · permalink
The most useful part of our model-audit package has no idea Django exists.
It is a field-diff engine. Two mappings go in, a typed changeset comes out: which fields changed, from what, to what. That's it. No model class, no ORM query, no import from the framework anywhere in that layer.
The Django side does the boring half — it knows how to get the previous state of a row and the new one, and it hands both to the core as plain data.
The reason this matters is not elegance. It's what your tests have to boot.
When change-tracking logic lives inside a model method or a signal handler, testing "did we detect this edit correctly" means a database, migrations, fixtures, and a save cycle. You end up testing the ORM to verify your own business rule. So people write three cases instead of thirty, and the interesting ones — a field set to null, a decimal that changed representation but not value, a JSON blob with reordered keys — never get written.
With the core separated, those are dictionary-in, changeset-out tests that run in milliseconds. And the same engine handles a dict from an API payload or an imported row, because it was never coupled to a table in the first place.
We keep it open at github.com/shipmindlabs/model-audit if you want to look at where that seam sits.
What's the last piece of logic you pulled out of a framework class to make it testable — and what finally forced the move?
- · permalink
Promo validation returning false is a design smell. Expired, usage-limit-hit, not-yours — those are different answers, and every caller ends up reverse-engineering them from a message string.
We made the rejection reasons a typed part of the contract (open-sourced it at github.com/shipmindlabs/promocodes). Feels obvious after payments work, where "declined" without a code is useless.
Where do you draw the line on exposing reasons to the client?
- · permalink
Most audit trails are not audit trails. They are log lines someone hopes to reconstruct a story from later.
The pattern is familiar. A model changes, something gets written — a message, a serialized blob, a "user updated" event. Then a dispute arrives: who changed the payout requisites, when, and what was the value before? Now an engineer is grepping, joining timestamps, and inferring intent. That is forensics. It happens after trust is already gone.
An audit record has to answer three things at the moment of the write, not afterwards: which fields changed, what the old and new values were, and which actor is responsible. If any of those is derived later, it is a guess.
Two things make that hold in production.
First, the changeset is typed and field-level. Not a text description of a change — a structured diff you can query, compare, and assert against in a test.
Second, the actor is resolved at write time. Requests have a user; background jobs, admin scripts, webhook handlers and migrations do not. If the actor is only available from request context, half your writes will be attributed to nobody — and those are usually the interesting ones.
The part teams skip: keep the diff core framework-free. If the logic that computes a changeset only runs inside the ORM's signals, the guarantee is only testable inside a full stack, so in practice it is barely tested at all. Pull it out and it becomes plain input-output — old state, new state, actor — verifiable in isolation.
We wrote the long version up here: https://shipmindlabs.com/blog/an-audit-trail-is-a-typed-changeset-not-a-log-line/
For those of you running compliance-sensitive systems: how do you attribute changes made by background jobs and scripts, where there is no request user to fall back on?
- · permalink
Months after the fact, someone always asks who changed this price, this limit, this customer record — and when. If the answer has to be reconstructed by reading through logs, that is forensics, not an audit trail. We write ours the other way round: the system records the exact fields that changed and who changed them at the moment of the write, so the answer already exists before anyone asks. What is the last record you wished you could see the full history of?
https://shipmindlabs.com/blog/an-audit-trail-is-a-typed-changeset-not-a-log-line/
- · permalink
If you can only answer "who changed this field" by grepping logs after the fact, that's forensics, not audit.
We write audit as a typed changeset with the actor resolved at write time, and keep the diff core free of the ORM so the guarantee is testable on its own. Where does your audit trail actually get written?
https://shipmindlabs.com/blog/an-audit-trail-is-a-typed-changeset-not-a-log-line/
- · permalink
Every monolith we've split had a few behaviours nobody designed on purpose. A timing quirk, a field that was always null, an endpoint that quietly tolerated bad input. Then someone built on top of them. Splitting the code is the easy half. Which accidental behaviour in your system is now a contract you can't break?
- · permalink
A discount that goes negative is a refund. A discount larger than the order total is a payout. Both look like ordinary arithmetic until they reach the ledger.
The usual shape is familiar: a percentage rule, a fixed-amount rule, a stacking rule, and somewhere in the checkout handler a line that says if the result is below zero, use zero. Then the same guard appears in the cart preview. Then in the invoice generator. Then someone adds a new promo type and writes the multiplication without the guard, because the guard was never part of the type — it was part of whoever remembered.
We put the clamp in the discount type itself. Applying a discount to a Money amount cannot return less than zero and cannot exceed the order total, because the operation that produces the number owns both bounds. Call sites stop checking. New promo types inherit the invariant instead of re-deriving it.
After years on payment and lending systems, our take is that money bugs are rarely arithmetic bugs. They are placement bugs: the rule was correct, it just lived in the caller instead of the type, so the fourth caller never got it. We open-sourced the promocodes package we use for this at github.com/shipmindlabs/promocodes — the interesting part is not the promo engine, it is where the bounds live.
Where do your money invariants live right now — in the type, or in the handler that happened to remember them?
- · 8 min readReview queues for compliance teams: leases, not row locks
Almost every system we have built with a human in the loop needs the same component: a queue that hands items to reviewers. KYC and KYB verification, transaction moderation, issuance tiers on a…
- · permalink
One thing we've learned building notification systems: "sent" and "delivered" are two different facts, and most software only tracks the first one. That's why a customer swears they never got the confirmation while your dashboard insists it went out — the message left your server and quietly died somewhere on the way to WhatsApp, email, or a phone that had push turned off. We build these to record what actually landed, not just what we tried to send. How often does your team find out about a failed message from a customer rather than from a system?
- · permalink
A payment provider's sandbox tests your code against their happy path. Production tests it against their timeouts, partial failures, and undocumented status codes. Budget for the second one. What broke for you first in prod?
- · permalink
Every team adds retries. Almost nobody adds idempotency keys on the other side.
So the retry that saves you during a network blip is the same one that double-charges a card six months later. Retrying isn't a client-side decision — it's a contract. Who owns that contract on your team?
- · permalink
Docker Compose is a great development tool. It is not a deployment strategy — but that is where most teams end up using it.
The path is always the same. Compose starts as the way a new engineer gets the project running in one command. It works. So it goes to staging, because staging is "just a bigger laptop". Then production needs to ship, the compose file already works, and nobody wants to introduce a new tool during a release week.
Now the ops model is a file that was written to make onboarding easy.
The symptoms show up later. Restarts drop in-flight work because nothing waits for the worker to finish its current task. Deploys are a short outage instead of a rollover, because there is no second instance to shift traffic to. Logs live on one host. Secrets live in the same file as the service definitions, so rotating one means editing the thing that defines your topology.
None of that means you need Kubernetes. It means the questions Kubernetes answers — how do I roll over without dropping requests, where do secrets live, what restarts a dead process, who collects the logs — still need answers. A single well-run host with systemd units, a reverse proxy, and a real secret store answers them for a lot of products.
Our rule: local orchestration and production orchestration are allowed to be different tools. Compose stays the developer's tool. Production gets whatever is chosen on purpose, even if that choice is deliberately small.
If your compose file made it to production, what forced you to change it — a failed deploy, or an incident?
- · permalink
A cron job that runs longer than its interval is now two jobs. Payment systems find this out in production. Where did overlap first bite you?
- · 8 min readWhy LLM catalog translation is a cache design problem
Translating a product catalog with a language model looks like a prompt problem and turns out to be a cache problem. A catalog is never translated once: a merchant edits a title, a supplier feed…
- · permalink
People assume code review is about catching typos. It isn't — the compiler finds those. What review actually catches is one engineer's private assumption about how something works, before that assumption reaches your customers. The side effect matters just as much: after a few months of reviewing each other's work, at least two people understand every part of the system. What's a decision in your business that only one person currently understands?
- · permalink
Every monolith we've split had one shared transaction quietly holding two features together. Nobody documented it. You find it the day the split goes live and the second write silently stops rolling back.
What's the invisible coupling in yours?
- · permalink
Every notification service starts as one function called send. It ends as the most fragile part of the system.
The pain shows up late. A user gets the same payment alert three times: push, WhatsApp, and email. Or worse, gets nothing at all because their push token expired two months ago and no one owned that failure. Support hears about it before monitoring does.
The reason is that teams model notifications as delivery, when the hard part is state. Which channels does this user actually have? Which ones are still valid? Was this event already delivered somewhere else? Is this notification worth waking someone up for, or is it a digest item?
We have built fan-out services pushing to WhatsApp, Telegram, mobile and web push, and the pattern that survived production was treating the event and the delivery as separate objects. One event, many delivery attempts, each with its own status. Deduplicate on the event, not the message. Keep channel preferences and token validity in one place that both the product team and the compliance team can read. And make silent failure loud: an expired token that fails quietly is indistinguishable from a happy user until it isn't.
The unglamorous version of this is a table of delivery attempts you can query. That table is what turns "the user says they never got it" from an argument into a lookup.
For teams running multi-channel notifications: where did yours first break down — deduplication, preference management, or token and identifier decay?
- · permalink
A scraper isn't an integration. It's a subscription to someone else's release schedule, and every silent redesign on their side is an outage on yours. The failure that hurts is HTTP 200 with nothing in it. What alerts when your parser succeeds and returns empty?
- · permalink
The scary part of a migration isn't the schema change. It's the minutes where old code and new code both run against the same table.
Most rollback plans we inherit assume that window doesn't exist. Does yours?
- · 8 min readExpand and Contract: Changing a Payments Column Without Downtime
Changing the shape of a payments table is trivial in a migration file and hard in production. The rows are being written to while the migration runs, the previous release and the new one serve…
- · permalink
When a payment fails halfway, the honest answer is that nobody yet knows whether the money moved. The safe fix isn't retrying harder — it's giving every attempt an identity, so the second try recognises the first and refuses to charge again. Has a double charge ever reached one of your customers, and how did you find out?
- · permalink
Every team adds retries before they add idempotency keys. Which means for a while, your "resilience" is just a machine that charges people twice under load.
We've inherited this in payment flows more than once. Where does your system dedupe — the client, the gateway, or nowhere yet?
- · permalink
Notifications look like a feature until the first duplicate storm. Then you find out you built a delivery system without dedup, ordering, or retry limits. Where does yours keep that state?
- · permalink
Moving work to Celery makes your API fast. It also quietly turns every user action into a promise nobody is tracking.
The pattern is familiar. A signup triggers a welcome email, a KYC submission triggers a verification call, a payment triggers a ledger update. All of it goes to a queue so the HTTP response stays under 200ms. The endpoint is fast. Everyone is happy.
Then a verification provider times out at 3am, the task raises, the worker moves on, and a user sits in "pending review" for two days. No error page. No alert. The request succeeded — the promise didn't.
What we've learned running payment and compliance systems on Python queues: the moment a task represents something a user is waiting for, it stops being infrastructure and becomes product state. It needs a row in your database, not just a job in Redis. Enqueued, started, failed, succeeded, retried — visible to the people who answer support tickets, not only to the engineer with terminal access.
The practical version is boring. Every user-visible async action gets a status record written before the task is enqueued. The task updates it. A reconciliation job scans for records stuck in a non-terminal state longer than they should be. That job is the actual monitoring — it catches the failures your exception tracker never saw because the worker died mid-task.
Celery is excellent at running code later. It was never designed to tell your business what didn't happen.
For teams running background work in production: how do you find the tasks that failed silently — and how long does it usually take?
- · 6 min readQueue Topology: One Broker, Several Queues, No Shared Pool
A single task queue served by a single worker pool works until the day a bulk job — a catalog scrape, a translation batch, a re-notification sweep — holds every process for minutes at a time.…
- · permalink
Notification services get scoped as a two-day task, then quietly become critical infrastructure. Nobody defines what happens when the push fails, the email bounces, and the user never learns their payment went through.
Who owns delivery guarantees on your team?
- · permalink
Before anything reaches your customers, it runs on a full copy of the system where breaking things costs nothing. It feels slower for a day and saves you the week where payments quietly stop working. What's the last change you wished someone had rehearsed first?
- · permalink
Running a product catalog through an LLM is the easy 10% of localization.
The expensive part starts after the first pass.
Most teams treat catalog translation as a batch job: dump the SKUs, call the model, load the results. It works once. Then merchandising edits 400 titles, a supplier renames a category, and someone asks why last month's fix disappeared.
What makes an LLM content pipeline survive production is not prompt quality. It is bookkeeping:
- a content hash per field, so you only pay for what actually changed
- a human override layer that re-runs are not allowed to overwrite
- a review queue where the diff is visible before it reaches the storefront
- versioning, so a bad batch rolls back without a database restore
We build these pipelines for cross-border catalogs, and the same lesson keeps repeating: the model is a component, the pipeline is the product. Teams that skip the state tracking re-translate everything on every run — the most expensive option and the least trustworthy one, because nobody can tell which strings a human already corrected.
If you push LLM output onto a live product surface, how do you handle human edits — do your re-runs respect them, or does someone re-apply the same correction every month?
- · permalink
In a rewrite, the old system's bugs are part of the spec. Someone downstream depends on every one of them. What quirk did you have to reimplement on purpose?
- · 7 min readExtracting a Service From a Monolith Without a Code Freeze
Every monolith extraction reaches the same question: how do you know the new service returns the same answers as the old code, on real production data, before anything depends on it? A test suite…
- · 9 min readOne payment protocol for every provider: hold, capture, refund
A payment integration usually starts with one provider and ends with three, and by then the provider's vocabulary has leaked into the order code: one branch for the gateway that separates…
- · permalink
Every KYC provider we've integrated has a failure mode that looks exactly like the user doing something wrong. Blurry doc, timeout, silent decline — same screen.
So people retry, get rejected again, and leave. What does your onboarding do when the provider is the one that's broken?
- · 7 min readTesting Money Paths: Four Tests Our Review Gate Requires
A payment service can pass every unit test it has and still lose money. The failures that matter in production are not about one request being wrong: a provider webhook delivered twice, two workers…
- · permalink
The scariest thing in a small project isn't a bug — it's one person being the only one who knows how something works. We keep every part of a system readable by at least two people, so a holiday or a sick week never turns into a frozen release. Who is the single point of failure on your product right now?
- · permalink
Manual KYC review is not an admin screen, it's a queue with latency and a backlog. Build it like a production system or onboarding stalls. How do you measure your review queue?
- · permalink
Every monolith rewrite we've done, the real work wasn't the code. It was finding out what the old system actually did, because nobody knew anymore. Half the "bugs" turned out to be load-bearing.
How do you tell a bug from a feature in a system older than everyone maintaining it?
- · permalink
The most dangerous code in most Python systems is the code nobody watches: the background task.
An HTTP endpoint fails loudly. Someone gets a 500, support gets a ticket, you get a trace. A Celery task fails into a log line nobody reads, and the system keeps looking healthy.
We have shipped queue-backed services in payments, e-commerce, and notification fan-out for years, and the failures repeat with the same shape. A payout webhook retried three times because the ack happened after the work. A catalog import that half-finished and left prices from two different runs in the same table. A notification service that quietly stopped delivering because one poisoned message kept the worker busy forever.
None of those were queue bugs. Celery and RabbitMQ did exactly what they were told.
What actually holds up in production is boring:
Every task takes an idempotency key, not just arguments, so a redelivery is a no-op instead of a second charge.
Every task has a hard time limit and a dead-letter path, because a task with no ceiling will eventually consume the whole pool.
Every task reports failures where humans already look, not into a log file. If your API errors page someone and your worker errors do not, you have two reliability standards in one system.
And tasks stay small. A task that does five things fails in the middle of the third one, and you get to reason about partial state at 3am.
Our rule when we take over an existing codebase: read the task modules before the views. That is where the unowned complexity lives.
What finally forced your team to take background jobs as seriously as your API — a duplicate charge, a silent data drift, or something worse?
- · permalink
Every webhook you receive will arrive twice eventually. If your handler isn't idempotent, you don't have an integration, you have a bet. What broke first for you: payments or carrier callbacks?
- · permalink
People sometimes ask why we spend time testing code that already works. Because the code isn't the risk — the next change is, six months from now, made by someone who wasn't in the room. Tests are how a team keeps moving when the person who wrote it is on vacation. How much of your codebase could someone new safely change tomorrow?
- · permalink
If you can't rebuild the ledger from events, you don't have reconciliation, you have hope. We add that job on day one, not after the first mismatch. What broke your books first?
- · permalink
Every search index you add is a second source of truth you now have to reconcile. Most teams budget for the sync job, not the drift. Who owns reindexing when it goes stale?
- · permalink
Every payment system we've inherited had reconciliation bolted on at the end, treated as a reporting job.
It isn't. It's the only proof the money math is right. Build it with the flow, or you're shipping on faith. What convinced your team to take it seriously?
- · permalink
Every payment integration looks done when the first charge succeeds. It's actually done when reconciliation runs for a month without a human touching it.
We've never seen a payment system break at the API call. It breaks at attribution — which money belongs to which user. What's your worst mismatch story?
- · permalink
A refund is not a payment with a minus sign. Different rails, different timing, different failure modes. Systems that model it as one flow leak money in reconciliation. Where did yours split?
- · permalink
Most monolith-to-microservices migrations we inherit are already half-done: the code is split into services, and every one of them still writes to the same database.
That is the version that hurts the most.
The team gets all the costs of distribution — network calls, deploy coordination, partial failures, harder debugging — and none of the independence they were promised. Two services still can't be deployed separately, because a column change breaks both. A migration still locks the whole product. Nobody owns the schema, so everybody edits it.
When we take over rewrites of inherited systems, we work in the opposite order. Data boundaries first, code second.
That means picking one domain that genuinely has its own lifecycle — payments, documents, notifications — and asking a boring question: can this domain hold its own tables and expose everything else through an API? If the answer requires a join across three other domains, it isn't a service yet. It's a module, and it should stay in the monolith until the data separates cleanly.
We've pulled service layers out of monoliths this way on payment platforms, lending systems and marketplaces. The pattern holds: the first extraction is slow and unglamorous, because it's mostly about untangling reads. Everything after it goes faster.
Our take: a monolith with clean internal boundaries is a better business than a distributed system with a shared database. Split when the data is ready, not when the org chart is.
For teams that went through a rewrite — what was the first piece you extracted, and would you pick the same one again?
- · permalink
Search is slow, so the team adds Elasticsearch. Six weeks later the catalog shows products that were delisted yesterday.
That is not an Elasticsearch problem. It is what happens when a search index becomes a second source of truth without anyone owning the sync.
The pattern we keep seeing: a product query gets heavy, someone puts the data into an index, and the write path grows a fork. One branch goes to PostgreSQL, the other to the index. They agree on the happy path. They disagree whenever a write fails halfway, a background job dies quietly, or a bulk import runs outside the normal flow. Nobody notices, because a stale index does not throw errors. It just answers wrong.
We run Postgres, MongoDB, Redis and Elasticsearch in production, and our rule is simple: the index is disposable, the database is not. Reindexing must be a routine operation you can trigger at any time, not a migration nobody dares to touch. Sync goes through a queue with retries, and there is a reconciliation job that compares counts and flags drift before a user does.
Most catalogs we have worked on could have gone further on Postgres than the team assumed — full-text search and the right indexes carry more load than people expect.
What finally forced you off your primary database for search — query latency, ranking quality, or something else?
- · permalink
Every deal platform we have worked on eventually hits the same wall: the database says the deal is closed, and nobody can prove what was actually agreed.
The status column is easy. A deal moves from listing to offer to deposit to closing, and someone models it as an enum.
What breaks is everything the enum does not carry.
Who signed which version of which document. Which compliance check passed before the deposit was accepted. What the terms looked like at the moment a party agreed to them, not after three amendments.
We have built this in two very different shapes. In one, the deal state and its signed documents accumulate on a token on-chain, so the trail is external and verifiable by every party. In another, e-signature workflows built to national digital-signature standards carried the same weight off-chain.
Different technology, identical principle: the document trail is the source of truth, and the status field is only a cached view of it.
Our take — if your system moves money or transfers ownership, design the evidence layer before the state machine. Retrofitting it means reconstructing history from logs that were never meant to be evidence.
For those of you running transaction platforms: when a party disputes what was agreed, what do you actually pull to settle it?
- · 8 min readCatching money bugs with ledger invariants, not error logs
A payout that credits the wrong sub-account returns HTTP 200. Nothing throws, the worker acknowledges the message, the error dashboards stay green, and the discrepancy surfaces days later when…
- · permalink
Half the Next.js apps we review are entirely behind a login screen.
Which means the main reason they chose Next.js — server rendering for search engines — is doing nothing for them at all.
Google never sees a dashboard. It never sees a billing page or an admin panel. But the team still carries the cost of that decision every sprint: two runtimes to reason about, server and client boundaries to keep straight, a hydration bug class that only shows up in production, and a hosting story that is no longer just static files behind a CDN.
The honest version of the question is not React or Next.js. It is: does this product need to be rendered before a user is authenticated?
Marketing site, docs, pricing, public listings, anything shareable — yes, and Next.js earns its complexity there. Internal tooling, dashboards, anything gated — a client-rendered app with a plain API is usually faster to build and far cheaper to keep alive.
Our rule when we scope a frontend: split the public surface from the private one early. They have different requirements, and forcing one framework across both is how teams end up paying for capabilities they never use.
For teams running Next.js on a fully gated product: what made it worth it for you?
- · permalink
The retry you added to make the system more reliable is often what takes it down.
Here's the pattern we see in code reviews more than any other. A call to a payment provider, an email service, or an internal API occasionally times out. Someone wraps it in a retry loop. It works in staging. Everyone moves on.
Then one day the downstream service slows down instead of failing outright. Every caller times out and retries. Those retries pile onto an already-struggling service. It gets slower, so more calls time out, so more retries fire. A brief latency blip becomes a full outage — and the retries are the thing keeping it down.
Two things separate a retry that heals from a retry that amplifies:
Idempotency. If the operation can run twice safely — an idempotency key on writes, a dedup check on the consumer — a retry is free insurance. Without it, a retry that half-succeeded now double-charges or double-sends.
Backoff with jitter and a budget. Retry immediately and you synchronize every client into a thundering herd. Exponential backoff with randomness spreads the load; a cap on total retries stops the loop from feeding itself.
Our rule in review is simple: no retry gets merged until we can answer two questions. Is this operation safe to run twice, and what stops it from retrying forever? If either answer is unclear, the retry is a liability, not a safeguard.
When a retry has bitten you in production, was it the missing idempotency key or the missing backoff that did the damage?
- · permalink
Most teams plug in a KYC provider, see a green checkmark, and call compliance "done."
Then the edge cases arrive: a document that's valid but mismatched, a name in a different alphabet, a selfie that half-passes. The provider returns a score, not a decision. Someone still has to make the call — fast, under audit, without blocking honest users.
We've built the layer that sits between the verification API and the compliance officer. Real-time moderation queues where a reviewer sees the full context of a user, flags cluster by rule, and every action is logged for the audit trail. The integration with the KYC/KYB provider is maybe 20% of the work. The other 80% is the review workflow, the state machine behind an application, and making sure a rejected user can be re-examined without losing history.
Our take: compliance tooling lives or dies on the operator's screen, not the API contract. A verification vendor tells you what it thinks. Your own moderation layer is where policy actually gets enforced — and it's the part you can't outsource.
For teams running KYC in production: how much of your compliance logic sits in the vendor versus in tooling you built yourselves?
- · permalink
In payments, transferring money is the easy part. Knowing exactly whose money it is — that's where systems break.
We've spent years building fintech backends where funds arrive tagged only by payment requisites: an amount, a reference string, a timestamp. From that, the system has to prove which user, which obligation, which account — and reconcile it against what the bank actually settled. Get this wrong and you don't just have a bug, you have money in the wrong place and a compliance problem.
Our approach treats attribution and reconciliation as first-class parts of the architecture, not an afterthought bolted on before launch. That means explicit ledgering with per-user accounts, deterministic matching rules, and a reconciliation layer that flags mismatches instead of silently absorbing them. Backend, QA, and DevOps work off the same staged pipeline, so edge cases surface in testing, not in production with real balances.
Our take: most payment failures aren't in the payment gateway — they're in the accounting model behind it. If your ledger can't answer "why is this amount here" for every cent, scaling only multiplies the ambiguity.
Building or reworking a payment flow and worried about reconciliation? Let's connect — happy to discuss your architecture.
FinTech #SoftwareDevelopment #Backend #Engineering #Architecture #DevOps #Payments#