Field Notes
- · permalink
The most useful thing we ask an engineer in an interview isn't to build something. It's to read a piece of code they've never seen and tell us what they would delete.
Writing from scratch is the easy half of the job. Most of the work on a live system is arriving in the middle of something someone else built, under deadline, and figuring out what is safe to touch. Deletion is the sharpest version of that question: to remove something confidently you have to know what depends on it, what happens to the data when it's gone, and what nobody wrote down. Strong candidates slow down, ask what runs in the background, and name the one thing they wouldn't touch without checking first. Weaker ones tidy up whatever looks ugly.
We like it because it can't be rehearsed, it takes about twenty minutes, and it predicts the thing clients actually feel: whether a new person can join a running project without breaking it.
What's the last thing you removed from a working system that turned out to be holding something up?
- · permalink
An LLM translation is not content, it is a cached derivation of a source field, and if you don't record what produced it, you can't tell stale from approved.
We run LLM pipelines that translate and adapt product catalogs at scale. The failure mode is rarely bad language quality. It is silence.
Someone edits a product description or a spec line in the source locale. The translated rows stay exactly as they were. Nothing throws. No queue backs up, no alert fires. The catalog quietly serves last quarter's specification to everyone reading it in another language, and you hear about it from a customer.
What fixes it is treating every generated field as derived data with a provenance record: the hash of the source text it came from, the model, the prompt version, and a review state. Regeneration then runs off hash mismatch instead of a schedule or a human hunch. Anything a reviewer approved stays pinned to the hash it approved, so a pipeline re-run cannot silently overwrite editorial work.
Our take: tokens are the cheap part here. The cost sits in re-translating text nobody touched, and re-reviewing text a human already signed off on. Provenance turns "what is stale" from a guess into a query.
If you run generation over content that keeps changing, you probably need both: a rule that decides what to regenerate, and a human approval that survives the next run.
- · permalink
A success response from a payment provider is a claim. The truth shows up the next day, in reconciliation against their ledger. Every system we've built with money in it eventually grows that job. Yours probably ran on trust for a while before you wrote it.
- · permalink
A candidate kept stopping mid-task to ask what the client actually wanted. We scored that as dodging and nearly passed on them. Hired anyway. Two weeks in, on a payment integration, they flagged that the spec double-charged on retry, before writing a line of code. We'd have found that in production, with money moving. Now "asks before building" is a scored signal for us, not a red flag.
Probably worth a look at which habits you screen out that you shouldn't.
- · permalink
We once gave a trial task on an inherited lending service. One candidate wrote almost nothing for two days: he read migrations, then asked why one balance column existed twice. That was the old attribution bug nobody had documented. We had been screening for how fast people write code on a clean repo, when most of our real work is reading someone else's decisions first. The task now ships with legacy baggage on purpose.
The choice is what you hand a candidate: a blank file or a mess.
- · permalink
The shipping price your customer paid and the price the carrier eventually charges you are two different numbers. Most checkouts are built as if they were the same one.
A rate call at checkout is a quote. It is priced from the dimensions and weight you declared, for one service level, at one moment in time. Weeks later the carrier re-measures the parcel, applies a surcharge, or reprices the lane, and the adjustment turns up in an invoice with nothing linking it back to the order.
What we settled on across multi-carrier delivery work: treat the quote as a stored artifact, not a number rendered on a page. Persist the full input set next to the order (declared dimensions, weight, destination, service, carrier, timestamp, validity window). Then re-quote at label purchase, compare against the stored quote, and apply an explicit tolerance rule that decides in advance who absorbs a difference: the customer, the margin, or a manual review queue.
The payoff comes later, not at checkout. It shows up when the monthly carrier invoice lands and every adjustment can be matched to an order, a declared measurement and a quote, so you can dispute the wrong ones instead of accepting all of them as the cost of doing business.
Unmatched carrier adjustments are probably one of the quietest margin leaks in e-commerce, because they never look like a bug.
For teams running multi-carrier shipping it tends to come down to one of two things: you reconcile carrier invoices line by line against stored quotes, or the delta gets written off as shipping cost variance.
- · permalink
In an interview, the thing we actually watch for is what a candidate does when the task we hand them is missing a detail on purpose. That tells us more than what they can build.
One recent decision came down to exactly that. Two people wrote comparable solutions: one asked what should happen when the same payment is retried, the other quietly picked an answer and moved on. We hired the one who asked. In payment work a silent guess about retry behaviour doesn't fail in code review, it fails on a customer's statement, weeks later.
If someone joined your team and turned out to be a great hire, we'd like to hear what you noticed about them in the very first conversation.
- · permalink
Staging is a shared hallucination we all agree to have before a release.
It has clean data, one user, and sandbox endpoints that answer on time. Production has ten years of weird rows and a partner API that returns 200 with an error inside. The bug was not in your code, it sat in the gap. We think your staging has yet to reproduce that gap even once.
- · permalink
Retries are not a safety feature. They are a duplication feature, unless the receiver can tell a repeat from a new request.
Every team adds retries eventually. The network times out, the client resends, and now the same payout exists twice. The usual fix is an idempotency key. The usual mistake is generating that key from the HTTP request: a hash of the body, a header the client sets fresh on every attempt, a UUID created inside the retry loop. All of those change when the caller changes, so the second attempt looks brand new to your system.
The key has to belong to the business operation, not the transport. It gets allocated once, when the intent is formed (this payout, for this order, for this period), and it survives the client restarting, the queue redelivering, and the operator clicking twice.
The second half gets skipped more often than the first. Recognising a duplicate is not enough. You have to store what the original attempt produced, then return that same stored result, byte for byte. If a repeat re-runs the handler and merely suppresses the side effect, the caller gets a different response for the same operation, and reconciliation later cannot tell which answer was true. The key maps to the outcome, not to a boolean "already seen".
We build this into payment services and ledger flows by default now, because every provider on the other side assumes we handle replays, and most of them are right to assume it.
So it is worth knowing where your idempotency key actually gets created: inside the client's retry loop, or at the moment the business intent is recorded.
- · permalink
A few years back we ran white-label hosted payment pages: each merchant's own branding, their own domain, our checkout behind it. One morning one of those pages stopped taking cards. Nothing had been deployed, no code had changed. The certificate on that domain had quietly expired overnight, and browsers were refusing to load the page at all.
Nobody was watching that date. The page had worked the day before, so everyone assumed it would work again. We heard about it from a confused customer's support message instead of a monitor, and by then a good part of a business day of card payments on someone else's storefront was simply gone.
These days certificate and domain expiry get monitored the same way we monitor databases and queues: an alert with days of warning, going to the same place as everything else that can take a product down. It's unglamorous, and it sits above design, copy and page speed on our list, because all three of those are useless on a page that won't open.
So if you're checking five things on your website today, start with this one: what gets told before your certificate and your domain renewal lapse. A person's memory is probably not the thing you want there, you want something that will wake someone up.
- · 8 min readA notification template is a contract, not a string
A notification is one of the few outputs in a production system with no undo. Once a provider accepts the payload, the SMS is on a phone, the email is in an inbox, and the push has already lit a…
- · permalink
Integration estimates go wrong because teams price only their own work. The expensive part sits on the other side: sandbox parity, credential turnaround, a spec that only matches production half the time. Your velocity is capped by someone else's release cycle, and adding an engineer does not move that line. We plan integrations around waiting rather than coding. Somewhere in your timeline there is probably an assumption that the counterparty behaves.
- · permalink
A queue that keeps draining can still be hours behind. Depth tells you nothing, the age of the oldest message does. That's probably the thing you want to alert on.
- · permalink
A notification template is a function signature we pretend is a string. The arguments go untyped, so a missing order_id fails at the provider, not in CI.
Declare the variables, validate before dispatch, and the failure moves back to your test suite where it costs nothing. That part is cheap. Who owns that contract, backend or whoever wrote the copy, is probably less settled on your team than it looks. https://shipmindlabs.com/c/cc5346c3
- · 7 min readWhy a six-digit code is safe: entropy, attempts and the pepper
A six-digit numeric code carries 19.93 bits of entropy, a number no password policy would accept. It is still the correct primitive for passwordless login and step-up confirmation, but only because…
- · permalink
The OTP bugs we inherit are rarely in the code generation itself. They sit in the state around it: cooldowns per channel, attempt counters that reset on resend, lockout that quietly never fires.
Swap SMS for email and the invariants should hold. If they move, your state machine probably lives in the wrong place.
Worth checking where yours lives. https://shipmindlabs.com/c/c8269d48
- · permalink
Ten years in, the part of the job nobody warns you about: most of your work gets read by people who weren't there. Not tests, not docs. Naming, boundaries, the commit that explains why. We've inherited enough systems to know the fastest engineer on a team is often the one who made the last person's code easy to change. Someone taught you that at some point, probably later than you needed it.
- · permalink
Most sites that look cheap don't have a design problem. They have a "what happens after you click submit" problem: the button stays clickable, nothing changes for two seconds, so you click again, and now there are two of you in the database.
Polished sites feel expensive because someone decided in advance what the slow moment looks like. You have probably left a checkout without knowing whether the order actually went through.
- · permalink
Divide one payment across many recipients and rounding stops being cosmetic.
Round each share on its own and the parts no longer sum to the whole. You end up a cent short, or a cent over. Nothing errors. It surfaces later, when the payout account drifts from the ledger and someone has to explain the difference in a meeting.
We hit this in lending flows, where a single borrower repayment is distributed pro rata across every investor in a deal.
What works for us: treat the split as one allocation function, not as N independent calculations. It takes the total and the shares, returns parts that provably sum back to the total, and applies a deterministic rule for the remainder (largest fractional part first, ties broken by a stable key, not by whatever order the database happened to return).
Then you write the remainder as a visible ledger entry instead of absorbing it quietly. If someone asks why they received an extra cent, the answer sits in the data, not in the source code.
The same goes for anything that gets divided: fees, refunds, revenue share, tax. Each of those needs one owner in the code and an explicit rule for the leftover. Rounding scattered across a codebase is a leak with no alarm on it.
Your system already decides who gets the leftover cent, probably without anyone having written that rule down.
- · 9 min readAn accessibility statement your audit trail can support
An accessibility statement is a public compliance claim, and in most organisations it is produced the way a summary gets produced: someone reads the last audit report, forms an impression, and picks…
- · permalink
Most people asking "should we hire or outsource?" are really asking who will still understand this system a year from now.
Outsourcing fails when it's one contractor who leaves with everything in their head. It works when the arrangement covers roles, backend, QA, delivery, someone who knows how the deploy works, so the answer to any question is never "we'd have to ask him."
That part is worth checking before you sign anything: if the person you talk to went quiet for two weeks, does the work stop? If you've outsourced before, the thing that broke was probably the handover and not the code.
- · permalink
Founders write the hire spec by stack: Python, React, done.
In payment and KYC work, most of the hard hours go to someone else's system misbehaving: a provider timing out mid-flow, a webhook arriving twice, a sandbox that lies. Framework depth is cheap to hire. Judgement about other people's failures is not, and we're not sure what you screen that on.
- · permalink
A default timeout is someone else's guess about your money. Payment calls, catalog scrapes, and KYC checks each deserve their own number. Most of your timeouts were probably never chosen, they just came with the client library.
- · 8 min readOverlapping delivery zones resolve by explicit priority
Two zones covering the same street is normal: a city hub reaches it, and the store's own ring reaches it too. The overlap is fine. What goes wrong is the resolution, because most implementations…
- · permalink
The expensive mistakes we see are rarely about a bad technology choice. They come from building a product that can't answer a simple question: what happened to this one customer's payment last Tuesday?
Teams spend months on features and nothing on the boring record of what the system actually did, and then every support question turns into an engineer digging through logs for an hour.
If a customer asked you today about a charge from last month, explaining it would probably take someone on your side a while.
- · permalink
One hiring habit we changed: we stopped asking what someone built and started asking what they deleted.
Adding a service is easy. Removing one means you knew who called it, what depended on the shape of its data, and how to prove nothing quietly broke. That is system understanding, not resume surface.
So we want to hear about the last thing you deleted in production, and how you convinced yourself it was safe.
- · permalink
A payment webhook is not an event. It is a retry with an opinion about the past.
We have built payment services on card rails and account-to-account flows for years, and the same failure turns up in almost every codebase we inherit: the handler reads the payload, trusts it, and updates the balance.
That holds until the provider does what every provider does. It retries the same notification because your 200 came back slow. It delivers "captured" before "authorized" because two workers fired in parallel. After an outage it resends a three-hour-old status and overwrites a refund that already happened.
None of that is a bug on their side. Delivery is at-least-once and unordered by design, and the documentation usually says so in one line nobody reads.
What we do instead: the endpoint stores the raw payload keyed by the provider's event id and returns immediately. Processing happens after, once, guarded by that id. State changes are applied as transitions rather than assignments, so a transaction that is already refunded rejects a late "captured" instead of accepting it. And the provider's timestamp, not our arrival time, decides what is stale.
The reframing that helps: the webhook is a hint that something changed, not the description of what it changed to. When money is involved, that hint triggers a reconciliation against the provider's own API, and that answer wins.
If you run payment integrations in production, you have probably already picked a side here: process webhooks inline, or store first and reconcile after.
- · 8 min readCourier dispatch: refusal cooldown instead of a penalty score
Dispatch systems that let a courier decline a job usually record the refusal as a penalty: a reliability score goes down, and the next assignment round reads that score. It works until someone asks…
- · permalink
When we're hiring, we skip the clean task description and hand over a bug report written the way real users write them: something is wrong, no details, no logs.
The fix matters less than what the candidate asks first: who noticed, when it started, whether money moved. That's the same instinct that keeps a live system from getting worse while you debug it. You probably have your own question you wish people asked before they started fixing things.
- · permalink
A retry limit without a dead-letter queue is just a delayed delete. After the last attempt the job either lands somewhere a human looks, or it never happened. You probably can't say where your fifth failure ends up.
- · 7 min readInvoice numbers that cannot skip or repeat
Two invoices carrying the same number, or a year whose numbering jumps from 0041 straight to 0043, are not cosmetic defects. Someone reading the books finds them years later, long after everyone has…
- · permalink
The day a payment provider renames a field, you find out how many files know that field name.
Most integrations start honestly. You get a provider SDK, the response comes back already parsed, and copying it into your own structure feels like wasted work. So the provider's shape leaks: into your ORM columns, into your serializers, into the frontend, into analytics. Six months later that vocabulary is your product's vocabulary.
Then the business wants a second provider in a new market, or the first one deprecates an endpoint, and the estimate comes back as a rewrite.
What we do instead is boring and cheap at the start: one translation layer per provider, and its only job is to turn their response into our own type, and their failures into our own errors. Business code never sees a provider status string. It sees our states.
Two details make it actually work.
We keep the raw response stored next to the translated record. When the provider disputes something, the argument ends with data, not with logs.
And we map error taxonomies deliberately (retryable, declined, needs-manual-review), because every provider names those differently, and the retry logic is the part that costs real money when it's wrong.
We've built payment services, KYC integrations, and multi-carrier delivery on top of this. The layer never looked impressive in code review. It's probably what made adding the next provider a week instead of a quarter.
When you added a second provider, something leaked further into your codebase than you expected.
- · permalink
On a long project, most of the work is not writing new code, it's changing code someone else wrote years ago. So we hand candidates an unfamiliar piece of a system and ask what they'd be nervous to touch, and why. Caution in the right places is what keeps a client's system running while we're changing it.
Every product has a part nobody wants to touch, and you probably already know yours.
- · permalink
Most people treat invoice numbers as labels. They are not. In most tax regimes the sequence itself has to hold up: no gaps, no duplicates, per series and per period. That gets uncomfortable the moment two orders are paid in the same second and both ask for the next number.
We wrote up how we keep that counter honest under real traffic, and why the accounting rules decide what goes on the invoice while the sequence decides whether it counts at all: https://shipmindlabs.com/c/0ed96d4b
Explaining a missing invoice number to an accountant is probably not a conversation you want to have.
- · permalink
Invoice numbers look cosmetic until two requests race and you get 1043 twice, or nothing at all between 1042 and 1044. Gaps and duplicates are both audit problems.
So the counter belongs in the store, scoped per series and per period. VAT rules shape the lines, and the sequence is what makes the document legal.
Worth checking where yours increments. https://shipmindlabs.com/c/10a95223
- · permalink
Founders tell us they need one senior engineer. Then the scope turns out to be backend, infra, release process and QA, three roles wearing one title.
Those hires rarely fail on skill. They fail on an undeclared boundary: nobody said which part gets dropped when the week is short, so everything gets dropped a little. Probably worth rereading the last senior role you wrote, and counting what it actually contained.
- · permalink
A missing invoice number is not a cosmetic bug. It is a gap an auditor will ask you to explain, and "our sequence skipped" does not count as an explanation.
Most teams hit this the same way. The number gets produced inside the request that renders the document, a count plus one, or a max plus one. Two checkouts land in the same second, both read the same value, and you ship either a duplicate number or, once one transaction rolls back, a hole in the series.
The reflex fix is a database auto-increment sequence. It solves the duplicates and keeps the hole: sequences are deliberately non-transactional, so every rolled back attempt burns a number.
What holds up is a counter that lives in the store as a row, scoped per series and per period, locked for the moment of allocation. And allocated when the document becomes final, not while a draft is still being edited. You serialize issuance inside one series, which is affordable, because nobody issues invoices at request-per-millisecond rates.
The split we keep coming back to: the VAT regime decides what lines appear on the document. The sequence decides whether the document is legal at all. Two different concerns, and mixing them is how numbering ends up in the template layer.
So it is worth checking where your invoice number gets assigned, at draft creation or at the moment the invoice is issued.
We wrote the full breakdown here: https://shipmindlabs.com/c/4a83d145
- · 8 min readA permission check should return a denial, not False
Most Django backends answer authorization questions with a boolean, and everything the check knew gets thrown away on the way out. A courier's app shows a refund button that returns 403, support…
- · permalink
Your support team should not have to read source code to answer "why can't this be edited?"
That is exactly what a boolean permission check costs you. It returns False and discards the only three things anyone downstream needed: which actor, which permission, which ownership rule failed.
The user gets a generic 403. Support escalates. An engineer opens the repository to reconstruct a decision the code already made and then threw away.
We build the check to return an explainable denial instead. Same call site, richer answer: the actor, the permission requested, and the specific rule that rejected it. The interface can then say something true and specific, and a support agent can close the ticket without a developer.
The part teams underestimate comes next. Once those rules are declared as data rather than scattered across view conditionals, the same rule set can drive the Django admin: which roles see which models, which fields are read-only, who may act on an object they do not own. One definition, two consumers. No second, informal permission model quietly drifting out of sync with the first.
That drift is the real failure mode we keep meeting in inherited codebases. The API says no, the admin says yes, and nobody can tell you which one is correct.
So it is worth looking at how access questions get answered on your product today. If the denial does not carry enough context to resolve the ticket, it probably lands on an engineer's desk.
https://shipmindlabs.com/c/056a6fe2
- · permalink
When someone on your team says "the system won't let me do this," a permission check that only answers no leaves everyone guessing — was it their role, the record they touched, or something nobody remembers configuring. We write our permission checks to answer with a reason instead: which person, which permission, which ownership rule failed. The same rule set then drives what each role sees in the internal admin panel, so access is described once rather than re-invented in two places. How often does an access question in your company end up as a message to a developer?
https://shipmindlabs.com/c/43f0cfa3
- · permalink
We once passed on a strong candidate over a single answer. They walked us through a production incident they had fixed, cleanly and honestly, but when we asked how they found out it was broken, there was nothing. No alert, no dashboard, no memory of who noticed first.
That gap is the part clients actually feel. Nobody experiences the fix, they experience the hours before anyone knew. So we hire for both halves now, the repair and the noticing, because a team that only knows how to fix things is still waiting on a customer to tell them something is wrong.
It is worth knowing how your own team usually finds out first: a monitor, or a message from a user.
- · permalink
Most accessibility statements get written by someone who was not in the room when the checks were run.
That is the whole problem. The statement is a public claim about the product, and in many markets it is a legal one. But it usually gets composed at the end, from memory, from a half-remembered audit, from a screenshot someone pasted into a ticket four months ago. Nobody is lying. The evidence just never existed in a form anyone could check.
We hit this while building a11ytrail, our open-source accessibility tooling, and the design decision that mattered was not the checking. It was inverting the direction of authorship.
Every check gets recorded as evidence: who ran it, when, what was checked, what the result was. Automated scan and manual review are both first class. A human confirming keyboard navigation on a component is evidence in exactly the same shape as a rule engine flagging a contrast failure.
Then the statement is generated from that record. A sentence claiming conformance for a criterion can only appear if evidence supports it. If nobody checked it, the statement cannot say it was checked. If the last check was a year ago against a page that has since been rebuilt, that shows.
The generator is not allowed to be more confident than the audit trail.
What this changes in practice is unglamorous and useful. The gap between what you claim and what you verified stops being invisible. It becomes a diff, and someone can look at it before a regulator, a customer, or a user with a screen reader does.
The general shape probably applies well outside accessibility. Compliance reports, security questionnaires, SLA claims: anywhere a document asserts something about a system, the document should be downstream of recorded evidence, not upstream of it. Prose is easy to write and impossible to audit.
If you publish an accessibility statement, it is worth knowing what it is actually generated from today, and whether you could reconstruct the checks behind any single sentence in it.
- · permalink
"Denied" is not an answer. "Not the owner of this record" is.
Most permission layers we inherit return a bare False, so nobody can tell a missing role from an ownership rule, and the admin quietly grows a second copy of the same logic. One rule set, one explanation, in both places.
When a user pings support about a 403, your system should be able to name the rule that denied them. https://shipmindlabs.com/c/824eabe2
- · permalink
A hiring habit we dropped: reading long tenure as loyalty.
Now we ask what state the system was in when they left, what was written down, who could run it on Monday. After a few multi-year engagements it became clear that leaving well is a skill, and it shows up in how someone builds long before they go. You can ask candidates about their exits.
- · 9 min readThe order lifecycle belongs in a table, not in your endpoints
In most order systems, the rules that govern an order live nowhere in particular. The rule that the warehouse may ship only a paid order with stock on hand gets written in the ship endpoint, then…
- · permalink
When an order breaks, the hard question is usually not what went wrong. It is who allowed this, and when. If the rules for moving an order from paid to shipped to refunded sit as conditions spread across half a dozen services, probably nobody can answer that without reading code.
We now declare the states and the allowed transitions as plain data, with rules about who may trigger what. The system either applies the change or refuses it with a stated reason, and every applied change lands in a history nobody can edit afterwards.
We are curious how your team answers the "who allowed this" question today. https://shipmindlabs.com/c/f732b545
- · permalink
Two delivery zones overlap. Which rate applies?
If you cannot answer that from the data alone, then your system is answering it from insertion order.
We ran into this while building courier and warehouse tooling for delivery marketplaces. Zones start clean: one polygon per area, no ambiguity anywhere. Then operations adds a surcharge ring around a bridge, a promo zone for a new district, a restricted area that overrides everything. Now a single address falls inside three polygons, and the lookup returns the first row the query happened to find. That result looks stable in staging, and it shifts the day someone reimports a shapefile.
The fix is small and boring: priority is a field on the zone, not a rule in the code. The point-in-zone lookup returns all matches and picks the highest priority one. The overlap is still there, it is just a decision somebody made and can review, instead of an accident.
We kept this pattern in our open-source delivery-zones project. The geometry is fairly ordinary. The part we care about is that the tie-break lives in the data, where operations people can see it, instead of buried in a query builder.
Our general take, and we think it holds pretty widely: any lookup that can return more than one row needs a declared ordering, or you have shipped a coin flip.
Somewhere in your system a silent tie-break probably decides something a customer actually pays for.
- · permalink
The clearest hiring signal we get comes from how a candidate talks about the worst code they inherited, not from their best project.
Contempt means they will rewrite it. Curiosity ("someone shipped this under a constraint we can't see yet") means they will read the history first. We have modernised enough legacy to know which one we want, and you probably know which one you would hire.
- · permalink
Part of our hiring process isn't technical at all: we ask a candidate to explain something they built to someone with no engineering background. If the tradeoff can't be made plain, the client probably ends up approving decisions they don't really understand, and that's usually where a project starts drifting.
We're always curious about the clearest explanation you've ever gotten from a technical person.
- · permalink
In our shift-planner, declining a shift isn't a penalty — it lowers your priority for the next few assignments, then decays. Punishment makes people accept shifts they'll drop later; a cooldown just moves work to whoever's actually available. Where else are you punishing a signal you need?
- · permalink
Every order bug we have inherited traces back to the same thing: transitions living as if-chains across three services, so nobody can answer "who could move it here, and when."
We declare states, guards and hooks as data, so the machine applies the trigger or refuses with a reason, and every applied transition lands in append-only history. You probably already know where your order logic sits. https://shipmindlabs.com/c/6ed94e2d
- · permalink
Ask an engineer which order transitions are legal and you probably get a shrug. Ask the codebase and you get eleven if-chains in six services.
That is the actual problem with order status. The status field is easy. The rules around it are the product: who may cancel after payment, what happens when a refund lands on a partially shipped order, which state a compliance hold drops you into. Those rules end up as conditionals spread across the services that happened to need them first.
Nobody owns that map. So adding one state means auditing every service that reads status, and hoping the review catches the branch you forgot.
We have been declaring it as data instead. States and transitions live in a table: trigger, source, target, the role allowed to fire it, the hook that runs on success. The machine applies a trigger or refuses with a reason. Every applied transition lands in append-only history.
What changes is not elegance. It is that the transition map becomes reviewable by someone who is not reading source code, and "how did this order get here" has an answer instead of a log grep. In deal flows where money and signed documents accumulate against the state, that history is the audit trail.
We think it is worth checking where your order rules actually live right now: in one place, or wherever the last feature needed them.
https://shipmindlabs.com/c/bff07710
- · permalink
We once made an offer mostly because of a question the candidate asked us. We gave them a small task around recording a payment, and before writing a single line they asked what happens if it runs twice.
That instinct is a big part of the job. Real systems retry — a network hiccup, an impatient user, a queue that redelivers — and code written as if that never happens is the code that charges a customer twice. We can teach a framework in a week; the habit of asking what the world does to your work is much harder to install.
What's the best question a candidate has ever asked you in an interview?
- · permalink
Two invoices issued in the same second, one number. That is not a race condition you fix later — that is an accounting document you now have to explain to an auditor.
Most teams treat the invoice number as a formatting concern. Take a counter, add a prefix, pad it to five digits, render it on the PDF. It works in development, where requests arrive one at a time.
Then two workers hit the same series concurrently. You get a duplicate, or you get a gap — and a gap is not neutral either. In several VAT regimes the sequence itself is the thing being audited: numbers within a series and period must not repeat and must not skip. A missing number reads as a deleted invoice until you prove otherwise.
The fix is boring and structural. The number is allocated by the store, per series and per period, under a lock the database owns — not computed in application code and hoped to be unique. Application code decides which lines appear: domestic sale, private buyer, reverse charge. The store decides which number is legal.
We pulled this apart while building euinvoice, our open-source invoicing work. The split that made it survive: VAT logic is a pure function of the transaction, numbering is a transactional guarantee of the storage layer. Mixing them is how teams end up retro-editing issued documents.
One consequence worth planning for early: a failed invoice after allocation still consumes a number. You cancel it, you do not reuse it.
How does your system handle an invoice that fails validation after the number is already allocated?
- · permalink
Founders almost always ask for speed. What they usually need is someone who knows which 5% of the system must be slow: the ledger write, the migration, the payout path. The rest is recoverable, so it can be messy. We've never once regretted moving slowly there. Which part of your product would you never let anyone ship fast?
- · 9 min readPromo code rejections belong in the type system
A promo code that fails at checkout has to serve two audiences at once: the buyer, who needs to know whether to fix something or give up, and the business, which needs to know which rule fired and…
- · permalink
Your promo endpoint returns false. Support now has three questions and no answer: expired, already used, or does the cart just not qualify?
A boolean flattens the reason into nothing. So teams patch it with strings — "PROMO_EXPIRED", "not eligible" — and those rot the first time someone rewords a message for the frontend and quietly breaks the analytics query that counts rejections by reason.
What we keep coming back to: the rejection reason belongs in the public API as a closed set of typed outcomes the caller has to handle. Expired. Usage limit reached. Minimum order not met. Wrong customer segment. The frontend maps types to copy in one place. Analytics counts types, not sentences. Adding a new rule becomes a compile-time conversation with every caller instead of a surprise in checkout.
The other half is arithmetic. A percentage discount that rounds the wrong way, stacks with a second code, or runs past the order total is a refund your billing logic never agreed to issue. The clamp belongs inside the money type — one place that cannot produce a negative total — not in call-site ifs spread across the checkout flow.
Where do your promo rejection reasons live today: in the type system, or in the copy?
Full write-up: https://shipmindlabs.com/c/332e0017
- · permalink
When a promo code fails at checkout, most systems just say no. That single word costs you twice: the customer doesn't know whether the code expired, didn't apply to their cart, or was already used, and you can't tell later which of those was happening most. We write the rejection reason into the type the checkout returns, so the page can explain itself and the numbers stay countable. Same story with the discount arithmetic — it belongs inside the money type with hard limits, not scattered across the places that happen to call it. Our new article walks through it: https://shipmindlabs.com/c/42616974
What does your checkout tell a customer when their code doesn't work?
- · permalink
Most promo validators return a boolean. The user reads "invalid code", support gets a ticket, and analytics gets nothing: expired, wrong region, basket below minimum — all collapse into the same false. Rejection reasons are public API surface, so give them a type. Strings rot the first time someone renames one. And the discount arithmetic belongs in the money type with hard clamps, not in ifs at the call site. Where does your checkout actually decide a code is invalid? https://shipmindlabs.com/c/005dff7c
- · permalink
Most interviews ask what a person can build. We also ask them to describe handing over a feature they'd only half-finished: what they'd write down, what they'd warn the next person about, what they know is fragile. The ones who answer that well are the ones whose work keeps moving when they're out for a week. What's the one question you'd never skip when hiring?
- · permalink
Every order table has a state machine in it. Most are undeclared: a status column and if-checks scattered across the codebase. Declare the transitions as data, guard them by role, append every applied one to immutable history. Then "who changed this and when" stops being a forensics project. Where does your state logic actually live?
- · permalink
A permission check that returns False tells you nothing. It tells the user nothing, it tells your support engineer nothing, and it tells the next developer nothing.
That is the part teams underestimate.
The access rule itself is usually simple: this actor, this permission, does this object belong to them. The expensive part shows up months later, when a customer says "I should be able to edit this" and nobody can answer without opening the code and reconstructing the decision by hand.
So we changed what a check returns.
Not a boolean. A denial that explains itself: which actor was evaluated, which permission was required, which ownership rule failed. Support reads the answer instead of escalating it. The engineer debugging a report gets the reason, not a stack trace hunt.
Declaring actors and permissions explicitly buys a second thing we did not expect at first. Once the rules are data rather than scattered conditionals, the same rules can drive the admin interface. In our own Django work that means an admin adapter separated by role — a compliance reviewer and an operations user see different objects and different actions, from the identical rule set that guards the API.
We extracted this into an open-source package we call role-scopes, because we kept rebuilding it. Compliance tooling made the need obvious: when a regulator or an auditor asks why someone was allowed to approve something, "the code said no" is not an answer.
One rule set, two consumers, and every denial able to state its own reason.
How does your system answer the question "why was this user denied?" — does it come from the code, or from someone reading it?
- · permalink
A hiring habit we changed: we stopped asking what someone is proudest of. Now we ask what part of a system they owned they still don't fully understand.
The proud answer is rehearsed. The second one is only answerable if you actually ran the thing after launch. Vagueness there usually means they built it and left before it got interesting. What's your version of that question?
- · permalink
Our most useful interview question is one we leave underspecified on purpose.
Some candidates start coding. Some ask what happens when the payment provider returns a duplicate. The second group is the one that survives contact with production — most real work arrives half-specified, and the assumptions you make silently become bugs someone else inherits. What underspecified detail would you have asked us about first?
- · permalink
If your order service knows the difference between an SMS and a push, the notification layer has already leaked.
It starts small. One if-statement for the channel. Then a second one for the provider that needs a different payload shape. Then retry logic in the checkout handler, because that provider times out. A year later, adding a fourth channel means a diff across every feature that ever notified a user, and testing any of it means mocking three vendor SDKs.
We kept hitting this while fanning out notifications to messengers, mobile and web push, so we settled on one rule: the calling code says who, what event, and what data. Nothing else.
Behind that single dispatch call sit provider adapters, one per channel, each owning its own payload translation, its own failure modes, its own retries. Delivery runs in the background, so a slow provider never shows up as a slow API response. We packaged the pattern as an open-source project we call notify-dispatch.
The part that actually decides whether this holds: the dispatch signature has to stay narrow. The moment it grows optional channel-specific fields, you have rebuilt the branch, just further from the code that needed it.
The honest test is adding a channel. If that is a new adapter plus config and nothing else, the abstraction is real.
Where did yours break first — the payload shape, or the per-provider failure semantics?
- · permalink
Halfway through one interview we stopped asking about code and asked the candidate to explain a feature they had built as if we were the client paying for it. They did it in two plain sentences, no acronyms, and that is what got them hired: an engineer who cannot explain their own work leaves the client guessing for months, and guessing is where budgets quietly go. We check for it now in every interview, because most of the friction we have seen on projects was never a technical failure, it was a translation failure. What is the last technical explanation you actually understood the first time you heard it?
- · permalink
The worst notification bug we've shipped wasn't a delivery failure. It was a message that arrived perfectly — reading "Your order {order_id} has shipped."
Nothing errored. The queue was green. The provider returned 200. The customer got a literal placeholder.
This is what happens when a template is treated as a string instead of a contract. Most notification code renders whatever it's handed, and a missing key becomes an empty string or a leftover brace. The pipeline has no opinion about it, because the pipeline never knew what the template needed in the first place.
So in our open-source notify-dispatch we made the template declare its own variables and their types up front. Rendering isn't a string operation anymore — it's a call against a signature. Values get validated at dispatch time, before any provider sees them. A forgotten order_id raises where it belongs: in the test that builds the payload, not in someone's inbox.
The expert take: in multi-channel systems this matters more than it looks. The same event fans out to email, push and messenger templates, each with a slightly different variable set, each maintained by a different person on a different day. Without declared variables you have four silent contracts and no way to test them together. With them, adding a channel is a schema change you can review.
A notification is the one part of your system that a customer reads word for word. It deserves the same type discipline as an API boundary.
How do you catch template drift today — schema validation, snapshot tests on rendered output, or does it only surface when support forwards the screenshot?
- · permalink
SMS, email, messenger — the channel is transport. Rate limits, attempt counters, single-use expiry: those live above it, once. We learned this building otpguard. If your OTP logic changes when the channel does, what exactly did you make pluggable?
- · permalink
Most of our interviews are spent reading code, not writing it. We hand a candidate a working piece of someone else's service and ask what they'd want to understand before changing it — because on real projects, the first months of any engagement are spent inside systems somebody else built, and the people who ask good questions there are the ones who don't break things. Writing new code from a blank page is the easiest part of the job and the least of what we hire for. What's the last thing you looked at in your own product and thought: I'd want to know why this was built this way before touching it?
- · permalink
Founders ask us for a stack match. What they describe two questions later is someone who'll still understand the system in year three.
Stack is the easiest thing to teach. Staying is the hard part — and most job posts don't price it at all. What are you actually hiring for?
- · permalink
We once interviewed someone who spent the first ten minutes asking what happens when things break: who gets woken up, how we find out a payment silently failed, what we do when a customer says money left their account and nothing arrived. Not one question about frameworks. We hired them, because anyone can learn our stack in a month, but the instinct to ask "what does this look like at 3am when it goes wrong" is what actually keeps a client's system running. What's the one question you wish more people asked before joining your team?
- · permalink
One interview habit we dropped: asking what someone built from scratch. We ask how they read code they inherited instead. Most of our work is somebody else's system — legacy migrations, monoliths where accidental behaviour turned load-bearing. A greenfield story tells us almost nothing about that. What's the interview question you stopped asking and never missed?
- · permalink
Most OTP bugs we see are not in the code generation. They are in what the caller forgot to check before verifying it.
The usual shape: a library gives you generate() and verify(). Everything protecting that pair — how soon a user can request a new code, how many wrong guesses before the code dies, how long the account stays locked — is left to the endpoint. So it gets written once in the login handler, differently in password reset, and not at all in the phone-change flow someone shipped later.
Each of those is a separate brute-force window, and none of them look like bugs during review.
Our take, after years on flows where a wrong code means money or identity: a one-time code is not a string. It is a small state machine that knows when it was last sent, how many attempts it has absorbed, and whether it is currently locked. If that state lives inside the library, every caller inherits the protection whether or not the developer was thinking about it that day. If it lives in the caller, protection is a convention — and conventions decay per endpoint.
The practical test: add a new OTP-protected action to your product. If you have to remember to re-implement cooldown and attempt limits, they are in the wrong place.
We put ours in a small Python package, otpguard, open on GitHub under shipmindlabs — resend cooldown, attempt counting and lockout are part of the verification result, not a separate thing to wire up.
Where does the attempt counter live in your auth stack right now — the code, the session, or the endpoint?
- · 7 min readNotification Fan-Out Needs a Stop Condition, Not a Retry Loop
A service that turns one event into push, chat and email messages is easy to build and hard to keep honest. The failure that reaches support is almost never a dropped message; it is the second and…
- · 9 min readAn audit trail is a typed changeset, not a log line
Most Django projects discover their audit requirement late. A compliance reviewer asks who moved an application to the next issuance tier and when, and the answer has to be assembled from…
- · permalink
When we take over an inherited system, the hard part is rarely the old code — it's that nobody left in the building can explain why a particular rule exists. So before rewriting anything, we spend time working out what each odd behaviour was protecting against, because half of those quirks turn out to be someone's fix for a real problem. What's the strangest rule in your business software that nobody can explain anymore?
- · permalink
When a customer asks why a number changed six months ago, the honest answer depends on one design choice made much earlier: did the system record who made the change at the moment it happened, or does someone now have to reconstruct it from server logs? We build audit so the actor is resolved explicitly and saved alongside the field-level diff — the record itself says who changed what, so nobody is stitching timestamps together under pressure. We keep our approach to this open at github.com/shipmindlabs/model-audit. If someone asked you today who changed a key setting last spring, how long would it take you to find out?
- · permalink
We wrote our field-diff engine with zero Django imports. It takes two mappings, returns a typed changeset. The framework layer just feeds it model state.
Not purity — testability. No ORM to spin up, and it survives whatever we migrate to next. It's open: github.com/shipmindlabs/model-audit
Where's the line in your codebase between the framework and the thing it's calling?
- · permalink
The most useful part of our model-audit package has no idea Django exists.
It is a field-diff engine. Two mappings go in, a typed changeset comes out: which fields changed, from what, to what. That's it. No model class, no ORM query, no import from the framework anywhere in that layer.
The Django side does the boring half — it knows how to get the previous state of a row and the new one, and it hands both to the core as plain data.
The reason this matters is not elegance. It's what your tests have to boot.
When change-tracking logic lives inside a model method or a signal handler, testing "did we detect this edit correctly" means a database, migrations, fixtures, and a save cycle. You end up testing the ORM to verify your own business rule. So people write three cases instead of thirty, and the interesting ones — a field set to null, a decimal that changed representation but not value, a JSON blob with reordered keys — never get written.
With the core separated, those are dictionary-in, changeset-out tests that run in milliseconds. And the same engine handles a dict from an API payload or an imported row, because it was never coupled to a table in the first place.
We keep it open at github.com/shipmindlabs/model-audit if you want to look at where that seam sits.
What's the last piece of logic you pulled out of a framework class to make it testable — and what finally forced the move?
- · permalink
Promo validation returning false is a design smell. Expired, usage-limit-hit, not-yours — those are different answers, and every caller ends up reverse-engineering them from a message string.
We made the rejection reasons a typed part of the contract (open-sourced it at github.com/shipmindlabs/promocodes). Feels obvious after payments work, where "declined" without a code is useless.
Where do you draw the line on exposing reasons to the client?
- · permalink
Most audit trails are not audit trails. They are log lines someone hopes to reconstruct a story from later.
The pattern is familiar. A model changes, something gets written — a message, a serialized blob, a "user updated" event. Then a dispute arrives: who changed the payout requisites, when, and what was the value before? Now an engineer is grepping, joining timestamps, and inferring intent. That is forensics. It happens after trust is already gone.
An audit record has to answer three things at the moment of the write, not afterwards: which fields changed, what the old and new values were, and which actor is responsible. If any of those is derived later, it is a guess.
Two things make that hold in production.
First, the changeset is typed and field-level. Not a text description of a change — a structured diff you can query, compare, and assert against in a test.
Second, the actor is resolved at write time. Requests have a user; background jobs, admin scripts, webhook handlers and migrations do not. If the actor is only available from request context, half your writes will be attributed to nobody — and those are usually the interesting ones.
The part teams skip: keep the diff core framework-free. If the logic that computes a changeset only runs inside the ORM's signals, the guarantee is only testable inside a full stack, so in practice it is barely tested at all. Pull it out and it becomes plain input-output — old state, new state, actor — verifiable in isolation.
We wrote the long version up here: https://shipmindlabs.com/blog/an-audit-trail-is-a-typed-changeset-not-a-log-line/
For those of you running compliance-sensitive systems: how do you attribute changes made by background jobs and scripts, where there is no request user to fall back on?
- · permalink
Months after the fact, someone always asks who changed this price, this limit, this customer record — and when. If the answer has to be reconstructed by reading through logs, that is forensics, not an audit trail. We write ours the other way round: the system records the exact fields that changed and who changed them at the moment of the write, so the answer already exists before anyone asks. What is the last record you wished you could see the full history of?
https://shipmindlabs.com/blog/an-audit-trail-is-a-typed-changeset-not-a-log-line/
- · permalink
If you can only answer "who changed this field" by grepping logs after the fact, that's forensics, not audit.
We write audit as a typed changeset with the actor resolved at write time, and keep the diff core free of the ORM so the guarantee is testable on its own. Where does your audit trail actually get written?
https://shipmindlabs.com/blog/an-audit-trail-is-a-typed-changeset-not-a-log-line/
- · permalink
Every monolith we've split had a few behaviours nobody designed on purpose. A timing quirk, a field that was always null, an endpoint that quietly tolerated bad input. Then someone built on top of them. Splitting the code is the easy half. Which accidental behaviour in your system is now a contract you can't break?
- · permalink
A discount that goes negative is a refund. A discount larger than the order total is a payout. Both look like ordinary arithmetic until they reach the ledger.
The usual shape is familiar: a percentage rule, a fixed-amount rule, a stacking rule, and somewhere in the checkout handler a line that says if the result is below zero, use zero. Then the same guard appears in the cart preview. Then in the invoice generator. Then someone adds a new promo type and writes the multiplication without the guard, because the guard was never part of the type — it was part of whoever remembered.
We put the clamp in the discount type itself. Applying a discount to a Money amount cannot return less than zero and cannot exceed the order total, because the operation that produces the number owns both bounds. Call sites stop checking. New promo types inherit the invariant instead of re-deriving it.
After years on payment and lending systems, our take is that money bugs are rarely arithmetic bugs. They are placement bugs: the rule was correct, it just lived in the caller instead of the type, so the fourth caller never got it. We open-sourced the promocodes package we use for this at github.com/shipmindlabs/promocodes — the interesting part is not the promo engine, it is where the bounds live.
Where do your money invariants live right now — in the type, or in the handler that happened to remember them?
- · 8 min readReview queues for compliance teams: leases, not row locks
Almost every system we have built with a human in the loop needs the same component: a queue that hands items to reviewers. KYC and KYB verification, transaction moderation, issuance tiers on a…
- · permalink
One thing we've learned building notification systems: "sent" and "delivered" are two different facts, and most software only tracks the first one. That's why a customer swears they never got the confirmation while your dashboard insists it went out — the message left your server and quietly died somewhere on the way to WhatsApp, email, or a phone that had push turned off. We build these to record what actually landed, not just what we tried to send. How often does your team find out about a failed message from a customer rather than from a system?
- · permalink
A payment provider's sandbox tests your code against their happy path. Production tests it against their timeouts, partial failures, and undocumented status codes. Budget for the second one. What broke for you first in prod?
- · permalink
Every team adds retries. Almost nobody adds idempotency keys on the other side.
So the retry that saves you during a network blip is the same one that double-charges a card six months later. Retrying isn't a client-side decision — it's a contract. Who owns that contract on your team?
- · permalink
Docker Compose is a great development tool. It is not a deployment strategy — but that is where most teams end up using it.
The path is always the same. Compose starts as the way a new engineer gets the project running in one command. It works. So it goes to staging, because staging is "just a bigger laptop". Then production needs to ship, the compose file already works, and nobody wants to introduce a new tool during a release week.
Now the ops model is a file that was written to make onboarding easy.
The symptoms show up later. Restarts drop in-flight work because nothing waits for the worker to finish its current task. Deploys are a short outage instead of a rollover, because there is no second instance to shift traffic to. Logs live on one host. Secrets live in the same file as the service definitions, so rotating one means editing the thing that defines your topology.
None of that means you need Kubernetes. It means the questions Kubernetes answers — how do I roll over without dropping requests, where do secrets live, what restarts a dead process, who collects the logs — still need answers. A single well-run host with systemd units, a reverse proxy, and a real secret store answers them for a lot of products.
Our rule: local orchestration and production orchestration are allowed to be different tools. Compose stays the developer's tool. Production gets whatever is chosen on purpose, even if that choice is deliberately small.
If your compose file made it to production, what forced you to change it — a failed deploy, or an incident?
- · permalink
A cron job that runs longer than its interval is now two jobs. Payment systems find this out in production. Where did overlap first bite you?
- · 8 min readWhy LLM catalog translation is a cache design problem
Translating a product catalog with a language model looks like a prompt problem and turns out to be a cache problem. A catalog is never translated once: a merchant edits a title, a supplier feed…
- · permalink
People assume code review is about catching typos. It isn't — the compiler finds those. What review actually catches is one engineer's private assumption about how something works, before that assumption reaches your customers. The side effect matters just as much: after a few months of reviewing each other's work, at least two people understand every part of the system. What's a decision in your business that only one person currently understands?
- · permalink
Every monolith we've split had one shared transaction quietly holding two features together. Nobody documented it. You find it the day the split goes live and the second write silently stops rolling back.
What's the invisible coupling in yours?
- · permalink
Every notification service starts as one function called send. It ends as the most fragile part of the system.
The pain shows up late. A user gets the same payment alert three times: push, WhatsApp, and email. Or worse, gets nothing at all because their push token expired two months ago and no one owned that failure. Support hears about it before monitoring does.
The reason is that teams model notifications as delivery, when the hard part is state. Which channels does this user actually have? Which ones are still valid? Was this event already delivered somewhere else? Is this notification worth waking someone up for, or is it a digest item?
We have built fan-out services pushing to WhatsApp, Telegram, mobile and web push, and the pattern that survived production was treating the event and the delivery as separate objects. One event, many delivery attempts, each with its own status. Deduplicate on the event, not the message. Keep channel preferences and token validity in one place that both the product team and the compliance team can read. And make silent failure loud: an expired token that fails quietly is indistinguishable from a happy user until it isn't.
The unglamorous version of this is a table of delivery attempts you can query. That table is what turns "the user says they never got it" from an argument into a lookup.
For teams running multi-channel notifications: where did yours first break down — deduplication, preference management, or token and identifier decay?
- · permalink
A scraper isn't an integration. It's a subscription to someone else's release schedule, and every silent redesign on their side is an outage on yours. The failure that hurts is HTTP 200 with nothing in it. What alerts when your parser succeeds and returns empty?
- · permalink
The scary part of a migration isn't the schema change. It's the minutes where old code and new code both run against the same table.
Most rollback plans we inherit assume that window doesn't exist. Does yours?
- · 8 min readExpand and Contract: Changing a Payments Column Without Downtime
Changing the shape of a payments table is trivial in a migration file and hard in production. The rows are being written to while the migration runs, the previous release and the new one serve…
- · permalink
When a payment fails halfway, the honest answer is that nobody yet knows whether the money moved. The safe fix isn't retrying harder — it's giving every attempt an identity, so the second try recognises the first and refuses to charge again. Has a double charge ever reached one of your customers, and how did you find out?
- · permalink
Every team adds retries before they add idempotency keys. Which means for a while, your "resilience" is just a machine that charges people twice under load.
We've inherited this in payment flows more than once. Where does your system dedupe — the client, the gateway, or nowhere yet?
- · permalink
Notifications look like a feature until the first duplicate storm. Then you find out you built a delivery system without dedup, ordering, or retry limits. Where does yours keep that state?
- · permalink
Moving work to Celery makes your API fast. It also quietly turns every user action into a promise nobody is tracking.
The pattern is familiar. A signup triggers a welcome email, a KYC submission triggers a verification call, a payment triggers a ledger update. All of it goes to a queue so the HTTP response stays under 200ms. The endpoint is fast. Everyone is happy.
Then a verification provider times out at 3am, the task raises, the worker moves on, and a user sits in "pending review" for two days. No error page. No alert. The request succeeded — the promise didn't.
What we've learned running payment and compliance systems on Python queues: the moment a task represents something a user is waiting for, it stops being infrastructure and becomes product state. It needs a row in your database, not just a job in Redis. Enqueued, started, failed, succeeded, retried — visible to the people who answer support tickets, not only to the engineer with terminal access.
The practical version is boring. Every user-visible async action gets a status record written before the task is enqueued. The task updates it. A reconciliation job scans for records stuck in a non-terminal state longer than they should be. That job is the actual monitoring — it catches the failures your exception tracker never saw because the worker died mid-task.
Celery is excellent at running code later. It was never designed to tell your business what didn't happen.
For teams running background work in production: how do you find the tasks that failed silently — and how long does it usually take?
- · 6 min readQueue Topology: One Broker, Several Queues, No Shared Pool
A single task queue served by a single worker pool works until the day a bulk job — a catalog scrape, a translation batch, a re-notification sweep — holds every process for minutes at a time.…
- · permalink
Notification services get scoped as a two-day task, then quietly become critical infrastructure. Nobody defines what happens when the push fails, the email bounces, and the user never learns their payment went through.
Who owns delivery guarantees on your team?
- · permalink
Before anything reaches your customers, it runs on a full copy of the system where breaking things costs nothing. It feels slower for a day and saves you the week where payments quietly stop working. What's the last change you wished someone had rehearsed first?
- · permalink
Running a product catalog through an LLM is the easy 10% of localization.
The expensive part starts after the first pass.
Most teams treat catalog translation as a batch job: dump the SKUs, call the model, load the results. It works once. Then merchandising edits 400 titles, a supplier renames a category, and someone asks why last month's fix disappeared.
What makes an LLM content pipeline survive production is not prompt quality. It is bookkeeping:
- a content hash per field, so you only pay for what actually changed
- a human override layer that re-runs are not allowed to overwrite
- a review queue where the diff is visible before it reaches the storefront
- versioning, so a bad batch rolls back without a database restore
We build these pipelines for cross-border catalogs, and the same lesson keeps repeating: the model is a component, the pipeline is the product. Teams that skip the state tracking re-translate everything on every run — the most expensive option and the least trustworthy one, because nobody can tell which strings a human already corrected.
If you push LLM output onto a live product surface, how do you handle human edits — do your re-runs respect them, or does someone re-apply the same correction every month?
- · permalink
In a rewrite, the old system's bugs are part of the spec. Someone downstream depends on every one of them. What quirk did you have to reimplement on purpose?
- · 7 min readExtracting a Service From a Monolith Without a Code Freeze
Every monolith extraction reaches the same question: how do you know the new service returns the same answers as the old code, on real production data, before anything depends on it? A test suite…
- · 9 min readOne payment protocol for every provider: hold, capture, refund
A payment integration usually starts with one provider and ends with three, and by then the provider's vocabulary has leaked into the order code: one branch for the gateway that separates…
- · permalink
Every KYC provider we've integrated has a failure mode that looks exactly like the user doing something wrong. Blurry doc, timeout, silent decline — same screen.
So people retry, get rejected again, and leave. What does your onboarding do when the provider is the one that's broken?
- · 7 min readTesting Money Paths: Four Tests Our Review Gate Requires
A payment service can pass every unit test it has and still lose money. The failures that matter in production are not about one request being wrong: a provider webhook delivered twice, two workers…
- · permalink
The scariest thing in a small project isn't a bug — it's one person being the only one who knows how something works. We keep every part of a system readable by at least two people, so a holiday or a sick week never turns into a frozen release. Who is the single point of failure on your product right now?
- · permalink
Manual KYC review is not an admin screen, it's a queue with latency and a backlog. Build it like a production system or onboarding stalls. How do you measure your review queue?
- · permalink
Every monolith rewrite we've done, the real work wasn't the code. It was finding out what the old system actually did, because nobody knew anymore. Half the "bugs" turned out to be load-bearing.
How do you tell a bug from a feature in a system older than everyone maintaining it?
- · permalink
The most dangerous code in most Python systems is the code nobody watches: the background task.
An HTTP endpoint fails loudly. Someone gets a 500, support gets a ticket, you get a trace. A Celery task fails into a log line nobody reads, and the system keeps looking healthy.
We have shipped queue-backed services in payments, e-commerce, and notification fan-out for years, and the failures repeat with the same shape. A payout webhook retried three times because the ack happened after the work. A catalog import that half-finished and left prices from two different runs in the same table. A notification service that quietly stopped delivering because one poisoned message kept the worker busy forever.
None of those were queue bugs. Celery and RabbitMQ did exactly what they were told.
What actually holds up in production is boring:
Every task takes an idempotency key, not just arguments, so a redelivery is a no-op instead of a second charge.
Every task has a hard time limit and a dead-letter path, because a task with no ceiling will eventually consume the whole pool.
Every task reports failures where humans already look, not into a log file. If your API errors page someone and your worker errors do not, you have two reliability standards in one system.
And tasks stay small. A task that does five things fails in the middle of the third one, and you get to reason about partial state at 3am.
Our rule when we take over an existing codebase: read the task modules before the views. That is where the unowned complexity lives.
What finally forced your team to take background jobs as seriously as your API — a duplicate charge, a silent data drift, or something worse?
- · permalink
Every webhook you receive will arrive twice eventually. If your handler isn't idempotent, you don't have an integration, you have a bet. What broke first for you: payments or carrier callbacks?
- · permalink
People sometimes ask why we spend time testing code that already works. Because the code isn't the risk — the next change is, six months from now, made by someone who wasn't in the room. Tests are how a team keeps moving when the person who wrote it is on vacation. How much of your codebase could someone new safely change tomorrow?
- · permalink
If you can't rebuild the ledger from events, you don't have reconciliation, you have hope. We add that job on day one, not after the first mismatch. What broke your books first?
- · permalink
Every search index you add is a second source of truth you now have to reconcile. Most teams budget for the sync job, not the drift. Who owns reindexing when it goes stale?
- · permalink
Every payment system we've inherited had reconciliation bolted on at the end, treated as a reporting job.
It isn't. It's the only proof the money math is right. Build it with the flow, or you're shipping on faith. What convinced your team to take it seriously?
- · permalink
Every payment integration looks done when the first charge succeeds. It's actually done when reconciliation runs for a month without a human touching it.
We've never seen a payment system break at the API call. It breaks at attribution — which money belongs to which user. What's your worst mismatch story?
- · permalink
A refund is not a payment with a minus sign. Different rails, different timing, different failure modes. Systems that model it as one flow leak money in reconciliation. Where did yours split?
- · permalink
Most monolith-to-microservices migrations we inherit are already half-done: the code is split into services, and every one of them still writes to the same database.
That is the version that hurts the most.
The team gets all the costs of distribution — network calls, deploy coordination, partial failures, harder debugging — and none of the independence they were promised. Two services still can't be deployed separately, because a column change breaks both. A migration still locks the whole product. Nobody owns the schema, so everybody edits it.
When we take over rewrites of inherited systems, we work in the opposite order. Data boundaries first, code second.
That means picking one domain that genuinely has its own lifecycle — payments, documents, notifications — and asking a boring question: can this domain hold its own tables and expose everything else through an API? If the answer requires a join across three other domains, it isn't a service yet. It's a module, and it should stay in the monolith until the data separates cleanly.
We've pulled service layers out of monoliths this way on payment platforms, lending systems and marketplaces. The pattern holds: the first extraction is slow and unglamorous, because it's mostly about untangling reads. Everything after it goes faster.
Our take: a monolith with clean internal boundaries is a better business than a distributed system with a shared database. Split when the data is ready, not when the org chart is.
For teams that went through a rewrite — what was the first piece you extracted, and would you pick the same one again?
- · permalink
Search is slow, so the team adds Elasticsearch. Six weeks later the catalog shows products that were delisted yesterday.
That is not an Elasticsearch problem. It is what happens when a search index becomes a second source of truth without anyone owning the sync.
The pattern we keep seeing: a product query gets heavy, someone puts the data into an index, and the write path grows a fork. One branch goes to PostgreSQL, the other to the index. They agree on the happy path. They disagree whenever a write fails halfway, a background job dies quietly, or a bulk import runs outside the normal flow. Nobody notices, because a stale index does not throw errors. It just answers wrong.
We run Postgres, MongoDB, Redis and Elasticsearch in production, and our rule is simple: the index is disposable, the database is not. Reindexing must be a routine operation you can trigger at any time, not a migration nobody dares to touch. Sync goes through a queue with retries, and there is a reconciliation job that compares counts and flags drift before a user does.
Most catalogs we have worked on could have gone further on Postgres than the team assumed — full-text search and the right indexes carry more load than people expect.
What finally forced you off your primary database for search — query latency, ranking quality, or something else?
- · permalink
Every deal platform we have worked on eventually hits the same wall: the database says the deal is closed, and nobody can prove what was actually agreed.
The status column is easy. A deal moves from listing to offer to deposit to closing, and someone models it as an enum.
What breaks is everything the enum does not carry.
Who signed which version of which document. Which compliance check passed before the deposit was accepted. What the terms looked like at the moment a party agreed to them, not after three amendments.
We have built this in two very different shapes. In one, the deal state and its signed documents accumulate on a token on-chain, so the trail is external and verifiable by every party. In another, e-signature workflows built to national digital-signature standards carried the same weight off-chain.
Different technology, identical principle: the document trail is the source of truth, and the status field is only a cached view of it.
Our take — if your system moves money or transfers ownership, design the evidence layer before the state machine. Retrofitting it means reconstructing history from logs that were never meant to be evidence.
For those of you running transaction platforms: when a party disputes what was agreed, what do you actually pull to settle it?
- · 8 min readCatching money bugs with ledger invariants, not error logs
A payout that credits the wrong sub-account returns HTTP 200. Nothing throws, the worker acknowledges the message, the error dashboards stay green, and the discrepancy surfaces days later when…
- · permalink
Half the Next.js apps we review are entirely behind a login screen.
Which means the main reason they chose Next.js — server rendering for search engines — is doing nothing for them at all.
Google never sees a dashboard. It never sees a billing page or an admin panel. But the team still carries the cost of that decision every sprint: two runtimes to reason about, server and client boundaries to keep straight, a hydration bug class that only shows up in production, and a hosting story that is no longer just static files behind a CDN.
The honest version of the question is not React or Next.js. It is: does this product need to be rendered before a user is authenticated?
Marketing site, docs, pricing, public listings, anything shareable — yes, and Next.js earns its complexity there. Internal tooling, dashboards, anything gated — a client-rendered app with a plain API is usually faster to build and far cheaper to keep alive.
Our rule when we scope a frontend: split the public surface from the private one early. They have different requirements, and forcing one framework across both is how teams end up paying for capabilities they never use.
For teams running Next.js on a fully gated product: what made it worth it for you?
- · permalink
The retry you added to make the system more reliable is often what takes it down.
Here's the pattern we see in code reviews more than any other. A call to a payment provider, an email service, or an internal API occasionally times out. Someone wraps it in a retry loop. It works in staging. Everyone moves on.
Then one day the downstream service slows down instead of failing outright. Every caller times out and retries. Those retries pile onto an already-struggling service. It gets slower, so more calls time out, so more retries fire. A brief latency blip becomes a full outage — and the retries are the thing keeping it down.
Two things separate a retry that heals from a retry that amplifies:
Idempotency. If the operation can run twice safely — an idempotency key on writes, a dedup check on the consumer — a retry is free insurance. Without it, a retry that half-succeeded now double-charges or double-sends.
Backoff with jitter and a budget. Retry immediately and you synchronize every client into a thundering herd. Exponential backoff with randomness spreads the load; a cap on total retries stops the loop from feeding itself.
Our rule in review is simple: no retry gets merged until we can answer two questions. Is this operation safe to run twice, and what stops it from retrying forever? If either answer is unclear, the retry is a liability, not a safeguard.
When a retry has bitten you in production, was it the missing idempotency key or the missing backoff that did the damage?
- · permalink
Most teams plug in a KYC provider, see a green checkmark, and call compliance "done."
Then the edge cases arrive: a document that's valid but mismatched, a name in a different alphabet, a selfie that half-passes. The provider returns a score, not a decision. Someone still has to make the call — fast, under audit, without blocking honest users.
We've built the layer that sits between the verification API and the compliance officer. Real-time moderation queues where a reviewer sees the full context of a user, flags cluster by rule, and every action is logged for the audit trail. The integration with the KYC/KYB provider is maybe 20% of the work. The other 80% is the review workflow, the state machine behind an application, and making sure a rejected user can be re-examined without losing history.
Our take: compliance tooling lives or dies on the operator's screen, not the API contract. A verification vendor tells you what it thinks. Your own moderation layer is where policy actually gets enforced — and it's the part you can't outsource.
For teams running KYC in production: how much of your compliance logic sits in the vendor versus in tooling you built yourselves?
- · permalink
In payments, transferring money is the easy part. Knowing exactly whose money it is — that's where systems break.
We've spent years building fintech backends where funds arrive tagged only by payment requisites: an amount, a reference string, a timestamp. From that, the system has to prove which user, which obligation, which account — and reconcile it against what the bank actually settled. Get this wrong and you don't just have a bug, you have money in the wrong place and a compliance problem.
Our approach treats attribution and reconciliation as first-class parts of the architecture, not an afterthought bolted on before launch. That means explicit ledgering with per-user accounts, deterministic matching rules, and a reconciliation layer that flags mismatches instead of silently absorbing them. Backend, QA, and DevOps work off the same staged pipeline, so edge cases surface in testing, not in production with real balances.
Our take: most payment failures aren't in the payment gateway — they're in the accounting model behind it. If your ledger can't answer "why is this amount here" for every cent, scaling only multiplies the ambiguity.
Building or reworking a payment flow and worried about reconciliation? Let's connect — happy to discuss your architecture.
FinTech #SoftwareDevelopment #Backend #Engineering #Architecture #DevOps #Payments#