Why LLM catalog translation is a cache design problem

25 August 2026 · 8 min read

Translating a product catalog with a language model looks like a prompt problem and turns out to be a cache problem. A catalog is never translated once: a merchant edits a title, a supplier feed re-imports overnight, a new locale gets added, and the pipeline runs again over a corpus that is mostly identical to the last run. A pipeline that sends every field to the model on every run pays for unchanged text forever and, worse, returns slightly different wording each time, so the storefront quietly churns and nobody can tell an edit from a re-roll.

We have built these pipelines for cross-border retail catalogs, and the constraint that shapes everything is that the corpus is large, the churn is small, and the model is not deterministic. Those three facts together mean the interesting engineering is not in the prompt. It is in deciding what a unit of work is, what identifies it, and what has to be true about an output before it is allowed to become a cached fact.

The unit of work is a field, not a product#

The first design decision is granularity. It is tempting to send a whole product as one JSON blob: title, short description, long description, attribute values, care instructions. It is one call, the model sees full context, and the output maps neatly back onto the record.

It is also the wrong unit. A merchant who fixes a typo in the long description invalidates the entire product, so the title and forty attribute values get retranslated and rewritten with new phrasing that nobody asked for. Product-level batching also makes partial failure total: one field that violates a constraint fails the blob, and the retry re-spends on the fields that were already fine.

So the unit is a single field of a single product for a single target locale. It is small, independently cacheable, independently retryable, and independently reviewable. Context loss is real but recoverable — the field name is passed to the model as a role hint (a title is translated as a title, an attribute value as a short noun phrase), and product-level context that genuinely matters, such as the category, goes into the prompt as metadata rather than as sibling text to be translated.

The cache key is the whole design#

Once the unit is a field, the cache key defines what "already translated" means. The key must cover every input that can change the correct output, and nothing else. Source text, obviously. Target locale, obviously. But also the prompt version and the glossary version, because both change the output and neither is visible in the text.

import hashlib
import unicodedata

PROMPT_VERSION = "catalog-v7"


def normalize(text: str) -> str:
    text = unicodedata.normalize("NFC", text)
    return " ".join(text.split())


def segment_key(
    text: str,
    field: str,
    source_locale: str,
    target_locale: str,
    glossary_version: str,
) -> str:
    payload = "\x1f".join(
        [
            PROMPT_VERSION,
            glossary_version,
            source_locale,
            target_locale,
            field,
            normalize(text),
        ]
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

Normalization is not cosmetic. Feed imports arrive with trailing spaces, doubled spaces, and mixed Unicode composition for accented characters, and each variant is a distinct cache miss for text that is identical to a human reader. Normalizing before hashing collapses that noise, and it is the cheapest cost reduction in the whole system.

Putting the prompt version in the key has a consequence worth being explicit about: bumping PROMPT_VERSION invalidates the entire catalog in every locale. That is correct — a changed prompt means the cached strings were produced by a system that no longer exists — and it makes prompt edits a budget decision made deliberately rather than a config tweak that silently triggers a full re-spend. It also gives you a diffable artifact: the old rows are still there under the old key, so before promoting a new prompt version you can retranslate a sample, put the two versions side by side, and see what actually changed.

Masking the parts that must not be translated#

A catalog field is not prose. It contains SKUs, model numbers, units of measurement, template placeholders, and occasionally a URL. Models translate all of these when they feel prose-like — a model number gets localized punctuation, a unit gets converted, a placeholder gets helpfully rendered into words. Any of those silently corrupts data downstream.

The fix is to take those spans out of the model's reach before the call and put them back after.

import re

PROTECTED = re.compile(
    r"""(?x)
      (?P<sku>\b[A-Z0-9]{2,}(?:-[A-Z0-9]+)+\b)
    | (?P<measure>\b\d+(?:[.,]\d+)?\s?(?:mm|cm|m|g|kg|ml|l|W|V|Hz)\b)
    | (?P<placeholder>\{[a-z_]+\})
    | (?P<url>https?://\S+)
    """
)


def mask(text: str) -> tuple[str, dict[str, str]]:
    tokens: dict[str, str] = {}

    def replace(match: re.Match) -> str:
        token = f"[[{len(tokens)}]]"
        tokens[token] = match.group(0)
        return token

    return PROTECTED.sub(replace, text), tokens


def unmask(text: str, tokens: dict[str, str]) -> str:
    for token, original in tokens.items():
        text = text.replace(token, original)
    return text

The masked form is also what gets hashed into the cache key in the more advanced version of this pipeline, because two products that differ only by SKU then share a cache entry for an otherwise identical description. That is a meaningful hit-rate improvement on catalogs with variant-heavy families, and it costs one extra step: the tokens are stored per product, not per cache entry.

Validation is what makes a cached string trustworthy#

Caching a model output means promoting it from "something the model said" to "a fact the storefront serves". That promotion needs a gate, because a cached bad string is worse than an uncached one — it never gets a second chance to be different.

The checks that earn their place are structural, not semantic. Every mask token must appear in the output exactly as often as it appeared in the input. The output must be non-empty. The length must be in band, because a model that starts explaining itself instead of translating produces output several times longer than the source. If the field is an enumerated attribute value, the output must be in the target locale's allowed set.

class TranslationRejected(Exception):
    pass


def validate(masked_source: str, output: str, tokens: dict[str, str]) -> str:
    mismatched = [
        token
        for token in tokens
        if output.count(token) != masked_source.count(token)
    ]
    if mismatched:
        raise TranslationRejected(f"protected token mismatch: {mismatched}")
    if not output.strip():
        raise TranslationRejected("empty output")
    if len(output) > 3 * len(masked_source) + 40:
        raise TranslationRejected("output length out of band")
    return output


def translate_field(text, field, source_locale, target_locale, glossary):
    key = segment_key(text, field, source_locale, target_locale, glossary.version)
    cached = cache.get(key)
    if cached is not None:
        return cached

    masked, tokens = mask(text)
    output = model.translate(
        masked,
        field=field,
        target_locale=target_locale,
        glossary=glossary.terms_for(text),
    )
    validate(masked, output, tokens)
    result = unmask(output, tokens)
    cache.set(key, result)
    return result

A rejected output is not retried in a loop until it passes. One retry with the same inputs is reasonable, since these failures are usually sampling noise. After that the segment goes to a review queue with the source, the raw output, and the rejection reason, and the storefront keeps serving the previous translation or the source string. This is the same shape as the moderation tooling we build for compliance work: an automated decision that fails its checks becomes a human's queue item with full context, not a silent drop and not an infinite retry.

What it costs to run#

One row per segment per locale, holding the key, the output, the prompt version, and a timestamp. That table is large but boring — it is append-mostly, read by primary key, and never joined in a hot path. Old prompt versions are kept until the new one is promoted, then swept.

Steady-state cost is driven entirely by hit rate, which is why normalization and masking matter more than model choice. After the first full pass, a nightly feed import pays only for genuinely changed fields, and a new locale pays for a full pass exactly once. The two things that reset the meter are a prompt version bump and a glossary change, and both are now explicit, versioned, reviewable events rather than accidents.

The general shape here is not specific to translation. Any pipeline that puts a non-deterministic component in front of stored data has the same requirement: define the unit, make the key cover everything that changes the answer, and put a structural gate between the model's output and the cache. The prompt is the part that is easy to change later. The cache key is the part you live with.