Shipmind Labs

A reviewer pasted our finding back into the page and told us the phrase was not there. It was. A zero-width space sat right in the middle of it.

This was early work on screenfence, our open-source page scanner. We had done the obvious thing: strip invisible characters, fold everything with NFKC, lowercase it, then run the phrase patterns over the cleaned string. Detection was fine. The quote we handed the reviewer was the problem, because it came from the cleaned string too. The evidence had already been washed. The exact artifact that made the page suspicious was missing from the proof we showed.

It cost us a review round, and worse, an argument we could not win, because our output and the reviewer's page disagreed about what the page said.

What we do now is normalization that remembers where it came from.

We walk the source one code point at a time. Characters flagged as deliberately invisible (zero-width space, word joiner, bidi override) get dropped from the folded text. A run of whitespace collapses to a single space whose span still covers the entire run. Everything else is NFKC-folded and lowercased individually, so a fullwidth letter becomes ASCII, and every unit of the folded output records the start and end offset of the one source character it came from.

Matching runs on the folded text. Each match index and length is mapped back through those offsets, and the quote is sliced out of the original source, invisibles included. Benign typography, curly quotes and similar, we leave untouched, since it was never the signal.

The rule we took away from it: match on normalized text, quote from raw text, and keep the mapping between the two instead of trying to recompute it afterwards. Any pipeline that cleans input before making a decision about it owes the human reviewer the uncleaned version.

If your system normalizes before it decides, it is worth knowing what it shows the person who has to sign off on the decision.

Was this useful?

Building something similar?

or email hello@shipmindlabs.com