Persian language engineering
Search, text processing and interfaces that work in Persian because the script was designed for, not translated into.
Everything fails quietly
An English-language search stack that cannot find a document raises no error. It returns an empty list, which looks exactly like a correct answer to a question with no matches. Persian produces that empty list far more often than anyone expects, for reasons that are mechanical rather than linguistic.
Two letters have Arabic lookalikes. ی and ک have near-identical Arabic counterparts, ي and ك, and both spellings occur throughout real corpora, often inside the same document. To a database they are different characters. To a lawyer they are the same word.
Three digit systems coexist. Persian digits, Arabic-Indic digits and ASCII digits all appear in real Persian text, frequently in the same paragraph, frequently inside the same citation.
A zero-width joiner decides where words end. It is invisible, it is semantically necessary, and code that strips it or inserts it inconsistently produces two spellings of a word that every reader considers identical.
And PostgreSQL ships no Persian stemmer. Not in the standard distribution, not in the widely used search extensions. This is not a gap we can configure our way around; it is a gap that has to be filled in the ingestion layer.
Any one of these silently removes a large share of a corpus from results. Together they mean a search feature can look fine in a demo, on whichever documents someone happened to test.
"Why does searching for a word that is visibly on the page return nothing?"
How we approach it
One normaliser, run twice. Letter folding, digit unification, diacritic and tatweel removal, joiner handling and Jalali date parsing live in a single implementation, and that implementation runs both when the index is built and when a query arrives. If those two paths ever diverge, the lexical arm is a lie — it will match documents nobody searched for and miss documents everybody expects.
Normalise into columns, not into memory. Normalised text is materialised and indexed beside the original. The original stays intact — it is what a user reads and quotes; the normalised copy is what the index sees.
Tune against real queries. A lexical arm that requires every term to match returns nothing for a real sentence. We test on text people type, not fixtures.
Right-to-left as the document default. Direction is set on the document so the first paint is already correct, layout uses logical properties rather than left and right, and numbers, phone fields and identifiers sit in explicit left-to-right islands. Retrofitting direction onto a left-to-right layout produces a long tail of defects that each look trivial and collectively make a product feel foreign.
Dates displayed Jalali, stored unambiguously. Users read the calendar they use. The wire format stays boring.
What you get
A normalisation specification and its implementation. Indexed, normalised columns. A search path running the same code as the index. A right-to-left interface built as one. Jalali display with sane storage. And a test suite that pins the actual characters, so a lookalike cannot slip back in unnoticed.
When this is the wrong choice
- If your text is already clean, single-script and consistently entered, most of this is unnecessary. Measure your own corpus first.
- If the problem is domain vocabulary rather than script, normalisation will not help. That needs a lexicon, a gold query set, and someone who knows the domain.
- If you only need the interface translated, a localisation library and a good translator will finish faster than an engineering engagement.
- If you are waiting for a Persian stemmer to appear in your database, it will not. Morphology gets handled before the text reaches the index, and any vendor implying otherwise has not looked.
- Lexical search alone has a low ceiling in Persian. If you cannot run a dense retrieval arm alongside it, expect that ceiling and plan for it rather than tuning against it forever.