Building a 1.8 GB Persian legal corpus
A legal corpus and platform built to the point where retrieval becomes an engineering problem instead of a guess.
A legal research assistant is only as good as the body of law underneath it. Everyone builds the assistant first. We built the corpus first, measured what was wrong with it seven times, and did not switch retrieval on until the evaluation set to judge it existed.
The part everyone skips
The demo is easy. A chat box, a language model, a few statutes pasted into a prompt, and a lawyer nodding politely. The demo does not survive the first question that matters, because the question that matters is not what does the law say — it is which law, in force when, and does anything above it say otherwise.
That question is answered by the corpus, not by the model. So the corpus was the project.
"Where does this answer come from, and was that provision still in force on the day my client acted?"
What was built
A single-jurisdiction Persian legal corpus of 1,826 MB, across 39 base tables, 4 views and 146 indexes:
| Content | Rows |
|---|---|
| Laws, including the constitution | 1,020 |
| Articles, including 177 constitutional articles | 32,103 |
| تبصره — subordinate provisions modelled separately | 14,471 |
| Court decisions | 35,185 |
| Advisory opinions | 13,462 |
| Judicial panel records | 6,406 |
| Scholarly works | 21,870 |
| Commentary passages | 82,540 |
| Topics, and links from documents to them | 23,567 · 361,167 |
| Judges | 1,401 |
| Article-to-article edges · article-to-decision edges | 34,773 · 39,712 |
| Search units in the retrieval pool | 151,216 |
| Search units with an embedding | 0 |
Around it: a production platform with authentication, subscription packages, two payment gateways behind one interface, an administrative surface, and an audit log — 33 route decorators across 11 routers, 14 application tables, and roughly 3,043 automated tests. Twenty commits over six days, sixteen planned tasks, sixteen green.
The crawl
183,725 pages in about eighty minutes, from eight of the cheapest droplets available, with zero bans, for one to two dollars.
The interesting part is not the speed. It is the two invariants the fleet was built around:
- No droplet is ever left running. Teardown on exit, on exception, on interrupt, on termination, plus a standalone reconciler that finds and destroys anything the controller missed.
- A worker never evades a block. Rate limited per address, jittered, and on a sustained refusal it reports the ban and exits rather than rotating to a new address.
Nothing about that is clever. It is just honest, and honest is what makes it repeatable.
The raw landing zone — the compressed pages exactly as fetched — is treated as the durable asset. Six of the seven cleanup rounds that followed were re-parses of that archive. None of them cost a second crawl.
Seven rounds, 56 findings, and the one where we were wrong
The verdict on the first load, written down at the time, was that it was a strong first-pass draft and not yet product-grade. Seven audit-then-fix rounds produced 56 findings, 18 of them critical. A sample of what a legal corpus actually gets wrong:
- The constitution was missing entirely. The parser recognised «ماده N» and not «اصل N», so every constitutional article had been silently dropped. Recovered from the raw archive with no gaps.
- Authority ranking was completely flat. Every decision loaded at the same tier with nothing marked binding, so binding precedent ranked exactly like a first-instance judgment.
- The citation graph was mostly navigation. 24,517 of 34,773 edges were the same-law N to N±1 pattern produced by previous-article and next-article links on the source pages.
- Documents in the wrong class. Theses ingested as statutes with a fabricated article number. A foreign constitution loaded as domestic law. Placeholder pages loaded as decisions.
- Layered markup residue in text columns, including a zero-width non-joiner encoding that breaks Persian rendering, cleaned by four reusable idempotent SQL functions rather than by a one-off script.
Round four: the diagnosis was wrong
Round four had recorded law "fragmentation" as a defect — the same law apparently split across several records — and scheduled a merge to fix it. The check before executing it showed the opposite. Two records sharing all of their article numbers shared almost none of their article text. They were successive versions of one law, correctly stored as two. The eleven laws flagged as "empty" were real laws whose content was genuinely missing, which is a different problem with a different fix.
The merge was cancelled. It would have destroyed real legal text in order to fix something that was not broken. That reversal is in the log next to the finding that caused it, because a cleanup log that only records successful fixes is a marketing document.
Retrieval, as a design rather than a claim
The retrieval design ships as a PostgreSQL function and a schema, not as a running feature. Two arms — a dense arm over half-precision 1024-dimension vectors with an HNSW index, and a lexical arm over a simple-configuration text-search vector — fused with reciprocal rank fusion at k=60, over-fetching 150 candidates per arm, with in-force and quotability filters baked in and the authority tier breaking ties.
Three things about it are decisions rather than defaults:
- The lexical query is OR-composed, because AND semantics returned zero rows for real case descriptions. Fewer, better-ranked results beat none.
- PostgreSQL has no Persian stemmer. Not in the stock configurations, not in the popular extensions. So normalisation is done in the ingest layer and materialised into dedicated columns, and the same normaliser has to run at query time or the lexical arm is a lie.
- Embeddings are self-hosted, on a permissively licensed model at 1024 dimensions — the same model and dimension chosen independently for our analytics platform. Hosted embedding APIs were not evaluated on quality. They were unavailable, which is a shorter conversation.
What it means for a buyer
If you are buying legal AI, the questions worth asking are about the corpus and not the model. How many documents, from where, under what licence. What proportion of the citation graph is real. Whether the authority hierarchy is encoded or assumed. Whether anyone has measured retrieval, and against what.
We can answer all of those for this corpus, including the answers that are unflattering. That is the whole point of publishing it at this stage rather than after the chat screen ships.
Decisions
Quarantine the bad citation edges with a flag instead of deleting them.
Alternatives: DELETE the 24,517 suspect rows, Leave the graph as ingested
Roughly seven in ten edges in the citation graph were artifacts of previous-article and next-article navigation links, sitting in the same table as about ten thousand genuine ones. A DELETE makes the diagnosis unfalsifiable — if we were wrong about the pattern, the evidence is gone. Marking them lets retrieval filter them out today and lets the call be re-argued tomorrow.
Retrieval and graph expansion consume only the edges established by extraction, never the flagged ones.
Cancel the planned merge of laws that looked fragmented, after the diagnosis turned out to be wrong.
Alternatives: Merge the apparently duplicated law records as planned
The fourth cleanup round had recorded law fragmentation as a defect and scheduled a merge. Checking it properly showed the opposite: two records sharing every article number shared almost none of their article text. They were legitimate successive versions of the same law, not fragments of one. The merge would have destroyed real legal text to fix a problem that did not exist, so it was cancelled and the reversal written into the log.
The same round also refused a proposed deletion filter for duplicate-looking rulings after most of the sampled rows turned out to be real settled decisions.
Rank on authority, not only on similarity.
Alternatives: Pure vector similarity, Similarity plus recency
A trial-court judgment that happens to be a perfect textual match must never outrank binding precedent on the same question, and vector similarity has no way to know the difference. The source hierarchy is encoded as an eleven-level authority tier that breaks ties inside a relevance band, and it was found completely flat on first load — every decision at the same tier, nothing marked binding — which is exactly the kind of defect a similarity score hides.
Write the audit row in the same transaction as the action it records.
Alternatives: A post-commit hook, An outbox, Best-effort logging with a warning on failure
Each alternative reintroduces the failure it exists to prevent: an action that happened with no record of it. The audit write only joins the session; it commits if and only if the action commits. A database trigger makes UPDATE and DELETE on the log raise, and a coverage test walks the live route table and fails when a new mutating endpoint appears without declaring what it audits. There is no off-switch in any environment.
Do not generate the 151,216 embeddings yet.
Alternatives: Embed the corpus and ship the chat screen
Embedding the corpus is a few hours of compute. Knowing whether the result is any good is the hard part, and the evaluation set that would answer it does not exist yet. Switching retrieval on before then produces a system that cannot be told apart from a broken one. The chat screen is parked in the backlog on purpose.
The published Persian embedding benchmarks contain no legal dataset, so leaderboard scores do not predict this domain and a project-specific evaluation set is not optional.
Limitations
- Retrieval is not switched on. The retrieval pool holds 151,216 search units and zero embeddings. This is a corpus and a platform, not a working legal assistant, and describing it as the latter would be false.
- No latency, accuracy, nDCG or recall figures exist for this system, anywhere. The gold evaluation set is a scheduled deliverable, not a result. Any number you are quoted about how well it retrieves has been invented.
- The corpus was assembled from public legal reference sources and its licensing position is not settled. Primary text is scheduled to migrate to official sources, and the corpus itself is not distributed.
- Some source pages carry the source's own machine-generated summaries. They are captured, flagged, and excluded from the citable set by default — but that is a filter applied to a label the source supplied, not a guarantee we generated ourselves.
- Most edges between articles and decisions reflect page-level links rather than a holding that actually cites the article. They are usable as ranking features and are never auto-included as results.
- Point-in-time law is only as complete as the version history behind it. Where the history is known to be incomplete the function returns nothing rather than passing today's text off as the text in force then, which is correct and also means it often returns nothing.