Data platform engineering

The number is downstream of everything you did not check

A model, a dashboard and a report all have the same property: they will produce output regardless of whether the data underneath them is right. Nothing raises an exception when a table is missing a third of its rows. The pipeline runs, the chart renders, and the meeting proceeds.

This is why data-platform work is undersold. It looks like plumbing and it is actually the part that decides whether anything above it can be trusted.

Some of what we have found doing it, in one corpus, in seven passes:

103,423 content rows had never been indexed. The search index looked healthy. It was missing a third of the material and nothing in the system knew.
A ranking field was entirely flat. Every record carried the same authority value, so the most binding sources in the corpus ranked identically to the least.
Around seventy per cent of a citation graph was navigation. Previous-page and next-page links had been ingested as citations. The graph had structure, statistics, and no meaning.
Documents were quietly misclassified. Academic theses had been loaded as statutes. Another country’s constitution had been loaded as domestic law.

None of these announce themselves. All of them were found by going looking, with counts, against the live database.

"Has anyone actually counted this?"

How we approach it

Make collection durable and correction cheap. Raw collected data lands once and is kept. Every subsequent fix is a re-parse of what is already on disk, never a second crawl or a second export from a system that may have changed since.

Quarantine rather than delete. When we found the bad citation edges, we marked them instead of removing them. Retrieval ignores them, the diagnosis stays reversible, and the day someone challenges the finding the evidence is still there. A cleanup that destroys its own evidence cannot be reviewed.

Be willing to cancel the plan. One of those seven rounds was supposed to merge duplicate records and did not, because the duplication diagnosis turned out to be wrong — records that shared all 124 of their identifiers had matching content in only eight. Merging would have destroyed real history to make a count look tidier. A cleanup pass that deletes nothing is not a failed pass.

Make the audit trail a property of the transaction. The record of an action is written in the same transaction as the action. Not afterwards, not through a queue, not inside a try-except that logs a warning. It commits if and only if the action commits, and the database itself refuses updates and deletes to the log.

What you get

A schema and a readable migration history. Ingestion you can re-run. A written data-quality audit with severities and counts. Cleaners that are safe to run twice. An audit trail that cannot be quietly disabled. Backups proven by restoring them.

When this is the wrong choice

If the data fits in a spreadsheet, use the spreadsheet. Platform engineering has a fixed cost that small data never repays.
If the source system is the problem, fix the source. A downstream cleaner that repairs bad entry forever is a permanent tax, and it hides the defect from the people who could remove it.
If nobody will fund the remediation, do not commission the audit. It will find things. Findings that are documented and unfunded are worse for an organisation than findings nobody has written down.
If the errors come from people entering wrong values, engineering measures the problem and does not solve it. That is a process change with a training budget attached.
If you need dashboards and governance tooling that vendors already sell, buy them. We build the layer underneath, not a competitor to the layer above.