Building a clean Landsat corpus for mineral exploration
Caught an 80% duplicate rate in a supplied dataset before a single annotator hour was spent on it.
A terrain classifier is a dataset problem long before it is a modelling problem. The supplied imagery included a ready-made RGB directory that looked like the obvious starting point, a second sensor that looked equivalent, and thousands of frames that were mostly sensor no-data. Each of those would have cost annotator time and produced a corpus that quietly could not train anything.
The dataset is the project
Everyone wants to talk about the model. The model is downstream of nine decisions about data that were each made once, quietly, and are each expensive to undo.
Here is the one that mattered most.
The imagery arrived with a ready-made directory of RGB images, 6,095 files, already composited and apparently ready to upload. It was the obvious starting point and it would have cost nothing to use.
Hashing it first took minutes. It held 1,202 unique images. Every scene had been written five times under different sensor prefixes, byte for byte identical. Using it as delivered would have meant showing each annotator the same picture roughly four times, an agreement statistic that measured the annotators’ memory rather than their judgement, and a training set with a fourfold duplication bias baked in.
"How do you know your training set doesn't have the same image in it twice?"
We know because every one of the 9,732 images in the final corpus carries a recorded content hash and all of them are distinct. That is a property that can be re-verified, not a claim.
From 10,348 scenes to 9,732 images
The usable source was a set of single-band greyscale rasters, one file per spectral band per scene. The pipeline is short and each stage rejects something:
- Compose — merge the red, green and blue bands into a natural-colour composite; reject any scene whose bands disagree on raster size.
- Reject no-data — 491 scenes were more than half sensor no-data and were dropped on an alpha test.
- Reject degenerate frames — all-black, all-white and flat single-colour frames, caught on the original pixels by a variance test and a histogram-pinning test together.
- Stretch — a per-channel percentile contrast stretch with bounded gain, because raw composites are too dark to read terrain in, and bounded so a well-exposed frame is barely touched.
- Trim — 125 edge slivers from the tiling grid removed, some of them a few pixels tall.
- Encode and hash — progressive JPEG under a size budget, SHA-1 recorded per output.
Result: 9,732 images, 741 MiB, every one content-hash unique, with a per-scene provenance record carrying status, rejection reason, dimensions, output size and hash.
The second sensor, rejected in one sentence
A second archive covered the same 10,348 scenes and looked like a free doubling of the corpus.
It has no blue band. Its green and red channels are strongly correlated, so a natural-colour render comes out nearly monochrome, and a human being asked to distinguish a quarry from a mountain from a river in near-greyscale is being set up to disagree with themselves. Two further directories turned out to be copies of that same archive in which about half the frames were single-pixel placeholders.
Three of the four candidate sources were rejected on measured evidence. That is the normal ratio, and it is worth budgeting for.
The annotation protocol
Eight single-choice Persian classes, hotkeys 1 to 8, with the vocabulary hard-coded in the worker as well as in the labelling configuration and validated on submit — so a typo is rejected loudly instead of stored silently. Formal spelling was chosen over colloquial for the class names, once, in a single source of truth.
The agreement protocol is three rules:
- Every image collects two independent labels.
- Identical labels are the final label.
- Different labels route the image to a third annotator who did not see it, into their own arbitration project, and their choice is final.
The tool’s community edition has no per-project membership and no automatic agreement routing, so all three rules live in code. That turned out to be an advantage: the third annotator is selected by task id modulo the candidate list, which is deterministic — re-running reconciliation never re-routes an image somewhere new — and spreads the load evenly instead of concentrating it on one person.
The community edition also means every annotator can open every project. Nothing physically stops someone labelling in another person’s arbitration queue. So the reconciler counts an arbitration label only when the person who made it owns that queue, and the case was tested live: a label placed in the wrong queue was refused and the image stayed awaiting arbitration.
What this case study is not claiming
The corpus is complete. The protocol is designed, deployed and verified end to end against the live instance. Full annotation is pending, and the only measured inter-annotator agreement figure — 82.7%, or 124 of 150 — comes from a pilot round on 150 images. There is no model, no training run and no accuracy number. Where a headline figure would go, there is a pilot figure and a note explaining what it is.
Why the honest version is the better sales document
A buyer evaluating an imagery pipeline can check duplication, no-data handling, class definitions and agreement protocol in an afternoon. Those checks are cheap and they are the first thing a competent technical reviewer does.
A corpus that passes them is worth more than a corpus with a bigger number attached to it.
Decisions
Hash every candidate image before uploading any of it.
Alternatives: Use the supplied ready-made RGB directory as delivered
The supplied directory held 6,095 files and 1,202 unique images. Every scene had been written five times under different sensor prefixes, byte for byte. Using it would have shown each annotator the same picture about four times, inflated every count in the project, and produced agreement statistics that measured nothing. The hashing pass cost minutes and ran before a single annotator was invited.
Content hashes are recorded per scene in the build report, so uniqueness is a property that can be re-checked rather than asserted.
Composite from single-band rasters rather than adopting the second sensor's archive.
Alternatives: A second multispectral archive of the same 10,348 scenes, Two further derivative directories
The alternative sensor has no blue band, and its green and red channels are strongly correlated, so a natural-colour render comes out almost monochrome — unusable for a human classifying terrain by eye. The two derivative directories were duplicates of it in which roughly half the frames were single-pixel placeholders.
Run the degenerate-frame test on the original pixels, before the contrast stretch.
Alternatives: Stretch first, then filter the results
Order matters here in a way that is easy to get backwards. A percentile stretch applied to a no-data tile amplifies sensor noise into something that looks like content, and the filter then passes it. Testing first — on both a per-channel variance test and a histogram-pinning test — means an empty frame cannot be promoted into the corpus by the very step meant to make frames readable.
Select the third annotator deterministically, by task id modulo the candidate list.
Alternatives: Random selection, Always the lowest-numbered available annotator, The tool's built-in agreement routing
Random routing is not idempotent: re-running reconciliation would send the same disputed image to a different person and duplicate the work. Lowest-id selection dumps every dispute on one person. The tool's community edition has no automatic agreement routing at all, so the rule had to live in code either way — and once it lives in code, making it deterministic and load-spreading costs one line.
Keep the striped scenes instead of filtering them out.
Alternatives: Delete every image with significant no-data banding
The scan-line corrector on this sensor failed in 2003, so wedge-shaped no-data gaps are a permanent property of the archive, not a defect in our processing. The banding was measured per image and recorded as a per-sample quality value, so it can be used later as a training weight or a filter. Deleting the images throws away real data and a decision that cannot be revisited — and once annotation begins, deleting a task deletes its annotations with it.
Limitations
- Full annotation is pending. The corpus is built and the protocol is verified end to end, but the only measured agreement figure is 82.7% on a 150-image pilot. We would rather publish a pilot number labelled as a pilot than a full-corpus number we cannot stand behind.
- No model exists. There is no training code, no train/validation/test split, no model artifacts and no evaluation results. The deliverable is the corpus and the annotation protocol, and any claim about downstream accuracy would have to come from work that has not been done.
- This is three visible bands, not hyperspectral imagery. Natural-colour composites support terrain classification by eye. They do not support mineral identification, and the two should never be quoted as if they were the same capability.
- Scene identifiers in this pipeline are bare integers with no geolocation attached, so the corpus cannot be joined to ground truth or field data without deriving footprints separately.
- The annotation tool ran on its built-in single-file database, where every write takes an exclusive lock. With several concurrent annotators that is a real ceiling. Two fixes were written up and neither was applied — the retry counter was instrumented so the decision could be made on evidence, and it came back at zero, so the ceiling has been measured but not removed.
- Roughly one image in thirteen carries substantial no-data banding. Those images are real data and mostly still classifiable, but they are harder, and a model trained on this corpus inherits that.