The class distribution chart finally looked right.
It was not perfect. Some variance between species remained, but the dominant classes were no longer towering over the rare ones. The ratio was sitting at about 4:1, down from 68:1. I could look at the bar chart without the sinking feeling in my stomach.
Loss was going down, accuracy was going up, and the pipeline didn’t crash.
I sat back and thought, that’s it, sorted. I moved on to the next problem.
The next run showed it was not sorted.
How I cleaned the dataset
The plan felt straightforward. I had found measurable problems in the dataset, so I treated the work as a cleanup job.
I removed the corrupt files, partial decodes, and missing references. That cleared 28 files.
Then I tackled duplication. Perceptual hashing across the full dataset found 438 duplicates, about 24% of everything I’d collected. What had looked like scale was the same images reused at different resolutions and compression levels. I kept the highest-quality copy of each image and deleted the rest.
Then I tackled imbalance. Some species had dozens of images. Others had a handful. The model wasn’t learning plants. It was learning frequency. I augmented the minority classes with flips, rotations, crops, and distortions, generating 162 synthetic images across 17 under-represented species. That pulled the ratio from 68:1 down to roughly 4:1.
I re-scored everything and ran training again. The chart looked right. The training runs held. I marked it resolved and moved on. It felt like progress.
Where augmentation failed
In the augmented folders, I kept seeing the same leaf seven times, each rotation a few more degrees off the last. Same texture, slightly distorted. Same background, slightly cropped. The variation was real in the strict technical sense because the pixel values were different. Anyone looking at the folder would still have called it one photograph repeated with minor variations.
The rare classes still behaved badly. I pulled out one of the species I’d boosted from a handful of images to forty through augmentation, then ran it through inference on a clean test image. The model predicted it correctly maybe one time in ten. The other nine times, it defaulted to one of the common species, the ones it had seen most often.
I checked another rare species, then another. The same thing happened.
So why didn’t augmentation fix it? Because augmentation is not new evidence. I had made more versions of the same photographs. I had not shown the model new plants, new angles, new lighting, or new growth stages.
The dataset had more files, but not more independent examples.
It had got bigger without getting richer.
Clean data is not enough
I didn’t know what to do about that at first.
I spent a couple of days moving between the dashboard and the augmented folders, watching the metrics go up while the predictions stayed wrong. The gap between the two stopped looking like a problem to fix and started looking like a problem I’d misunderstood.
I had cleaned the data, but I had not added much information.
Cleaning removed noise. Deduplication removed redundancy. Augmentation created variation, but not the kind of variation the model would see from real photographs.
A useful rare-species dataset does not mean seven rotated copies of one photograph. It means that species photographed at the seedling stage, the vegetative stage, and the flowering stage. It means photographs in full sun and in shade, in the north of its range and in the south, on a phone camera in 2019 and a different phone camera in 2024.
It also means photographs from above, from the side, and from below. The model needs to see the space the species actually occupies, not one point in that space rendered at slightly different angles.
I had made the dataset larger. I hadn’t made it more informative.
I’d been working from a simple idea: if the data is messy, clean it. The real problem was that the data wasn’t strong enough to begin with. There was not enough real variation, not enough coverage of how plants appear in the wild, and not enough signal for the model to learn from.
What I’d treated as a data quality problem was a data sufficiency problem.
Those are not the same thing. The fixes for one don’t fix the other.
This distinction matters outside ML as well. Cleaning bad records makes a system less noisy. It does not fill in missing experience. If the system has never seen seedlings, winter growth, poor lighting, or rare species photographed badly, a cleaner dataset will still leave it guessing in exactly those places.
Running, not working
By this point, the system ran.
It didn’t crash. It produced outputs. The metrics said it was improving. The chart on the dashboard sloped the right way.
I still didn’t trust it, but I couldn’t yet explain why.
The problem was still in the data, but not in the way I thought.
What I hadn’t yet realised was that some of the parts I was measuring weren’t doing what I thought they were doing. The pipeline I’d been measuring against was telling me something different from the pipeline that was actually running.
I wouldn’t find that out for another few months.
To be continued.