The third image wouldn’t open either.
The training run had already died once. I reran it, thinking it was a transient issue, maybe CUDA, maybe memory. Then I stepped through it batch by batch to isolate the failure.
The first image opened.
The second opened.
The third was corrupt.
I pulled the file directly. It existed, but it would not decode. I tried another. Same thing. Another loaded, then threw an error halfway through. Another was zero bytes.
The failure was not isolated to one bad file. The dataset itself was unreliable.
Two days earlier, everything had looked like it was working.
I had a full pipeline running end to end: collection, storage, training. I could kick off a run and watch it progress. The loss went down, the metrics updated, and the batches kept moving.
At one point, I had a run complete cleanly. It produced predictions without crashing. It was not good, but it looked like a working training pipeline.
I treated the next step as a scale problem. Add more data, train again, improve the model.
This is a common engineering trap. When a system fails loudly, you investigate. When it runs badly, you are tempted to optimise it. I had skipped the boring question: was the thing I was optimising the system I thought I had built?
The crashes were just the most visible problem.
Once I started looking at the dataset itself, other issues surfaced quickly.
The first issue was corrupted files.
I found twenty-five corrupt files scattered across the dataset. Some existed in the database but could not be opened. Some partially decoded and then failed. Three more were referenced in the database but missing on disk entirely, leaving orphaned rows that pointed at files deleted somewhere along the line.
Most of the corruption came from collection-time write errors, where a download had been interrupted or a file had been written incompletely. Training runs had been hitting these files for days and crashing. None of the failure messages had said the word corrupt. They had all said CUDA.
The training run could not recover from those inputs. It stopped.
The second issue was duplication.
I started inspecting the dataset manually, folder by folder. It wasn’t a sophisticated diagnostic. I was opening directories and scrolling through thumbnails.
One species looked well represented, with dozens of images. As I scrolled, the same image kept appearing. Sometimes it was identical, sometimes resized, sometimes slightly compressed, but it was clearly the same original.
I checked more species. Same pattern.
That afternoon I wrote a perceptual hash check. It compares images and flags those that are visually identical even when they’ve been resized, recompressed, or saved in different formats.
The problem was systemic. Around 24% of the dataset was repeated content.
The cause was straightforward once I traced it. Multi-phase collection had been pulling the same images from different sources, sometimes pulling the same image twice from the same source in different runs, and nothing in the pipeline had been checking. The dataset hadn’t been growing. It had been duplicating.
The apparent scale was mostly copies.
The useful lesson is annoyingly plain: count the thing you mean to count. I had been counting files. The model needed independent examples. Those are not the same measurement, and the gap between them was big enough to fool the whole pipeline.
The third issue was class imbalance.
I ran a class distribution check.
At first glance, it looked uneven but manageable. Some species had more images than others, which is expected in scraped data.
The class ratio came out at 68:1 across 33 species. The most represented species had 68 times more images than the least.
I reran the numbers. Same result.
I picked one of the least represented species and checked its predictions. The model did not recognise it. It defaulted to one of the dominant classes, the ones it had seen most often.
It wasn’t learning the space of plants. It was learning the frequency of my dataset.
Why would it do anything else? From the model’s point of view, the common species were the safest bet. If the data says one answer appears sixty-eight times more often than another, the model has every incentive to learn that shortcut. It is not being lazy. It is doing exactly what the evidence rewards.
The dataset did not yet behave like training data. At this point, the system was not even getting to the stage where the model mattered.
I spent the rest of the week on a four-stage cleanup. I removed bad files, deduplicated the rest with the perceptual hash check, generated synthetic images to balance the under-represented classes, and re-scored everything for quality.
By the end of the week, the dataset was clean. The imbalance was down to 4:1. The training pipeline ran without crashing.
A larger model would not have fixed any of this. It would have made the problem harder to see. A more powerful model would have memorised the 438 duplicates more confidently, learned the 68:1 frequency bias more sharply, and produced predictions that looked more authoritative without being more correct. The failure was in the data, and in the system around it.
Before the model could improve, the dataset and the pipeline had to improve.
The cleanup made the data look fixed. The next training run showed otherwise.
To be continued.