I went on holiday on April 3rd.
I’d kicked off a new training run the day before. ArcFace, B2, balanced sampling. The architecture I had spent weeks researching and thinking about. The plan was that it would train across the ten days I was away. By the time I got home, the model would either be at or near the accuracy I needed, or it would be far enough along that I could tell whether the architectural reframe had worked. Either way, I’d have nine days of GPU time on it that I wouldn’t have to be at the desk for.
I went to Bratislava, Budapest, and Vienna with my family. The holiday was great. It was busy and full and I didn’t think about the project much. When I did, I was quietly confident. The reframe of the project had been months of thinking landing into a specific decision, and the decision had felt right. The data was as good as I could get it. The architecture was the right one. The run would do what it was meant to do. I came home on the evening of April 12th expecting to find a model that was almost finished, just about plateauing somewhere near where I needed it to be.
The first thing I did was open the dashboard.
Zero
The training run had exited early. Seven days in, having plateaued. The training loss had decreased smoothly across the run, the optimiser had converged, the curves looked healthy, the system had done what training systems do. The validation accuracy was zero.
Zero…
I stared at the number for a long time.
The training metrics showed a model that had been learning. The loss had come down. The gradient norms were reasonable. The throughput was as expected. From every internal indicator, the run had been doing what it was supposed to do for a full week. And the validation set, the only thing that actually measured whether the model had learned anything useful, said it had learned nothing.
This was the third time this project had taught me the same lesson. Once when the data had looked fixed and the model had known better. Once when the pipeline had run end to end and the critical path hadn’t been executing. Now when the training had looked clean for a full week and the model had learned nothing.
A system that runs is not a system that works. A system that produces metrics is not a system you can trust. A loss curve that decreases is not a model that has learned. I had known these things. I had written them down. I had not kept them in front of me when I rushed to launch the run before flying out.
The diagnosis
The next morning I started looking for what had happened.
I’d expected to find a single dramatic failure. A wrong file path, a corrupted dataset, a configuration override that had silently inverted something. Instead I found something stranger.
Every output the model produced was limited to half the space it was supposed to be using, the non-negative half of a 256-dimensional embedding space. Every component of every vector was zero or positive. None of the negative half was being used.
I knew what this meant before I finished reading the histogram.
The shared layers of the network had a standard filtering block that zeroed out negative values, a ReLU plus Dropout that had been there for the previous architecture. The block did nothing harmful with the old loss function. It hadn’t occurred to me to check whether it would still work with the new one.
It didn’t. ArcFace’s whole mechanism is angular. It works by spreading classes apart on a full hypersphere, and it needs the full sphere, including the negative components, to discriminate classes by angle. With every output clipped to the non-negative half, the classes were being asked to separate inside one corner of a space they should have had all of. The geometry of what the loss function was trying to do, and the geometry of what the model was producing, didn’t agree.
The fix was a single block change. Replace the ReLU + Dropout with one that centred the distribution without clipping it. Two lines of code.
That was the first bug. I found it within an hour of starting the diagnosis. The face-palm was immediate. I knew this. The ArcFace paper discusses it. ArcFace integration guides discuss it. Anyone with even a little experience with angular-margin losses would have caught it the moment they looked at my model architecture. I had not caught it because I had not looked. I did not look because I was rushing to get the run set up before my holiday. But I also did not think to look, because I didn’t have any real experience with angular-margin losses. Reading articles and papers for weeks does not translate to real-world experience. I had just learned that the hard way.
The other ten
Once I’d found the first one, I started looking properly. Over the next three days I found ten more.
The second headline bug was gradient crushing. The ArcFace weight matrix has one row per species, five thousand rows by 256 dimensions, and accumulates gradients about twenty times larger than any other parameter group in the network. I’d been applying a single global clip_grad_norm across the whole model. Clip-norm rescales proportionally. The ArcFace gradients were being held to a sensible magnitude, but the backbone and the auxiliary task heads were being crushed to roughly five percent of their intended size. The model was barely able to update anything except the ArcFace head itself, and the ArcFace head couldn’t do its job without the backbone learning useful features for it to organise.
The fix was per-parameter-group gradient clipping. The ArcFace head got a loose clip that allowed its naturally-large gradients through. The backbone and the auxiliary heads got clips at their natural scale. Half a day of work to refactor the optimiser configuration.
The other nine were smaller in individual significance and larger in total drag. Weight decay being applied to BatchNorm gamma and beta parameters, which causes training instability that’s well-known and easily fixable but I’d never explicitly disabled it. The training loop logging only the backbone learning rate, which had hidden the fact that the head learning rate was misconfigured. Checkpoint resume not saving the species-to-index mapping, which meant a resumed run could silently apply old species weights to a new species ordering (a correctness-breaking failure mode that would not have surfaced as an error, just as quietly wrong predictions). The best-model save criterion still using validation loss as its trigger, when ArcFace’s validation loss does not correlate monotonically with validation accuracy, meaning the system would save the wrong checkpoint as “best” even when later epochs were doing better.
None of these were difficult problems individually. Each had a known fix and a known cause. What made them collectively serious was that I had introduced all of them at the same time, in the same commit, when I’d switched architectures before my holiday.
This was the cost of solo engineering shown plainly. The maths during the reframe had been right. The architectural choice had been right. The implementation, in isolation, had been right. What I had not done, and what I had no way to do, working alone, was have a competent ML engineer review the code. A pull request review by anyone with prior ArcFace experience would have caught the ReLU bug in seconds. The gradient clipping would have come up the moment a reviewer saw a single global clip_grad_norm applied to a model with ArcFace weights in it. Most of the other nine would have been caught in the same review.
I didn’t have a reviewer. I had only the run, which had failed silently for nine days.
The DataLoader crashes
After the eleven bugs were fixed I launched a test run to verify the architecture was working.
The training loop crashed.
This was a different class of problem. The system that loads images into memory was failing at a level deeper than Python. Somewhere inside the image-decoding libraries it depends on. The cause was corrupt or truncated images in the dataset. Some had been corrupt for months. The previous architecture’s data loading path had been more forgiving and had quietly tolerated them. The new pipeline ran faster and harder, and the bad images broke it.
The fix was layered. Catch what could be caught at the image-decoding level. Validate file headers before passing anything to the decoder, so files that weren’t actually images got rejected early. Force a full read inside a controlled context so any remaining corruption surfaced as a normal error that the pipeline could handle and skip.
The second fix was a smaller engineering improvement that mattered for what was about to come. Until then, if a worker crashed in the middle of an epoch, the entire epoch would restart from the beginning. Eight hours of GPU time lost per crash on a long run. The fix made the system remember where it had been, rebuild the worker, and resume from the next batch. No epochs lost.
This work took another day. By then it was mid-April. The new architecture was working. The data pipeline was robust. The eleven training bugs were fixed. The crashes from corrupt images were handled.
I’d lost more days and the entire holiday’s worth of compute. The stage gate was uncomfortably closer than it had been before I’d left.
The new run
I launched the 256-pixel run the same day.
The maths I did before launching it: the run would take about eight hours per epoch. To reach 75% on five thousand species, I likely needed somewhere between ten and twenty epochs depending on how much of the architectural reframe had landed. That meant five to seven days of training. If the run hit a number I could work with, I might have time for one more run after it, a higher-resolution variant that could push the accuracy further if needed. The stage gate was three weeks away.
It was tight. It hadn’t been tight before the holiday. It was now.
I didn’t know whether the architecture would do what I thought it would. I didn’t know whether the eleven bugs were really all of them, or just the ones I’d found. I didn’t know whether the run would exit early like the last one had, or train cleanly for ten days, or do something else I hadn’t anticipated.
I started the run. I left it running overnight.
The next morning the validation accuracy on the first proper epoch came back at 60%.
The reframe had worked. The bugs had been the bugs. The new architecture was doing what I’d designed it to do.
The accuracy was ticking up. The clock was ticking down. I didn’t know which one was about to run out first.
To be continued.