I went on holiday on April 3rd.
I’d kicked off a new training run the day before, using ArcFace, EfficientNet-B2, and balanced sampling. It was the architecture I had spent weeks researching and thinking about. The plan was that it would train across the ten days I was away. By the time I got home, the model would either be at or near the accuracy I needed, or far enough along that I could tell whether the architectural reframe had worked. Either way, I’d have nine days of GPU time on it that I wouldn’t have to be at the desk for.
I went to Bratislava, Budapest, and Vienna with my family. The holiday was great. It was busy and full, and I didn’t think about the project much. When I did, I was quietly confident. The reframe had taken months of thinking and had landed in a specific decision. The data was as good as I could get it. The architecture felt right. I came home on the evening of 12 April expecting to find a model that was almost finished, just about plateauing somewhere near where I needed it to be.
The first thing I did was open the dashboard.
Zero
The training run had exited early, seven days in, after plateauing. The training loss had decreased smoothly across the run. The optimiser had converged. The curves looked healthy. The system had done what training systems do.
The validation accuracy was zero.
I stared at that number for a long time.
The training metrics showed a model that had been learning. The loss had come down. The gradient norms were reasonable. The throughput was as expected. From every internal indicator, the run had been doing what it was supposed to do for a full week. The validation set, the only thing that actually measured whether the model had learned anything useful, said it had learned nothing.
The project had reached the same lesson from a new direction. The data had looked fixed and hadn’t been. The pipeline had run end to end while the critical path wasn’t executing. Now the training run had looked clean for a full week, and the model had learned nothing useful.
A system that runs is not a system that works. A system that produces metrics is not a system you can trust. A loss curve that decreases is not a model that has learned. I knew these things. I had written them down. I had not kept them in front of me when I rushed to launch the run before flying out.
This is where dashboards can mislead you. They answer the question you instrumented, not necessarily the question you care about. My dashboard could tell me whether training was busy. It could not, by itself, tell me whether the model was learning the representation the app needed.
The diagnosis
The next morning I started looking for what had happened.
I’d expected to find a single dramatic failure. A wrong file path, a corrupted dataset, a configuration override that had silently inverted something. Instead I found something stranger.
Every output the model produced was limited to half the space it was supposed to be using. It was confined to the non-negative half of a 256-dimensional embedding space. Every component of every vector was zero or positive. None of the negative half was being used.
I knew what this meant before I finished reading the histogram.
Or rather, I thought I did. Experience has taught me that this is usually the point where I am about to discover a second problem hiding behind the first one. This time the first problem was embarrassing enough on its own.
The shared layers of the network had a standard filtering block that zeroed out negative values, a ReLU plus Dropout that had been there for the previous architecture. The block did nothing harmful with the old loss function. It hadn’t occurred to me to check whether it would still work with the new one.
It didn’t. ArcFace’s whole mechanism is angular. It works by spreading classes apart on a full hypersphere, and it needs the full sphere, including the negative components, to discriminate classes by angle.
Why does that matter? Because ArcFace is not trying to learn “positive features good, negative features bad”. It is trying to learn directions. If every embedding is forced into the non-negative half of the space, the model has fewer directions available. That makes the angular margin much less useful.
With every output clipped to the non-negative half, the classes were being asked to separate inside one corner of a space they should have had all of. The geometry of what the loss function was trying to do, and the geometry of what the model was producing, didn’t agree.
The fix was a single block change. Replace the ReLU + Dropout with one that centred the distribution without clipping it. Two lines of code.
That was the first bug. I found it within an hour of starting the diagnosis. The face-palm was immediate. The ArcFace paper discusses this. ArcFace integration guides discuss it. Someone with prior angular-margin experience would probably have caught it as soon as they looked at my model architecture.
I had not caught it because I had not looked closely enough. I was rushing to get the run set up before my holiday, and reading articles and papers for weeks had not given me real integration experience. I learned that the hard way.
The other ten
Once I’d found the first one, I started looking properly. Over the next three days I found ten more.
The second headline bug was gradient crushing. The ArcFace weight matrix has one row per species, five thousand rows by 256 dimensions, and accumulates gradients about twenty times larger than any other parameter group in the network.
I’d been applying a single global clip_grad_norm across the whole model. Clip-norm rescales proportionally. The ArcFace gradients were being held to a sensible magnitude, but the backbone and the auxiliary task heads were being crushed to roughly five percent of their intended size. The model was barely able to update anything except the ArcFace head itself, and the ArcFace head couldn’t do its job without the backbone learning useful features for it to organise.
The fix was per-parameter-group gradient clipping. The ArcFace head got a loose clip that allowed its naturally-large gradients through. The backbone and the auxiliary heads got clips at their natural scale. Half a day of work to refactor the optimiser configuration.
The other nine were smaller in individual significance and larger in total drag. Weight decay was being applied to BatchNorm gamma and beta parameters, which can cause a well-known training instability. I had never explicitly disabled it.
The training loop was logging only the backbone learning rate, which hid the fact that the head learning rate was misconfigured. Checkpoint resume was not saving the species-to-index mapping, which meant a resumed run could silently apply old species weights to a new species ordering. That would not have surfaced as an error, just as quietly wrong predictions.
The best-model save criterion was still using validation loss as its trigger. ArcFace’s validation loss does not correlate monotonically with validation accuracy, so the system could save the wrong checkpoint as “best” even when later epochs were doing better.
None of these were difficult problems individually. Each had a known fix and a known cause. What made them collectively serious was that I had introduced all of them at the same time, in the same commit, when I’d switched architectures before my holiday.
This was the cost of solo engineering shown plainly. The maths during the reframe had been right. The architectural choice had been right. The implementation, in isolation, had been reasonable. What I had not done, and what I had no way to do while working alone, was have a competent ML engineer review the code.
There is a useful distinction here. A wrong idea fails even when implemented well. A good idea can still fail if the integration is wrong. I had spent weeks checking the idea and far less time checking how the new architecture connected to the old training code.
A pull request review by someone with prior ArcFace experience would likely have caught the ReLU bug quickly. The gradient clipping would have come up when a reviewer saw a single global clip_grad_norm applied to a model with ArcFace weights in it. Most of the other nine bugs belonged in the same review.
I didn’t have a reviewer. I had only the run, which had failed silently for nine days.
The DataLoader crashes
After the eleven bugs were fixed I launched a test run to verify the architecture was working.
The training loop crashed.
This was a different class of problem. The system that loads images into memory was failing at a level deeper than Python, somewhere inside the image-decoding libraries it depends on. The cause was corrupt or truncated images in the dataset. Some had been corrupt for months. The previous architecture’s data loading path had been more forgiving and had quietly tolerated them. The new pipeline ran faster and harder, and the bad images broke it.
The fix was layered. I caught what could be caught at the image-decoding level, validated file headers before passing anything to the decoder, and forced a full read inside a controlled context so any remaining corruption surfaced as a normal error that the pipeline could handle and skip.
The second fix was a smaller engineering improvement that mattered for what was about to come. Until then, if a worker crashed in the middle of an epoch, the entire epoch would restart from the beginning. Eight hours of GPU time lost per crash on a long run. The fix made the system remember where it had been, rebuild the worker, and resume from the next batch. No epochs lost.
This work took another day. By then it was mid-April. The new architecture was working. The data pipeline was robust. The eleven training bugs were fixed. The crashes from corrupt images were handled.
I’d lost more days and the entire holiday’s worth of compute. The stage gate was uncomfortably closer than it had been before I’d left.
The new run
I launched the 256-pixel run the same day.
The run would take about eight hours per epoch. To reach 75% on five thousand species, I likely needed somewhere between ten and twenty epochs, depending on how much of the architectural reframe had landed. That meant five to seven days of training.
If the run hit a number I could work with, I might have time for one more run after it, a higher-resolution variant that could push the accuracy further if needed. The stage gate was three weeks away.
It was tight. It hadn’t been tight before the holiday. It was now.
I didn’t know whether the architecture would do what I thought it would. I didn’t know whether the eleven bugs were really all of them, or just the ones I’d found. I didn’t know whether the run would exit early like the last one had, or train cleanly for ten days, or do something else I hadn’t anticipated.
I started the run. I left it running overnight.
The next morning the validation accuracy on the first proper epoch came back at 60%.
The reframe had worked. The bugs had been the bugs. The new architecture was doing what I’d designed it to do.
The accuracy was ticking up. The clock was ticking down. I didn’t know which one was about to run out first.
To be continued.