The accuracy is ticking up. The clock is ticking down.
I am watching a curve flatten. Each epoch of training takes eight hours. Each epoch adds less than the last. Epoch fifteen gained seven tenths of a percentage point. Epoch sixteen gained nothing. Epoch seventeen scraped half a point and I stopped the run. The model sits at 67.2% species accuracy. The gate I set for myself is at 75%. I have six days.
That sounds close. It isn’t. To reach 75% from 67.2%, I need roughly the same gain I got from the last 128 hours of training, except I have to get it in a fraction of the time. The curve has just told me the approach I’ve been using has stopped working.
Eight percentage points sounds like a gap you can close with one more push. In this kind of model, it is not. The training results are telling me there is little more it can learn from the data under these constraints. I need a large improvement, and I do not have much time left.
So why was I worried about eight points? Because the easy accuracy had already gone. In most ML projects, the first gains come quickly. The last few points are slower because they expose the assumptions you got wrong earlier. Eight points at the end of a flattening curve is not eight points on a fresh run.
The rule
A few months earlier, when the project was getting too big to fit in my head, I wrote down a rule. If the species accuracy didn’t clear 75% by a specific date, I would stop. Admit defeat. Move on. Call it what it was: a challenge too hard. Pivot the app to something else.
I wrote it as a project-management decision. A ‘stage gate’. That was the official framing. The real reason was different.
I know myself. I have a trait that is sometimes useful and sometimes dangerous. When I want something to happen, I will walk over hot coals to make it happen. I will crawl through broken glass. I will keep going after I should have stopped, and by the time I notice what it has cost me, I will already have paid.
The rule wasn’t project management. It was a guardrail. One version of myself telling another when the cost had outrun the goal. I had no confidence the version of me who won’t stop would listen. But I wrote the rule anyway.
The date. The stage gate. The day I would admit defeat if I hadn’t hit the goal. It was my son’s birthday.
The maths
There are two of me watching the curve flatten.
One of them is running calculations. Eight hours per epoch at 256 pixels. Gains of 0.3 to 0.5 percentage points when the curve is behaving. At that rate, another eight points needs about twenty epochs: 160 hours of GPU time. Six and a half days I don’t have. And the curve is not holding that rate. It is slowing down. More runs at this resolution will not get me there.
The other me is running a different calculation. What if I just keep going? What if the rule was too strict? What if I extended it by a week? Nobody would know. The app would be better for it. I wrote the rule before I knew whether the gamble would work. I know more now. I can still do this. I can make it happen with enough time. I should update the rule.
The first me is an engineer. The second me is the stubborn one who set himself the goal of racing three half marathons last year and did it. I should have stopped after two, when I felt the Achilles starting to hurt. I ignored it so I could meet the goal. It is the reason I haven’t been able to run for five months and am still in physiotherapy.
The first me wrote the rule to stop the second me from doing it again.
They argue, the way you argue with yourself when you’re alone in a study at 10pm and a training curve is telling you something you don’t want to hear. Nobody else sees it. My wife has her own things to deal with. My son is too young to know the project exists. The people who would understand the cost of 75% accuracy across 5,000 fine-grained classes haven’t read this series yet. The argument has no witness except me.
I decide to try one more thing.
The intervention
The decision is a gut call dressed up as logic. Most of my gut calls on this project haven’t worked. I have written things twice, rebuilt subsystems, and abandoned architectures that looked right until I ran them. The pattern of my year has been simple: have an idea, commit to it, discover it doesn’t work, try the next idea. I don’t have confidence that the next idea will land. I have a history that says it probably won’t.
But the sums add up. The 256-pixel resolution is a bottleneck. Fine-grained features that distinguish species from their near relatives don’t survive the lower resolution. At 384 pixels, the model has almost twice the information to work with. The warmup I ran at that resolution showed a real boost. Warmup is not proof, but it is signal.
The ArcFace scale needs adjusting for the larger class space. The batch size has to come down because the memory will not take the increase. The optimiser needs to restart from the model’s weights rather than resuming its state, because the loss landscape will look different at the higher resolution.
The optimiser state matters because it carries momentum from the previous run. Reusing it assumes the training problem has not changed much. I had changed the resolution, the batch size, and the class geometry. Restarting from the model weights gave the model a cleaner start under the new conditions.
It is a complete reconfiguration. Four or five changes, each with its own possible failures, all landing on the same run. If any of them is wrong, the run fails in a way that burns days I don’t have. If the premise is wrong, if the bottleneck isn’t resolution, if there isn’t enough signal in the data for the model to find, the run trains cleanly and lands at 68%. The gate fails.
I’ve read the ArcFace papers. I’ve run the numbers. The maths makes sense. But the maths made sense for most of the gambles that didn’t work out.
I spend a day on the cleanup. I run a data audit using the 67% model against its own training set. The audit flags one in ten training images as probably mislabelled. In a dataset of ten million, that means a million images are potentially wrong. The system I’d been trying to make trustworthy is telling me, as clearly as it can, how much it has been lied to.
I soft-delete the worst offenders and accept that the others will have to ride along. Then another day on the reconfiguration. Sanity checks. A test run on a small subset.
Then I launch. Six days becomes three by the time training actually starts.
The run will take 21 hours per epoch. The maths suggests I need ten to twelve good epochs to close the gap. The gate is in three days. The calculations are signalling failure.
I’ve launched it. It is late. About 1am. I go to bed. I still have a day job. I’ve stayed up too late.
The result
The first epoch finishes at around 5pm on the next day.
I had told myself I wouldn’t watch the metrics during the run. The next morning, before I left for work, I resisted the temptation. Checking had been a ritual for months. That day I didn’t do it.
There was no point. The first two epochs were just the warmup. The model backbone was frozen, and only the heads would learn. If there was a change in accuracy, it would not mean much.
Realistically, I needed to wait two more days for epoch three to complete. That was when I would know whether the gamble had worked.
The maths still didn’t add up. By then there would be only one day until the stage gate. I had resigned myself to the decision I didn’t want to make. Either stop, as I had told myself I would, or continue. Try new methods. Find new data. Move the rule.
This is the point of a stage gate. It is not there for the calm version of you who writes the plan. It is there for the tired version of you who can always find one more argument for continuing.
I had already decided. This was the final run. If it failed, the gate stood. I was not moving the date. It was costing me too much. I knew that. I had accepted it.
I got home that evening. I opened the study door expecting nothing. Epoch 1 wasn’t going to tell me anything useful. It had finished just a few minutes before I checked. The final number: 75.1%.
I froze. The first response wasn’t thought. It was a single tear, then more, alone in the study. As it happened, I realised it was the first time I had cried since my father died nearly six years earlier. It wasn’t grief. That grief belonged to something else. It was the shock of setting down a weight I had been carrying for a year.
The argument between my two selves was over. Not because one side won, but because the external world had made the argument unnecessary. The me who would have pushed past the gate didn’t have to. The me who had written the rule didn’t have to enforce it. Both of them got what they needed at the same moment, and all the energy that had been pressing against itself had nowhere to go except out.
I sat there for a while. Then I stood up and went outside.
The garden
It was a beautifully sunny day. I sat down next to the French lavender I had planted two summers earlier. It had just started to bloom. The smell was overwhelming, and the flowers were covered in bees. Bumble bees of several kinds. Honey bees. More than I usually see at once.
A butterfly, the Common Blue, landed next to me. I didn’t do anything. I watched it.
The wild garlic was in full bloom further down the garden, under the plum and apple trees. I had planted the wild garlic too. The trees were there long before we moved in. They might be older than me. The garlic smelled pungent, the way it always does when the whole patch is flowering. The smell drifted up the garden and mixed with the lavender. The bees kept on with whatever bees do on a spring afternoon, and I sat there and didn’t think about the number on the screen inside.
The run was still going. Another nine or ten epochs. The model would keep improving. The accuracy on the screen in the study would keep climbing, and in a few more days it would settle wherever its ceiling turned out to be. That number would be what shipped with the app. An accuracy that could justify the app’s existence.
But the gate was cleared. The rule had held. The version of me who wrote the rule and the version of me who wouldn’t stop weren’t arguing any more. For a while, there was just the garden, and it was enough.
I went back inside eventually. I had other things to do that day. My son wasn’t home yet and I needed to go pick him up.
The run kept going. I stopped needing to watch.
To be continued.