The accuracy is ticking up and the clock is ticking down.
I am watching a curve flatten. Each epoch of training takes eight hours. Each epoch adds less than the last. Epoch fifteen gained seven tenths of a percentage point. Epoch sixteen gained nothing. Epoch seventeen scraped half a point and I stopped the run. The model sits at 67.2% species accuracy. The gate I set for myself is at 75%. I have six days.
That’s not close. That’s a chasm. To reach 75% from 67.2%, I need roughly the same gain I got from the last 128 hours of training, except I have to get it in a fraction of the time, and the curve has just told me the approach I’ve been using has stopped working.
It feels close, only 8 percentage points. Anyone who has tried to train one of these models knows that isn’t close. The training results are telling me there is little more it can learn from the data under these constraints. I have a mountain to climb but I’m out of rope, and time.
The rule
A few months ago, when the project was getting too big to fit in my head, I wrote down a rule. If the species accuracy didn’t clear 75% by a specific date, I would stop. Admit defeat, move on. Call it what it was, a challenge too hard. Pivot the app to something else.
I wrote it as a project-management decision. A ‘stage gate’. That was the official framing. The real reason was different.
I know myself. I have a trait that is sometimes useful and sometimes dangerous. When I want something to happen, I will walk over hot coals to make it happen. I will crawl through broken glass. I will keep going after I should have stopped, I just refuse to give up, and by the time I notice what it has cost me, I will have already paid. I have paid that price before on this project already.
The rule wasn’t project management. It was a guardrail. One version of myself telling another when the cost had outrun the goal. I had no confidence the version of me who won’t stop would listen. But I wrote the rule anyway.
The date. The stage gate. The day I would admit defeat if I hadn’t hit the goal. It was my son’s birthday.
The maths
There are two of me watching the curve flatten.
One of them is running calculations. Eight hours per epoch at 256 pixels. Gains of 0.3 to 0.5 percentage points when the curve is behaving. At that rate it’s twenty epochs to get another eight points, which is 160 hours of GPU time, which is six and a half days I don’t have. Even if the curve holds its current rate, which it isn’t; it’s decelerating, the maths doesn’t close. More runs at this resolution will not get me there.
The other me is running a different calculation. What if I just keep going. What if the rule was too strict. What if I extended it by a week. Nobody would know. The app would be better for it. The rule was something I wrote when I didn’t know if the gamble would work. I know more now. I can still do this. I can make it happen with enough time. I should update the rule.
The first me is an engineer. The second me is the stubborn one who set himself the goal of racing three half marathons last year and did it. I should have stopped after two, when I felt the Achilles starting to hurt. That’s a signal the second me ignored to meet the goal. It’s the reason I haven’t been able to run for the last five months and am still in physiotherapy. The first me wrote the rule to stop the second me from doing it again.
They argue, the way you argue with yourself when you’re alone in a study at 10pm and a training curve is telling you something you don’t want to hear. Nobody else sees the argument. My wife has her own things to deal with. My son is too young to know the project exists. The people who could understand the cost of 75% on 5,000 classes of fine-grained classification haven’t read this series yet. The argument has no witness except me.
I decide to try one more thing.
The intervention
The decision is a gut call dressed up as logic. Most of my gut calls on this project haven’t worked. I have written things twice, rebuilt subsystems, abandoned architectures that looked right until I ran them. The pattern of my year has been: have an idea, commit to it, discover it doesn’t work, try the next idea. I don’t have confidence that the next idea will land. I have a history that says it probably won’t.
But the sums add up. The 256-pixel resolution is a bottleneck. Fine-grained features that distinguish species from their near relatives don’t survive the lower resolution. At 384 pixels, the model has almost twice the information to work with. The warmup I ran at that resolution showed a real boost, though warmup isn’t proof; it’s signal. The ArcFace scale needs adjusting for the larger class space. The batch size has to come down because the memory won’t take the increase. The optimiser needs to restart from the model’s weights rather than resuming its state, because the loss landscape will look different at the higher resolution.
It’s a complete reconfiguration. Four or five changes, each with its own possible failures, all landing on the same run. If any of them is wrong, the run fails in a way that burns days I don’t have. If the premise is wrong, if the bottleneck isn’t resolution, if there isn’t enough signal in the data for the model to find, the run trains cleanly and lands at 68%, and the gate fails. The dream is dead.
I’ve read the ArcFace papers. I’ve run the numbers. The maths makes sense. But the maths made sense for most of the gambles that didn’t work out.
I spend a day on the cleanup. Data audit using the 67% model against its own training set. The audit flags one in ten of my training images as probably mislabelled. In a dataset of ten million, a million images were potentially mislabelled. The system I’d been trying to make trustworthy is telling me, as clearly as it can, how much it had been lied to. I soft-delete the worst offenders and accept that the others will have to ride along. Another day on the reconfiguration. Sanity checks. A test run on a small subset.
Then I launch. Six days becomes three by the time training actually starts.
The run will take 21 hours per epoch. The maths I have run suggests I realistically need ten to twelve of them to go well to close the gap. The gate is in three days. The calculations are signalling a failure.
I’ve launched it. It is late. About 1am. I go to bed. I still have a day job. I’ve stayed up too late.
The result
The first epoch finishes at around 5pm on the next day.
I had told myself I wouldn’t watch the metrics during the run. The next morning, before I left for work, I resisted the temptation. Checking had been a ritual for months. That day I didn’t do it. There was no point, the first two epochs are just the warm up. The entire model backbone was frozen during those epochs only the heads would learn. There wouldn’t be a significant change in the accuracy if any.
I need to wait realistically two more days for epoch three to complete. That is when I would know if the gamble would pay off.
The maths still didn’t add up, that would only be one day until the stage gate. I had resigned myself that this wasn’t going to be it. I was going to be forced with a difficult decision, one I didn’t want to make. One that had been causing internal conflict for weeks. Either stop, as I had told myself I would. Or continue. Try new things. Other methods. New data.
I had already decided. This was the final run. If it failed, the gate stood. I was not moving the date. It was costing me too much, I knew that, I had accepted it.
I got home that evening. I opened the study door expecting nothing. Epoch 1 wasn’t going to tell me anything useful. It had finished just a few minutes before I checked. The final number: 75.1%.
I froze. The first response wasn’t thought. It was a single tear, then more, alone in the study, and I realised as it was happening that this was the first time I had cried since my father died nearly six years ago. The magnitude wasn’t grief. That grief was dealt with long ago. It was that I hadn’t held anything at this volume in my mind since then, and the thing I’d been holding for a year had just been set down.
The argument between my two selves was over. Not because one side won. Because the external world had made the argument unnecessary. The me who would have pushed past the gate didn’t have to. The me who had written the rule didn’t have to enforce it. Both of them got what they needed at the same moment, and all the energy that had been pressing against itself for a year had nowhere to go except out.
I sat there for a while. Then I stood up and went outside.
The garden
It was a beautifully sunny day. I sat down next to the French lavender I had planted two summers ago. It had just started to bloom. The smell was overwhelming, and it was covered in bees. Bumble bees of several kinds, honey bees, more than I usually see at once. I love bees. I especially love bumble bees. I think they are so cute.
A butterfly, the Common Blue, landed next to me. I didn’t do anything. I watched it.
The wild garlic was in full bloom further down the garden, under the plum and apple trees. I had planted the wild garlic too. The trees were there long before we moved in. They might be older than me. The wild garlic smelled pungent. It always does when the whole patch is flowering, and the smell drifted up the garden on the air and mixed with the lavender, and the bees kept on with whatever bees do on a spring afternoon, and I sat there and didn’t think about the number on the screen inside.
The run was still going. Another nine or ten epochs. The model would keep improving. The accuracy on the screen in the study would keep climbing, and in a few more days it would settle into whatever its ceiling turned out to be, and that number would be what shipped with the app. An accuracy that could justify the app’s existence.
But the gate was cleared. The rule had held. The version of me who wrote the rule and the version of me who wouldn’t stop weren’t arguing any more. For a while, there was just the garden, and it was enough.
I went back inside eventually. I had other things to do that day. My son wasn’t home yet and I needed to go pick him up.
The run kept going. I stopped needing to watch.
To be continued.