While I was deep in the valley fixing collectors, I was also reading papers.
The data grind took about six weeks of explicit work. What it didn’t take was all of my attention. I fixed collectors in the morning and the evening. I read about model architecture whenever I had spare time. In bed, on the train to work, on my lunch break.
I had assumed, at the start of the data work, that the 42% model just needed more data. By early March I had stopped assuming that. I had a growing suspicion that the architecture I’d been training wasn’t going to work on 5,000 species no matter how much data I gave it.
The suspicion wasn’t founded on anything I could prove. It was founded on the mismatch between what the model was doing and what the app would eventually ask it to do.
The mismatch
The model was being trained to classify. Given an image, assign it to one of 5,000 species. Loss function: softmax cross-entropy, the standard choice for multi-class classification, which rewards the model for pushing the correct class’s probability up and the wrong classes’ probabilities down. This is how you train a classifier. It is what classifiers are for.
The app, at runtime, wasn’t going to do classification. The app was going to do similarity search. Given an image, compute an embedding. A 256-dimensional vector representing the image’s position in learned feature space, and find the nearest known plant in a precomputed reference set. This is a different question from classification. It asks which known example does this look most like, not which fixed label applies to this.
The two objectives sound similar. They aren’t. A model trained to classify correctly can produce embeddings that are terrible for similarity search. The classification objective rewards decision boundaries between classes. The similarity objective rewards distances between classes. You can satisfy the first without the second. You can have a model that classifies at 95% accuracy and produces embeddings where intra-class variation is larger than inter-class variation. Where two images of the same species land further apart than an image of that species and an image of something else entirely.
I had been training for classification. The app would use embeddings. I had been training the wrong thing.
The search
Once I saw the mismatch clearly, I started looking for what to do about it. I spent most of March reading. Papers, blog posts, model zoos, benchmark results. I used AI tools to help me map the space, asking them for similarity-learning approaches I hadn’t considered, then going to the primary sources to evaluate each one.
There are several families of approaches to similarity learning. Some try to teach the model to place known pairs at correct distances from each other. Some pull similar examples together and push different ones apart. Some learn distance functions directly. Some use text descriptions of images to learn what similar things look like.
Each had their own weaknesses. Some were notoriously unstable in training. Some produced collapsed solutions where every embedding ended up in the same region of space. Some required data I didn’t have. Some worked but plateaued at the scale I needed.
What kept coming up in my reading, and what kept looking more relevant, was ArcFace.
The choice
ArcFace was developed for face recognition. The problem it was built to solve is structurally the same as mine. Thousands of classes, each with significant intra-class variation. The same face in different lighting, different expressions, different angles. A runtime objective of similarity search: does this new photo match someone we know? A requirement that classes be not just separable but distant, because at runtime the decision isn’t which class is highest, it’s is this close enough to any known class to call it a match.
The mechanism ArcFace uses to do this is angular margin. Instead of optimising linear decision boundaries between classes, it places all embeddings on a hypersphere and pushes classes apart by angle. The margin is measured in radians. Two classes that are hard to separate don’t just get a wider decision boundary between them; they get rotated to different regions of the sphere entirely. At inference, cosine distance between any two embeddings corresponds directly to how the model sees their similarity. The training objective and the serving objective become the same function.
The SubCenter variant, which I ended up using, allows each class to have up to K=3 embedding clusters rather than being forced into a single point. This matters for species. Sage photographed in the wild, sage photographed in cultivation, and sage photographed in winter might legitimately occupy different regions of feature space. Standard ArcFace would be forced to average them into a compromise centre that fitted none of them well. SubCenter lets the species have three centres, with the model deciding which is most relevant for any given image.
There was a second benefit I hadn’t anticipated when I started reading. ArcFace’s training setup uses a compact embedding head: a 256-dimensional vector per image, produced by a small final layer, instead of a full 5,000-class softmax classifier. The softmax head, at 5,000 classes, takes a substantial amount of the model’s parameter budget. The ArcFace head takes almost none of it.
This changed what was possible with the backbone. In February, I had ruled out EfficientNet-B2 because the combined size of B2 plus a 5,000-class softmax head broke the mobile model size budget I’d set in January. With ArcFace, the head shrank dramatically, and the budget reopened. B2 was now feasible where before it hadn’t been. The larger backbone, which I wanted anyway for capacity reasons, was suddenly affordable.
The mobile size budget needed recalculating, not breaking. When I’d set 10MB in January, I’d been conservative. It was an early number, written before the rest of the app’s size footprint was clear. By March I had better information. The full mobile app, with all its bundled features, had more headroom for the ML model than the January budget had assumed. I revised the budget to 20MB. B2 fitted inside it.
The commit
On March 28th I made one commit. 0253b2c: “Add ArcFace loss, EfficientNet-B2 backbone, and balanced sampling for training overhaul.” Three changes bundled because they were one decision. 744 lines of code across six files, including a new 144-line ArcFace module and 281 lines of unit tests written alongside it. Substantial engineering, not a small commit.
But the lines of code weren’t the hard part of those few days, or of the weeks before them. The hard part was the recognition that the training objective and the serving objective were different functions; the search through similarity-learning approaches; the choice of ArcFace from a list of options that all had reasonable claims; the decision to bundle the backbone change with the loss change rather than ship them separately; the recalculation of the model size budget that made B2 affordable. None of that work shows up in git diff. It shows up in the commit existing at all, on that date, with those specific decisions inside it.
The rule I’d set myself: build it alone, write every line. It had been about code. The work that decided whether the project would succeed wasn’t code. It was the thinking that produced the code. I’d spent nearly a year guarding the wrong thing.
What I was betting on
That the mismatch I’d identified was real. That ArcFace was the right answer for it. That the facial-recognition analogy translated cleanly to plants. That thousands of species with intra-class variation were structurally equivalent to thousands of faces with intra-person variation. That the angular margin would do for sage what it had done for faces across different conditions.
I’d spent weeks reading. The reasoning held up. But I had spent weeks on this kind of work before, and the reasoning had not always translated into results. The pattern of the year had been familiar: have an idea, commit to it, discover it doesn’t work, try the next idea.
I didn’t know whether ArcFace was another one of those. I would find out.
The first epoch of the new run started the evening of March 28th. I left it running overnight.
To be continued.