While I was deep in the valley fixing collectors, I was also reading papers.
The data grind took about six weeks of explicit work, but it didn’t take all of my attention. I fixed collectors in the morning and the evening. I read about model architecture whenever I had spare time, in bed, on the train to work, and on my lunch break.
I had assumed, at the start of the data work, that the 42% model just needed more data. By early March I had stopped assuming that. I had a growing suspicion that the architecture I’d been training wasn’t going to work on 5,000 species, no matter how much data I gave it.
I couldn’t prove that yet, but the mismatch was becoming hard to ignore. The model was being trained to do one thing. The app would eventually ask it to do another.
The mismatch
The model was being trained to classify. Given an image, it had to assign the image to one of 5,000 species. The loss function was softmax cross-entropy, the standard choice for multi-class classification. It rewards the model for pushing the correct class’s probability up and the wrong classes’ probabilities down. This is how you train a classifier. It is what classifiers are for.
The app, at runtime, wasn’t going to do classification. It was going to do similarity search. Given an image, it would compute an embedding, a 256-dimensional vector representing the image’s position in learned feature space, then find the nearest known plant in a precomputed reference set. This is a different question from classification. It asks which known example this looks most like, not which fixed label applies.
The two objectives sound similar. They aren’t. A model trained to classify correctly can produce embeddings that are terrible for similarity search. The classification objective rewards decision boundaries between classes. The similarity objective rewards distances between classes. You can satisfy the first without the second. A model can classify at 95% accuracy and still produce embeddings where two images of the same species land further apart than an image of that species and an image of something else entirely.
This is the point I had to slow down and understand properly. A classifier asks, “Which label should this image get?” A search system asks, “Which known image is this closest to?” Those are related questions, but they are not the same. I had built the first and needed the second.
I had been training for classification. The app would use embeddings. I had been training the wrong thing.
The search
Once I saw the mismatch clearly, I started looking for what to do about it. I spent most of March reading papers, blog posts, model zoos, and benchmark results. I used AI tools to help map the space, asking them for similarity-learning approaches I hadn’t considered, then going to the primary sources to evaluate each one.
I needed a training objective that made the embeddings useful at inference. Some approaches teach the model to place known pairs at correct distances from each other. Some pull similar examples together and push different ones apart. Some learn distance functions directly. Some use text descriptions of images to learn what similar things look like.
Each had their own weaknesses. Some were notoriously unstable in training. Some produced collapsed solutions where every embedding ended up in the same region of space. Some required data I didn’t have. Some worked but plateaued at the scale I needed.
What kept coming up in my reading, and what kept looking more relevant, was ArcFace.
Why ArcFace fitted
ArcFace was developed for face recognition, and the problem it was built to solve was structurally close to mine. It deals with thousands of classes, each with significant intra-class variation. The same face can appear in different lighting, different expressions, and different angles. At runtime, the question is not just which class scores highest. It is whether this new photo is close enough to a known person to call it a match.
ArcFace does that with angular margin. Instead of optimising linear decision boundaries between classes, it places all embeddings on a hypersphere and pushes classes apart by angle. The margin is measured in radians. Two classes that are hard to separate don’t just get a wider decision boundary between them; they get rotated to different regions of the sphere entirely.
The useful part is the angle. ArcFace puts embeddings on a sphere and separates classes by direction from the centre. If two species are too close together, the loss pushes their directions apart. That matches the similarity search the app needs later.
At inference, cosine distance between any two embeddings corresponds directly to how the model sees their similarity. The training objective and the serving objective become the same function.
The SubCenter variant, which I ended up using, allows each class to have up to K=3 embedding clusters rather than being forced into a single point. That matters for species. Sage photographed in the wild, sage photographed in cultivation, and sage photographed in winter might legitimately occupy different regions of feature space. Standard ArcFace would be forced to average them into a compromise centre that fitted none of them well. SubCenter lets the species have three centres, with the model deciding which is most relevant for any given image.
There was a second benefit I hadn’t anticipated when I started reading. ArcFace’s training setup uses a compact embedding head, a 256-dimensional vector per image produced by a small final layer, instead of a full 5,000-class softmax classifier. The softmax head, at 5,000 classes, takes a substantial amount of the model’s parameter budget. The ArcFace head takes almost none of it.
This changed what was possible with the backbone. In February, I had ruled out EfficientNet-B2 because the combined size of B2 plus a 5,000-class softmax head broke the mobile model size budget I’d set in January. With ArcFace, the head shrank dramatically. B2 was now feasible where before it hadn’t been. The larger backbone, which I wanted anyway for capacity reasons, was suddenly affordable.
The mobile size budget needed recalculating, not breaking. When I’d set 10MB in January, I’d been conservative. It was an early number, written before the rest of the app’s size footprint was clear. By March I had better information. The full mobile app, with all its bundled features, had more headroom for the ML model than the January budget had assumed. I revised the budget to 20MB. B2 fitted inside it.
The commit
On 28 March I made one commit. 0253b2c: “Add ArcFace loss, EfficientNet-B2 backbone, and balanced sampling for training overhaul.” Three changes were bundled because they were one decision. The diff was 744 lines across six files, including a new 144-line ArcFace module and 281 lines of unit tests written alongside it.
The larger work was the decision behind the diff. I had recognised that the training objective and the serving objective were different functions, searched through similarity-learning approaches, chosen ArcFace from a list of options that all had reasonable claims, bundled the backbone change with the loss change, and recalculated the model size budget that made B2 affordable. None of that work shows up well in git diff. It shows up in the commit existing at all, on that date, with those specific decisions inside it.
The rule I’d set myself had been about code. Build it alone, write every line. This decision reminded me that the work that decided whether the project would succeed was the thinking that produced the code. I had been guarding the wrong thing.
That was a useful realisation. Code is the visible artefact, so it is easy to treat it as the work. Here, the important work was choosing the right problem. Once I stopped treating the app as a classifier and treated it as a search system, the code had a clearer direction.
What I was betting on
I was betting that the mismatch I’d identified was real, that ArcFace was the right answer for it, and that the face-recognition analogy translated cleanly to plants. Thousands of species with intra-class variation had to be close enough to thousands of faces with intra-person variation. The angular margin had to do for sage what it had done for faces across different conditions.
I’d spent weeks reading, and the reasoning held up. But I had spent weeks on this kind of work before, and the reasoning had not always translated into results. The pattern of the year had become familiar. Have an idea, commit to it, discover it doesn’t work, try the next idea.
I didn’t know whether ArcFace was another one of those. I would find out.
The first epoch of the new run started on the evening of 28 March. I left it running overnight.
To be continued.