The Crop That Was Costing Me 15 Points of Recall

OpenVisionSearch is a self-hosted API for finding products by photo. You index product images, a shopper uploads a picture, and the closest matches come back. Under the hood it is simple: OpenCLIP turns each image into a vector, Qdrant finds the nearest neighbours.

The interesting part turned out to be one line of preprocessing.

What the usual setup throws away

The common way to prepare an image for CLIP is to resize the short side and then centre-crop to a square. That's what most examples do, and it works fine on square catalogue shots.

Shoppers don't take square catalogue shots. Phone photos are portrait, the product is rarely centred, and sometimes the part that matters sits near the top or the bottom of the frame.

A centre crop on a portrait photo keeps the middle square and drops the rest. That's about 40% of the picture, often including part of the product. The model never sees it, so it can't match on it.

Measure first

I didn't want to change a default because it felt better, so I set up a benchmark before touching anything:

• 300 portrait products in the index
• a degraded query photo for each product, closer to a real upload than a catalogue image
• Recall@1 (is the right product the first result?) and Recall@5 (is it in the top five?)

Then I compared ways of framing the image before embedding it.

The numbers

framing        views   recall@1   recall@5   index   query
crop (stock)   1       75.0%      96.2%      7 ms    17 ms
pad            1       90.0%      100%       6 ms    16 ms
pad            3       95.0%      100%       16 ms   25 ms

Padding scales the whole image to fit inside the square and fills the leftover space, instead of cutting anything off. The model sees everything the shopper photographed.

That one change took Recall@1 from 75% to 90%. It's also fractionally faster, because there is less image to resample. On images that are already square it does nothing, and the results are byte-identical.

The three-view option averages the padded frame with a centre crop and a mirrored copy. It buys another 5 points of Recall@1 for roughly double the indexing work, so it's off by default. Turn it on with EMBED_VIEWS=3 if indexing time doesn't matter to you.

The part that's easy to get wrong

Changing preprocessing changes the vectors. Index half a catalogue with crop and half with pad, and the scores stop being comparable. Search gets worse and nothing tells you why.

So every collection stores the framing it was built with and keeps using it. Changing the server default doesn't touch collections that are already indexed. To adopt a new setting you create a new collection and re-index. It's a bit more work, and nothing breaks silently.

The same rule applies to the model. Vectors from two different models mean nothing to each other, so a collection is pinned to the embedding model it was created with.

Text search for free

OpenCLIP puts images and text in the same vector space, so the images you already indexed can be searched by description with no second index. You can also mix the two: a photo of a shoe, nudged by the words "but in blue".

One thing to know: text-to-image scores sit much lower than image-to-image scores, roughly 0.2 to 0.35 for a good match against 0.8 and up. Tune min_score per endpoint instead of sharing one threshold.

What I took away

• Check what your preprocessing throws away. Defaults are written for someone else's data.
• Build the benchmark before the fix. Without that table this would have been an argument, not a decision.
• Store the settings that shaped your vectors right next to the vectors.

The code, the benchmark and the API docs are on GitHub: hoysengleang/images-analystic-search. There's a longer write-up of the project on its case study page.