Skip to content
Houy Sengleang
← Projects

OpenVisionSearch

A self-hosted visual search API. Index product images from URLs, uploads, folders or S3, then search by photo, by text, or both, over plain HTTP.

May 2026 – now · Open source · Python, FastAPI, OpenCLIP, Qdrant, SQLite, Docker, GitHub Actions

Recall@1 went from 75% to 90% on 300 portrait product photos after replacing CLIP's centre-crop with padded framing.

Overview

OpenVisionSearch is an open-source, self-hosted visual search platform. It turns images into vectors with OpenCLIP, stores them in Qdrant, and returns the closest matches for a photo, a sentence, or a photo nudged by words ("this shoe, but in blue").

Images can come from URLs, multipart uploads, local folders, base64 payloads, or signed S3/R2 URLs. Only vectors and metadata are stored, never the original files.

Searching part of a picture is supported two ways: the caller can send a normalised crop box, or the server can detect objects itself and search each region, keeping the best score per image.

The project ships with CI and CodeQL, written decision records, and a spec for the next milestone: a multi-tenant product-level API that returns one result per product instead of per image.

The problem

Shoppers take product photos on phones. A stock CLIP pipeline centre-crops portrait photos and throws away about 40% of the frame, often including part of the product.

What I did

I designed and built the service end to end: the ingestion API, embedding pipeline, Qdrant integration, configuration, tests, CI and documentation.

  1. Measured stock CLIP against alternatives on 300 portrait products with degraded query photos before changing anything.
  2. Switched the default to padded framing and stored the framing per collection so old indexes keep working.
  3. Added text and hybrid search, region search with crop boxes or object detection, and an ONNX runtime option.

The result

A visual search API anyone can self-host with one Docker command, with measured quality instead of guessed quality.

  • Benchmark table and method are published in the README.
  • Public repository with CI, CodeQL, decision records and a roadmap.

Features

  • Image, text and hybrid search from one index (OpenCLIP shares a vector space for both)
  • Padded framing keeps the whole photo in view, lifting Recall@1 from 75% to 90%
  • Optional three-view embedding adds another 5 points of Recall@1
  • Each collection is pinned to the model it was built with, so vectors never get mixed
  • Runs without PyTorch through an ONNX path for lighter deployments
  • A search-quality harness for measuring recall before changing defaults

Try it

$ docker compose up --build

Run inside a clone of the repository.