Visual product search app

I’ve wanted to develop this for a long time. Recently, I came across Google’s MediaPipe Image Embedder — that was my starting point, but not where the journey ends.

The basics: I need to generate digital fingerprints (embeddings) of all my product images and store them in a vector database or index. I chose FAISS. Then, on the mobile app, when the user takes a picture, I need to generate its embedding and send it to my endpoint, where I calculate its Cosine Similarity against all the indexed items and return the most similar. So, the architecture is: an indexer (in Python), a comparison endpoint (Python), and the mobile app (in Kotlin).

The MediaPipe Image Embedder is based on the MobileNetV3 model. It was a nice start, but I soon noticed that changes in the image backgrounds affected the results so much that it became unusable. The images in my catalog have white or grey backgrounds, but the test images I took in an actual retail store have store shelf backgrounds.

I tinkered with removing the image backgrounds using a segmentation model (mobile-device-focused models like MobileSAM or EdgeSAM), but, dissatisfied with the latency and results, I started to research other embedding models.

I found suitable versions of some models for use on mobile devices and apps (low latency, low to medium model size): DINOv2, DINOv3, MobileCLIP/MobileCLIP2, MobileNetV4, SigLIP2, and EVA02. Each of them has variations, and some are more or less suitable for mobile device deployment (latency and model size in megabytes).

I implemented all of the suitable variations. It was difficult because for each variation I needed to implement the indexer, the endpoint, and the mobile app embedder. I also worked on “validators” — scripts where I compared the implementation of the indexer (Python) with the implementation of the mobile app embedder (Kotlin). The image transformations I execute on the Python indexer need to be exactly the same or extremely similar to the mobile app (Kotlin) implementation.

And that is where it ends for now. I got some amazing (sometimes magical) results with MobileCLIP and SigLIP2. And I expect that you will too!