# Visual Product Search Across 3.5M Images, Answering in 21ms

Client work · Sole engineer · in production

Upload a photo of an outfit and get matching products for every item in it separately - top, trousers, dress, bag, shoes, belt, jewellery. It runs entirely on one laptop GPU: no server, no cloud, no per-query cost.

- **3.48M** vectors indexed
- **~15–21 ms** retrieval

## How it works

Measured with the query photo’s own vector hidden, so a product must be found through its other angles.

- Outfit photo → Per garment
- Per garment → Embedding
- Embedding → GPU router
- Verifier → 3.48M vectors
- 3.48M vectors → GPU router
- GPU router → Scoring
- Scoring → Matches

## The detail

3,475,255 image vectors across 617,020 products. Two retrieval backends are selected automatically by available GPU memory: exact cosine similarity over the full resident matrix, or a two-stage approach that uses a compressed coarse pass to shortlist candidates and then re-ranks them exactly.

The accuracy came from the details: aggregating scores per product with a bonus when a second gallery angle also matches, treating category and colour agreement as a small nudge rather than an override, deduplicating colourways, and removing 34,000 junk vectors such as fabric swatches, logos and size charts.

Accuracy is measured honestly - the evaluation hides the query photo's own vector, so a product has to be found through its other angles. An index verifier re-embeds a random sample and proves alignment.

The embedding model was chosen by benchmarking two candidates on this specific catalogue rather than trusting a public leaderboard.

## What I did

- Built the entire visual search system - embedding pipeline, index, retrieval engine, detection and evaluation
- Embedded 3.48M product images using a parallel pipeline with atomic checkpointing
- Implemented two retrieval backends selected automatically by available GPU memory
- Designed the product-level score aggregation with a multi-hit bonus across gallery angles
- Built multi-garment detection so each item in a photo is searched independently
- Removed 34,000 junk vectors and fixed a whole class of taxonomy misclassification
- Wrote the index verifier that re-embeds a sample and proves row-to-metadata alignment
- Benchmarked two embedding models on the real catalogue and chose on evidence rather than reputation

## Stack

Machine Learning, Python, PyTorch, Vector Database, Computer Vision, NumPy, SigLIP, Image Processing, GPU Optimization, Benchmarking

Case study: https://www.siddhvasudev.com/projects/visual-search
