Visual Product Search Across 3.5M Images, Answering in 21ms
Client work · Sole engineer · in production
Upload a photo of an outfit and get matching products for every item in it separately - top, trousers, dress, bag, shoes, belt, jewellery. It runs entirely on one laptop GPU: no server, no cloud, no per-query cost.
- 3.48M vectors indexed
- ~15–21 ms retrieval
How it works
Measured with the query photo’s own vector hidden, so a product must be found through its other angles.
- Outfit photo → Per garment
- Per garment → Embedding
- Embedding → GPU router
- Verifier → 3.48M vectors
- 3.48M vectors → GPU router
- GPU router → Scoring
- Scoring → Matches
The detail
3,475,255 image vectors across 617,020 products. Two retrieval backends are selected automatically by available GPU memory: exact cosine similarity over the full resident matrix, or a two-stage approach that uses a compressed coarse pass to shortlist candidates and then re-ranks them exactly.
The accuracy came from the details: aggregating scores per product with a bonus when a second gallery angle also matches, treating category and colour agreement as a small nudge rather than an override, deduplicating colourways, and removing 34,000 junk vectors such as fabric swatches, logos and size charts.
Accuracy is measured honestly - the evaluation hides the query photo's own vector, so a product has to be found through its other angles. An index verifier re-embeds a random sample and proves alignment.
The embedding model was chosen by benchmarking two candidates on this specific catalogue rather than trusting a public leaderboard.
What I did
- Built the entire visual search system - embedding pipeline, index, retrieval engine, detection and evaluation
- Embedded 3.48M product images using a parallel pipeline with atomic checkpointing
- Implemented two retrieval backends selected automatically by available GPU memory
- Designed the product-level score aggregation with a multi-hit bonus across gallery angles
- Built multi-garment detection so each item in a photo is searched independently
- Removed 34,000 junk vectors and fixed a whole class of taxonomy misclassification
- Wrote the index verifier that re-embeds a sample and proves row-to-metadata alignment
- Benchmarked two embedding models on the real catalogue and chose on evidence rather than reputation
Stack
Machine Learning, Python, PyTorch, Vector Database, Computer Vision, NumPy, SigLIP, Image Processing, GPU Optimization, Benchmarking
Case study: https://www.siddhvasudev.com/projects/visual-search