Product Catalog Engine: 485 Brands, 650K+ Products Nightly
Client work · Sole engineer · in production
An AI shopping assistant is only as good as its catalogue. Buying product feeds is expensive and slow to expand, and writing a separate scraper per brand does not scale past a few dozen.
The detail
I built a self-growing catalogue engine that discovers candidate stores, vets them, onboards them automatically, fingerprints which e-commerce platform each one runs, and fetches through a tiered strategy - browser-free by default using TLS impersonation rather than headless browsers, which is dramatically cheaper and faster.
Around 94% of volume flows through a single universal path with no per-brand code, including a workaround for the platform's own page ceiling so large catalogues are not silently truncated. The rest is covered by five platform adapters and a generic sitemap fallback, so it can be pointed at almost any store with no configuration.
Every product carries full lifecycle state - first seen, last seen, delisted - plus per-size stock, verified currency, mined attributes and a quality score. A daily digest flags every brand that broke, dropped, recovered or is new, because a scraper fleet without drift alerting degrades invisibly.
485 brands, 650,000+ active products, 600,000+ curated records, refreshed nightly.
What I did
- Built the entire pipeline - discovery, vetting, onboarding, platform detection, fetching, normalisation and curation
- Designed the universal platform path that carries 94% of volume with no per-brand code
- Worked around the platform's own pagination ceiling so large catalogues are not silently truncated
- Chose TLS impersonation over headless browsers, cutting cost and time dramatically
- Built five platform adapters plus a generic fallback so it works on almost any store with no configuration
- Implemented full lifecycle tracking so delisting is a recorded fact rather than a silent disappearance
- Built the daily drift digest that classifies every brand as broken, dropped, recovered or new
- Documented the data ceilings honestly in the README rather than hiding them
Stack
Python, Web Scraping, Data Extraction, API Integration, Database Design, Data Pipelines, SQLite, ETL, Data Quality
Case study: https://www.siddhvasudev.com/projects/product-catalog-engine-485-brands-650k