Catalogue Enrichment Pipeline on AWS With Budget-Admitted Runs
Client work · Engineer (contract) · in production
A product-catalogue pipeline on AWS where quality and cost are part of the processing contract.
The detail
A scraper that works on one laptop is not a data platform. In the cloud it can fail quietly three ways: a job that succeeds and returns nothing, a paid model request repeated after a timeout, and a release nobody can trace to its source.
Every run is tied to immutable source artifacts and a qualified worker image, and carries an explicit budget and admission evidence. A record is acquired, then a validated candidate, then a released item, and the three are never counted as one. Unchanged products are skipped, verified work is reused, and an uncertain paid request stays uncertain until reconciled.
Styling-attribute coverage on the live catalogue went from 0% to 99.5%. Releases require the exact merged source, an immutable image identity, a clean vulnerability scan and a rollback plan. The record keeps what went wrong as well, including a cutover that was rolled back.
Python, AWS Batch, Step Functions, Lambda, ECS, S3, DynamoDB, Docker.
What I did
- Turned a laptop scraper into a governed cloud pipeline: acquisition, classification, embeddings, validated release
- Tied every run to immutable source artifacts, a qualified worker image and an explicit budget
- Kept uncertain paid model requests in an uncertain state instead of repeating them
- Took styling-attribute coverage on a live catalogue from 0% to 99.5%
- Built release controls: exact source, immutable image identity, scan gates and rollback plans
- Ran cost audits and kept an incident record, including a rolled-back cutover
Stack
Python, AWS, Docker, Data Pipelines, Machine Learning, Embeddings, CI/CD, MongoDB, GitHub Actions
Case study: https://www.siddhvasudev.com/projects/catalogue-enrichment-pipeline-on-aws-with