Job Aggregation Service Across 30+ Sources, Callable by AI
Developer tools · Sole engineer · complete
Job data is fragmented across dozens of applicant tracking systems, job APIs, community boards and feeds, each with a different schema, rate limit and freshness profile. Anyone building job tooling rebuilds this aggregation layer badly and ends up with duplicates and stale postings.
The detail
One call aggregates 30+ sources into ranked, deduplicated, structured results. The pipeline is built around thin per-source adapters, so adding a source means writing an adapter rather than touching the pipeline - which is what keeps 30+ sources maintainable by one person.
Every posting gets a market region classification, a freshness tier, a quality score, and deduplication by canonical URL that persists across runs in a local database with full-text search.
Three deployment surfaces share one store: a command-line watcher with adaptive pacing that backs off on errors, an HTTP API, and a tool server so an AI agent can call it directly.
The source research is a versioned document grading every endpoint as verified or unverified with field mappings, rather than tribal knowledge in someone's head.
Python, httpx, SQLite, FastAPI, Docker.
What I did
- Designed the adapter-per-source pipeline so new sources need no pipeline changes
- Built adapters for 10 applicant tracking systems, 9 job APIs and community sources
- Implemented region classification, freshness tiering, quality scoring and cross-run deduplication
- Built three deployment surfaces over one store including an AI-callable tool server
- Wrote adaptive watch-mode pacing with jitter and exponential backoff
- Produced a versioned source-research document grading every endpoint
Stack
Python, API Integration, Web Scraping, FastAPI, Database Design, httpx, SQLite, FTS5, MCP, Async Programming, Docker, Rate Limiting
Case study: https://www.siddhvasudev.com/projects/job-aggregation-service-across-30-sources