Job Aggregation Service Across 30+ Sources, Callable by AI

Developer tools · Sole engineer · complete

Job data is fragmented across dozens of applicant tracking systems, job APIs, community boards and feeds, each with a different schema, rate limit and freshness profile. Anyone building job tooling rebuilds this aggregation layer badly and ends up with duplicates and stale postings.

The detail

One call aggregates 30+ sources into ranked, deduplicated, structured results. The pipeline is built around thin per-source adapters, so adding a source means writing an adapter rather than touching the pipeline - which is what keeps 30+ sources maintainable by one person.

Every posting gets a market region classification, a freshness tier, a quality score, and deduplication by canonical URL that persists across runs in a local database with full-text search.

Three deployment surfaces share one store: a command-line watcher with adaptive pacing that backs off on errors, an HTTP API, and a tool server so an AI agent can call it directly.

The source research is a versioned document grading every endpoint as verified or unverified with field mappings, rather than tribal knowledge in someone's head.

Python, httpx, SQLite, FastAPI, Docker.

What I did

  • Designed the adapter-per-source pipeline so new sources need no pipeline changes
  • Built adapters for 10 applicant tracking systems, 9 job APIs and community sources
  • Implemented region classification, freshness tiering, quality scoring and cross-run deduplication
  • Built three deployment surfaces over one store including an AI-callable tool server
  • Wrote adaptive watch-mode pacing with jitter and exponential backoff
  • Produced a versioned source-research document grading every endpoint

Stack

Python, API Integration, Web Scraping, FastAPI, Database Design, httpx, SQLite, FTS5, MCP, Async Programming, Docker, Rate Limiting

Case study: https://www.siddhvasudev.com/projects/job-aggregation-service-across-30-sources