Voice AI Debt-Collection Agent With Full-Duplex SIP Calls

AI products · Creator, sole engineer · shipped

An outbound voice agent that places real phone calls, negotiates in real time, and can be interrupted mid-sentence like a real conversation.

Watch: https://www.youtube.com/watch?v=VsEfOfwh8XM

How it works

Outbound calls over SIP. When a call ends, a 70B model scores risk, legal flags and the outcome.

  • Caller → LiveKit
  • LiveKit → Deepgram STT
  • Deepgram STT → Groq LLM
  • Groq LLM → Deepgram TTS
  • Deepgram TTS → LiveKit
  • Groq LLM → Risk review

The detail

A risk engine analyses each call's transcript as soon as it is saved. It flags regulatory triggers - bankruptcy, legal representation, cease-and-desist - classifies the person's situation, and tags the outcome as a payment promise, refusal or dispute.

A dashboard shows each call's transcript with its risk indicators and aggregates recovery rates across calls.

Built as three deployable services so the voice agent's event loop never shares a process with the API.

Python, FastAPI, LiveKit, Deepgram, Groq, React, Docker.

What I did

  • Built the complete system - voice agent, API, risk engine and dashboard
  • Implemented outbound telephony over SIP with full-duplex interruption
  • Built the compliance engine that flags regulatory triggers from each call's transcript
  • Built automatic outcome classification from the conversation itself
  • Architected it as three separate services so the voice event loop never shares a process with the API
  • Built the dashboard showing transcripts with risk indicators

Stack

Voice Assistant, Python, Speech Recognition, AI Agent Development, FastAPI, LiveKit, SIP, Groq, Deepgram, Docker, Real-Time Systems

Case study: https://www.siddhvasudev.com/projects/vaani