# Agentic RAG for PDFs With Tables, Charts and Figures

AI products · Sole engineer · shipped

Most retrieval systems only handle text. In research papers, medical reports and technical datasheets the answer is often inside a table, a bar chart or a figure caption - so a text-only pipeline either finds nothing or invents a number.

Code: https://github.com/vasu-devs/Waldo

## The detail

Ingestion uses a high-fidelity document parser for structure, OCR as a fallback for scanned pages, and a vision model that transcribes figures and charts into searchable text before embedding into a vector database.

Retrieval is agentic: it routes between answering directly and searching, grades every retrieved passage for relevance, rewrites failed queries and retries within a limit, and automatically pulls in figure transcriptions when the question is visual.

Guardrails matter here. Out-of-scope questions are refused rather than answered, and if the vision model's quota runs out the system falls back to the parser's own captions instead of inventing a description.

Python, FastAPI, LangGraph, Qdrant, React.

## What I did

- Built the full ingestion pipeline including structure parsing, OCR fallback and vision transcription of figures
- Implemented the agentic retrieval loop with relevance grading, query rewriting and bounded retries
- Added visual-query detection that pulls figure transcriptions in automatically
- Designed the refusal behaviour for out-of-scope questions
- Built graceful degradation to parser captions when the vision model quota is exhausted
- Built the React frontend

## Stack

Python, Retrieval-Augmented Generation, LangChain, Vector Database, FastAPI, Qdrant, LangGraph, OCR, Document Processing, Sentence Transformers

Case study: https://www.siddhvasudev.com/projects/agentic-rag-for-pdfs-with-tables
