A Forensic Report on 14 Months of My Own AI Coding-Agent Sessions
AI products · Creator, sole engineer · complete
A local pipeline that reads every AI coding-assistant session on one machine and turns it into a report on how its owner works.
The detail
Over fourteen months it covers 936 sessions, 6,854 prompts, about 1,274 active hours and 155 projects across eight assistants.
The pipeline extracts each session, reconstructs what the agents did, mines the commits in every repository and links sessions to commits. Sessions are then scored against a written rubric with the dates and tool names removed, so a grader cannot judge by era, and sixty sessions are scored twice to report agreement.
The findings are specific: a third of all prompts were typed at night, session quality was lowest then, and about one scored session in eight was a loop. A self-check validates invariants and scans for secrets before anything renders. Nothing is uploaded.
Python, JavaScript.
What I did
- Read the local session stores of eight coding assistants into one dataset
- Linked sessions to the commits they produced across every repository
- Scored sessions against a written rubric, blind to date and tool, with 60 scored twice for agreement
- Computed behaviour and outcome statistics with confidence intervals
- Added a self-check for invariants and leaked secrets before rendering
- Built an interactive report, a playbook and a PDF
Stack
Python, Data Analysis, LLM Evaluation, Data Visualization, JavaScript, Technical Writing, Git
Case study: https://www.siddhvasudev.com/projects/a-forensic-report-on-14-months