- Capstone
- AI security
- Research
Eight specialists,
one verdict.
A smart-contract auditor that runs entirely on a laptop, and what measuring it honestly taught us.
- Role
- Owner of the verdict engine, in a team of four
- Project
- Final-year capstone, PES University
- Output
- A working tool, a 2,250-contract benchmark and a manuscript in preparation
- Stack
- React, FastAPI, Ollama, ChromaDB

React · FastAPI · Ollama · ChromaDB
01 · Context
A first pass before a human auditor looks.
ThirdEye reads a Solidity contract and returns one verdict: GO, NO-GO or INCONCLUSIVE. It runs on local models only: no API key, no network call, no contract source leaving the laptop. It is a cheap filter that runs before a person looks, not a replacement for an audit.
- 8specialist roles
- 3model families
- $0per scan
02 · How it works
Route, accuse with evidence, then pool.
- 01A router picks the readersA static router decides which of the eight specialist models should read a given contract.
- 02Every accusation needs a quoteA specialist must quote, word for word, the line it is accusing. A finding without that quote is dropped.
- 03The survivors are pooledThe findings that pass are combined with noisy-OR risk pooling into a single verdict.
03 · My part
I owned the verdict engine.
- Built
The static router that chooses which specialists read each contract.
- Built
The verbatim-quote evidence gate that throws out findings with nothing to point at.
- Built
The noisy-OR pooling that turns many small findings into one verdict.
- Ran
The threshold sweep and the held-out-seed evaluation behind every number on this page.
04 · Results
The evidence gate cut false alarms and kept most of the catches.
| Scoring rule | Recall | False-alarm rate |
|---|---|---|
| Before any gating | 95.9% | 72.2% |
| Evidence gate (ours) | 89.1% | 45.9% |
| Noisy-OR at τ = 0.99 | 84.9% | 40.0% |
The same stored verdicts on the same 2,142 contracts, balanced between safe and vulnerable. Only the rule that reads them changes.
05 · The honest number
We lead with the bad number on purpose.
A 45.9% false-alarm rate means roughly one in two clean contracts gets flagged. That is the first thing the project site tells you. Every finding ships with the line of code it rests on, precisely so that a wrong one takes seconds to dismiss.
- 89.1%of vulnerable contracts caught
- 45.9%of safe contracts wrongly flagged
- 2,142contracts scored

06 · The finding
The headline number is not a property of the tool.
On the same contracts, with the same rule, the false-alarm rate climbs from 12.2% to 90.1% as the router fires more specialists. Nobody chooses that number, and it is rarely reported. The result held on a seed the system had never seen, and we are writing it up.
The test suite reflects the same worry: 54 tests, most of them aimed at catching our own measurement errors.

07 · What I took from it
Measuring is a design decision.
- Lesson
A model's claim is worth what it can point to. Demanding the quoted line did more than any prompt tweak.
- Lesson
Report the number that makes you look worse. It is the one people can actually plan around.
- Lesson
Hold out a seed. A result you have not replicated is still a hunch.
Next case study
ADA: an adaptive RAG assistant for research papers
Need a developer
who ships?
Talk to me.
Open to SDE internships and 2027 full-time roles. Remote or relocation.
Download resume1{2 "name": "Tarun Basavaraju",3 "title": "Software engineer",4 "email": "tarunbasavaraju25@gmail.com",5 "phone": "+91 98458 59937",6 "location": "Bengaluru, India"7}