All work
  • Capstone
  • AI security
  • Research

Eight specialists,
one verdict.

A smart-contract auditor that runs entirely on a laptop, and what measuring it honestly taught us.

Role
Owner of the verdict engine, in a team of four
Project
Final-year capstone, PES University
Output
A working tool, a 2,250-contract benchmark and a manuscript in preparation
Stack
React, FastAPI, Ollama, ChromaDB
Open the project site

React · FastAPI · Ollama · ChromaDB

01 · Context

A first pass before a human auditor looks.

ThirdEye reads a Solidity contract and returns one verdict: GO, NO-GO or INCONCLUSIVE. It runs on local models only: no API key, no network call, no contract source leaving the laptop. It is a cheap filter that runs before a person looks, not a replacement for an audit.

  • 8specialist roles
  • 3model families
  • $0per scan

02 · How it works

Route, accuse with evidence, then pool.

  1. 01A router picks the readersA static router decides which of the eight specialist models should read a given contract.
  2. 02Every accusation needs a quoteA specialist must quote, word for word, the line it is accusing. A finding without that quote is dropped.
  3. 03The survivors are pooledThe findings that pass are combined with noisy-OR risk pooling into a single verdict.

03 · My part

I owned the verdict engine.

  • Built

    The static router that chooses which specialists read each contract.

  • Built

    The verbatim-quote evidence gate that throws out findings with nothing to point at.

  • Built

    The noisy-OR pooling that turns many small findings into one verdict.

  • Ran

    The threshold sweep and the held-out-seed evaluation behind every number on this page.

04 · Results

The evidence gate cut false alarms and kept most of the catches.

Scoring ruleRecallFalse-alarm rate
Before any gating95.9%72.2%
Evidence gate (ours)89.1%45.9%
Noisy-OR at τ = 0.9984.9%40.0%

The same stored verdicts on the same 2,142 contracts, balanced between safe and vulnerable. Only the rule that reads them changes.

05 · The honest number

We lead with the bad number on purpose.

A 45.9% false-alarm rate means roughly one in two clean contracts gets flagged. That is the first thing the project site tells you. Every finding ships with the line of code it rests on, precisely so that a wrong one takes seconds to dismiss.

  • 89.1%of vulnerable contracts caught
  • 45.9%of safe contracts wrongly flagged
  • 2,142contracts scored
The ThirdEye project site's 'Where it stands' panel: 89.1% recall, 45.9% false-alarm rate, 2,142 contracts scored and $0 cost per scan, above a note headed 'We lead with the bad number on purpose'
Where it stands, from the live project site. The bad number sits beside the good one.

06 · The finding

The headline number is not a property of the tool.

On the same contracts, with the same rule, the false-alarm rate climbs from 12.2% to 90.1% as the router fires more specialists. Nobody chooses that number, and it is rarely reported. The result held on a seed the system had never seen, and we are writing it up.

The test suite reflects the same worry: 54 tests, most of them aimed at catching our own measurement errors.

A gauge from the ThirdEye project site showing a measured false-alarm rate of 80.7% when 5 of the 8 specialists run
The gauge on the live project site: the false-alarm rate on the same safe contracts, here with 5 of the 8 specialists fired.

07 · What I took from it

Measuring is a design decision.

  • Lesson

    A model's claim is worth what it can point to. Demanding the quoted line did more than any prompt tweak.

  • Lesson

    Report the number that makes you look worse. It is the one people can actually plan around.

  • Lesson

    Hold out a seed. A result you have not replicated is still a hunch.

Next case study

ADA: an adaptive RAG assistant for research papers

Need a developer
who ships?

Talk to me.

Open to SDE internships and 2027 full-time roles. Remote or relocation.

Download resume
contact.json
1{2  "name": "Tarun Basavaraju",3  "title": "Software engineer",4  "email": "tarunbasavaraju25@gmail.com",5  "phone": "+91 98458 59937",6  "location": "Bengaluru, India"7}