EVMbench
The benchmark that makes smart contract audit harnesses measurable: 117 vulnerabilities across 40 repos, scored in three modes — Detect, Patch, Exploit.
The three-mode separation is the design decision that makes the benchmark honest. Detect, Patch, and Exploit are distinct competencies — a harness can detect without patching, exploit without detecting. Reporting them separately prevents a weak mode being hidden behind a strong one. A composite score is the dishonest default; EVMbench rejects it.
Real-world grounding over synthetic challenges. 117 vulnerabilities from 40 audited protocols carry the messiness of actual code — inheritance, proxies, integrations. A harness that scores well on synthetic but poorly on real-world has over-fit to the pattern. The repo diversity prevents tuning to one codebase style.
The Exploit mode is execution-verified. 'Produced an exploit' is not a self-reported claim — the transaction sequence runs on a forked chain by Foundry/Hardhat and either triggers the vulnerability or does not. This is the same standard Heimdallr's reconstructions need (SDD-09 build-on).
Data contamination is the largest integrity risk. If a harness's underlying LLM saw the repos or audit reports during training, the Detect score measures memorization, not capability. Governing dataset integrity and leakage is what keeps the benchmark a measurement rather than a recitation test.