OpenAI Launches EVMbench to Detect, Patch, and Exploit Vulnerabilities in Blockchain Environments
OpenAI, in collaboration with Paradigm, has released EVMbench, a benchmark designed to assess AI agents' capabilities in identifying, repairing, and exploiting high-severity vulnerabilities in smart contracts. This benchmark is crucial for evaluating AI…
OpenAI, in collaboration with Paradigm, has released EVMbench, a benchmark designed to assess AI agents' capabilities in identifying, repairing, and exploiting high-severity vulnerabilities in smart contracts. This benchmark is crucial for evaluating AI performance in economically significant environments, given that smart contracts secure over $100 billion in crypto assets.
EVMbench leverages 120 curated vulnerabilities from 40 security audits, mostly sourced from open code audit competitions. It also includes scenarios from the Tempo blockchain's security audit process, targeting payment-oriented smart contract code.
EVMbench evaluates AI agents through three modes, each focusing on a different phase of the smart contract security lifecycle:
Detect: Agents audit smart contracts and are scored on the recall of vulnerabilities and audit rewards. Patch: Agents modify vulnerable contracts to preserve functionality, verified through tests and exploit checks. Exploit: Agents perform fund-draining attacks in a sandboxed environment, graded through transaction replay and verification.
EVMbench leverages 120 curated vulnerabilities from 40 security audits, mostly sourced from open code audit competitions.
A Rust-based harness supports reproducible evaluations by deploying contracts deterministically and restricting unsafe RPC methods. All exploit tasks are conducted in an isolated environment.
Performance on EVMbench varies across tasks. In exploit mode, GPT‑5.3‑Codex scored 72.2%, significantly higher than GPT‑5's previous score of 31.9%. While agents excel in exploit tasks, detect and patch modes remain challenging due to incomplete audits and difficulty in removing subtle vulnerabilities.
EVMbench does not fully represent real-world smart contract security complexities and cannot distinguish between true vulnerabilities and false positives beyond the human-auditor baseline. OpenAI has allocated $10 million in API credits to its Cybersecurity Grant Program to advance defensive security research.
Additionally, OpenAI announced an expansion of its security research agent, Aardvark, through a private beta program. EVMbench's components are publicly available for further research into AI-driven cybersecurity capabilities.
Based on reporting by Cyber Security News.
