The data point landed on my desk like a grenade. CyberGym, a security firm with a name that sounds like a startup from a 2018 accelerator, claims its AI detects vulnerabilities with over 90% accuracy. The number was published on Crypto Briefing, a venue that usually covers token launches and regulatory theater. No methodology. No benchmark. No falsifiability. Just a number that would, if true, rewrite the economics of smart contract auditing. But I have spent 20 years in this industry—first auditing Zcash’s shielded transactions in 2018, then managing a $2M DeFi fund through the 2020 yield farming frenzy, and later designing data integrity frameworks for autonomous agents in 2026. I have learned one immutable law: code does not lie, only developers do. And when a single source claims to have solved a problem that has resisted every major security vendor for a decade, I reach for my on-chain data tools, not my applause.
This is not about CyberGym. It is about the broader pattern of AI-over-hype in blockchain security, where marketing narratives replace technical rigor. The stakes are high: smart contract vulnerabilities still cost billions annually, and the promise of AI-driven audit could either democratize security or create a false sense of safety that leads to even larger exploits. I will dissect this claim through the lens of my own empirical framework—the same one I used to detect the inflated reserves in Terra’s algorithmic stablecoins in 2022, and the same one that saved my fund from 80% losses by liquidating before the collapse. Let the data speak.
Context: The State of Smart Contract Vulnerability Detection
To understand whether 90% is plausible, we must first understand the battlefield. Smart contract vulnerabilities fall into categories: reentrancy, oracle manipulation, integer overflow, access control, and logic errors. Each requires a different detection approach. Static analysis tools like Slither and Mythril can catch common patterns with high recall but suffer from false positive rates above 40%. Fuzzing tools like Echidna stress test invariants but miss logic flaws. Formal verification, used by protocols like Maker and Compound, provides mathematical guarantees but is expensive and slow—typically reserved for critical functions.
Enter AI. Since 2023, large language models (LLMs) fine-tuned on code have been touted as the next generation of auditors. GPT-4, CodeLlama, and specialized models like WizardCoder have been tested on benchmarks like SmartBugs and EvalPlus. The best published results show true positive rates around 60-70% on CWE Top 25 categories, with false positive rates exceeding 30%. That is not 90%. And those benchmarks are on curated datasets, not production codebases where obfuscation, complex state machines, and upgradeable contracts complicate detection.
CyberGym’s claim, if valid, would represent a 30% improvement over the state of the art, with no published architecture, no open-source code, and no independent replication. That is not impossible—it is improbable. Every gas fee tells a story of intent, and the intent behind this announcement is likely fundraising, not scientific contribution.
Core: Why the 90% Number Fails the On-Chain Test
Let me apply the same forensic discipline I used to trace the Zcash shielded transaction bug in 2018. That bug was a zero-knowledge proof implementation flaw that could have allowed balance inflation. I found it by systematically testing the consensus rules against the mathematical specification. The data did not lie—it revealed a mismatch that the whitepaper had obscured.
For CyberGym, I need three pieces of information that are missing from the announcement:
- False Positive Rate (FPR). If the model detects 90% of vulnerabilities but produces 40% false positives, the practical utility is zero. Security teams are already drowning in alerts. A tool that adds noise is worse than no tool. In my 2020 Curve pools analysis, I learned that volume-to-liquidity ratios matter more than raw yield. Similarly, precision matters more than recall in security. A 90% recall with 50% precision means every true vulnerability is buried under a false positive. The cost of triage outweighs the benefit.
- Test Set Composition. Was the test set sourced from open-source repositories like Solidity CTF challenges, or from real-world audited contracts? The difference is night and day. CTF challenges are designed to be solvable—they have clear vulnerabilities and no edge cases. Real-world contracts have upgradeable proxies, complex inheritance, and flash loan interactions. I have audited over 200 contracts in my career. The easiest vulnerabilities to detect are reentrancy and integer overflow. The hardest are business logic errors that require understanding the protocol’s economic incentives. If CyberGym tested only on easy categories, 90% is trivial. If they tested on all categories, they would need to publish their results for verification.
- Time to Detection. Did the AI detect vulnerabilities in live code, or in a static snapshot? In 2026, I built a framework for AI agents executing on-chain transactions. One of my key findings was that 30% of AI-driven trading errors came from manipulated oracle data—not code bugs, but data integrity issues. Vulnerability detection is not just about reading code; it is about understanding the runtime environment. A smart contract may be safe in isolation but vulnerable when interacted with a specific oracle or bridge. The context matters. The claim does not specify whether the AI considered on-chain state dependencies.
Based on my own experience with the 2024 ETF inflow correlation study, I know that institutional claims require multiple independent data sources. I aggregated data from ten custodians and wallet trackers before concluding that ETF inflows correlated with long-term holder accumulation. CyberGym provides a single data point from a single source. That is not a signal—it is noise.
Contrarian: Even If True, It Is a Double-Edged Sword
Let us assume, for the sake of argument, that CyberGym’s AI actually achieves 90% detection on real-world smart contracts. What then? The article itself warns of the risk: "exacerbating the risks of automated exploitation and patch verification." This is not a footnote—it is the main event.
In 2023, researchers demonstrated that GPT-4 could autonomously exploit known CVEs with over 80% success rate. The pipeline was simple: query the vulnerability description, generate exploit code, test it, and iterate. If CyberGym’s model can detect vulnerabilities with 90% accuracy, it can also be used to generate exploits with similar accuracy. The defense becomes the offense. The same tool that a security team uses to find bugs in their own code can be used by an attacker to find bugs in any code.
This is not theoretical. In 2025, I witnessed a case where an AI agent exploited a zero-day in a DeFi lending protocol within hours of the vulnerability being disclosed in a private audit. The attacker used a model fine-tuned on Solidity code to generate a valid exploit. The victim had not yet applied the patch. The window between disclosure and exploitation shrunk from days to hours. The industry is not prepared for this.
Moreover, the "patch verification" risk is real. If the AI can verify that a patch fixes a vulnerability, it can also verify that a patch does not fix it—or find a bypass. The security community is entering an arms race where both sides have access to the same tools. The only differentiator is the quality of the data and the speed of iteration.
I have seen this movie before. In 2022, when Terra collapsed, the narrative was that algorithmic stablecoins were the future. The data showed otherwise: the on-chain reserves were inflated, and the liquidity was fragmented. I liquidated 80% of my fund’s exposure in 48 hours because the data pointed to a structural failure. The same principle applies here: the data points to a structural failure in the AI security narrative. The claim is too good to be true, and the risks are too high to ignore.
Takeaway: The Next Signal to Watch
The market is currently in a bull phase, where euphoria masks technical flaws. Investors are FOMOing into AI security tokens and projects. My job is to remind them that every gas fee tells a story of intent, and the intent behind this announcement is likely to raise capital, not to improve security.
What should you watch? Three signals:
- Independent replication: If CyberGym publishes a paper with a reproducible benchmark, or if a third party like MITRE or SANS validates the 90% claim, the story changes. Until then, treat it as marketing.
- False positive rate: If the model is deployed in a real CI/CD pipeline and the security team reports a manageable alert volume, that is a positive sign. But the default assumption should be that the FPR is high.
- Attack surface increase: Track the number of AI-driven exploit attempts in the wild. If the number rises, the industry will be forced to adopt AI defense at scale. That is the real opportunity—not detection, but automated defense.
Standardization survives the chaos of collapse. The standardization of on-chain verification, combined with rigorous third-party auditing, is the only way to trust AI in security. The data does not lie. The 90% claim? It is a hypothesis that has not yet been tested. I will wait for the evidence before I adjust my risk models.
Ledger lines reveal what noise obscures. The noise is the hype. The ledger is the on-chain data. Until CyberGym opens its ledger, I remain skeptical. And so should you.