A startup claiming to fix AI’s most embarrassing failure—the widespread contamination of public benchmarks—just raised $40 million from a16z at a $400 million valuation. Vals AI promises to evaluate models on real-world tasks pulled from GitHub pull requests, not static datasets. It sounds like a savior for a field drowning in inflated metrics. But tracing the code back to its genesis block, I see a familiar pattern: a centralized oracle with a conflict of interest, dressed in algorithmic objectivity.
Context: The Benchmark Crisis and the Rise of Evaluation-as-a-Service
The AI industry has a dirty secret. Benchmarks like GSM8K, HumanEval, and MMLU have been systematically poisoned—model trainers simply memorize the test sets. This is well-known among researchers. I’ve seen it in crypto audits: smart contracts rigged to pass test vectors but fail in production. The same phenomenon now plagues AI. Vals AI’s solution is to treat evaluation like a continuous security audit: generate tasks from live codebases, hide the tests, and score models on actual developer problems. They claim OpenAI, Anthropic, Google, Meta, and xAI already cite their results in model cards. That’s a big claim. Where liquidity flows, truth eventually pools—but only if the pool is deep enough.
Core: The Mechanism Behind the Hype
Vals AI’s technical innovation is not in model architecture but in infrastructure. They extract historical pull requests from any GitHub repository, use hidden tests to judge completion quality, and cover domains like finance, law, and medicine. This is essentially SWE-bench turned into a product. It’s a clever engineering play—composability is a double-edged sword. On one hand, it lowers the barrier for enterprises to evaluate models on their own code. On the other, it inherits all the risks of relying on public repositories.
From my experience auditing smart contracts, I immediately flash-red on data contamination. If the training data for a model included public GitHub repos, then tasks extracted from those same repos are not truly out-of-sample. Vals claims to use “hidden tests” and possibly private repositories, but they haven’t disclosed their methodology for avoiding overlap. Decoding the signal hidden in the noise requires more than a slide deck. The $400 million valuation implies a16z is buying a category, not a company. Third-party evaluation is a necessary infrastructure layer, but the market is still unproven. The revenue claim—“8x growth this year”—is phrased ambiguously. In my 2017 audit days, I learned to treat any unaudited self-reported metric as noise until proven otherwise.
Contrarian: The Oracle Problem of AI Evaluation
Here’s the uncomfortable truth: Vals AI is a centralized oracle for AI quality. It evaluates models for the same companies that could become its paying customers. OpenAI, Anthropic, and Google are both clients and subjects. This is a structural conflict of interest worse than any audit firm auditing a company it also consults. In crypto, we learned that oracles must be decentralized to be trusted. Chainlink succeeded because no single party controlled the data feed. Vals AI, by contrast, sits in the middle, deciding what constitutes a “real-world task” and how to score it. If a model fails, can Vals afford to publish that result when the model maker is a major revenue source? Follow the smart contract, ignore the whitepaper—the real incentive structure is hidden in the cap table. a16z’s portfolio companies will be pressured to use Vals, creating a walled garden of “verified” models that excludes competitors. This is not independent evaluation; it’s strategic compliance.
Moreover, the technical risk of benchmark contamination remains unsolved. Vals’s tasks are derived from GitHub, which is public. If a model’s training data includes the same codebase, the test is compromised. The company hasn’t proven it can generate truly private, non-leakable tasks at scale. Human review for legal and medical domains is expensive, and they haven’t disclosed their quality control process. Bubbles burst, but architecture remains—the architecture of centralized evaluation is fragile.
Takeaway: The Fork in the Road
The AI industry needs a trust layer for evaluation, but Vals AI’s centralized model is a temporary fix. The real solution might be a blockchain-based registry of evaluation tasks, where each test is hashed, timestamped, and provably never seen by the model. Vals could become the user interface for such a decentralized protocol, but that’s a pivot they haven’t announced. As an analyst, I’m watching whether they embrace verifiability or double down on proprietary control. The next narrative shift will be from “trust us, we’re independent” to “trust the code, not the company.” Until then, treat every model card citation with cryptographic skepticism.