New AI System Automates the Math Behind Software Safety Checks

The era of manually auditing every line of critical software code may be ending as artificial intelligence begins to handle the mathematical proofs that guarantee its safety. This development marks a significant shift in how developers approach software reliability, moving beyond simple testing toward rigorous, automated verification.

Anthropic, the developer behind the Claude large language model, has partnered with Theorem to deploy an AI system capable of generating formal proofs for existing code. Unlike previous AI coding tools that primarily wrote new software, this system focuses on verifying that established code works correctly under all possible conditions. The project represents a major step in automating one of the most labor-intensive aspects of software engineering.

Formal verification has historically been reserved for high-stakes applications like avionics or operating system kernels because it is so resource-intensive. For instance, a team of specialists previously required 20 person-years to formally verify the seL4 operating system kernel. By applying Claude to this task, Anthropic and Theorem have dramatically compressed that timeline, suggesting that rigorous safety checks could become a standard part of the software development lifecycle.

In the recent trial, the AI system processed test programs from 123 open-source libraries within Google’s OSS-Fuzz security project. Over a period of just a few weeks, Claude generated approximately 180 million lines of mathematical proof code. This output was not arbitrary; the model translated existing crash tests into mathematical problems and analyzed compiled machine code to produce extensive proofs that demonstrate the code’s correctness.

The process is designed to be largely autonomous. An independent verification program called Rocq checks the proofs generated by Claude, requiring minimal human intervention. While the underlying formal foundations and tools were built by human engineers at Theorem, the actual generation and checking of the proofs are handled by the AI. This level of automation addresses a key bottleneck in software safety, where human reviewers often struggle to keep up with the volume of code in modern systems.

A significant benefit of this approach is the discovery of hidden defects. While generating proofs, Claude identified hundreds of bugs that had evaded conventional testing methods. Many of these issues only manifest under specific conditions, such as unusually large input files or particular sequences of data access. In one notable instance, the team found as many security bugs in a widely used PDF reader in a single week as are typically reported for that software in an entire year.

However, the results are not without limitations. Anthropic notes that "formally verified" does not equate to "error-free" in a general sense. The proofs are valid only under strict constraints, including specific processor architectures, single-threaded execution, and input sizes under 64 MB. In real-world scenarios, these libraries often run on diverse hardware and configurations, meaning the verification covers a subset of potential use cases rather than every possible environment.

The computational cost of this automation is substantial, with individual proofs consuming hundreds of millions of tokens. Despite this, the efficiency gains are evident in newer models; Opus 5.5, for example, solves half the required proofs with roughly 40 percent fewer tokens than its predecessor. As models become more efficient, the cost of verification may decrease, making it viable for a broader range of software projects.

If this technology matures, it could transform the software industry by shifting the focus from writing code to proving its safety. Anthropic expects formal verification with AI to become as routine as standard testing in the near future. Theorem’s roadmap already targets critical infrastructure components like glibc, OpenSSL, and cURL, indicating that this method is poised to play a central role in securing the digital backbone of the internet.

For California’s tech ecosystem, which relies heavily on open-source software, this advancement offers a path to more robust and trustworthy systems. By automating the most difficult parts of quality assurance, developers can spend less time on manual review and more time on innovation, potentially accelerating the pace of software development across the region.