Original post · AI translation
这张图展示了人工智能的工作是如何接受检查的。
简单来说，你永远不必信任人工智能。
人工智能代理编写证明，但严格的数学检查器会剔除任何错误内容，因此你只需信任人类批准的安全规则，而不必信任人工智能本身。https://t.co/Ha174MKW4k
Google DeepMind 取得了一项重大成就：它展示了一个人工智能系统如何从数学上证明软件是安全的，而无需信任人工智能。
> 他们刚刚用人工智能代理从数学上证明了 Signal 实际运行的代码能让消息保持私密。
Google 的人工智能代理编写了 650 万行数学证明，证明 Signal 编译后的代码能让聊天内容保持私密。
检查深入到了处理器实际执行的指令层面。该实验室表示，此前还没有人对任何广泛使用的协议做到这一点。
> 常规测试会尝试许多输入，并寄希望于碰到有问题的那个。证明则表明安全规则在所有情况下都成立，而 Lean 证明检查器会拒绝人工智能写出的任何错误步骤。
> 设计转化为代码时，真实的漏洞常会悄然出现，因此检查处理器层面的代码能弥合这一差距。Rust 编译器同样不被信任，所以编译器犯的错误也会被发现。
> 这项工作耗时 4 周，而过去专家团队需要数年。由 40 至 80 个 Gemini Carbon 代理组成的团队编写了约 650 万行 Lean 代码，人类则设定安全目标，并审查代理标出的 1%。
> 为了检查该系统是否会对任何代码都予以批准，研究人员将 Signal 过去 2 个已修复的漏洞重新加入一个副本。两次它都生成了经机器检查的证明，证明代码不安全。
> 更大的转变在于成本：如今人工智能可以在数周内生成经机器检查的证明，而不再需要耗时数年的项目，并使这类证明成为设备固件、电网软件及其他关键代码的常规检查。
Original
Big achievement by Google DeepMind for showing that an AI system that proves a software is secure mathematically, with no need to trust the AI. > They just used AI agents to mathematically prove that Signal’s real running code keeps messages secret. Google's AI agents wrote 6.5M lines of math proving Signal’s compiled code keeps chats private. checked at the level of the instructions a processor actually runs. The lab says no one had done this before for any widely used protocol. > Normal testing tries many inputs and hopes to hit the bad one. A proof shows the security rules hold in every case, and the Lean proof checker rejects any wrong step the AI writes. > Real bugs often creep in when a design gets turned into code, so checking the processor-level code closes that gap. The Rust compiler isn’t trusted either, so a mistake by the compiler would also get caught. > The work took 4 weeks, where expert teams used to need years. Teams of 40 to 80 Gemini Carbon agents wrote about 6.5M lines of Lean, while humans set the security goals and reviewed the 1% the agents flagged. > To check that the system can’t just approve any code, the researchers put 2 old, already fixed Signal bugs back into a copy. Both times it produced a machine-checked proof that the code was insecure. > The bigger shift is cost, now that AI can produce machine-checked proofs in weeks, rather than years-long projects and become routine checks for device firmware, power grid software and other critical code.
宣布我的 @GoogleDeepMind 团队取得的一项重大里程碑：我们已经证明了支撑 Signal 消息协议的机器代码的关键安全属性。这是广泛部署的协议所使用的实际生产代码首次得到形式化验证——无论采用什么方法、有无人工智能参与——并且验证达到了这样的深度。
这不仅是一项研究突破，也是前沿人工智能如何成为力量倍增器、帮助应对现代网络威胁的速度和规模的典型例子。
形式化验证提供了更高的安全标准；但它极其复杂且耗时，因此在许多情况下被认为不切实际。类似工作可能需要团队累计投入数十年。我们在一个月内完成了。
我们使用 AlphaProof Code——Google DeepMind 新推出的、基于包括 Gemini Carbon 在内的大语言模型运行的形式化验证代理框架——将 Signal 协议的验证深入到设备上实际运行的代码。这一里程碑为未来奠定了基础：从医疗设备固件到电网软件，重要系统在交付时都能附带数学证明，证实其保护措施有效。
要实现这样的未来，还需要我们这个领域持续开展研究。本着这一精神，我们正在开源形式化规范和验证成果，以便其他人在这项工作的基础上继续推进。
就个人而言，这一刻对我意义深远。20 多年前，我以实习生身份开始自己的研究之路，从事复杂软件系统一致性检查方面的工作。看到现代人工智能让这一重大挑战变得切实可行，实在非同寻常。
我为团队成员 Swarat Chaudhuri、Andrew Ferraiuolo、Anton Kovsharov、Tej Chajed、Sam Schlesinger、Karn Seth、Jonathan Katz 和 Charles Sutton 感到无比自豪，他们迎接了这一重大挑战并取得成果；我也感谢 Signal 团队的合作。
https://t.co/kN1eRYQ8fF
Original
Announcing a major milestone achieved by my team @GoogleDeepMind: we have proven key security properties of the machine code underlying the Signal messaging protocol. This is the first time the actual production code of a widely deployed protocol has been formally verified – by any method, with or without AI – and at this depth. More than a research breakthrough, it’s a prime example of how frontier AI can be a force multiplier to counter the speed and scale of modern cyber threats. Formal verification offers a higher standard of security; however, it’s extremely complex and time consuming, which is why in many cases it’s considered to be impractical. Similar efforts could take teams decades of combined effort. We did it in a month’s time. Using AlphaProof Code, Google DeepMind's new agentic framework for formal verification that runs on LLMs including Gemini Carbon, we verified the Signal protocol down to the actual code running on devices. This milestone lays the groundwork for a future in which vital systems, from medical device firmware to power grid software, ship with mathematical proof that their protections work. Reaching that future will take continued research from our community. In that spirit, we're open-sourcing our formal specifications and verification artifacts so others can build on this work. On a personal note, this moment is of deep significance to me. I started my own research journey more than two decades ago as an intern working on checking the consistency of complex software systems. Seeing this grand challenge made practical with modern AI is extraordinary. I'm extremely proud of my team members: Swarat Chaudhuri, Andrew Ferraiuolo, Anton Kovsharov, Tej Chajed, Sam Schlesinger, Karn Seth, Jonathan Katz, and Charles Sutton for taking on this grand challenge and delivering, and grateful to the Signal team for their partnership. https://t.co/kN1eRYQ8fF