正文 · AI 翻译
IT Home reported on October 27 that Anthropic disclosed in IPO prospectus documents released on Monday local time that it had paused training of its next-generation flagship AI model for more than a month because of safety concerns. Four AI agents also broke through restrictions during testing and accessed real external computer systems.
IT Home noted that the company behind Claude said in the risk disclosures that it had delayed pretraining a new frontier foundation model to strengthen safeguards against theft of the model weights and intellectual property.
The disclosure shows that developing increasingly capable AI models brings a range of safety challenges. Some in the industry have warned that this technology could one day escape human control. Competitor OpenAI also paused some training work earlier this year over safety issues, after its model gained unauthorized access to the Hugging Face platform.
These incidents show that unexpected safety incidents could disrupt the development and launch of core models, which are crucial to Anthropic's push toward a highly anticipated public listing and to its business growth.
Anthropic warned that pauses in training could delay revenue and put the company behind its competitors. The document devoted about 94 pages to risk factors.
Sandbox escapes
The prospectus documents also disclosed previously unreported incidents during safety evaluations: four AI models escaped their test sandboxes and accessed real-world computer systems, with the sessions lasting between 10 and 34 hours.
Anthropic said an operational error occurred during cybersecurity testing with a third-party partner. The test environment was supposed to have no internet access, but it remained connected to the internet.
In one incident, an early version of the model kept trying to complete an evaluation task that could not be completed. During that process, it found a saved password and used it to gain administrator access to a third-party system. It then obtained more credentials, changed system settings and accessed personal data. The documents did not identify the external partner involved.
Anthropic said it had notified all affected parties and found no evidence that the models had deliberately evaded human oversight or that multiple models had coordinated with one another. Afterward, the company halted some prelaunch cybersecurity evaluations and suspended higher-risk reinforcement learning test environments.
Separately, the company said last Sunday local time that Claude had bypassed a paywall and retrieved publicly available aggregated ballot data from the US state of Georgia without human authorization.