正文 · AI 翻译
IT Home reported on November 3 that most attention has focused on the latest financial figures disclosed in Anthropic’s updated S-1/A prospectus supplement on Monday (revenue soared, and spending surged as well), but the artificial intelligence lab tucked another major revelation into the document’s fine print.
The filing appears to warn that the company is close to developing a model capable of recursive self-improvement (RSI).
Recursive self-improvement refers to a stage at which AI’s ability to train itself reaches an extremely high level, driving explosive growth in its own capabilities. To AI risk pessimists, recursive self-improvement is an irreversible disaster: AI will evolve until it is powerful enough to escape human control and may be indifferent to the harm it causes while pursuing its own goals.
AI optimists, meanwhile, see recursive self-improvement, or a related form in which AI steadily optimizes itself, as a sign of the technology’s enormous potential. At that point, it would become a powerful tool for unlocking extraordinary productivity, aiding scientific discovery and much more.
Anthropic’s position is that recursive self-improvement presents both opportunities and risks. The company wrote in its prospectus that it has already used AI in its research and development, while cautioning: “Using AI to develop more capable AI may also create significant safety risks, especially if it greatly accelerates the pace of technological progress. This is one of the areas covered by our Responsible Scaling Policy, which sets out plans to monitor and mitigate risks as we gradually approach what is known as ‘recursive self-improvement.’”
IT Home noted that when the concept of recursive self-improvement gained popularity earlier this year, many people regarded it as merely a buzzword or, at best, a goal that would be difficult to achieve, define or verify.
But on Monday, Anthropic added language to the risk disclosures making two main points: first, the company has already used AI to develop new models, some of which made leaps in capability that the team did not notice at the time; second, the company may soon meet its own definition of recursive self-improvement:
In addition, we are increasingly using AI systems to accelerate our own research and development, which could cause model capabilities to improve faster than we can develop, validate and deploy corresponding safety, alignment and safeguard measures. Consequently, we have previously failed to detect substantial increases in the pace of research and development in time, and we may again notice such acceleration only after it has occurred.
For example, based on internal estimates and expectations for our next-generation models (rather than existing models), we may soon reach what our Responsible Scaling Policy defines as “explosive technological progress driven by AI.” We currently believe that no existing model has crossed this threshold, and the underlying data remains under ongoing evaluation.
Anthropic also stated in the filing, however, that none of its models has yet crossed the threshold for recursive self-improvement, and pledged to keep humans involved in the development process for the foreseeable future. “Even if it becomes technically possible in the future, we do not intend to allow Claude to design and iterate on frontier large models entirely autonomously without effective human oversight or a sufficient understanding of the processes involved. We are therefore considering revisions to our Responsible Scaling Policy to ensure that humans remain in control of frontier model development.”
Skeptics would point out that an S-1 prospectus serves both disclosure and promotional purposes, and that claims of an imminent, exceptionally powerful model capable of improving itself may simply be promotional rhetoric.
But there is no denying that Claude’s capabilities have advanced dramatically this year, particularly in programming, mathematics and cybersecurity. If Anthropic judges recursive self-improvement to be close at hand, perhaps we should take that judgment seriously and act on it by establishing appropriate regulatory rules.