OCOPD Anthropic pushes AI speed limits as capability race intensifies
Anthropic CEO Dario Amodei is calling for AI speed limits and stronger government intervention, arguing that frontier systems could soon improve their own successors faster than companies, regulators and researchers can maintain control. In the warning described by The Decoder, Amodei said recursive self-improvement could pose a threat to the broader internet within six to twelve months.
The proposal would not amount to a permanent halt to artificial-intelligence research. Instead, it seeks a mechanism that would allow leading laboratories and governments to slow or temporarily pause development when systems reach a level of risk that existing safeguards cannot reliably manage.
Amodei’s intervention arrives as Anthropic remains one of the small group of companies at the leading edge of AI development—and as the company approaches what could become the largest initial public offering in history. That timing gives the appeal both greater visibility and a potential commercial complication: Anthropic is asking an industry in which it competes to accept constraints on the same advances that underpin its valuation and growth prospects.
Recursive self-improvement is the central concern
Anthropic defines the risk around a potential threshold at which AI systems can design, train or deploy improved successors with limited human input. If capability gains become rapid and self-reinforcing, safety testing and institutional decision-making could fall behind.
The company has pointed to the cyber-security implications of advanced models as an early indicator of the stakes. In testing, Anthropic’s Mythos model reportedly identified and exploited thousands of bugs and software defects in major operating systems and browsers. The system was not released widely, but Amodei said a hostile state could have used comparable capabilities to attack other countries or support military operations.
In an interview with ABC News, Amodei said the government should have the authority, in a narrow and targeted way, to block or deter deployment when independent assessment finds that a model presents unacceptable risks. “The onus primarily falls on us,” he said, while also arguing that voluntary company commitments are not sufficient on their own.
Proposed safeguards would extend beyond voluntary pledges
Amodei’s recommendations include several layers of oversight:
- Embedded auditors inside major AI companies to monitor development and deployment decisions.
- Shared safety standards that make evaluations more consistent across competing laboratories.
- Third-party assessments capable of identifying unacceptable risks before public release.
- International agreements modeled loosely on arms-control arrangements such as the SALT treaties.
- A government option to pause or block unsafe deployments under narrowly defined conditions.
Anthropic has also announced a $200 million investment in research on AI’s effects on society, according to ABC News. The company’s broader position is that governments and industry need time to build alignment research, monitoring systems and emergency procedures before capabilities advance beyond effective oversight.
Coordination is the biggest political obstacle
A coordinated slowdown would be difficult to enforce. The leading AI companies operate in several countries, compete for the same customers and researchers, and face pressure from investors and governments to improve models quickly. Any agreement would also need credible verification so that participants could demonstrate they had actually limited development.
Scientific American described Anthropic’s proposal as technically significant but politically fraught, noting that no treaty currently obliges frontier laboratories or governments to accept a pause. Noah Giansiracusa, a mathematics professor at Bentley University, questioned whether the proposal represents a genuine desire to slow progress, suggesting that Amodei may instead be seeking to shape how the race proceeds while Anthropic continues to advance.
The international dimension adds another layer of complexity. Amodei has said he does not trust China to observe a unilateral pause, raising the risk that safety measures could be viewed as a strategic disadvantage if major powers do not participate together.
Implications for competitors, users and regulators
For rival laboratories, the proposal increases pressure to disclose safety practices and accept more formal external scrutiny. It could also intensify debate over whether companies developing powerful models should help write the standards governing them—or be subject to rules set independently by governments.
For users, stronger pre-deployment testing could reduce exposure to cyber, fraud and other misuse risks, but a slowdown could delay new AI products and productivity tools. For regulators, Amodei’s warning reinforces the case for mandatory reporting, model evaluations and deployment controls rather than reliance solely on corporate promises.
The immediate effect is unlikely to be a global pause. Its larger impact may be political: Anthropic is making the possibility of capability-based limits part of the mainstream debate at a moment when the industry’s commercial incentives still favor speed.
Source
Original source: the-decoder.com