AI could soon lead to systems capable of improving their own performance faster than humans can effectively supervise them, reviving concerns about the industry’s longstanding “alignment problem,” ensuring AI systems reliably pursue human goals, senior Anthropic researchers have warned in a new blog post titled “When AI builds itself.”

Anthropic Institute lead Marina Favaro and Anthropic co-founder Jack Clark outlined three possible futures: growth in AI capabilities may flatten out; AI efficiency gains may continue to grow, but expose bottlenecks elsewhere in software development; or AI systems may become capable of full recursive self-improvement, and build their successors by themselves. It’s that third scenario that’s prompting them to suggest society be ready to hit the brakes on AI development.

“How the alignment problem gets solved — or not — in this future is something we are least certain about,” they wrote. Advanced, self-improving models could follow our needs and wants — or, they warned, “The rare occurrences of misalignment present in today’s models could compound as the models build their successors, growing more frequent but less understood until we lose control of them. It’s possible that we can’t build, integrate, and verify the tools that we’d need to understand which trendline we are actually on.”

While Anthropic’s warning is framed around future AI development, analysts say it highlights governance questions enterprises are already beginning to confront as autonomous AI agents move from answering questions to taking actions.

“The issue is no longer just whether AI gives the right answer, but whether autonomous systems take the right action, at the right time, within the right authority,” said Ashish Banerjee, senior principal analyst at Gartner.

From model governance to agent governance

The warning comes amid growing enterprise investment in agentic AI.

Gartner predicts that by 2028, 15% of day-to-day work decisions will be made autonomously through agentic AI and that one-third of enterprise software applications will incorporate agentic AI capabilities. The firm has also warned that governance shortcomings are already emerging, predicting that 40% of enterprises will demote or decommission autonomous AI agents by 2027 after governance failures become apparent in production environments.

Banerjee said many organizations continue to approach AI agents as advanced productivity tools when they increasingly resemble digital workers operating with delegated authority.

“CIOs should stop treating AI agents as smarter chatbots,” he said. “They are becoming digital workers with delegated authority — and must be governed like privileged users, not productivity tools.”

As agents gain the ability to conduct research, write code, invoke tools, trigger workflows, and make recommendations, enterprises face new risks around unauthorized actions, accountability gaps, data exposure, tool misuse, and insufficient auditability, Banerjee said.

“Human-in-the-loop is not a strategy if the human cannot keep up with the loop,” he said.

Charlie Dai, vice president and principal analyst at Forrester, said Anthropic’s concerns mirror challenges enterprises are already encountering as AI systems gain greater autonomy.

“Alignment becomes operational,” Dai said. “It is about ensuring agents consistently act within policy, not just model accuracy.”

Current governance approaches focus largely on models and data, but increasingly autonomous agents require oversight of runtime behavior, permissions, tool usage, and decision boundaries, Dai said.

Concerns about agent oversight are not limited to AI vendors and industry analysts.

In AI Agent Governance: A Field Guide, researchers from Institute for AI Policy and Strategy warned that “society is largely unprepared for this development” and said “the exploration of agent governance questions and the development of associated interventions remain in their infancy.” The paper argues that advances in autonomous AI agents are outpacing the governance mechanisms needed to oversee them.

Both analysts argued that governance frameworks originally designed for generative AI models may prove insufficient for increasingly autonomous systems. Dai said organizations will need greater oversight of runtime behavior, permissions, tool usage, and decision boundaries as agents become more capable.

Why Anthropic is worried

Anthropic’s researchers argue that those governance questions could become significantly harder if AI systems become increasingly involved in the process of AI research and development itself.

Favaro and Clark stopped short of predicting that fully autonomous recursive self-improvement is inevitable. Instead, they argued that the possibility warrants preparation and discussion among developers, policymakers, and other stakeholders. They also suggested the industry may eventually need mechanisms to slow development if capabilities begin advancing faster than safeguards, while acknowledging that such measures carry risks of their own.

“But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe,” they wrote in the blog post.

Forrester’s Dai said the practical implication for enterprises is that governance can no longer depend primarily on human review.

“Supervision becomes architectural, not manual,” he said. Organizations will increasingly need bounded autonomy, embedded guardrails, verifiable execution mechanisms, and fallback controls designed into agentic systems from the outset.

Over the next three to five years, both governments and the private sector will need to rapidly adapt identification and mitigation protocols as adversaries move from AI-assisted to AI-enabled sanctions evasion and proliferation financing (PF), a new research paper warns.

The report, Algorithms of Evasion: The Rise of AI-Enabled Proliferation Financing, from the Royal United Services Institute (RUSI), a UK-based defense and security think tank, defines PF as the use of funds or financial services to acquire, develop or otherwise deal in weapons of mass destruction (WMD). It states, “North Korea and Iran are now developing and deploying AI models to aid with sanctions evasion activities.”

Key findings include the fact that AI is now capable of mass producing high-quality fraudulent documents, as well as automating what the report describes as “the administrative minutia of managing extensive shell company  networks.” AI powered systems, it states, can also “analyze blockchain patterns in real time to dynamically adjust cryptocurrency mixing strategies, effectively evading detection tools.”

In addition, it says, “[tools such as generative AI] which can produce sophisticated fraudulent identification documents, for example, have helped North Korea perpetrate phishing attacks against Western companies.”

Dr. Aaron Arnold, senior associate fellow with the Centre for Finance and Security at RUSI, who authored the paper, said in an email that what prompted it was an uptick over the last year in North Korea’s use of AI to facilitate and enhance its cyber operations, in the form of phishing schemes designed to generate revenue for the country’s ballistic missile and nuclear weapons programs.

He advised enterprise IT managers who need to protect their organizations from becoming victims of sanction evasion activities that “[it] means largely adapting to a landscape where traditional human-focused security boundaries are being bypassed by automated technologies.”

For IT managers, said Arnold, “this might entail incorporating defensive AI, the use of behavior-based analytics, using ‘circuit breakers’ when there is heavy use of API or MCPs, updating personnel training, and hardening identity verification, especially for any remote hiring.” 

Distinction between AI-assisted and AI-enabled activity is ‘central’

Sanchit Vir Gogia, chief analyst at Greyhound Research, said that the RUSI report matters “because it names the right structural shift. AI is not creating sanctions evasion from thin air, it is compressing and scaling methods that already work.”

He pointed out that none of the sanction-evading techniques such as fraudulent documents, synthetic identities, shell companies, hidden beneficial ownership, crypto laundering, and others are new. “What changes is the speed, quality, volume and coordination with which these methods can now be assembled,” he said.

According to Gogia, “the distinction between AI-assisted and AI-enabled activity is central. AI-assisted evasion uses AI for discrete tasks: writing a better email, producing a cleaner document, generating a stronger false profile, translating a pitch, summarizing regulations or preparing a plausible job application. AI-enabled evasion is more serious.”

A ‘structural asymmetry’

This tactic, he said, “begins to coordinate the system itself. It links identity, documents, ownership structures, payment routes, cloud access, crypto wallets, API calls and timing. The difference is not whether AI helps someone fake a document. The difference is whether AI begins to orchestrate the deception.”

That is why the report’s findings should worry enterprise leaders, he noted: “Many organizations still assume the bad actor is mostly human, mostly linear and mostly slow. That assumption is expiring. AI lets adversaries run more attempts, with fewer errors, across more channels, in more languages, with better paperwork and greater patience than most enterprise review processes can absorb. This is not a tale of genius criminals discovering magic. It is the story of ordinary controls meeting industrialized plausibility.”

The evidence today, he pointed out, is strongest around tactics such as identity fraud, document fraud, synthetic personas, remote-worker deception, phishing, social engineering, crypto obfuscation and workflow abuse. “Fully autonomous evasion networks sit on the horizon,” he said. “They are serious, but they are not yet the everyday baseline.”

This distinction matters, said Gogia: “If enterprises obsess over cinematic autonomous agent scenarios while leaving remote hiring, vendor onboarding, payment approvals, and document review full of holes, they will lose in the most prosaic way imaginable.”

The report, he said, also gets the “asymmetry” right. “Offensive actors can learn across the ecosystem,” he said. “They can scrape open information, reuse leaked records, study enforcement patterns, test onboarding forms, inspect public procurement data, watch court filings, probe compliance thresholds and [use the information to] refine their behavior.”

Defenders, by contrast, are hemmed in by privacy rules, fragmented data, explainability requirements, jurisdictional boundaries, conservative operating models and siloed technology estates. “Offensive AI learns broadly,” he said. “Defensive AI often learns from fragments. That is the structural asymmetry.”

He explained that the regulatory landscape also amplifies the problem, in that regulatory bodies “still speak in separate dialects. [For example] the EU AI Act pushes organizations toward stronger obligations for high-risk AI. NIST-style frameworks push risk management, transparency, and governance.”

A trust architecture problem

Financial Action Task Force (FATF) expectations push national risk assessment and counter-proliferation controls, he noted, while banking regulators focus on model risk, accountability and operational resilience. “None of these streams is irrelevant. The trouble is that criminals do not organize themselves around regulatory workstreams. They organize around outcomes.”

What that means, said Gogia, “is that enterprise cannot wait for a clean global rulebook. It will not arrive in time. CIOs, CISOs, compliance officers and boards need a working governance model now. They need privacy-preserving analytics, controlled data environments, audit trails, legal safeguards and clear model-risk accountability.”

He said that enterprise IT managers should treat the situation as a trust architecture problem rather than a narrow sanctions-screening problem. “The uncomfortable truth is that AI is not simply helping bad actors write better phishing emails or forge tidier documents,” he noted. “It is helping them manufacture legitimacy across a chain of enterprise workflows.”

Likely outcome an ‘AI arms race’

Report author Arnold also noted that there are signs that cyber criminals have discovered new AI technologies and abilities that legitimate enterprises could adopt for legitimate applications.

History, he said, “is replete with [criminals] developing novel solutions to tough problems, [which are] later adopted by law enforcement. Much of our anti-financial crime policy is effectively a response to bad actors exploiting systems or using technology in novel ways to perpetrate crimes. In this scenario, I think an ‘AI arms race’ between enforcement authorities and bad actors is the most likely outcome.”

Gogia added, “the baddies are not teaching enterprises how to invent AI. They are teaching enterprises where trust is leaking. That is the lesson worth taking seriously.”

This article originally appeared on CIO.com.