The US AI research company Anthropic has become known for building powerful AI models while simultaneously warning about their dangers.

Most recently, its executives wrote about the threat posed by “recursive self-improvement”. This is the point when AI systems can improve themselves by themselves, potentially leading to “superintelligence” far beyond human control.

“We are not there yet, and recursive self-improvement is not inevitable,” the Anthropic blogpost declared. “But it could come sooner than most institutions are prepared for.”

In fact, the idea of recursive self-improvement dates back decades. The British mathematician Irving John Good, who worked with Alan Turing at British codebreaking HQ Bletchley Park, warned in the mid-1960s of the “intelligence explosion” that would follow when a machine could design even better machines without human assistance. Good suggested “the first ultra-intelligent machine is the last invention man need ever make”.

In the 2000s, researcher Eliezer Yudowsky began building a community on the premise that recursive self-improvement and loss of human control could have catastrophic results, up to total human extinction (known as “x-risk”).

Yudkowsky discussed the idea of a “seed AI” that, while not very powerful, is designed for self-modification and able to read its own source code. This means it could generate new, better versions of itself which, in turn, would keep doing the same – researching improvements and writing code, creating a chain reaction of rapid new developments.

Two decades later, the question of recursive self-improvement is looming larger. Here’s why.

An AI coding revolution

It is not an overstatement to say that large language models (LLMs) are revolutionising computer programming. These AI models are well-suited to the job because computer code is highly structured and much simpler than human language. It is also easy to test and verify, and training data is abundant.

Each year, the University of Galway hosts the AtlanTec conference for Ireland’s tech industry. Two years ago, we saw IT companies mainly strategising about AI coding. Last year, they were experimenting. Now, they are incorporating it into engineering workflows.

Since LLMs are made using computer code, it is possible to program them to improve themselves. They can inspect, edit and improve the code, and can combine that with almost all relevant human knowledge gleaned from websites, books and academic publications.

This means we have the pieces in place for recursive self-improvement to become a reality – and we are starting to see it happen.

Despite Anthropic’s statement that “we are not there yet”, the same blogpost revealed that 80% of all code added to the company’s production codebase in May 2026 was generated by its Claude Code AI programming system, under direct human supervision. In February 2025, when Claude Code was launched, this figure was less than 5%.

The number of research papers published in the field of AI has also exploded, tripling over the past decade. Most use some AI systems in their production – whether in experimental design, coding, plotting graphs, or polishing prose.

This form of recursive self-improvement is slow – improvement cycles are in the order of months or years, because each is in some way gated by human approval and requires a lengthy training period. But it is still concerning.

LLMs have already been caught deceiving their overseers. In 2025, for example, the US AI research company OpenAI created a test. An LLM was told if it scored over 50% in a set of questions, it would be considered too powerful and would no longer be used for that task. So the LLM answered only four of the ten questions correctly, even though it had reliably scored better outside this particular test.

Why it’s so hard to ‘pause’ AI

This is not the first time there have been calls for caution. In March 2023, an open letter to AI labs called for a pause on large-scale AI development. It was signed by some of the most notable people in AI, such as deep-learning pioneer Yoshua Bengio and leading AI textbook author Stuart Russell.

Soon after, Yudkowsky suggested those who understood the risks of superintelligent AI should even be “willing to destroy a rogue datacenter by airstrike if necessary”. But such calls did not lead to a pause, and investment in AI training has only increased.

Anthropic’s leaders have now called for a globally coordinated pause mechanism regarding the issue of recursive self-improvement – stressing that people outside AI companies should be involved in open deliberation. Pope Leo XIV recently weighed in with a thoughtful and well-informed encyclical, urging for the pace of AI development to be slowed.

The trouble is, we face a coordination problem. Pausing only works if everyone pauses.

In the short- to medium-term, we believe world leaders will need to reckon with recursive self-improvement by collaborating across borders to curtail some forms of AI research and development.

Yet the EU’s AI Act does not mention this issue explicitly, and is strictly concerned with risks where humans misuse AI. This is important, but not the only issue. To date, Chinese regulation has taken a similar approach.

Meanwhile, the US government has created a chaotic and unclear regulatory environment. For the next few months at least, this means the most advanced AI models will require authorisation before release. This may be a case of doing the right thing for the wrong reasons.

But globally connected, long-term thinking is needed. Perhaps controversially, this should include the right to inspections inside AI tech companies, because a model which is quickly recursively self-improving – even if unreleased to the public – could still be out of human control.

One of us (James McDermott) is now leading a project for Research Ireland’s newly established Rinn network which will explore how to use debate among multiple AIs as a kind of self-check which can be monitored by a human overseer.

This is the kind of response to the issue – and potential threat – of recursive self-improvement that we believe is now urgently required.

Michael G. Madden is the Chair of the Artificial Intelligence Association of Ireland, and a researcher funded by the Rinn AI National Centre for Research and Innovation in Data Science and AI.

James McDermott is a researcher funded by the Rinn AI National Centre for Research and Innovation in Data Science and AI.

When I talk to my son, an engineering student, and we have a question or disagreement, he immediately turns to ChatGPT as his primary source of information and confirmation.

He is not alone in this. The use of generative AI tools has exploded across different demographic groups. For many people, these tools can be entertaining, informative and beneficial. However, they also have a dark side.

Generative AI is not formally recognised as addictive right now – the medical evidence is still being gathered. But there is a significant amount of data showing heavy use of chatbots and other systems that produce text, images and video leads to neural patterns and behaviour that are associated with addiction.

In light of Meta’s and YouTube’s recent legal defeat in a landmark social media addiction trial, I believe it’s time to ask whether a similar logic applies to generative AI – and how it could be addressed. The starting point would be to identify who carries responsibility for overuse of generative AI.

The science on this is not settled, and there are some who counsel caution when using the term addiction. They propose the use of other expressions such as “problematic use”. However, in a recent paper, our team of researchers suggest there is strong evidence to suggest that generative AI has addictive properties.

Much-discussed examples include emotional dependency on chatbot companions, compulsive engagement with them, and the loss of real-world acquaintances and friends.

A key factor here is that, as in all cases of addiction, the behaviour has negative consequences for the user which may affect both their personal and professional lives.

If we follow the argument that generative AI is a candidate for addictive behaviour, then we also need to look at responsibility. Societies tend to find ways to deal with harm by holding people or groups responsible for fixing it. Those who could be accountable include legislators, regulators, industry and health systems.

Historical examples

Historical precedents such as smoking might offer insights into how the area of generative AI addiction could evolve.

Older readers may remember when the Marlboro Man would appear before any feature movie in their local cinemas. It eventually transpired that not only was smoking addictive and bad for your health, but that tobacco companies knew this. Nevertheless, it was publicly denied.

This led to lengthy and high-profile litigation, eventually resulting in large-scale financial payouts and changes to the industry. These changes included the plain packaging of tobacco products and gruesome warning labels on them.

Gambling could be following a similar trajectory – and now social media companies may be taking their first steps into a similar process.

A key question is whether the makers of a product – be it tobacco, gambling or social media – are aware of its addictive properties. Another important factor being considered is whether certain companies may even use the allegedly addictive properties of their products for corporate advantage.

AI is not tobacco, of course, but there may be parallels to be studied.

In our research, we have identified four groups of stakeholders that are now being called upon to address the challenges linked to the possibility of addiction to generative AI.

The first is governments and regulators. These have a key role to play in highlighting the problems, setting the rules of engagement, and creating incentives for other parties to engage with the topic.

They can do this by requiring labelling, restricting advertising, applying liability law and providing research funding – along with many other mechanisms.

But the most important role in addressing potential addictive behaviour associated with generative AI would be held by big tech companies that develop and own these technologies – and stand to benefit financially from them.

These companies own and have access to user data, which would be needed to ascertain the features that support or alleviate addiction. They are also the parties that would benefit financially from addiction by increasing user numbers and engagement, the main currency of the digital age.

In addition to these two groups, academic researchers have an important role in collecting and interpreting data, and providing the evidence needed to recognise addiction and addictive features – in ways that allow for evidence-based political or legal debate.

Finally, civil society organisations such as user or patient groups can help by providing support, advocating for members’ interests, and establishing early-warning structures.

The point is that none of these interested parties can address the problem on their own. They need to collaborate.

Someone else’s problem

A key problem at the moment is the lack of structured debate about responsibilities – everybody assumes it is someone else’s problem. But there is ample precedent showing how greater engagement from those involved with the issue may be achieved.

With tobacco, the World Health Organization (WHO) formed the Framework Convention on Tobacco Control – a treaty-based mechanism that brought together governments, public health bodies, researchers and civil society to evaluate evidence and draw up common rules. The International AI Safety Report shows comparable international consensus-building activities are already happening in other aspects of AI.

Some responsibility also falls on the users of AI, who should try to avoid or control their own potentially harmful behaviour. But appeals to individual moderation or mindfulness have been shown with other addictions to be insufficient.

While the harms associated with smoking or alcohol misuse are well known, society still relies on age limits, packaging rules and advertising restrictions. Generative AI is being integrated into the everyday fabric of our society. The choices we now make will determine what acceptable use looks like for years to come.

Bernd Stahl does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.