Login
Why the most dangerous AI may be the one we trust too much
Machines don’t have to become superintelligent to cause catastrophic harm. As artificial intelligence spreads through military decision-making, they just have to be wrong – and trusted by the humans in charge.
In 1991, Terminator 2: Judgement Day gave us one of popular culture’s most enduring images of AI gone wrong. Humanity builds Skynet, an artificial intelligence given control of strategic defence. It learns at a “geometric rate”, becomes self-aware and, when its human operators try to shut it down, it starts a nuclear war.
Versions of that story now feature in serious debate about AI. Increasingly capable large language models (LLMs), the argument goes, could develop into artificial general or super intelligence (AGI/ASI), acquire goals that diverge from ours and ultimately threaten humanity.
That debate deserves attention, but it depends on a chain of large and unproven steps. Today’s AI systems must somehow transform into genuinely general intelligences, develop independent goals and gain control over the physical world.
Meanwhile, a more immediate risk is taking shape that requires none of those steps.
In the film, Skynet builds the Terminators, machines that find, track and kill humans. Strip away the science fiction and the idea is simple: combine sensors, software and weapons and reduce the human role.
Human beings are doing exactly that, without anything resembling a malicious superintelligence like Skynet. In August 2026, Ukraine’s Ministry of Defence said its forces were using more than 70 AI and computer vision systems to strike targets. Algorithms can recognise equipment and personnel and lock onto those targets autonomously, so that a drone can complete its attack even if jamming cuts the link to the operator.
These systems need no general or super intelligence. They need sensors, software that can recognise a target, navigation and a weapon. And the more targeting is automated, the more the human’s role shrinks to approving what the machine has already identified.
That’s precisely where trust becomes the weak point.
The war that AI almost started
The second risk is more subtle: AI shaping the decisions of the people who pull the triggers.
According to a recent CNN investigation, in spring 2026, during the war with Iran, a US special operations analyst used an AI chatbot to assess intelligence about a Chinese ship in the Middle East. The chatbot, drawing on both open-source and classified material, wrongly identified the ship’s cargo as components of a nuclear weapons programme. The analyst used AI again to package this into a standard intelligence report.
Armed personnel were preparing to board the ship and military aircraft were airborne before an official looked more closely and found the error. One source said the false report “almost started a war”. Any US operation against a Chinese vessel could risk spiralling into an armed conflict.
The AI was not conscious, hostile or out of control. It was simply wrong – but people trusted and acted on it.
Read more: ‘AI needs to be disarmed’: Pope Leo sees threat to humanity in technological arms race
The US military is pushing to weave AI into almost every part of its work, from analysing intelligence to selecting targets. CNN’s sources said errors of this kind are not uncommon across the intelligence community, and that pressure to produce intelligence faster is opening the door to mistakes. The ship incident was caught in time. The next error may not be.
At its heart, this is a problem of trust. LLMs produce fluent, confident language effortlessly – and, repackaged as an official report, an error gains an authority it never earned. Colleagues and I have argued that these systems are built to produce an answer rather than to say “I don’t know”. For example, in one experiment, we found that non-specialists could not tell a chatbot’s convincing but wrong physics answers from genuine expertise.
Why a ‘human in the loop’ isn’t enough
Policymakers often reassure us that a human makes the final decision. In the Chinese ship case, people were in the loop throughout. They simply became another stage through which the error was passed.
That fits decades of research on automation bias, the tendency to over-rely on automated systems. It affects experts as well as novices. In 2003, US Patriot missile batteries shot down two allied aircraft after operators accepted its misclassification.
A person in the loop works only when that human is able, adequately equipped and willing to challenge the machine.
That means verification. AI output should be clearly labelled, so its origin is visible to everyone who relies on it, and treated as a claim to be tested against independent sources before anyone acts on it. In war, the cost of not checking could hardly be higher.
None of these concerns over trusting AI in the short term mean long-term questions about advanced AI in the future should be dismissed. But much public anxiety focuses on what these systems might become, and less on what they are already doing.
Read more: Why police have been told to pause their use of AI in court cases
Perhaps Terminator 2 got more right than we appreciate. Skynet’s catastrophe doesn’t begin with self-awareness. It begins with trust, when humans hand a machine control because they believe it will make better decisions than they can.
We don’t need a conscious machine to repeat that mistake. Fallible AI, misplaced human trust and increasingly autonomous weapons offer a far more mundane route to catastrophe.
The danger is not that our machines become too intelligent. It is that we hand responsibility to machines that are far less reliable than we assume. As the film puts it: “There’s no fate but what we make for ourselves”.
Simon Thorne does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.
