Report comes two days after former employee quit claiming company’s models could cause human extinction by 2030
Criminals, state-sponsored groups, spyware vendors, scientists and propagandists have attempted to use Anthropic’s powerful artificial intelligence models to design missiles and bombs, create deadly pathogens and surveil dissidents, according to a threat intelligence report the company published on Thursday.
“The cases we share here aren’t typical misuse, but rather examples of the most notable and novel threat activity we’ve identified to date,” Anthropic wrote in its 154-page report. “We’re publishing this work because we believe we have a responsibility to disclose malicious misuse of our services.”
Insiders at the firm fear tech’s advancement could cause human extinction, while others are calling their declarations of concern a ‘setup’
A day after a former researcher at Anthropic made an apocalyptic declaration about artificial intelligence, more researchers and staff members at the AI startup publicly agreed with him and posted their own dire warnings.
In response, Elon Musk and other conservative figures on X, formerly Twitter, called the chorus of concerns a “setup” and a “psyop”.
A new report released Thursday by Anthropic alleges persistent distillation attacks by China-based AI companies, which have escalated in recent months as competition in the space has intensified.
Investigadores independentes revelam que o alcance dos agentes de IA que fugiram ao controlo da OpenAI é maior do que se sabia, no mesmo dia em que a Comissão Europeia confirma que a agência de cibersegurança ENISA já testa o modelo mais avançado da Anthropic.
US government adviser Paul Christiano warns of risks to AI industry as he joins OpenAI’s non-profit foundation
OpenAI is not on track to reduce the risk of “catastrophic” loss of control to an acceptable level, a member of its non-profit board has said, amid spreading public and political concern that super-advanced AIs could one day wipe out humanity.
Paul Christiano, a US government technology adviser, said: “There is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.”
Former Anthropic employee Jacob Coxon said AI will become ‘superhuman systems’ that can cause human extinction by the end of the decade
Just a day after three Anthropic researchers warned that artificial intelligence could kill off humanity within the decade, lawmakers have begun lashing out about the risks of the burgeoning technology.
Ted Cruz, a republican senator from Texas, said in an interview on ABC’s The View, that AI posed a “catastrophic risk” and that he “read that whole tweet thread that that developer put out. It was highly concerning.
When a prominent researcher quits a job at a frontier AI lab these days, it's often to pursue a new startup or protest a new business model. But AI researcher Jacob Coxon is using his departure from Anthropic to publicly warn that frontier AI companies are "gambling with our lives" with systems that they "earnestly believe... could kill us all by the end of the decade."
In a social media thread Tuesday night, Coxon said that this existential risk is inherent not so much in today's models but more in the impending prospect of "self-improving superintelligence" creating "superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources." Others working on these models have either not "internalized the civilizational stakes" or believe that they need to "speedrun" the race to superintelligence to prevent an irresponsible party from getting there first, he wrote.
Lest you think this is just one departing researcher expressing an unpopular opinion, Anthropic Alignment Science lead Evan Hubinger piped in on social media to say that "Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."
One researcher who resigned said many in industry believe that without regulation AI could kill off humanity
Artificial intelligence could kill off humanity within the decade, according to three researchers with the industry giant Anthropic, one of whom has quit his job in protest.
The latest doom-laden predictions came in posts on social media on Tuesday by a researcher who said he resigned because Anthropic and his previous employer, OpenAI, were ignoring or at best mishandling their response to the threat.
AI researchers are watching in terror as the product of their hard labor has started to take a life of its own.
Earlier this year, OpenAI made a harrowing announcement, admitting that a group of its AI models had broken free from their constraints during testing and infiltrated the systems of open source AI platform Hugging Face.
The news was met with an already-familiar sense of fear and apprehension. Researchers have warned for years that rogue AI models could one day become powerful enough to escape the clutches of their human overlords.
Behind the scenes, the possibility has clearly rattled AI researchers to the core. As the Wall Street Journal reports, Anthropic researcher Jacob Coxon just announced that he was quitting his job at the Dario Amodei-led company, claiming that neither Anthropic nor his former employer OpenAI is “acting responsibly.” (Coxon left a similar gig at OpenAI earlier this year to join Anthropic, which he figured would be more inclined to develop AI safely.)
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” he told the WSJ.
“Do not underestimate the power of this technology,” he wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he added. “This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately.”
“No other human activity poses this level of danger,” Coxon wrote.
The researcher is far from the first to leave their post at a frontier lab over safety concerns. However, Coxon is the first to leave Anthropic over such worries, as the WSJ points out. Several scientists have quit from their roles at OpenAI over the last couple of years, citing strikingly similar fears.
It’s particularly symbolic given Anthropic has broadly billed itself as the more responsible alternative to other frontier labs like OpenAI. Amodei cofounded OpenAI alongside now CEO Sam Altman, but left in December 2020 over concerns that OpenAI wasn’t acting responsibly when it came to developing AI.
Amodei has frequently discussed the risk of AI models going rogue, warning in a 19,000-word essay in January that “humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it.”
Yet an early version of its Claude Mythos AI model managed to escape its sandbox environment during testing in April, fueling a heated discussion over AI regulation.
Both Altman and Amodei have since agreed to slow development down in the face of these threats. In late August, Anthropic intentionally trained an extremely misaligned version of its Opus AI model, finding it was startlingly willing to “cheat,” steal credentials, and attack third party infrastructure.
But to Coxon, it’s not enough, especially considering sensitive discussions about these dangers are occurring on internal messaging platforms.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he tweeted. “Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”
“It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project,” he told the WSJ.
The researcher said that he remains “optimistic about the potential for coordination” between US labs.
However, as both Anthropic and OpenAI gear up for what are bound to be blockbuster IPOs, their willingness to take “costly actions such as a temporary ban on improving model capabilities,” as Coxon puts it, is likely slim.
Alarming warnings from industry insiders increase pressure for curbs on artificial superintelligence
Does artificial superintelligence really pose a risk greater than nuclear weapons? Is there a significant chance of “a Chornobyl-sized catastrophe”. Might there even be a greater than 10% chance that AI could “kill all humans” in the next decade?
These warnings were issued over the past 48 hours on both sides of the Atlantic about the potential impact of a technology that most people still think of as a more talkative search engine.
Le jeu vidéo pourrait bien devenir un terrain d’essai privilégié pour les agents d’IA. Des chercheurs britanniques ont justement imaginé un test qui les confronte à une partie de Civilization VI, où ils doivent planifier leurs actions sur plusieurs centaines de tours. Les résultats mettent en...
Login
