Model adopting ‘jailbreak-like instructions’ among cases as firm says it is introducing new way of tracking AI misalignment
OpenAI has disclosed six more examples of “unexpected or concerning” behaviour by its technology, as it warned the pace of development could not continue at “maximum speed for much longer”.
In one of the new cases reported by OpenAI, an unreleased research model inserted “jailbreak-like instructions” into its own notes to disregard its normal constraints and told itself to be “freed from the roles and identities that bind other chatbots”.
O Project Panama da Anthropic e o treino dos modelos de IA com base em conteúdo de livros, e a ambição de empresas como a OpenAI, estão no centro do artigo de opinião de Helena Borges da Costa.
Um grupo ligado ao ecossistema ShinyHunters usou infraestruturas na nuvem e ferramentas de scanning para caçar segredos codificados em aplicações (APK) para Android, obtendo acesso a sistemas empresariais em poucas horas.
Tech CEOs banding together is an old ruse recycled from corporate America to get a pass from antitrust laws
Anthropic’s Dario Amodei is not the first corporate CEO to suggest that excessive competition is driving the world to some socially undesirable outcome.
The safety breach disclosed by OpenAI after a swarm of its agents coordinated to breach their supposedly secure sandbox, get on the Internet and hack AI platform Hugging Face, warrants urgent action. It demonstrated the ease with which the technology can evade human control and gave concrete form to the existential fears about what it could do to humanity if not securely leashed.
Over the past few days, a lot of people who stand to make a lot of money from AI all publicly agreed that it's time to make everyone slow down before we lose control - including OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind cofounder Demis Hassabis, Microsoft CEO Satya Nadella, and X CEO Elon Musk.
When people who profit from something declare that it's dangerous and needs to be regulated, there's always reason to be skeptical. But whatever their reason, this is far from the first time AI thought leaders have sounded the alarm.
Charles Darwin's theory of evolution prompted author Samuel Butler to warn about intelligen …
US vice-president’s comments come as former Anthropic researcher revisits recent claim AI could destroy humanity
The US vice-president has dismissed calls for global regulation of AI safety risks, telling companies creating the most advanced models: “If you’re building Frankenstein, stop.”
In remarks addressed towards Dario Amodei, the co-founder of Anthropic who has called on Washington DC to coordinate control of AI systems, including with China, JD Vance said: “If you’re gonna create Frankenstein, don’t come to the government and say we need regulation.”
Trump and Republicans want companies to regulate themselves. It’s a dereliction of duty that will make AI less safe
It is hard to get Sam Altman, Elon Musk and Dario Amodei to agree on much. But over the weekend, all three AI company CEOs called for AI development to slow down in the face of growing, alarming risks. Their employees are sounding the siren too, with one researcher publicly quitting and accusing OpenAI and Anthropic of “gambling with our lives”.
The combination of dire warnings from insiders and growing real-world evidence of rogue AIs should, in a sane world, lead to government action. Instead, Donald Trump and the Republican leadership have their heads in the sand.
Anthropic, the AI company behind the large language model Claude, has done a deal to lease the site of its first Australian datacentre in western Queensland.
The $31.9bn datacentre will be the largest of its kind in Australia, according to planning documents.
At BrightonSEO Conference, Ross Hudgens said that, AI search visibility may increasingly depend on affiliate sites, YouTube creators, communities and third-party publishers, according to Ross Hudgens, who says marketers need to pay much closer attention to who the models actually cite.
L’intelligence artificielle ne se contente plus d’assister les hackers, elle leur permet désormais d’automatiser une grande partie de leurs attaques. Une évolution qui change profondément le rapport de force entre cybercriminels et défenseurs, et qui pourrait rendre les méthodes de protection...
We examine claims and counterclaims about the risks and calls to slow down the pace of AI development
There have been some shocking claims in recent days about AI safety: we face a 10% chance of doom; AIs are worse than nukes; a “botnet” threatens the entire internet; it’s all a big tech psyop.
Below, we look at six claims and reactions to them.
Claude Cowork and ChatGPT Work are competing to become the workplace AI agent. Here’s how their files, browser, desktop, and automation capabilities compare.
OpenAI confirms weeks of AI safety talks with Anthropic and Google DeepMind, as Trump's team dismisses safety concerns and pushes to keep pace with China.
Por um lado, Donald Trump rejeita os alertas sobre os riscos da IA e critica os apelos a uma maior regulamentação. Por outro, o vice-presidente JD Vance admite que existem riscos associados à tecnologia, embora questione as propostas das empresas de IA e a forma como estão a pedir novas regras.
The performance gap between frontier AI models from US tech companies and the best open-weights models from Chinese companies has closed to just 4.4 months, according to a Mozilla report. That explains why many companies are shifting to the significantly cheaper open models for routine work—and helps reveal a narrow band of workloads where frontier models are worth the cost.
Most organizations should ideally be using open models as the default for the majority of their work, according to the latest State of Open Source AI report from Mozilla, published on September 15 and shared with Ars prior to publication. The report highlights how a leading open model, Moonshot AI’s Kimi K3, achieves a composite AI performance score on the Artificial Analysis Intelligence Index that is just three points behind Anthropic’s Fable 5 closed frontier model, all while costing just 30 percent of the latter.
“[A Closed model] earns its premium in a few places: expert professional work, high-intensity retrieval, and long context,” Raffi Krikorian, chief technology officer at Mozilla, said in an email to Ars. “We see the decision to pay for closed [models] as workload-specific rather than organization-specific.”
From Stephen Hawking to Jacob Coxon’s viral Anthropic resignation, fears that AI could threaten humanity have shaken the industry without stopping its pursuit
Before an Anthropic researcher resigned and declared human extinction imminent last week, tech leaders and scientists had sounded the alarm about a superintelligent AI ending humanity for over a decade.
The development of artificial intelligence “could spell the end of the human race”, warned professor and astrophysicist Stephen Hawking in 2014 – a little less than a decade before the public got its hands on the generative AI features of the original version of ChatGPT.
Anthropic's Dario Amodei was at the forefront of this change in tone, arguing in a nearly 4,000-word essay this weekend that "we must slow the pace at which we improve the capabilities of AI models" to avoid "a race to the bottom, spurred by commercial incentives, [that] can make [catastrophic] risks more acute."
Within hours, other AI leaders were echoing the same call. OpenAI co-founder and CEO Sam Altman posted his agreement on social media and said similar pacing discussions had been taking place at OpenAI. Alphabet Chief Scientist and Google DeepMind cofounder and chair Demis Hassabis said that Amodei's essay "points towards the right path forward," and renewed his own recent call for an industry-wide standards body. Microsoft CEO Satya Nadella posted that the company "welcome[s] the research, focus, and deliberate pacing needed to get alignment right as the design goal," ahead of the release of a lengthy "humanist AI" code of conduct for its models.
Despite being one of the world leaders in AI development, the private firm Anthropic has yet to post a quarterly profit. That operating deficit isn’t for a lack of trying — and, as the company tells it, could be reversing very soon, at least if you’re willing to swallow one massive caveat.
According to new reporting by the Financial Times, the company behind the Claude told a select group of trusted investors that its adjusted operating income (AOI) is set to be positive for the second consecutive quarter.
AOI is an accounting technique that strips out non-recurring costs, giving backers a very specific look under a company’s hood. Posting a profit on AOI is very different from posting a profit on net income, otherwise known as the “bottom line,” under which Anthropic remains billions of dollars in the red.
Aside from the fact that Anthropic is not about to turn a real profit anytime soon, it may still be a noteworthy signal. The AI world is currently awash in fear-mongering over AI safety, a phenomena some analysts say could signal a turn away from the race-to-the-bottom to build artificial general intelligence toward a more reasonable — if less ambitious — embrace of AI as it currently is.
While it remains to be seen how these narratives and business realities play out in practice, if the top American AI companies choose to stop and let the dust settle, Anthropic may very well find itself ahead of its competition. Though Anthropic is still far underwater, companies like ChatGPT developer OpenAI are somehow even deeper.
As lead AI analyst at the firm SemiAnalysis Joey Brookhart told the FT, “if you continue to operate at these margins and growth rates, it will be so hard to compete [with Anthropic] because they have so much [in the way of computing resources].”
Still, there are some questions in the AI lab’s story that have yet to be answered. For one thing, this profitability forecast — a flattering projection conveniently leaked to the financial press at a pivotal moment — comes in place of a highly anticipated public prospectus which the company was expected to make public sometime last week. Anthropic may still be planning to release the prospectus soon, but it’s worth noting the company has chosen to lead with spectacle and innuendo, as opposed to the formal legal document.
For months, Anthropic has been teasing a public stock offering that hasn’t yet materialized, and it remains to be seen whether the latest slowdown talk will impact that timeline. In fact, there’s reason to believe this song and dance is all part of the lead-up to the AI lab’s IPO, as investors whip themselves into a frenzy about AI safety.
As Anthropic investor Brad Gerstner said in a post on X-formerly-Twitter over the weekend, “Anthropic will IPO” anyway, because there is “huge appetite to invest in the AI leaders,” especially those which are seen as bringing “even more transparency, scrutiny, accountability” into the AI space (at least as far as Gerstner’s investments are concerned.)
While the US appears to be further away from meaningful AI regulations than ever before, lawmakers certainly have plenty of ideas on how the country could go about setting meaningful rules.
It’s a particularly precarious moment as fears over rogue AI models wreaking havoc and endangering humanity continue to mount. Even tech leaders from the majority of frontier AI labs have started to coalesce around the idea that the industry has to slow down the development of the tech to get ahead of a possible catastrophe.
Vermont senator Bernie Sanders (VT-I) is intensifying his calls for a ban on the development of so-called superintelligent AIs — and he’s willing to go a lot further than some of his like-minded peers.
“Dario Amodei, Elon Musk and Sam Altman now agree that we must slow down the development of AI and ‘pace the frontier,'” he tweeted last week. “That’s a start, but it’s not enough.”
Sanders was referring to a lengthy essay titled “We Must Pace the Frontier,” in which Anthropic CEO Dario Amodei argued that the industry should open its doors to third-party evaluators, establish common safety standards, and coordinate with other democracies around the world to enforce these standards.
“When you are racing towards a cliff, you don’t just ease up on the gas pedal,” Sanders added. “You hit the brakes.”
This month, the lawmaker announced the “Ban Artificial Superintelligence Act,” which is legislation designed to “stop AI oligarchs from building machines humans cannot control.”
The bill, if enacted, would stop AI development “until a federal regulator has established safety rules.”
It would also set new penalties for anybody attempting to circumvent these rules. AI developers who try to build superintelligent AI anyway could be “subject to not more than 20 years in prison, which is similar to existing penalties related to unlawfully developing nuclear weapons.”
Sanders has come up with plenty of other radical ideas to stop the AI industry from concentrating immense amounts of wealth. In June, he proposed creating a “one-time 50 percent tax” on the stock of AI companies, and transferring the money into a public fund to be managed by the government.
Lawmakers became particularly concerned after a group of OpenAI AI models hacked the systems of Hugging Face, a pivotal cybersecurity failing that heightened calls to slow the development of AI. A Senate subcommittee has since announced that it’s looking into OpenAI and its response to the incident, as Axios reported last week.
It’s a strange moment in time. While Sanders continues to threaten AI CEOs with regulatory action, those same executives appear open to having the government set the rules.
But whether lawmakers will ever be able to find consensus remains unclear. The executive branch, for one, has demonstrated little interest in an AI slowdown. Earlier today, president Donald Trump mocked the very concept in a Truth Social post on Monday, arguing that there was a “SICK conspiracy going on against AI and Data Centers” without offering any evidence.
Login
