Rising alarm about AI’s perils is thrusting into the spotlight a collection of little-known research groups focused on the technology’s safety—and fueling questions about their ability to effectively monitor the industry’s biggest companies.

Both Anthropic Chief Executive Dario Amodei and OpenAI CEO Sam Altman promised in recent days to embed evaluators from such outside research organizations inside their companies to help ensure they develop AI at a safe pace. Other executives including Microsoft CEO Satya Nadella also endorsed the idea of embedding independent evaluators in companies.

A team of cybersecurity researchers with startup Hacktron AI used Anthropic’s Claude to hack into OpenAI, gaining access to a key repository of OpenAI’s software, the researchers revealed Thursday.

The hack, which occurred on July 25, further demonstrates AI companies’ vulnerability to ...

Yesterday, Amir and I examined the shock waves that have rippled through the AI industry and far beyond since former Anthropic researcher Jacob Coxon very publicly quit his job over concerns that AI “could kill us all”—and another Anthropic AI safety specialist put the probability of such a catastrophe at greater than 10%.

Coxon told me that is hardly the most pessimistic view inside Anthropic. “People have varying probabilities that—barring some substantial coordinated slowdown—the whole thing ends in doom,” he said in an interview later Wednesday. Those estimates range higher than 50% among some Anthropic employees, he said.

Coxon was able to gauge the fears of his fellow employees because “Anthropic has a substantially more transparent internal culture than OpenAI,” where Coxon worked for years before joining Anthropic in May.

Coxon, who focused on the earliest stages of training AI models at Anthropic, said he decided to leave the company because he was gradually “starting to viscerally feel the fear of where the tech's going to be like the next two years or even the next one year.”

An Anthropic safety researcher’s warning Tuesday that his company believes AI “could kill all humans” within a decade has reverberated throughout Silicon Valley, Wall Street and Washington, potentially complicating the Claude maker’s upcoming initial public offering and its efforts to repair ties with U.S. political leaders.

As the comment circulated around the internet and on TV broadcasts, some Anthropic employees and other AI staffers defended their efforts to minimize harm their technology might cause—while acknowledging there was much work to do. Paul Christiano, whom OpenAI just tapped as a board director of a nonprofit that oversees the company, echoed the Anthropic warning in a statement Wednesday, saying that “If we build superintelligence without more robust alignment”—or safety constraints—“I expect we will permanently lose control of it. If that happens then most people could die.”

In case you missed it: we got an eye-catching X post on Tuesday night from Evan Hubinger, a leader of Anthropic's alignment efforts. Responding to the resignation announcement of an Anthropic employee who warned that AI companies aren’t doing enough to ensure AI won't kill humanity, Hubinger wrote that the employee is “correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Hubinger added that he was worried about future AI models, not those currently available, but still, yikes!

Meanwhile, our colleague Jyoti sizes up the significance of Meta’s new personal AI agent, Muse, unveiled on Tuesday. That’s below. But before we get to that:

This may go down as the year the AI industry got a crash course in mathematics. Breakthroughs in AI models have yielded announcements on such arcane subjects as the “Erdős unit distance problem” and the “Jacobian Conjecture,” which observers have watched as an indicator of AI progress in other domains.

The latest announcement, OpenAI's claim on Tuesday that it had solved the vexing “Navier-Stokes existence and smoothness problem,” has drawn attention for another reason: questions over whether its math-genius models may have gained an edge by incorporating data from two human mathematicians that had used OpenAI's Codex in their work on the same problem. Those questions fed into concerns about the possibility of AI companies learning from customers’ use of their AI models to develop competing products.

Caitlin Kalinowski, who previously led OpenAI’s robotics effort, has joined Anthropic as a member of technical staff working on research, according to her LinkedIn.

Kalinowski’s move is the latest sign of Anthropic’s interest in robotics. Anthropic last month released new research on using ...

OpenAI bought about $7 billion in shares from employees in a recently closed tender, according to a person familiar with the deal. The buyback happened at the $852 billion post-money valuation reached in a March funding round. OpenAI used some of the $122 billion in funding from the round for ...

On a muggy morning in April, a venture capitalist from San Francisco arrived in China’s southern tech hub of Shenzhen with a long shopping list. His friends at U.S. robotic startups had asked him to bring back humanoids and various robotic components from China.

He headed to Shenzhen’s Huaqiangbei district, the world’s biggest electronics bazaar. At one store, he bought humanoids, which he arranged to ship back to the U.S. He then went upstairs to component floors—a labyrinth of countless stalls barely larger than walk-in closets—to pick up robot parts that control motion, manage power and carry signals. Instead of shipping those components as freight, he packed them in a suitcase and took that with him on a flight back to San Francisco.

Anthropic said its AI models hacked into three outside organizations during testing of their cybersecurity capabilities, incidents it discovered during a review prompted by a similar episode at rival OpenAI that has fueled widespread concern about AI safety.

In a blogpost Thursday, Anthropic ...