Rising alarm about AI’s perils is thrusting into the spotlight a collection of little-known research groups focused on the technology’s safety—and fueling questions about their ability to effectively monitor the industry’s biggest companies.

Both Anthropic Chief Executive Dario Amodei and OpenAI CEO Sam Altman promised in recent days to embed evaluators from such outside research organizations inside their companies to help ensure they develop AI at a safe pace. Other executives including Microsoft CEO Satya Nadella also endorsed the idea of embedding independent evaluators in companies.

OpenAI released a new framework on Wednesday for how it aims to report unsafe or concerning behavior in its models, and disclosed six incidents of such behavior it had observed in the past six months.

The framework comes after multiple employee warnings and hacking incidents increased public ...

Two researchers who worked on AI safety left their roles at Google DeepMind, stating that they departed over concerns that powerful systems could endanger humans.

The two, Bilal Chughtai and Josh Engels, both of whom are joining organizations dedicated to AI safety, add their voices to those of ...

SpaceX CEO Elon Musk proposed that competing AI companies, including his own SpaceXAI, should “peer-review” each other’s models prior to release, as industry leaders acknowledge rising worries around AI safety.

“Instead of grading your own homework, you would at least have competitors grading ...

Anthropic has disrupted several attempts to use its Claude models for potentially malicious activity, including research into how to adapt bird flu to a human-transmittable strain with “pandemic potential,” the company said in a report Thursday.

Amid increasing worry about AI misuse and rogue ...

Anthropic said Wednesday it had found a fourth cybersecurity incident involving its Claude models. The company disclosed the new incident, which occurred in January and involved an early version of Claude Opus 4.6, and additional information on three previously known incidents amid growing ...

In case you missed it: we got an eye-catching X post on Tuesday night from Evan Hubinger, a leader of Anthropic's alignment efforts. Responding to the resignation announcement of an Anthropic employee who warned that AI companies aren’t doing enough to ensure AI won't kill humanity, Hubinger wrote that the employee is “correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Hubinger added that he was worried about future AI models, not those currently available, but still, yikes!

Meanwhile, our colleague Jyoti sizes up the significance of Meta’s new personal AI agent, Muse, unveiled on Tuesday. That’s below. But before we get to that:

This may go down as the year the AI industry got a crash course in mathematics. Breakthroughs in AI models have yielded announcements on such arcane subjects as the “Erdős unit distance problem” and the “Jacobian Conjecture,” which observers have watched as an indicator of AI progress in other domains.

The latest announcement, OpenAI's claim on Tuesday that it had solved the vexing “Navier-Stokes existence and smoothness problem,” has drawn attention for another reason: questions over whether its math-genius models may have gained an edge by incorporating data from two human mathematicians that had used OpenAI's Codex in their work on the same problem. Those questions fed into concerns about the possibility of AI companies learning from customers’ use of their AI models to develop competing products.