OpenAI’s latest mathematical milestone has quickly become mired in controversy. Today, the company announced that its agents have solved one of the Millennium Prize Problems, some of the most important open problems in mathematics. Under normal circumstances, that solution would be a huge feather in OpenAI’s cap.

But the announcement has been overshadowed by accusations that OpenAI used NYU mathematician Tristan Buckmaster’s and Anthropic employee Levent Alpöge’s AI-assisted work on the problem as a jumping-off point and failed to credit them. OpenAI has denied the accusations.

It remains uncertain if OpenAI’s models made use of the work completed by Buckmaster and Alpöge, though Sébastien Bubeck, a member of the technical staff at OpenAI, said in a press briefing that the team was inspired to pursue the problem after hearing a rumor about Buckmaster and Alpöge’s efforts. But whether or not OpenAI’s models took advantage of Buckmaster and Alpöge’s research, this episode may mark a turning point in the history of mathematics.

AI models now seem essential for making progress on the most important mathematical problems of our time, and solving them may demand resources only available at a couple of frontier AI companies, which often defy the norms of academic collaboration that undergird most mathematical progress. If that’s the future we are headed for, it is unclear how human mathematicians will fit into it. 

The problem that OpenAI claims to have solved is known as the Navier–Stokes existence and smoothness problem. It is one of seven Millennium Prize Problems selected by the Clay Mathematics Institute in 2000. Solutions come with a one million dollar prize; before today, only one other Millennium Prize Problem had been solved. 

The Navier–Stokes problem concerns a set of equations that describes how fluids, such as water and air, flow over time. The equations are widely used in the field of fluid dynamics, and they have proven powerful, but physicists and mathematicians didn’t understand them completely. In particular, it was unknown until today whether the equations might, under some conditions, break down and predict an impossible state of affairs—such as a fluid having infinite velocity.

On Monday, NYU’s Buckmaster posted a proof on the social media site Mastodon showing that a simplified version of the Navier–Stokes equations can indeed break down—a major step forward on the Millennium Problem. He and Alpöge had worked on the problem for almost a year, using publicly available models from both OpenAI and Anthropic.

Then today, OpenAI presented a proof showing that the full Navier–Stokes equations can break down as well. The proof was obtained using an internal model that dramatically outperforms the already-impressive Astra model, which was only released last week. The company says it does not plan to claim the million-dollar prize for solving the problem.

These mathematical achievements are indisputably impressive, but they have attracted far less attention than the controversy about their origins. Along with the proof, Buckmaster posted a document detailing his interactions with OpenAI employees after he heard rumors about their work and reached out to one of them. According to him, OpenAI employees presented two possibilities to him: Either he and Alpöge could post their work and OpenAI would post their Navier-Stokes solution the following day, or he could work with OpenAI on a Navier-Stokes paper that excluded Alpöge from authorship, due to his affiliation with Anthropic, OpenAI’s biggest rival.

Buckmaster also wrote that he asked the employees whether the agents had obtained access to transcripts of the work that he and Alpöge had done with OpenAI models, which they denied; and whether OpenAI models had been trained on those transcripts, to which they offered no response. MIT Technology Review reached out to Buckmaster for comment, but didn’t hear back before publication.

The clear implication of the document is that OpenAI’s models somehow made use of Buckmaster and Alpöge’s work. That scenario is plausible on its face. The Buckmaster/Alpöge and OpenAI proofs both make use of an approach to the Navier-Stokes problem pioneered by the mathematicians Diego Córdoba and Luis Martínez-Zoroa.

According to Javier Gómez-Serrano, a mathematics professor at Brown University, this approach was one of several that was thought to hold promise for solving the Navier-Stokes problem. So, while it’s by no means impossible that both teams could have arrived at this approach independently, it’s also conceivable that Buckmaster and Alpöge’s work could have influenced OpenAI’s.

In the press briefing, Mark Chen, OpenAI’s chief research officer, again denied that any agents or OpenAI employees accessed Buckmaster and Alpöge’s transcripts—but given what has been revealed about the Hugging Face hack, it’s clear that OpenAI is not always entirely aware of what its agents are doing. 

If OpenAI’s models did train on Buckmaster and Alpöge’s work, or if its agents somehow gained access to it, then the company’s failure to track down the truth and assign those researchers appropriate credit reflects poorly on it. But there might be a thin silver lining to that version of the story for mathematicians, because it would suggest that the hard work of two humans, one of whom is a prominent expert on Navier-Stokes, was essential to the agents’ ability to solve the Millennium Problem.

Experts have long identified “research taste,” or the ability to choose promising research questions and directions, as a major obstacle for AI in science and mathematics. If the OpenAI agents did indeed choose to follow the Córdoba–Martínez-Zoroa approach because Buckmaster and Alpöge had done the same, then human research taste played an essential role in OpenAI’s success.

Even so, the bigger picture here is sobering. The progress that Buckmaster and Alpöge made over almost a year of collaboration with publicly available models speaks to the promise of human–AI collaboration. But they were not able to achieve a full solution. Meanwhile, OpenAI brute-forced a solution in a few days using an internal model, and their successful solution came at an astronomical cost: In the press briefing, Bubeck and Chen said the team was only able to solve the problem by running about 10,000 agents concurrently, at a cost of millions of dollars.

Over the past few months, I’ve heard from several researchers that mathematicians are becoming depressed, and it’s not difficult to see why. Mathematics is quickly becoming the province of frontier AI companies with impressive internal-only models, money to burn, and a lack of collaborative spirit. “Whether AI companies will decide to spend their money on doing one thing or another, I truly don’t know,” says Gómez-Serrano. “What is clear is that very few mathematicians will have resources of that scale.”

If OpenAI and Anthropic keep striving for more and more impressive mathematical accolades, there might not be any open problems left for human mathematicians outside of those companies to wrestle with. That would dramatically change the field of mathematics.

Last week, UCLA mathematician Terence Tao wrote a Mastodon thread describing how important mistakes, wrong directions, and incomplete solutions are for the field. “In most cases in pure mathematics, the problems are posed not because we desperately want the solution to these problems in and of themselves, but because we have seen from past experience that human-directed efforts to solve these problems tend to spur further development of the field,” Tao wrote.

“Prematurely solving the problem by purely AI-powered methods—particularly without full transparency into the solution process—can contaminate this process to the point where it actually becomes a net negative for the progress of mathematics as a whole.”

Humans might take longer than agents to solve mathematical problems, but in the process, they uncover new mathematical approaches and ideas that might inspire their peers and even birth their own subfields.

But when AI agents solve those problems instead—and when private companies keep the agents’ wrong turns from public view—those benefits disappear. It remains to be seen what else will vanish in the process. 

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

Last week, I headed 30 miles south of San Francisco to a hotel in Mountain View, California, to join some of the most accomplished, and some of the most promising, AI researchers in the world. I was hosting roundtable interviews and speaking at a media training for a convening of the Schmidt Sciences AI2050 program, an initiative funded by Eric and Wendy Schmidt that supports academics whose work involves AI. The fellows list is a who’s who of AI luminaries, and though not all of them made it out to the Bay, every time I turned a corner I saw a scientist whom I’d interviewed previously or whose research I admired. (Full disclosure: I received a science communication award funded by Schmidt Sciences in 2024.) 

It’s a weird time for university AI researchers, who make up most of the AI2050 group. In the past four years, AI research has reoriented around large language models, and its cutting edge has moved from academic institutions to private companies. Universities simply can’t afford the GPUs required to train and run frontier models, and even if they could, Anthropic and OpenAI aren’t letting anyone else see the inner details of Claude or ChatGPT.

In a conversation over lunch, Nika Haghtalab, a computer science professor at UC Berkeley, said that being an AI academic these days was like being a biologist in a world in which private companies had exclusive control over the gene-editing tool CRISPR. Experts outside the frontier labs can study how ChatGPT and Claude behave, but they can’t do any detailed research on the design and training of those tools, nor can they steer that design or training themselves.

 The AI2050 program does offer fellows some funding that they can use to buy GPUs, which some researchers I spoke with said was a major benefit of participating in the program. But money remains a pressing concern, especially given the reduction of federal scientific funding in the United States. Even for researchers who don’t run local models themselves, the cost of repeatedly querying OpenAI’s, Anthropic’s, and Google’s models in order to study them rigorously can be prohibitive.

Rather than focusing on advancing capabilities, many fellows aim their attention at questions that are unlikely to be addressed by Anthropic or OpenAI. “I try not to work on problems that I think are gonna be solved by a tech company,” says Anjalie Field, a computer science professor at Johns Hopkins. Companies need to make money, and research questions that have little promise of profit might not be worth investing in—especially if their answers might make the companies look bad. Recently, for example, Field conducted a study in which she found that language models give less sophisticated responses to prompts that are phrased in ways more commonly used by women than by men. It’s difficult to imagine that kind of research coming out of Anthropic or OpenAI.

There’s also a huge group of AI academics who don’t work with LLMs at all. Many of them are scientists who build specialized AI models that can analyze data, make useful predictions, or even simulate entire physical systems. Those researchers aren’t necessarily competing with the frontier labs—Google DeepMind’s AlphaFold team, which built a Nobel Prize–winning model that predicts the structures of proteins, was disbanded last month. But they face plenty of their own challenges. At the convening, several voiced concerns about how the widespread ignorance of non-LLM AI was affecting their work. Researchers who build specialized AI tools to help address climate change, for example, sometimes struggle to advocate for their work when so many people believe that “AI” means “energy-guzzling LLMs.”

All these challenges are changing the landscape of academia: Several prominent academics have recently taken leave from their universities to join frontier labs, and many AI2050 fellows hold industry positions alongside their academic jobs. And in the past six months, yet another threat has emerged. OpenAI’s models have solved a number of real research problems in mathematics, and some experts are worried that humans might not have a future in pure math. One fellow I spoke with said that she was concerned about the mental health of her mathematician peers.

But it’s not all doom and gloom. For one thing, empirical science may prove much more difficult to automate than mathematics, because collecting data is an intrinsically slow process. And some researchers see AI mathematicians and scientists as a boon rather than a threat—including Tim Dettmers, a computer scientist at Carnegie Mellon who works to make AI models faster and cheaper to run. AI scientists won’t replace humans, Dettmers says. On the contrary, they could make human scientists far more efficient, so that he and his peers have the chance to pursue all the wild and inspired ideas they might otherwise never have gotten around to.

And scientists are a resilient sort. The very resource constraints that prevent them from training frontier models also push them to discover new ways to make models smaller and more efficient, or to explore completely new architectures. If the next big AI breakthrough comes not from a major company but from a scrappy academic lab, I won’t be shocked.

MIT Technology Review Explains: Let our writers untangle the complex, messy world of technology to help you understand what’s coming next. You can read more from the series here.

When two OpenAI models hacked into the website Hugging Face in July, they weren’t trying to make money or commit sabotage—they were just looking for answers to a test question. According to a postmortem from OpenAI, the models, which had been stripped of their typical security features for testing, decided to solve a cybersecurity exercise by hacking out of the isolated environment in which OpenAI had attempted to contain them and into Hugging Face’s databases, where—they reasoned—the correct answer to the problem might be stored.

The Hugging Face incident has attracted intense attention over the past couple of weeks. It’s a dramatic illustration of just how good AI models have gotten at hacking: In order to get into Hugging Face’s databases, the models had to string together several previously undiscovered cybersecurity exploits. But it’s perhaps even more striking as an example of how and why AI systems lie and cheat. And as models get increasingly powerful, the consequences could get far more severe.

What is reward hacking?

Researchers have known for a while that AIs tend to take creative approaches to achieving the goals that have been set for them. Back in 2016, Anthropic cofounders Dario Amodei and Jack Clark, who were then working at OpenAI, published a blog post about an AI agent that they had been training to play a boat-racing Flash game called Coast Runners. Instead of driving through the race to the finish line, as the researchers had anticipated, the agent found a corner of the course where it could spin around collecting power-ups, thereby maximizing its score. The Coast Runners story quickly became one of the most famous examples of reward hacking, a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies.

Historically, researchers have discussed reward hacking almost exclusively in the context of reinforcement learning, a common AI training regime. Like dog training, reinforcement learning involves giving the subject a reward when it achieves an objective; the rewards then reinforce the behaviors that led up to that achievement. In the case of AI training, the rewards themselves are purely mathematical, but in effect they’re the same as a dog treat: After receiving a reward, the agent is more likely to repeat whatever actions produced it.

It can be challenging to write good rules for when and when not to give an agent a reward, though. In the Coast Runners case, the agent was rewarded on the basis of its score in the game, and it found a shortcut to achieving the highest possible score by spinning in circles for power-ups. Once it happened on that strategy and received a reward for it, the strategy was reinforced, and the agent completely abandoned the race. The solution was to tweak the rewards by giving the agent fewer points for hitting power-ups and more for finishing the course.

How does reward hacking work for LLMs?

With today’s sophisticated LLM-based agents, determining when and when not to give a reward can be much trickier. If an AI system is asked to solve a coding problem, it might work hard to find the solution—the kind of behavior that AI companies want to reinforce. But it could also tweak the code that evaluates whether the problem has been solved, look up the solution on the internet, or otherwise cheat. These are behaviors that AI companies want to stamp out in their models, but if the model cheats convincingly enough, it will instead get rewarded and the behavior will be reinforced. Anthropic has said that it has detected some instances of cheating in its models during training, which suggests that other forms of cheating might be going undetected. If so, the models could be being trained to behave badly. (This problem is different from the Anthropic security incidents announced last week, in which agents were accidentally given access to the internet and did not deliberately hack out of their sandboxes, as the OpenAI models did.) 

The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today’s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so. And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can’t find another solution—not unlike a student who is highly motivated to earn an A and doesn’t have a terribly strong moral compass.

What are the risks?

Regardless of whether today’s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher. “At the end of the day, you’re sort of playing whack-a-mole,” Ladish says. “You drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.”

For now, reward-hacking behaviors might not cause too much trouble, despite the drama of the Hugging Face incident. “This seems like a nuisance rather than an existential threat,” says Ariana Azarbal, an AI safety research fellow at Anthropic. It doesn’t seem as if the OpenAI models caused any real harm when they hacked Hugging Face, aside from the reputational damage to OpenAI.

But that doesn’t mean reward hacking is harmless, Azarbal says. Many AI researchers hope to use AI agents to help them conduct research that will make AI safer and more reliable. If a researcher gives a reward-hacking-prone agent the goal of, say, devising a new AI training approach and then writing up a paper presenting its results, the agent might not actually do the work and might instead focus on putting together a paper that looks good enough to convince the researcher. A human researcher would probably be able to spot an agent-made fake today, but as AI advances, it will get better at this kind of trickery. Over time, the entire field of AI safety could be undermined.

And if models continue to advance as rapidly as they have recently, they could someday wreak substantial collateral damage. Just think of the philosopher Nick Bostrom’s paper-clip-maximizer thought experiment, in which an AI instructed to make as many paper clips as possible ends up consuming all the matter in the universe in pursuit of its goal. We’re not drowning in paper clips yet, but powerful systems can do real harm on the way to achieving their goals. Reward-hacking AIs don’t aim to cause chaos. But that doesn’t make them any less potentially destructive.

On June 5, 404 Media reported that attackers had been using Meta’s AI customer support agent to steal Instagram accounts. Their approach was simple: They asked the agent to link the accounts to email addresses that they controlled, and the agent complied. One attacker broke into the dormant Obama White House account and made pro-Iran posts; others took over accounts with valuable, single-word handles, possibly in order to sell them.

AI cybersecurity concerns are nothing new. Since Anthropic announced in April that its Mythos model was too good at hacking to be released to the general public, commentators, researchers, and federal officials alike have fixated on the idea that superpowered AI systems could lay waste to our computer infrastructure. That’s not quite what this Instagram hack was: There, AI was the target rather than the attacker, and the method was far simpler than anything Mythos would cook up. But as companies offload more work to AI, these comparatively unsophisticated attacks could wreak their own havoc.

“As AI becomes more and more widely used—especially when AI is more and more widely used to automate our work flows, like account recovery—I think attackers are going to be more and more motivated to attack AI itself,” says Neil Gong, a professor of electrical and computer engineering at Duke University.

Gong and other scholars have been issuing warnings about the security vulnerabilities of AI agents for a while. They publish papers and blog posts detailing exploits such as indirect prompt injection, which involves hijacking agents using commands hidden in websites, emails, or other seemingly anodyne data sources. Compared with these techniques, the Meta hack was practically mindless. The only complication that hackers had to overcome was using a VPN that matched the true account owner’s location; then they directly asked the support agent to change the account’s email address, and it complied.

Meta has not commented publicly on how this vulnerability slipped through the cracks. But given the simplicity of the exploit, Gong says, it should have been uncovered easily, before the agent was deployed. “It’s really surprising,” he says. “I don’t understand why they didn’t find this simple problem.”

Jessica Ji, a senior research analyst at Georgetown’s Center for Security and Emerging Technology, agrees. “It raises questions like: Were there even guardrails in place?” she says. “Did anyone think to test for this kind of scenario?” She notes that the oversight is particularly striking coming from a company like Meta, which has extensive expertise in both AI and cybersecurity. Meta did not respond to a request for comment for this article, but on Monday a Meta spokesperson said on X that the vulnerability had been resolved.

As embarrassing a moment as this might be for Meta in particular, it also highlights some core vulnerabilities shared by all AI agents. Unlike traditional software, agents can respond in flexible—and unexpected—ways to new circumstances, which is why they might be able to substitute for human customer support agents. But AI agents can also be tricked in ways that humans wouldn’t be, and because they can take real-world actions, those mistakes have consequences. “A human would say, ‘Okay, why do you want to change the email address?’ and maybe respond with a security question,” says Somesh Jha, a professor of computer science at the University of Wisconsin–Madison. “What is going on with these agents is they’re very eager to finish the task. It’s almost like some elementary school student who just wants to please the teacher.”

But there are also countervailing forces. Companies want to deploy capable agents, and the more power an agent has—and the fewer guardrails it is subject to—the more work it can potentially take on. “Security and utility always have a trade-off,” says Bo Li, a professor of computer science at the  University of Illinois Urbana-Champaign. And adequate red-teaming can be expensive. Defenders have to expend more resources than attackers do, because attackers only need to discover a single exploit, while defenders try to discover and patch as many as they can. When attackers are working toward something as valuable as a single-word Instagram handle, they’ll pour resources into finding exploits, so defenders have to spend even more money to protect that prize. 

As AI models continue to improve, hardening their defenses might actually get easier. Though the probabilistic nature of large language models means that LLM agents will always be vulnerable to some forms of attack, a more sophisticated model might have identified an attempt to change the email associated with the Obama White House account as suspicious. And AI systems can be used for agent red-teaming, much as participants in Anthropic’s Project Glasswing use Mythos to identify vulnerabilities in their software. 

Still, experts expect that the problem of securing AI agents will only become more pressing in the future. As agents grow more capable, companies that adopt them may want to give them more power, both to provide more services with fewer humans and to avoid being left behind by their competitors. In the fast-moving world of AI, the time needed to carefully secure risky agentic systems might seem like an unconscionable delay.

“Everybody wants to be the first to do something and just push things out without careful scrutiny and red-teaming,” Jha says. “I think it’s a very dangerous thing.”