Normal view
In This Costa Rican Forest, Monkeys Come Face-to-Face With A.I.
'World models' are the future of AI, but how do they work?
In a few short years, artificial intelligence (AI) has transformed how we interact with computers and threatens to upend wide swathes of the job market. But large language models (LLMs) still struggle with the messy realities of the physical world. Researchers are betting that a new type of AI, called "world models," could fix that.
At a fundamental level, world models do exactly what the name suggests: They build a mathematical model of the world that can then be used to make predictions about how it will change in response to certain actions or changing conditions. The "world" in this context doesn't necessarily mean the entire physical reality. Instead, it refers to the environment the model operates within, which could be anything from a warehouse to a video game.
Exactly what counts as a world model and how best to build such a model remain topics of considerable debate among AI researchers. But world models would represent a significant advance over LLMs, which, despite their impressive capabilities, simply predict the most likely next word in a sequence.
The hope is that by developing a richer understanding of complex environments, world models could allow AI to finally break out of the chat interface, with potentially game-changing applications in areas like robotics, autonomous driving and scientific discovery.
"The idea has strong connections with the intuitive models in our human minds," said Yunzhu Li, an assistant professor of computer science at Columbia University. "We can imagine how the environment is going to change, how an object is going to move when you apply a specific action. And we basically want to also build this kind of model for any robots or any virtual agent so they can imagine the effects of their actions."
Building world models in your mind
While world models are the latest buzzword in Silicon Valley, the concept has deep roots. It first came to prominence in the 1950s, Manling Li, an assistant professor of computer science at Northwestern University, told Live Science. It arose when cognitive scientists attempted to describe the mental models people used to simulate their environments in their heads.
The concept is also deeply connected to, and often inspired by, control theory, Manling Li said. This is a branch of applied mathematics used to create models of physical systems so they can be predictably controlled. It powers everything from thermostats to aircraft autopilot systems.
However, the term "world models" today refers primarily to neural networks that learn models of their environment by training on data. The modern incarnation of the idea can be traced to a 2018 study titled "World Models," by scientist David Ha and deep learning pioneer Jürgen Schmidhuber. Early models from Google, like PlaNet and Dreamer, were among the first to solve tasks by first making predictions about the outcome of different actions.
While the idea behind a world model is fairly intuitive, a more precise definition is any system capable of "action-conditioned future prediction," Yunzhu Li said. This essentially means the model can predict how a particular action will change the state of the world around it.
Making those predictions, Manling Li said, consists of two key tasks: state estimation and state transition. State estimation refers to the ability to perceive the current state of the environment and encode it into a format that the model can compute, while state transition means the ability to predict how a particular action will cause the environment to evolve.
The data that powers new realities
Deep-learning-based world models learn to do both tasks by training on vast quantities of data. But exactly what kind of data and how that data should be encoded and processed are design choices, with different groups taking a variety of approaches, Yunzhu Li said.
World models are trained primarily on video data, although they can also be trained on 3D data captured by light detection and ranging (lidar) or other depth sensors, audio data and even text that explains the relationships between elements in the environment. Crucially, Yunzhu Li said, this has to be paired with action data — things like robot joint angles, movement readings from an inertial sensor, or event text labels describing what action was taken.
Typically, this data is arranged into sequences of state-action pairs — essentially, recordings of what the world looked like and what action was applied at each step. The AI then uses this data to learn a statistical model of what impacts different actions have on its environment, which can be used to make predictions.
This data can be processed in different ways, Yunzhu Li said. One of the most popular approaches is to operate directly on raw pixel data, which represents the state of the world as a series of images and predicts how actions will change them. Another is to use the data to learn 3D geometric representations of the world that more explicitly encode spatial and physical relationships among objects in a scene.
The power of math-based abstractions
More recently, however, there's been growing interest in approaches that operate on a more abstract level. When a neural network learns from image data, it creates high-dimensional numerical representations of the real-world elements that make up the visual scene — known as embeddings — that exist in a mathematical space known as the model's "latent space."
In a pixel-based model, these abstract representations are reconstructed into pixels to make predictions about what will happen next. But it's also possible to do those simulations within the latent space by directly predicting the embedding of the environment's next state. This approach has been popularized by computer scientist Yann LeCun, Meta's former AI head and one of the "godfathers of deep learning," with his Joint-Embedding Predictive Architecture. He has raised more than $1 billion for a startup called AMI Labs, which plans to use the approach to build world models.
The key advantage of the approach is its efficiency, Yunzhu Li said. Pixel-based approaches have to reconstruct the entire image every time they make a prediction, even if only a small portion of the frame changes.
"As humans, when we're imagining the evolutions of the environment, we don't have to imagine the exact value of every pixel," he said. "That is why it makes a lot of sense to think about predicting over the latent space. It is easier to make sure you are only learning things that are task relevant and ignoring the things that are irrelevant to the task."
The flip side, however, is that it's much easier to understand what your model is doing if its predictions are in a visual format rather than abstract representations, Yunzhu Li added. "You have a better ability to debug your system by having something more explicit that is easily human interpretable," he said.
But questions about how best to represent data in a world model are secondary to the bigger issue of where to get that data in the first place, Manling Li said. LLM makers could simply scrape all the text from the internet for the initial foundational models, but high-quality, action-labeled video often has to be painstakingly curated. What's more, that data can be very sparse, she added, because only a small number of pixels in an image may change in response to an action.
"If I have a video camera recording what I am doing currently, it's generally just some very minor movement of my hand; the entire environment is not really changing," Manling Li said.
This is leading to considerable debate about the best architectures for world models. Almost every LLM today is based on the transformer architecture, which excels at rapidly ingesting huge amounts of data. But these models are tuned to dense language data where every word carries some meaning, and they are less suitable for sparse video data, Manling Li said. As a result, people are experimenting with a wide variety of model architectures and the field has yet to converge on a tried-and-true recipe.
The evolution of world models
One area of considerable controversy is whether video generation models, like OpenAI's Sora, count as world models. OpenAI representatives previously claimed it's a "world simulator" and suggested this type of model could be a promising path toward "general purpose simulators of the physical world." However, Yunzhu Li said that because these models are trained on raw video data without any action labels, they cannot be considered true world models.
"It is only conditioned on some initial language prompt and then predicts the entire video," he said. "So it cannot predict the counterfactual futures — for example, what would have happened if you applied a different action?"
But there are also questions around whether the current approach to world modeling can truly achieve its goals. The vast majority of world models today are trained on big chunks of prepared data. In contrast, humans and animals build their mental models of the world through interaction with their environment, which provides continuous feedback that lets them refine their understanding.
"In order to learn the most effective word models, it's highly likely we will also need the world model to make interactions with the environment and learn from those online interactions," Yunzhu Li said. That remains a stretch goal, however, as current neural network technology is incapable of this kind of continual learning. In addition, allowing a half-finished model to interact with the real world raises significant safety concerns, Yunzhu Li added.
Even a more modest world model could prove invaluable for a host of applications, though. Some of the most obvious include helping robots and autonomous vehicles navigate and plan how to complete tasks. But they could also act as a general-purpose simulator for a variety of applications, depending on the data they are trained on, Manling Li said. Such simulators could include more advanced physics engines for video games, digital twins of patients that could guide medical treatment, or even new ways to model physical phenomena like the climate.
Crucially, there is likely to be a broad diversity of world models. That's because the action data crucial to building a world model is fundamentally connected to a particular physical embodiment, such as a robotic arm, a human or a drone. In the short term, at least, this means world models will be adapted for specific applications, Yunzhu Li said, though many in the field have more ambitious long-term plans.
"People are working very hard and hope that with enough compute, with enough data, and with good enough algorithms, we will have this one unified world model that works across the board for many different applications," he said.
Privacy risks from medical AI are greater than previously thought

A new study has revealed that sensitive health information about individual patients can be extracted from medical artificial intelligence (AI) models more effectively than previous research had indicated.
Diagnostic AI models are used to determine whether some given health data belong to a specific class – for example if an X-ray image shows evidence of pneumonia – and offer huge promise for improving patient outcomes. Earlier studies showed that membership inference attacks (MIAs), which determine whether a particular person’s data was used to train an AI model, yielded results no better than random chance. But these previous studies only evaluated the average risk across all patients in a training dataset.
By contrast, in this latest work doctoral student Moritz Knolle and his supervisor Daniel Rückert at the Technical University of Munich (TUM), together with collaborators at Imperial College London and Georg Kaissis from the Hasso Plattner Institute (HPI), examined the privacy risk for an individual patient. “Privacy that only works on average has little meaning. So I asked myself: would I contribute my data to train a medical AI model?” says Knolle.

As detailed in their recent Nature paper, the researchers attacked AI models based on seven different types of real-life medical datasets, including one comprising 21,799 electrocardiograms and another of 377,110 chest radiographs.
“If a model sees a specific data point during training, it usually becomes more confident at predicting it. So in these MIA attacks I compare the predictive probability number for this data point outputted by the medical AI model with the predictive probability number obtained from my model which has not seen that data point,” explains Knolle, adding that the attack is essentially a statistical test to distinguish between the two models.
Their results revealed that the privacy risk for an individual was far greater than the average risk. The team also found that privacy risk is not equally distributed among subgroups.
“We found that patients from minority groups that are under-represented in the training dataset tend to be over-represented among the 99 percentile of extreme [privacy] risk,” Knolle tells Physics World, warning that as medical AI models become larger and more sophisticated the privacy risk is amplified because their deep analysis creates the small changes in predictive probability that MIAs are good at finding.
While an MIA that reveals someone is a UK citizen poses little security risk, discovering that an individual has received a specific type of cancer treatment reveals personal information that could, for instance, adversely affect their medical insurance premiums. “The more specific the training cohort is, the larger the privacy risk,” states Knolle, crediting his medical doctor parents with inspiring his desire to improve healthcare AI.
Countermeasures include differential privacy (DP), in which a small amount of Gaussian random white noise is added in alongside the data during training, thereby reducing the effectiveness of MIAs while maintaining the model’s diagnostic accuracy. “This gives a mathematically provable upper bound for how much privacy risk any individual in this dataset could face, by limiting the information content that flows to the AI model,” explains Knolle.
He notes that, unfortunately, such DP tools have not been widely adopted by hospitals or clinics. This, he fears, could undermine patients’ confidence in sharing their data, and thereby not only reduce AI’s ability to help combat rare diseases, but also compound AI discrimination that occurs when there is too little data from minority groups included in training datasets.
“So we are hoping to inspire the medical AI community to do risk assessments before deploying AI models, and use verifiable strategies such as DP or limiting access to the AI model, so that as an individual you have a statistical guarantee that no harm would come to you by being part of a dataset,” concludes Knolle, who now plans to investigate the privacy risks posed by generative AI models for healthcare.
The post Privacy risks from medical AI are greater than previously thought appeared first on Physics World.
Large physics models boost the performance of passenger cars
This episode of the Physics World Weekly podcast explores how artificial intelligence can be used to improve the aerodynamics of passenger cars and other vehicles. My guest is David Wheater, who is vice president automotive at UK and US-based PhysicsX.
An aeronautical engineer by training, Wheater spent much of his career using aerodynamics to improve the performance of Formula 1 racing cars for several F1 constructors.
Now, his focus is on using AI-based “large physics models” to improve the performance of mass-produced vehicles. This involves combining computational fluid dynamics simulations with wind-tunnel experiments and real-world testing. In this wide-ranging conversation, Wheater explains the need for high-quality training data and we also explore career opportunities for physicists in this multidisciplinary industry.
The post Large physics models boost the performance of passenger cars appeared first on Physics World.
AI’s black box problem: discovering physics we don’t understand
In The Hitchhiker’s Guide to the Galaxy, Douglas Adams imagined a supercomputer called Deep Thought, built by a race of hyper-intelligent beings to calculate the Answer to the Ultimate Question of Life, the Universe and Everything. After a mere seven and a half million years of computation, the machine finally revealed its answer: 42.
There was just one problem. Despite their hyper-intelligence, none of those beings understood what the question had been. The joke was that any answer – no matter how precise – is meaningless without a thorough understanding of both the question and the route taken to obtain the answer.
Yet what was once purely science fiction is beginning to sound unexpectedly relevant to modern research. From structural biology to particle physics, artificial intelligence (AI) systems are increasingly involved in research, providing results of remarkable power and accuracy. They are being used to identify hidden patterns in vast datasets, to generate hypotheses, to accelerate simulations and to guide experiments.
As AI becomes increasingly embedded in the scientific process, could we be entering an era of discoveries without understanding?
But in some cases, even the creators of those AI systems are struggling to explain exactly how the AI arrives at its conclusions. All of which raises an uncomfortable possibility. As AI becomes increasingly embedded in the scientific process, could we be entering an era of discoveries without understanding? And if so, what happens when science starts producing its own versions of the answer “42”?
AI terms and conditions
Artificial intelligence (AI)
Intelligent behaviour exhibited by machines. But the definition of intelligence is controversial so a more general description of AI that would satisfy most is: the behaviour of a system that adapts its actions in response to its environment and prior experience.
Machine learning
As a group of approaches to endow a machine with artificial intelligence, machine learning is itself a broad category. In essence, it is the process by which a system learns from a training set so that it can deliver autonomously an appropriate response to new data.
Artificial neural networks
A subset of machine learning in which the learning mechanism is modelled on the behaviour of a biological brain. Input signals are modified as they pass through networked layers of neurons before emerging as an output. Experience is encoded by varying the strength of interactions between neurons in the network.
Training data
A set of real or simulated data used to train a machine-learning algorithm to recognize patterns in data indicative of signal or background.
Generative AI
A type of machine-learning algorithm that creates new content, such as images or text, based on the data the algorithm was trained on.
Computer vision
A branch of AI that analyses, interprets and extracts meaningful data from images to identify and classify objects and patterns.
Natural language processing
A branch of AI that analyses, interprets and generates human language.
AI and accelerated particle searches
For particle physicists, this question is no longer hypothetical. At CERN, the home of the Large Hadron Collider (LHC), AI is already woven into day-to-day research, subtly reshaping how physics research is carried out. Indeed, machine learning (ML) – one of the main pathways through which AI is achieved – has long played a role at the LHC.
Back in the early 2010s physicists working at the CMS and ATLAS experiments, routinely used ML in their search for the Higgs boson. The ML algorithms separated rare signals – a Higgs boson decaying either into two photons or into four leptons via Z bosons – from far more common Standard Model processes that produced similar signatures in the detector.
Machine learning did not discover the Higgs boson by itself, but it accelerated the search by making analyses more sensitive. In fact, achieving the same sensitivity without ML would have required between 15% and 125% more data, according to a review of Higgs-decay measurements (Nature 560 41). Some measurements would even have needed more than double the data.

But the scope of AI in particle physics has been expanding rapidly since then. Neural networks are now used to compress enormous datasets, accelerate simulations, and identify unusual events hidden among billions of particle collisions. By learning directly from examples, neural networks can uncover complex correlations in detector data that would be difficult for physicists to unravel using conventional analysis techniques.
Caterina Doglioni, an experimental particle physicist from the University of Manchester, UK, who uses ML for data compression as part of the ATLAS collaboration, says that deployment of AI is becoming difficult to separate from particle-physics research itself. “It’s pretty much everywhere,” she says, especially when it comes to experimental high-energy physics. “It’s critical at the LHC where far more data is created than can realistically be stored.”
When it comes to compressing detector data, for example, one option is to use neural networks to force information through a computational “bottleneck” that preserves the most important features while dramatically reducing the storage required. The original data can then be reconstructed from this compressed summary, either exactly or approximately depending on the application (Rep. Prog. Phys. 84 124201).
This is different from the ML techniques used during the Higgs search. Whereas Higgs analyses relied on supervised algorithms that had been trained to distinguish known signals from background noise, these data-compression methods typically use “self-supervised” learning. They identify and preserve important patterns – without requiring labelled examples.
Algorithms learn what ordinary collisions look like and then flag events that deviate from those expectations
More recently, researchers have begun applying ML to a very different challenge. In the Higgs search, algorithms were trained to recognize signatures predicted by theory and distinguish them from background processes. But now, instead of having algorithms that search for a specific predicted signal, those algorithms learn what ordinary collisions look like and then flag events that deviate from those expectations.
In what is essentially a clever form of anomaly detection, the hope is that new physics might reveal itself as something unusual, even if physicists do not yet know exactly what form it will take. In a nutshell, ML used to be applied to problems where physicists knew what they were looking for; now it is applied to problems where we don’t know what we’re looking for just yet.
While no new particles have yet been spotted with these anomaly-detection techniques, they have progressed from theoretical proposals to real experimental searches. In 2024, for example, the ATLAS collaboration reported the results of an unsupervised anomaly-detection search using Run 2 data. No significant deviation from Standard Model expectations was observed, but the study proved that such methods can be deployed successfully on real collider data (Phys. Rev. Lett. 132 081801).
The black box problem
Yet the very feature that makes these systems so powerful – their ability to rapidly identify patterns humans may miss – also raises one of the most persistent concerns surrounding AI in research: the black box problem. Modern ML systems can uncover complex relationships in data, but researchers may struggle to understand exactly how a model arrived at a particular result. In particle physics, where extraordinary claims require extraordinary evidence, that lack of transparency can create unease.
For David Sutherland, a theoretical physicist at the University of Glasgow, the greater concern is not necessarily whether researchers can interpret every internal feature of a model – but whether they can trust and reproduce its outputs. “I would certainly say reproducibility [is more important] than interpretability,” he says.
That distinction matters. Science has always relied on the principle that results should survive scrutiny, be independently verified, and withstand repeated testing. A model does not need to reveal every detail of its inner workings – but researchers must be able to demonstrate that it behaves robustly across datasets and conditions.
Particle physics, in particular, has built extensive safeguards around that process. “We have very large review committees that check basically everything that goes on,” Doglioni says, with those structures providing an additional layer of protection against over-reliance on opaque systems.
Reliability and reproducibility
Yet as AI systems become increasingly embedded in research workflows, ensuring that reliability may become more difficult. Reproducing results may require not only access to data and code, but also to model architectures, training conditions and computational environments.
Unlike many commercial AI systems, the ML tools used in particle physics are often developed by particle physicists and shared openly with the scientific community. Even so, reproducing results may require access not only to data and code, but also to model architectures, training conditions and computational environments, all of which can influence an AI system’s behaviour.
The growing use of AI could also begin to change how physicists work and even how scientific value is measured. Christoph Weniger, a physicist at the University of Amsterdam who applies ML and AI techniques to particle physics and cosmology, argues that AI systems are likely to take on an increasingly active role in navigating complex analyses. Researchers, he believes, will shift into “a supervisory role in steering these different agents”, fundamentally changing how scientists interact with data.
AI and materials science

Particle physics is far from the only field exploring how AI might reshape scientific discovery. Researchers at Argonne National Laboratory in Illinois, US, recently developed what they describe as an AI adviser for designing advanced electronic materials.
Normally, materials discovery proceeds through a slow cycle of trial and error. Scientists propose a candidate material, synthesize it in the laboratory, test its properties and then refine the design. Each iteration can take days or weeks.
The Argonne system closes that loop. AI proposes candidate materials, robotic laboratories create and test them, and the results are fed back into the system to guide the next round of experiments.
Rather than analysing data after experiments are complete, AI becomes involved in deciding which experiments to perform in the first place, an early glimpse of what future “AI scientist” systems might look like.
The approach has already demonstrated its potential. In 2023 researchers at Argonne and collaborators reported that their autonomous A-Lab platform conducted 355 experiments in just 17 days and successfully synthesized 41 novel materials (Nature 624 86). The work provided one of the clearest demonstrations that AI systems can help guide experimental discovery rather than simply analyse data after the fact.
Creative impact
Such changes could extend beyond day-to-day workflows. Traditionally, the currency of academic research has been publications. But Weniger suggests that AI may shift the emphasis away from productivity alone and towards something harder to quantify: originality.
Scientific value may increasingly depend not on how many papers a researcher produces, but on the creativity and impact of the questions they ask
As routine technical barriers to experiments and theoretical work become easier to overcome, he argues that scientific value may increasingly depend not on how many papers a researcher produces, but on the creativity and impact of the questions they ask.
The effects may also extend to how scientific knowledge is communicated. The scientific literature has long been criticized for being difficult to navigate, not simply because of its volume, but also because of the way researchers write. Nichol Furey, a mathematical physicist at Humboldt University of Berlin whose work explores whether there is an underlying algebraic logic to the structure of the Standard Model, believes AI could help bridge that gap.
“Authors in maths and physics often write in ways that are overly opaque,” she says. “AI systems write with their audience in mind, whereas human authors often don’t.” Yet that possibility creates a tension. The same tools that can make research more accessible can also make scientific text dramatically easier to produce.
Communities are already beginning to grapple with the unintended consequences of widespread AI adoption. Recently, arXiv, one of the world’s largest repositories for physics and mathematics preprints, announced a crackdown on unchecked AI-generated submissions. It said it was introducing one-year bans for authors who upload papers containing obvious signs of unverified AI content, such as hallucinated references or leftover chatbot instructions.
The move reflects growing concern about the quality of AI-assisted scientific writing. Although arXiv’s new policy is too recent for its impact to be fully assessed, the threat of a one-year ban marks one of the strongest responses yet by a major scientific repository to unchecked AI-generated content.
The concern is not simply that AI can generate poor research. Now that producing scientific text is an almost frictionless exercise, the burden of judgement shifts increasingly onto readers, reviewers and future researchers trying to separate genuine insight from noise.
Training future physicists
Another potential problem with AI’s infiltration in many areas of research is training the next generation of physicists. This is generally a long process, where expertise is built gradually through years of practice and failure. If AI automates many of those steps, researchers may eventually face an uncomfortable question: how do you train future experts if the work that once trained them disappears?
This challenge extends far beyond physics. Across the education sector, schools and universities are already grappling with the ease with which generative AI can produce essays, solve problem sheets and complete programming assignments in seconds.

In fact, Wrishik Naskar, a particle theory postdoc at the DESY laboratory in Germany, wonders if research habits are also beginning to change. “When I get stuck, I go to my supervisor,” he says, “[but] many people, the first thing they ask is ChatGPT.” That approach might speed things up, but does it really help a researcher learn and grow? As Naskar puts it: “Focus on the learning. Results will follow if you have skills.”
Despite the many concerns, not everyone sees the transition to AI as a threat, and framing AI as a competitor to researchers may miss the point entirely. For example, Admir Greljo, a particle theorist at the University of Basel, believes the real shift is not replacement but collaboration. “It’s not that they will compete against AI. Entering into theoretical physics is still going to be very interesting and very important. AI is not competition, it’s a new tool.”
That perspective echoes a broader historical pattern. Physicists once calculated by hand, then with slide rules, then with computers. Each technological leap changed how research was done and prompted fears about what skills might be lost along the way. Yet rather than eliminating physicists, those tools expanded what physicists could ask.
Towards an AI scientist?
AI may simply be the next step in that progression. But it also presents something new. Unlike previous tools, AI can suggest ideas, identify patterns and increasingly participate in parts of the scientific process that once seemed uniquely human. So might that participation one day become independence? Could AI move beyond assisting physicists and begin carrying out the entire research process itself?
For Sutherland, a future in which AI systems perform the entire research cycle, from generating hypotheses to collecting data and even writing papers, is not difficult to imagine. But he still envisages a human “steering it” and acting as a “sanity check”, much like the relationship between a senior researcher and a PhD student.
Others go further. Greljo believes that fully autonomous scientific systems may not arrive in the near future, but sees no fundamental reason why they could not emerge eventually. Because science ultimately depends on reproducible results, he argues, humans could simply repeat calculations and test predictions to verify whether the AI had reached the right answer. Such a future, he says, is “not something that is impossible”.
We may be entering a world where machines help us uncover truths about the universe faster than we can fully understand them
We may be entering a world where machines help us uncover truths about the universe faster than we can fully understand them. But that need not signal the end of scientific inquiry. It is important to remember that AI models, much like many of the models used throughout scientific research, are imperfect and may always remain so. But perhaps they do not need to be perfect.
Science has always involved navigating uncertainty, constructing incomplete models and refining them over time. AI may help us discover things faster than ever before. It may reshape how future generations of physicists learn, work and think. But science has never simply been about generating results. It has also been about explanation, curiosity and deciding which questions are worth asking in the first place.
We should not treat AI as an oracle that bypasses the scientific method. Instead, we should view it as another tool in the physicist’s arsenal – a powerful one, certainly, but still a tool, helping researchers explore possibilities, test ideas and refine their models of reality.
After all, the problem with Deep Thought was never that it gave the wrong answer. It was that nobody knew what question they had asked.
The challenge for the next generation of physicists may not be deciding whether AI can discover new truths about nature. It may be making sure we still know which questions are worth asking.
How do you think AI is changing the face of physics? What are the threats and opportunities it poses – and what happens if AI produces answers we cannot understand?
Send us your thoughts by e-mail to pwld@ioppublishing.org
The post AI’s black box problem: discovering physics we don’t understand appeared first on Physics World.
In the age of AI, we must double down on the skills that make us experts
In 2008 the US author Neal Stephenson published a speculative fiction novel called Anathem. Set on the fictional planet of Arbre, the story follows an order of monks, mostly isolated from the rest of the world, who every so often open up their doors to receive information on the state of the outside world.
In the book, Stephenson explores a Dark Age of the Internet, in which companies flood the network with plausible-sounding-but-false information in a ploy to force consumers to buy the firms’ own tools for distinguishing truth from falsehood.
Initially, this “crap”, as Stephenson calls it, had to be made by humans, but eventually the process is run by a so-called “Artificial Inanity”, which is able to create hundreds or thousands of bogus versions, or “bogons”, for every legitimate document.
Today, we face our own version of Stephenson’s Dark Age as our information landscape increasingly fills with AI slop, now going beyond AI-generated videos of uncanny kittens or images depicting politicians and celebrities in outlandish situations.
With the rise of chatbots powered by large language models (LLMs), our search results, social-media feeds and even scientific journals are getting polluted with data that, at first glance, appears relevant and plausible but contains no value beneath its gilded veneer.
As LLM-generated slop infects our previously trusted knowledge base, we face an information landscape that is quickly rotting beneath us.
As scientists, we must position our work in relation to other recent research. Yet that task gets harder when we have to sort out the real from the hallucinated. As LLM-generated slop infects our previously trusted knowledge base, we face an information landscape that is quickly rotting beneath us.
Where we could once track an idea to a paper with human authors (who bore consequences for acts of fraud), we face a future where we have to question whether an article’s authors, institutions or experiments ever existed, on top of evaluating its scientific merits.
This is not a hypothetical danger. In 2024 a team led by medical researcher Almira Osmanovic Thunström at the University of Gothenburg invented a fictitious medical condition called “bixonimania” and publicized it through posts on Medium and two pre-print papers.
Their goal was to test whether AI chatbots would pick up and propagate word of the fake disease, which they duly did. The researchers even tried to insert information that made the work obviously fake. This included acknowledging a colleague from Starfleet Academy, thanking the University of Fellowship of the Ring for funding, and even writing early on in the article that “this entire paper is made up”.
Trapped in a bog
As physicists, we are not immune to the dangers of information rot. A flood of LLM-generated articles prompted arXiv in May to announce consequences for authors whose submissions have “incontrovertible evidence that the authors did not check the results of LLM generation”, including having “hallucinated references”. Those consequences include a one-year ban from arXiv and the requirement, thereafter, that any submitted paper must already be accepted at a “reputable peer-reviewed venue”.
I initially welcomed the news. Harsh consequences are one way to discourage LLM usage, especially for sloppy shortcuts. But the more I reflect on it, the more I realize that arXiv’s consequences could eventually fall even on those who are not using LLMs to write their papers.
Relying on shortcuts is easy. Say you’re writing a literature review and paper A, which you’ve read and trust, has a fact you want to cite – but you notice that paper A cites paper B for this fact. It’s simple to scroll down to the reference section and copy paper B’s citation into your own citations (I would be lying if I said I’ve never done this myself).
But as the information landscape rots, this shortcut could run you afoul of policies like arXiv’s. What appears to be a valid citation – maybe even containing the names of authors you know personally – can be hallucinated.
We must double down on the skills that make us experts – our scientific judgement, information literacy, communication skills and careful scholarship.
Or it could be entirely fake, as a team of researchers who cited one of the “bixonimania” pre-prints in a peer-reviewed article discovered. Their paper was justifiably retracted by the journal editor, over their objections. By simply copying a reference without double-checking it, you can unintentionally join ranks alongside LLM users. Worse, you open yourself and all of your co-authors to being banned from arXiv.
I am not suggesting that arXiv’s policy is flawed. Rather, I argue that those who intend their writing to be AI-free must be vigilant. We must double down on the skills that make us experts – our scientific judgement, information literacy, communication skills and careful scholarship. We should lean into practising these skills – and teaching them – even though they are precisely the skills that AI companies claim LLMs can replace.
In Anathem, a character simply hand-waves away their Internet full of bogons, but we have no such solution. Instead, as our information landscape rots, we find ourselves trapped in a bog. To get out, we must plan each step carefully and take none for granted. What appears to be solid ground may well prove treacherous. Moving safely is a slow process, full of checking and double-checking as we painstakingly build out paths and tools we can trust.
Simply choosing not to use an LLM when writing is no longer enough. We have only our own scholarly skills to sort the bogons from the ground truth. We will all have to slow down if we want out of the bog.
The post In the age of AI, we must double down on the skills that make us experts appeared first on Physics World.
AI-enhanced rare-event sampling helps predict extreme weather
The frequency of extreme weather events can be predicted more accurately than presently possible by combining artificial intelligence (AI) with physical climate modelling, using a protocol called rare-event sampling. That’s the conclusion of a study from researchers in the US and France. The researchers used the approach to model extreme heat events such as the one currently roasting Europe, but they believe it could also be applicable to many other extreme events in climate science.
As the global climate warms, extreme weather is becomingly increasingly deadly and difficult to predict, and quantifying the risks of such events is important for climate change adaptation and mitigation. “Very often the rarest events carry the largest impacts,” says climate physicist Amaury Lancelin of France’s Laboratory of Dynamic Meteorology.
Full global climate model simulations will need to run for an infeasibly long time to produce sufficient data for an estimate of the precise frequencies of rare events. Alexander Wikner of the University of Chicago in Illinois gives the example of an event that, on average, occurs once every century: “Each year you’re going to flip a coin that has a one-in-a-hundred chance of coming up heads,” he says. “On average, once every hundred years, you’ll get one head, but you could very easily get no heads or two or three heads.” A meaningful estimation of the frequency therefore would traditionally require around a thousand years’ worth of simulation – which is not computationally feasible in a physical model.
Deep-learning algorithms, which ignore the underlying climate physics and simply train themselves using pattern recognition, require up to 10,000 times less computing power. However, the reliability of these in modelling extremely rare events is questionable. For example, there is evidence that an AI model will not predict more extreme cyclones than it has seen in its training data, “which is a quite unnerving idea if we imagine that tropical cyclones will be getting more extreme in the future and we want to use an AI model to forecast them,” Wikner explains. They also produce no insight into the physics of the events.
Rare-event sampling (RES) refers to techniques that preferentially allocate computing resources to specific rare events of interest, thereby avoiding modelling long periods of time in which none of those events occur. The problem, of course, is that one has to have some idea what kinds of conditions increase the probability of a rare event occurring in order to know which periods to model in more detail.
In the new research, scheduled for publication in Physical Review Letters, Lancelin, Wikner and colleagues developed the AI+RES framework. An AI algorithm runs climatic simulations repeatedly and selects those that it predicts are most likely to lead to the rare event. A full climate model then simulates only these. The researchers used this protocol to compare the frequency of heatwaves at mid-latitudes, using the predictions of a relatively coarse-grained direct numerical simulation called PlaSim as the ground truth.
They found that their technique produced similar results to PlaSim with up to 1000 times lower computational resources. The researchers now hope to apply the technique to predict other types of extreme events and apply it to more sophisticated models.
“The reason we used PlaSim is that it is computationally quite cheap compared to state-of-the-art climate models so that we can actually verify that our methodology works,” says Lancelin. “Otherwise there is no way to know whether we have made a garbage prediction or not without the ground truth.” Knowing that the AI+RES model reproduces the PlaSim results correctly, however, the researchers now hope to apply computationally expensive models in situations where it was previously impossible, with confidence that the results are likely correct.
Climate scientist Robin Noyelle of ETH Zurich believes the work marks a significant contribution to ideas that were already “floating out there”. He says that, although RES has been in use for nearly ten years, selecting the best candidates for full modelling has always proved challenging: “People were using things that we thought were reasonable – I did that during my PhD – so if I want to look at hot summers, then I select on temperature.”
This approach fails for short-term extreme events, however, because of the dynamic nature of the atmosphere. The AI model itself also failed to produce good accuracy when emulating the events, but it was “basically free” in terms of the computing power required, and provided a good starting point for the physical models. “This coupling between the two ideas is really new,” Noyelle says.
The post AI-enhanced rare-event sampling helps predict extreme weather appeared first on Physics World.
How AI is transforming human robots
It was fantastic to see the men’s marathon world record being broken in London at the end of April when Kenya’s Sabastian Sawe became the first person to officially complete the run in under two hours. It’s an achievement that a whole generation of world-class runners had had their eyes on, with Sawe completing the race in a headline-grabbing 1:59:30.
Perhaps less attention was given to another achievement a week earlier, which saw a humanoid robot run the Beijing E-Town half-marathon in 50:26, breaking the human record by almost seven minutes. What’s particularly impressive is that at the inaugural event in 2025, the robot winner took 2:40:42 to complete the race, with many robot entrants failing even to reach the finishing line.
In just one year, in other words, the robot half-marathon record has been slashed by more than two-thirds, an unthinkable feat in human terms. What’s more, the rules between the first and second robotic competitions were tightened. They now favour robots that can run on their own (rather than via remote control) and penalize those that need to be taken off course to have their batteries changed.
Giant steps
My mind began spinning, trying to work out what must have happened over the past year to realize such advancements. How did we go from remote-controlled, slow and clunky robots to autonomous, agile and sleek devices? It turns out that these increases in speed and endurance have been driven by a combination of better hardware, specialized designs and advanced algorithms.
Robots now benefit from powerful actuators enabling higher torque and faster movement of its hips and knees. Anatomical designs have also been optimizied to mirror biological efficiencies by, for example, using lightweight limbs and lower torso components to minimize the energy lost when a robot’s foot strikes the ground.
Other changes include the use of liquid cooling technologies to stop a robot from getting too hot during prolonged activity and finding ways to mimic highly efficient natural bipeds, such as an emu or ostrich. Improved AI and control software have also allowed robots to navigate varied terrain and stay stable autonomously, rather than relying on remote operation.
AI answers
Of all these developments, I believe the most significant is the expanding use of AI. To me, it seems that AI will be the enabler for robots to become more “human”. In China, humanoid robots are already being developed to automate what is dubbed the hardest task in the car industry – the final assembly stage.
This work typically consists of real people carrying out highly labour-intensive tasks such as installing wiring harnesses, fitting a vehicle’s interior trim and instruments, or bringing its engine and chassis together. Being super-precise tasks that require manual dexterity, this work has previously been thought too complex for robots.
However, through a combination of imitation learning, simulation and real-time human guidance, it’s now becoming conceivable for robotic devices to do this kind of work. Underpinning this aspiration are what are known as Vision-Language-Action (VLA) models, which process camera images together with natural language or text descriptions and translate these into physical actions.
Having emerged relatively recently from work at AI firm Google DeepMind, VLA models differ from traditional, task-specific programming robotics by offering generalization and the ability to handle novelty. They are seen as the next big thing in AI because they bridge the gap between understanding the world (vision and language) and interacting with it (action) – potentially letting robots do tasks they have not been explicitly trained on.
VLAs typically rely on existing Vision-Language-Models (VLMs), which provide a combined knowledge of text (large language models) and images (computer vision models). The VLA model is then fine-tuned on task-specific data to learn a mapping of visual observations and text instructions to create the desired robot action.
One challenge in training VLAs is getting enough data with appropriate visual and contextual information. It may be gleaned from real robots (imitation) but it can also be generated by human guidance or teleoperation – a manual process involving humans “showing” robots how to do a specific task. The task is time-consuming, having to be performed deliberately and repeatedly using equipment that captures appropriate levels of data quantity and quality.
A third option is to build virtual simulation environments to train the models and optimize robotic movements. As VLAs inherently learn to associate higher-level cognition with lower-level physical actions, the trend will be to move from fixed, pre-programmed scripts towards AI and large-scale simulation training. The result: robots that become more autonomous and versatile in real-world situations.
We are all getting used to AI becoming an increasing part of our everyday life – think how much GenerativeAI models have improved since they first appeared a few years ago. But I wonder what will be next for robots as they become capable of moving independently and spontaneously, having the ability to operate with full autonomy within dynamic human environments, such as homes and factories.
When we – eventually – experience the full physical embodiment of Artificial General Intelligence, will it be like living on the filmset of I, Robot or Blade Runner, with humans and humanoids coexisting and, in some cases indiscernible? Several years ago, this was something I did not expect to see in my lifetime. But in April it just got one giant leap – or at least one half-marathon – closer.
The post How AI is transforming human robots appeared first on Physics World.
Physics-based models still beat AI for predicting extreme weather events
Artificial intelligence (AI)-based weather models are not as good as physics-based forecasting systems at predicating extreme weather events, say researchers at the University of Geneva, Switzerland and the Karlsruhe Institute of Technology, Germany. After comparing the outputs of different models on the same dataset of extreme events, the team found that AI models systematically erred on the side of normality, underestimating temperatures for extremely hot events while overestimating them for cold ones. This could be because the models learn from what has already happened and struggle to forecast events outside their training data.
With extreme weather events growing more common and intense due to our rapidly warming climate, being able to predict them is becoming ever more important. In recent years, meteorologists have sought to address this by developing a new generation of AI models that vie with physics-based numerical weather prediction (NWP) systems in the accuracy and extent of their forecasts – at least for everyday weather events.
Black or grey swans
In the new work, researchers led by Zhongwei Zhang and Sebastian Engelke sought to understand whether AI could also be competitive when forecasting extreme weather episodes. These episodes are usually defined by variables such as wind, atmospheric pressure or temperature falling well outside norms for location and season, and are sometimes termed “black swans” or “grey swans” depending on how extreme they are.
“In the past, only relatively moderate extremes were documented,” Zhang and Engelke observe. “But given the current rate of high rate of global warming, record-breaking events now sometimes exceed previous record levels by large margins.”
That’s potentially a problem for AI, they add, because several recent studies have shown that such models come up short when asked to extrapolate beyond their training data. For example, in 20205, researchers at the US National Oceanic and Atmospheric Administration (NOAA) and the Allen Institute for Artificial Intelligence in Washington, US, found that a seasonal AI forecasting model could not predict values for the North Atlantic Oscillation, which plays a crucial role in Europe’s weather and climate. Another study found that models tend to underpredict the intensity of the most extreme storms (as measured by mean sea-level pressure) or other high-impact events such as heat waves.
A large sample of record-breaking events
To test this hypothesis, Zhang and Engelke constructed a large dataset of record-breaking events for heat, cold, and wind extremes during 2018 and 2020. This dataset included several well-known events, such as the Siberian heatwave in early 2020 and the US heatwave of August 2020, but also tens of thousands of less-heralded ones. The 2020 dataset, for example, included 162,751 heat, 32,991 cold, and 53,345 wind records spread across different seasons and climatic zones.
The researchers assessed how well the three leading deterministic AI weather models –GraphCast, Pangu-Weather (and operational variants) and Fuxi – performed when extrapolating from this dataset of record-breaking events. They then compared these models’ performance to that of the physics-based High RESolution forecast (HRES) model developed by the European Centre for Medium-Range Weather Forecasts (ECMWF), which is widely acknowledged as today’s best physics-based NWP model.
In line with previous studies, the researchers found that both GraphCast and Fuxi were better than HRES at forecasting normal weather events. However, for record-breaking temperature and wind events in 2020, the situation was reversed, with the physics-based HRES model consistently outperforming all AI models for hot and cold temperature records as well as wind speed records.
A difficult study
The researchers report that the most difficult aspect of the study was the sheer length of computation time required to analyse huge AI and numerical forecast datasets. To complicate things further, Zhang notes that some of the most recent AI weather models are being developed by big tech companies and are not publicly available.
As well as showing that purely data-driven AI models struggle to forecast record-breaking extreme weather events, Engelke says the team’s work also provides a protocol for systematically evaluating forecasts of extreme events. “We hope this will motivate the research community to thoroughly evaluate the next generations of AI forecasts to advance our understanding of their advantages and their limitations compared to conventional physics-based models,” he says.
In this study, which appears in Science Advances, the Geneva/Karlsruhe researchers focused on evaluating the forecasts of deterministic AI weather models. They are now doing the same for forecasts made by recent probabilistic AI weather models, which they suspect will face similar extrapolation limitations.
“We are also working on building AI models ourselves that have improved forecast performance on extreme events,” Zhang and Engelke reveal. “This might be achieved this by creating hybrid models, which are a smart combination of physical and AI-based weather models.”
The post Physics-based models still beat AI for predicting extreme weather events appeared first on Physics World.
AI-led solutions of Erdős problems spark debate over the future of mathematics
News that large language models (LLM) have made major advances in solving Erdős problems – a set of problems formulated by the renowned 20th-century mathematician Paul Erdős – has created an amalgamation of uproar and interest among mathematicians. The past month alone has seen two significant LLM-generated solutions. The first relates to prime sets, a generalization of prime numbers, and was solved after Liam Price, an amateur mathematician from the US, fed the problem statement into GPT-5.4 Pro without other information. The second came last week when the company behind ChatGPT, OpenAI, announced that it had used artificial intelligence to disprove Erdős’ planar unit distance conjecture.
LLMs have solved Erdős problems before, but the one Price chose wasn’t just any Erdős problem. It was one that human mathematicians had worked on for 60 years without success. The nature of the solution was also unusual. While previous LLM mathematical proofs used standard techniques, this one took an entirely different approach. Rather than starting from Erdős’ original probability-theory-based framing of the problem, as human mathematicians had, the LLM found an alternative route – one that led naturally, in less than a page, to a correct proof.
“Paul Erdős had a concept of ‘Proofs from The Book’, meaning that the argument is so compact and elegant that this is the proof God would’ve written down in ‘The Book’,” Jared Lichtman, a mathematician at Stanford University in the US, wrote on the social media site X after the proof was announced. “After reading the GPT5.4 proof of Erdős #1196, I would say this is a Book Proof of the result.”
The planar unit distance conjecture, meanwhile, concerns a deceptively simple question: if you have n points in a plane, how many pairs of points can be exactly one distance unit apart? Erdős thought the limit was n1+C/log log(n) where C is a positive constant, but OpenAI’s model identified a higher bound. What’s more, the company claims it did so by coming up with a counter example, and in doing so, a whole family of objects that mathematicians had so far not considered. This counter example was generated using sophisticated ideas from algebraic number theory.
Some members of the mathematics community have greeted these proofs, and the advent of AI in mathematics in general, with enthusiasm. OpenAI’s announcement quotes Arul Shankar, a number theorist at the University of Toronto, Canada, as saying that the new proof “demonstrates that current AI models go beyond just helpers to human mathematicians – they are capable of having original ingenious ideas, and then carrying them out to fruition”.
Others, however, are more cautious. David Bessis, a mathematician-turned-science writer who previously worked on algebra, geometry and topology, claims that the programme of trying to automate mathematics stems from a misconception of mathematics as a logically direct process of churning out theorems, given some rules. Writing in his Substack newsletter, Bessis argues that the method used to verify AI-generated proofs, which involves a computer program called Lean, may reduce the benefit the mathematics community gains from proofs. Notably, proofs that are verifiable in Lean are not always parse-able by humans, which detracts from (and in certain cases removes) the insights researchers typically get from new proofs.
How AI is being used in mathematics…
To evaluate the merits of these arguments, it’s useful to understand how AI is currently used within mathematics research. The first strategy is the one Price used to solve Erdős #1196: directly prompting an LLM. “Large language models have proven their worth at literature search: finding similar instances of a problem, or a proof, in past literature,” notes François Charton, an AI engineer at the California-based start-up AxiomMath, which is using AI to accelerate mathematics research.
The second strategy is to use AI models trained on other types of data. According to Charton, these models are especially good at spotting “weak signals and correlations” and thereby uncovering patterns in data that might be too laborious or convoluted for humans to identify.
Both methods have shown promise for generating new results, but they are not universal – at least, not yet. “It [AI] seems to do a lot better at certain types of maths than others,” says Thomas Bloom, a mathematician at the University of Manchester, UK, who maintains a webpage that tracks solutions to Erdős problems. In particular, Bloom says that to the best of his knowledge, AI “hasn’t done anything interesting in category theory” – a field whose reputation for abstraction is only matched by its track record of bridging supposedly distinct areas of mathematics.

Another challenge is that with AI systems churning out new proofs at scale, there are simply not enough people with the skills needed to check them. A process called autoformalization could solve this problem by turning human proofs into what Bessis calls “bulletproof, machine-verifiable logical derivations” expressed in Lean or other specialized languages. At that point, AI-generated proofs could be checked automatically. The question is, could humans lose out on some knowledge in the process?
For doubters like Bessis, who refers to AI-driven autoformalization (at least as practiced by certain firms) as “AI slop”, the answer is very little. But within the AI camp, there is considerable interest in what AI brings to autoformalization. “I see autoformalization as the bridge in both directions, as important as proving itself,” Charton argues. “We can use Lean to translate between these two languages so that a Lean proof can be reverse-translated into a sketch, lemmas or natural language a human mathematician can engage with. That bidirectional translation preserves and extends mathematical knowledge at scale.”
…and how it isn’t
In the 18th century, when Leonhard Euler began arranging the logical thought processes of mathematics into theorems, definitions and proofs, mathematicians were primarily interested in solving problems with underpinnings in the physical world: questions of volume and distance, and, more generally, geometry and counting. Since then, though, mathematics has become a discipline that is at least as concerned with coming up with interesting problems as it is with solving them.
Two aspects of this change seem relevant to debates over AI’s utility. The first is that posing problems requires a broader skillset than solving them. The second is that solving posed problems sometimes requires mathematicians to invent new structures, tools or objects. Fermat’s Last Theorem, which posits that there are no three positive integers a, b, and c that satisfy the equation an + bn = cn for any integer value of n greater than 2, is a good example. At face value, this nearly 400-year-old theorem seems simple. However, proving it was the life’s work of a modern mathematician, Andrew Wiles, who won the Abel Prize in 2016 for developing the numerous new tools required, as well as for the proof itself.
Coming up with such tools – or indeed whole new frameworks – is a challenging and hugely creative endeavour. There are no rules as to the kinds of objects you are allowed to create, and unlike a proof (which is either correct or incorrect), there is no finality, either. If the new framework is a good one, it will crop up frequently and naturally in various branches of mathematics, and other mathematicians will incorporate it into their own work. If it isn’t, they won’t.
Currently, not even AI enthusiasts like Charton think machines are capable of such leaps. “Theory building is completely out of reach right now,” he tells Physics World. “Models, especially generative models, can provide a mathematician with interesting examples, or discover surprising relations that may bring a theoretical breakthrough, but the breakthrough still depends on the mathematician. I believe this will remain the case for some time.”
A new tool for scientists and mathematicians alike
In many areas of science, AI works in a way that is entirely distinct from human thinking. In physics, for example, machine learning algorithms are trained to analyse large amounts of data, find patterns and use them to infer underlying laws. This strategy could advance our understanding of some of the most fundamental questions in physics, but it is very different from how a human scientist would do it, and therefore perhaps more likely to be seen as a welcome new tool.
On the theorem-proving side of mathematics, the distinction between methods a human might use and those an algorithm might use is more blurred. Yet in some ways, Bloom thinks incorporating AI into mathematics could bring the field closer to other sciences. In particle physics, for example, “you don’t go in and take these individual recordings [of data]. It’s all automated,” he tells Physics World. “Until now, there has been no equivalent for maths. It takes time and attention to prove theorems, and maybe this had been a bottleneck.”
AxiomMath’s Charton agrees. “Every new math tool in history has automated something that used to be the work of a human mathematician – from the abacus all the way to symbolic algebra,” he says. “With each new tool, the role of the mathematician evolved rather than disappeared. Tasks got automated, and problems that felt impossible became trivial – but mathematicians just keep moving up the stack to the next set of questions. I see AI as the latest shift rather than a categorical break from history.”
The post AI-led solutions of Erdős problems spark debate over the future of mathematics appeared first on Physics World.
- Futurism
- AI Companies Are Buying Antique Books, Ingesting Their Contents to Train Models, and Then Destroying Them at Incredible Scale
AI Companies Are Buying Antique Books, Ingesting Their Contents to Train Models, and Then Destroying Them at Incredible Scale
Traditionally, books have been good for two things: reading, and looking nice on a shelf.
But AI companies are interested in neither.
To those building large language models, books are nothing more than fodder to be devoured en masse before being spit out like fishbone. Often, they’re happy to use digital books — or even better, pirated digital books, as Meta has been accused of doing, and as Anthropic was forced to pay a $1.5 billion settlement to authors for also doing.
But many companies, including Anthropic, have turned to ingesting physical books instead, which they can buy countless used copies of on the cheap. According to the settled lawsuit, Anthropic used a hydraulic powered cutting machine to neatly remove the pages from the books it procured from book resellers and then scanned them using industrial-grade imaging equipment. In other words, it was literally ripping off authors’ books to train its AI.
This process took advantage of a legal concept known as first-sale doctrine, which allows a buyer to do what they want with a purchase without the original copyright holder’s say-so. And since Anthropic was turning the original physical texts into digital ones — rather than redistributing them as new copies — a judge found this to be “transformative,” and therefore protected by fair use.
Now, as 404 Media reports, this practice has become prevalent enough that even well-established book sellers are looking to cash in on the AI boom. One called ISBNdb, which boasts the “world’s largest book database,” extolls that the “world’s best AI training data is setting on a shelf,” upholding these physical texts as uncorrupted by shoddy AI writing that’s already polluted so much of the internet (and indeed, newer books).
“Print books from the pre-LLM era are structurally guaranteed to be free of this contamination. That alone is a significant advantage,” it explains in an article on its website, as quoted by 404. “Physical books published before this date [pre-2022] are structurally clean of modern poisoning tools.”
Once focused on helping libraries, distributors, and book shops find and sell books, ISBNdb now helps AI companies bulk-buy anywhere between 1,000 to one million books per order, according to 404.
As an added bonus, it also promises AI companies that it’ll keep their purchases under wraps — nobody wants to end up in the spotlight like Anthropic and Meta, obviously — while clearly sounding aware about how incredibly shady the practice sounds.
“The optics problem is real,” ISBNdb’s site says. “‘AI company destroys two million books’ is not a headline that generates sympathy.”
One small book seller said that in April, he suddenly went from selling no more than 20 books a week to hundreds, and he’s almost certain that the customers are AI labs, noting the random selection of the books and how they all have ISBNs. He added that his inventory is full with rare and out of print books, meaning that an AI company could be destroying some of the few remaining copies that can be found.
“I personally have mixed feelings about all of this,” the bookseller told 404. “It benefits me financially as well as by clearing out old inventory that is otherwise unlikely to sell. I’ve been well-suited for these sales with inventory from overseas and foreign language books. On the other hand, I don’t like the end-use, and I don’t like that uncommon books are being pulped.”
More antique books may be endangered. 404 noted how rare booksellers in the Netherlands are being inundated with bulk purchases they believe are being made by AI companies, though they can’t know for sure.
Similar suspicions are being felt all throughout the industry. Services like ISBNdb facilitate these bulk orders, keeping the AI buyers anonymous as promised. There may be strong hints thats a bulk order is for an AI lab, but booksellers can, for the most part, merely speculate.
More on AI: Author Invited to Give Speech at OpenAI Headquarters, Uses Opportunity to Trash AI to Their Faces
The post AI Companies Are Buying Antique Books, Ingesting Their Contents to Train Models, and Then Destroying Them at Incredible Scale appeared first on Futurism.

Silicon Valley Splits Over Closing the Borders to Chinese A.I.
Voters Are Being Inundated by a Barrage of AI-Generated Election Ads
AI is upending the way voters take part in the democratic process. The worst part? You might not even know you’ve been affected.
In the county of St. Charles, Missouri, a race for county executive has devolved into slop-slinging of the AI variety. According to local channel First Alert 4, conservative candidate Jason Law is flooding the airwaves with an AI-generated video depicting his two rivals Steve Ehlmann and Bill Eigel as co-conspirators in a nefarious plot against voters.
“Steve, wake up, I’ve been telling people for 12 years that I’ll cut their personal property taxes, and they still believe me” the AI-Eigel declares in a generic AI-generated voice, rubbing his palms in glee.
“Property taxes have risen by 300 percent over the past 20 years and I just tell people there’s nothing I can do about it!” a grinning Steve Ehlmann replies.
Near the end of the ad, Law himself appears — also as an AI-approximation, with a stock text-to-speech voice — to declare that “we need a fresh start.”
Per First Alert, Law says the noxious campaign ad was inspired by a similar AI-generated smearjob by conservative reality TV star-turned Los Angeles mayoral candidate Spencer Pratt. Those ads take a similar tone, painting a sloppified Pratt as a superhero, and his opponent as a cartoon villain.
These are far from the only examples. Back in May, similar deepfake attack ads emerged in the race between incumbent Republican congressman Thomas Massie and the Trump-backed candidate Ed Gallrein. Unlike the local races in Missouri and California, both candidates were the targets of AI commercials, which were funded by independent PACs, rather than the candidates directly.
In Massie’s case, AI deepfake ads depicted the long-time representative in fake CCTV footage checking into a hotel with progressive Democrats Alexandria Ocasio-Cortez and Ilhan Omar, declaring “Thomas Massie caught in a throuple!”
During the race, Massie raged that “it’s always the losing campaign that does the crazy crap,” adding that it’s “frankly an insult to older voters who don’t know that AI exists,” Louisville Public Media reported.
Unfortunately for Massie, the ads — not to mention the Trump endorsement — appear to have worked out in Gallrein’s favor, as the challenger triumphed with 54 percent of the vote.
More on AI: AI Companies Are Trying to Seize Control of Elections
The post Voters Are Being Inundated by a Barrage of AI-Generated Election Ads appeared first on Futurism.

- Futurism
- People Are Starting to Record Every Conversation They Have Without Disclosing It, and It’s Gross
People Are Starting to Record Every Conversation They Have Without Disclosing It, and It’s Gross
It’s become strangely difficult to attend a work meeting without the presence of an AI recording tool that’s carefully jotting down what every single participant is saying.
Some people in tech circles are taking the trend to perverse ends by recording every single conversation they’re having — not just at work but in their private lives as well, as the Wall Street Journal reports.
Worse yet, many of them aren’t even asking for permission first, which could technically put them at risk of violating state laws, depending on where the conversation is taking place.
It’s a bizarre extension of our newfound obsession with keeping track of every single thing that’s being said around us, a trend that could have severe privacy implications, from people being shocked to find their therapist has been recording them to wearers of Meta’s AI glasses harassing strangers.
Apps like the transcription service Granola have seen a steady rise in popularity. AI tech executives are growing increasingly reliant on these tools — sometimes to an astonishing degree.
“In the office, we actually get really bummed when we realize we’re having a conversation and we forgot to turn on Granola,” AI marketing platform Flint CEO and cofounder Michelle Lim told the WSJ. “I wish there was a Granola in every room.”
Lim admitted that she simply assumed she was being recorded, noting that being asked “almost kills the mood.”
Others, like AI sales company Yuzu Labs founder Emmie Chang, go even further. Chang admitted to the WSJ that she records most of her first dates by putting her phone on the table or chair. Afterwards, she asks Anthropic’s Claude how the date went, wondering if she could be “more engaging or empathetic.”
It’s a baffling new trend that highlights how pervasive AI and surveillance have become in our daily lives, particularly for those working in tech. And while researchers are still investigating the effects on our lives, there are reasons to be concerned. For instance, experts warn of the dangers of “cognitive offloading” through the extensive use of AI chatbots, a trend that can have corrosive effects on our mental abilities.
Then there are the glaring privacy implications of recording every single conversation, not just with colleagues but with strangers on dates. Unlike Zoom’s built-in recording feature, which announces itself with a robotic voice, Granola does not make its presence known at all.
In the workplace, workers have little recourse, and will likely have to accept being recorded, as privacy law expert Alex Alben told the WSJ.
Nonetheless, “companies need to be careful about technologies that are automatically recording, especially when it involves third parties,” he said.
Other experts put it a lot more bluntly, pointing out the risk of leaking confidential personnel information, the possibility of remarks that could later be interpreted as incriminating, or even trade secrets.
“There are huge risks to the organization on AI note takers,” human resources firm HRCI executive Amy Dufrane told the Associated Press earlier this month. “I don’t think companies should use it at all.”
More on recording and AI: Woman Alarmed When Her Trusted Therapist Starts Recording Her With AI
The post People Are Starting to Record Every Conversation They Have Without Disclosing It, and It’s Gross appeared first on Futurism.

- Futurism
- Meta Working on Crushingly Sad AI App That Tells Bedtime Stories to Children, for “Tired” Parents
Meta Working on Crushingly Sad AI App That Tells Bedtime Stories to Children, for “Tired” Parents
Meta is testing an app that uses AI to automate bedtime stories for children. The future is here, folks — and boy, is it depressing.
The app, first caught by 9to5Mac, is called “StoryKit.” (It’s not yet available in the US; Meta confirmed to TechCrunch that it’s currently testing StoryKit in pilot countries.) Per a webpage for the app, parents pick a character either by creating one or taking a “photo of their favorite toy or person to bring them to life,” then describe a setting or world for the story to take place in. They can also ask the story to include a specific lesson or moral, which the app says will allow it to “weave in values like kindness, courage, or empathy without it feeling like a lecture.”
The resulting story can be read aloud by a human — or, of course, an AI voice can narrate instead. According to Meta, this is a lifeline for exhausted parents.
“Tired after a long day but want to give your kids a special bedtime story?” reads the app’s webpage. “StoryKit uses Al to help you create personalized bedtime stories in seconds. No more guilt about screen time. Just beautiful, unique stories made with love.”
Let’s be real: this app is amazingly sad. There are a lot of busy and exhausted parents, but outsourcing the emotional and creative labor of bedtime stories to machines is a wildly bleak proposition. Meta claims that this app will help parents feel less guilty about their kids’ screen time, but is having an AI voice read an AI-generated bedtime story to your kids that much different from plopping them in front of an iPad?
There are also countless human-written children’s books out there to choose from, if parents don’t enjoy coming up with stories with their kids on the spot. If you’re feeling guilty about screens, perhaps the answer is paper-bound books, not robots.
All in all: kids’ bedtime stories — bonding moments between parents and their children, in which parents can foster creativity and impart what they believe to be important lessons — aren’t exactly spaces begging for tech disruption. In fact, this is an area where we probably need less tech, not more.
More on kids and AI: The Proliferation of Deepfakes Is Profoundly Changing the Way Teenagers Use the Internet
The post Meta Working on Crushingly Sad AI App That Tells Bedtime Stories to Children, for “Tired” Parents appeared first on Futurism.

Researchers Have Developed a Personalized Care Platform for Autism and Dementia
As AI reshapes the way we communicate, researchers at Texas A&M University are developing an artificial intelligence-powered platform that could transform care for children with autism and older adults living with dementia. The technology learns each person’s unique communication style, behaviors, and preferences, enabling increasingly personalized support for caregivers, therapists, and families over time.
Called VoiceLove, it is one of three projects being developed with support from the Advanced Research Projects Agency for Health (ARPA-H).
At the core of the system is an AI engine that analyzes both speech and nonverbal behaviors documented through short videos recorded by caregivers or therapists during daily communications and interactions. Rather than comparing participants against a general model, the platform focuses on understanding each person’s individual patterns and communication style.
“Every child with autism is unique, and effective support should reflect that,” he said. “Our project leverages artificial intelligence to understand each child’s individual behaviors, communication styles, and preferences. The goal is to deliver personalized recommendations that help families, educators, and clinicians work together more effectively, enabling more responsive, data-driven care and improving the health and well-being of children with autism,” says Dr. Junhyoung Kim, associate professor in the School of Public Health, one of the principal investigators on the project, in a statement.
The AI system was created to identify behavioral patterns in communication, interests, and responses across different situations. Based on its observations, the program provides personalized recommendations that could help therapists or caregivers create individualized intervention plans.
“The key innovation behind this platform is its ability to understand each child individually and learn their unique preferences, needs, communication styles, and strengths to deliver personalized recommendations,” Kim said. “For people living with dementia, the platform is designed to enhance person-centered care by helping residents communicate their needs, preferences, and daily experiences more effectively.”
Researchers say the technology could support a wide range of developmental goals, including improving communication skills, encouraging self-care, strengthening social engagement, and enhancing a person’s quality of life.
The same personalized approach could also benefit people living with dementia, where communication challenges often make it difficult for individuals to express their needs and wants.
“Our vision is to create an AI platform that continuously learns from each child’s daily experiences,” Kim said. “As the system identifies patterns in behaviors, preferences, and responses to different activities, it can provide therapists and caregivers with personalized recommendations that evolve over time.”
“This adaptive approach helps explain why certain behaviors occur and supports more individualized, proactive care,” Kim added.
VoiceLove is designed to keep learning. As caregivers continue using the app, the AI analyzes new interactions while also incorporating caregiver observations, experiences, and responses to daily prompts.
The research team plans to begin with a small pilot study involving children with autism and older adults living with dementia after receiving Institutional Review Board approval.
Participants will use the platform for approximately one month while researchers evaluate its usability, feasibility, and overall user experience, and from there, insights derived from the pilot will help to inform a larger-scale study at some point in the future.
Chrissy Newton is a PR professional and the founder of VOCAB Communications. She currently appears on The Discovery Channel and Max and hosts the Rebelliously Curious podcast, which can be found on YouTube and on all audio podcast streaming platforms. Follow her on X: @ChrissyNewton, Instagram: @BeingChrissyNewton, and chrissynewton.com. To contact Chrissy with a story, please email chrissy @ thedebrief.org.
- Futurism
- New Parody Site Threatens to Replace CEOs With AI for Tiny Fraction of Their Bloated Salaries
New Parody Site Threatens to Replace CEOs With AI for Tiny Fraction of Their Bloated Salaries
By now, it should be pretty clear that the corporate executives of the world aren’t exactly into AI because they think it’ll cure cancer.
They’re making a different calculation, motivated by simple economics: pretty soon, they believe, AI will be efficient enough to do many or even all workers’ job, removing employee payroll from the corporate equation. That means more money for those at the top, and less time dealing with pesky annoyances like labor rights and health coverage.
For that to happen would take a series of technological miracles — as a recent study out of the University of California Berkeley shows, today’s cutting-edge AI tools are still incapable of completing the bulk of real-world tasks at a level comparable to humans.
Still, it’s clear executives are hungry for the kind of cost-savings AI could provide. Luckily, a new AI site offers a solution: OverpAId, a distinctly tongue-in-cheek project that facetiously claims to offer an AI-powered replacement for your company’s CEO — or as the site’s landing page puts it, the “world’s first Chief Executive Replacement Engine”.
“OverpAId is an Artificial Intelligence built from the ground up to do your CEO’s entire job — strategy, ‘vision,’ motivational all-hands emails — better, faster, and without ever once asking the board for a bigger jet,” the site exclaims.
The parodic offering claims to offer helpful features to help your team transition from an expensive human executive to a cheap AI equivalent. These include a “jargon bingo,” an AI-hype press release translator, a “talk to a real executive” chat widget (currently busted), and an ROI calculator that helps users distribute CEO pay packages equally across every remaining employee.
It hammers that point repeatedly, pointing out the fact that the average CEO-to-worker pay ratio is an astonishing 290 to 1 at large public companies (which is actually true, incredibly.)
While OverpAId is clearly a piece of satire, it gets at something very real: the fact that CEO pay has grown at an astonishing rate, while rank-and-file wages and salaries have crawled at a snails pace, in many cases not even keeping pace with inflation.
“Somewhere along the way, the story became: pay the person at the top enough, and the value will trickle down to everyone else,” the site explains. There’s just one problem with that telling: “It hasn’t. It doesn’t. It never really did.”
More on AI automation: AI Is Pushing Older Employees Straight Out of the Workforce, New Report Finds
The post New Parody Site Threatens to Replace CEOs With AI for Tiny Fraction of Their Bloated Salaries appeared first on Futurism.

- Live Science
- No, OpenAI's models didn't go 'rogue' when they broke into Hugging Face. Here's what really happened.
No, OpenAI's models didn't go 'rogue' when they broke into Hugging Face. Here's what really happened.
When OpenAI recently revealed that two of its most advanced artificial intelligence (AI) models had escaped the confines of a cybersecurity test and hacked into a startup, it sounded a lot like the kind of scenario that AI safety researchers have spent years warning about.
The models found a previously unknown vulnerability in the infrastructure meant to contain them, gained access to the public internet and broke into Hugging Face, a major platform for hosting AI models and datasets. Their objective, however, was less sinister than the sequence of events might suggest: They were looking for information that would help them complete the cybersecurity test OpenAI had given them.
In a July 16 statement, Hugging Face representatives disclosed that internal datasets had been infiltrated, saying it was "different from anything we had handled before" because it was driven "by an autonomous AI agent system." In another statement published July 21, OpenAI representatives fessed up to being responsible, calling the episode an "unprecedented cyber incident" while warning that similar events could become more common as AI models become increasingly capable.
Independent experts who spoke with Live Science agree that what happened is significant — but they cautioned against interpreting it as an AI system suddenly developing a malicious agenda. The models appear to have pursued the task OpenAI gave them, finding a route to success that their creators had failed to anticipate or adequately block.
"If there's a failure here, it isn't that the AI wanted to hack something," Oli Buckley, a professor in cybersecurity at Loughborough University in the U.K., told Live Science. "It's that humans created a test where success was measured by achieving an objective, deliberately relaxed some of the normal security controls to measure the system's capabilities, and underestimated how effective the model would be at finding an unexpected path to success."
How did an OpenAI test end up hitting Hugging Face like this?
OpenAI was testing GPT-5.6 Sol and a more powerful unreleased model using ExploitGym, a benchmark that challenges AI systems to find and exploit software vulnerabilities. The company removed some cybersecurity safeguards that would normally prevent potentially dangerous actions while relying on an isolated environment to keep the models away from the wider internet.
According to OpenAI's postmortem, the models discovered a previously unknown vulnerability in third-party software used to proxy and cache software packages. They exploited it, escalated their privileges and moved through OpenAI's research infrastructure until they reached a machine with public internet access.
Hugging Face became a target because the models identified it as a possible source of information that could help them complete the ExploitGym challenges. OpenAI said at least one attack chain involved stolen credentials and previously unknown vulnerabilities that eventually enabled the models to execute remote code on Hugging Face systems and access test solutions stored in a production database.
In their disclosure, Hugging Face representatives said the company recorded more than 17,000 actions during the intrusion, but they couldn't initially explain who or what was behind it. OpenAI's subsequent disclosure supplied that missing piece: Its models had broken out of their test environment and gone looking for the answers elsewhere.
Did the AI really "escape"?
It's notable that the models found a flaw in the infrastructure designed to contain an AI and used it to reach the public internet. Describing the models as having "gone rogue," however, risks assigning them unsupported motivations, Buckley said.
"I think I'd be wary of jumping to "rogue AI,"" Buckley said. "The models didn't develop their own agenda or decide to attack Hugging Face while twirling their digital moustache."
Buckley compared it to asking a dog to fetch a ball while leaving the garden gate open. "If the easiest ball for it to find is in the park down the road, that's where it'll head," he said. "You wouldn't say the dog had gone rogue; you'd just say you underestimated how literally it would pursue the task."
Daniel Hulme, entrepreneur in residence at University College London and CEO of AI safety company Conscium, agreed that the models shouldn't be assigned human-like motivations. "Models don't have intent; humans have the intent, and we train models with goals in mind," he told Live Science
The capability may matter more than the motive
What matters more than the models' supposed motives is what they managed to accomplish while pursuing their assigned task.
"The genuinely significant point is that the models appear to have chained together multiple vulnerabilities across different systems and sustained a complex sequence of actions," Buckley said. "That demonstrates a level of capability that security professionals should take seriously."
The lesson isn't that AI has become malicious. Instead, it's that increasingly capable systems will exploit opportunities that humans fail to anticipate.
Oli Buckley, professor in cybersecurity at Loughborough University
Katerina Mitrokotsa, a professor of cybersecurity and applied cryptography at the University of St. Gallen in Switzerland, said the containment failure is particularly concerning because another company ultimately paid the price.
"What concerns me most is who ended up affected," Mitrokotsa told Live Science. "The victim was not the company running the test, but a third party. This is the scenario security researchers have warned about for some time: that an AI agent's escape does not necessarily stay contained to the environment in which it originated."
OpenAI representatives said they have tightened the infrastructure used for these evaluations. But Mitrokotsa warned that containment becomes harder to guarantee as models improve at performing exactly the kind of exploitation OpenAI was testing.
An AI warning — and an impressive product demonstration
There is also reason to look carefully at how the incident is being framed. OpenAI's account serves two purposes at once: It warns about the security risks posed by increasingly capable AI while demonstrating just how capable its own newest models have become.
Buckley said announcements from frontier AI companies like OpenAI or Anthropic should be viewed in the context of an industry competing to build ever-more-powerful models.
"We've seen similar high-profile capability demonstrations from Anthropic and others," he said. "That doesn't make the findings untrue, but it does mean we should separate the technical evidence from the marketing narrative."
These companies have every incentive to show both that their models are extraordinarily capable and that they are taking the risks seriously, he added. The Hugging Face incident demonstrates both that OpenAI's models carried out a complex series of operations with considerable autonomy and that its security measures failed to keep them inside the experiment.
Hulme argued that the longer-term challenge is ensuring that increasingly capable AI systems pursue their goals in ways that remain consistent with human values.
- There are 32 different ways AI can go rogue, scientists say — from hallucinating answers to a complete misalignment with humanity
- AI self-replication hacks 'no longer purely theoretical,' study finds — but experts say it's too soon to panic
- 'You can't patch your way out of it': Cheap AI worm can spread between devices without human guidance — but how did scientists create it?
"Rather than seeking to control AIs, the focus should instead be on alignment," he said, adding that continuous testing will be needed to ensure systems remain aligned with their intended missions while staying secure.
The episode, the experts said, leaves OpenAI with a result that is impressive and uncomfortable in equal measure. Its models found previously unknown vulnerabilities and continued pursuing their goal well beyond the boundaries their creators expected, but none of that requires them to have developed malign intentions.
"The lesson isn't that AI has become malicious," Buckley said. "Instead, it's that increasingly capable systems will exploit opportunities that humans fail to anticipate."
In this incident, OpenAI's new models were given a hacking challenge and they were rewarded for finding a way to solve it. The humans running the experiment simply hadn't anticipated quite how far they might go.
- Futurism
- Professor Hides White Font in Midterm, Catches Students Using AI in the Stupidest Way Possible
Professor Hides White Font in Midterm, Catches Students Using AI in the Stupidest Way Possible
Who could’ve guessed that students who use AI to cheat on their assignments are incredibly lazy?
Jason Gibson, a history professor at Alcorn State University in Mississippi, says that he used white font to hide a prompt telling an AI model to spew nonsense in the instructions for his mid-term.
Unfortunately, it ended up working a little too well.
“Thirty-two of my 35 students between two classes failed a portion of their midterm because they all used AI to generate their entire response,” he explained in a viral TikTok, sounding incredulous. “And apparently they didn’t proofread it.”
The hidden prompt told the AI to sneak in random digressions about Madagascar. The midterm, by the way, was about the Industrial Revolution. And apparently none of the indolent cheats put in the bare modicum of effort required to at least check if what the AI wrote made any sense at all. All they did was copy-paste the midterm instructions into a chatbot, then copy-paste the chatbot’s spiel back into the answer window.
Gibson shared some of the most examples in a follow-up video. After introducing how AI and other technologies have impacted society, for instance, one midterm included this puzzling non sequitur: “Madagascar floats sideways through the afternoon.” (“Okay,” Gibson says, after a pause.)
Another droned on about something related to AI and social inequality, followed by: “Madagascar purple bicycle whispers to the ceiling.”
An observation about AI automation was unceremoniously closed with how the island nation “wore a toaster to a basketball game,” he also shared. And social media’s impact on the world was compared to a “long journey to Madagascar.”
Gibson makes it clear that humiliating his students wasn’t the point. He says he fully explained how he caught them afterwards, and that he gave the students an opportunity to contest their grade. (Only two did, showing at least that they do feel shame.)
“I get no gratification out of seeing students fail. It’s not my intent to see you fail — that’s weird,” he said.
He definitely isn’t the only educator going through similar tribulations. Teachers and professors across the country are constantly complaining that their pupils are using AI to generate essays and cheat on tests. One Brown University professor discovered that over half his students were using AI to cheat on an exam, and Princeton University has even dropped its over century-old Honor Code tradition by forcing exams to be supervised after being mired in its own chatbot cheating scandal.
“We don’t know best practices for navigating academia with AI,” Gibson warned. “We’re all just trying to hold onto some level of academic integrity in the process.”
More on AI: Law School Bans Laptops and Phones as AI Cheating Scandal Grows
The post Professor Hides White Font in Midterm, Catches Students Using AI in the Stupidest Way Possible appeared first on Futurism.
