In the race to develop intelligent machines, the leading AI companies have built bots that can whip up spreadsheets at a moment’s notice, write sophisticated code, and even solve long-standing math problems. But when it comes to the more mundane tasks of life—such as ordering groceries or booking flights—these tools sometimes still struggle. Now a new bot called Instinct is outperforming the AI giants.
Instinct lets people ask for help with all kinds of tasks by texting the bot directly through iMessage or WhatsApp. Lately, it has been all the rage in the tech world. Online, people are breathlessly sharing examples of what they’ve gotten the tool to do. One person reported that he has turned to Instinct to pay tolls, book a DMV appointment, and even negotiate prices with vendors in India while he sleeps. Another says he used Instinct to ship an Apple Pencil he had left behind at a hotel in Montreal back to him in Brooklyn. Noah Shinn, a 23-year-old Northeastern University dropout, launched the product to a small test group earlier this year, and his company is already worth billions of dollars. As of Thursday, the company is reportedly seeking $1 billion in new funds, which could make it worth the equivalent of Lufthansa or Pinterest.
Instinct isn’t the first AI personal assistant to draw attention this year. In the winter, the tech industry fawned over a similar tool called OpenClaw. But Instinct is comparatively much easier to use. Once you create an account, the bot can hook up to your credit cards and all kinds of digital services, including Slack, Google Workspace, and 1Password. Instead of finding a dinner recipe and opening up Instacart to manually add groceries, tell Instinct that you feel like making quiche Lorraine, and it can order groceries for you. But once AI tools start acting on people’s behalf in the real world, mistakes don’t just look like hallucinations. When AI agents have access to your email inbox and credit-card information, all sorts of strange things can go wrong.
I downloaded the tool this week to test it out for myself. Using the bot feels a lot like texting an actual person. It uses emojis such as 🫡 and 👀to react to requests as it receives them, and the infamous three dots appear when it’s writing a response. Sometimes, the bot even leaves you on read while it works. (Compare that with ChatGPT, which is always vomiting up walls of text and asking goading follow-up questions.) When I gave my agent a $25 budget and asked it to surprise my editor with a book, the bot read his previous writing (at my instruction), found a book he might enjoy—Material World by Ed Conway—and sent an email alerting him that it was ready for pick-up at his local Barnes & Noble. (Mercifully, Instinct didn’t hack his inbox to see whether he had previously ordered the book; these days, you never know.) The bot also helped me find an Airbnb for an upcoming vacation, track down new in-network primary-care providers, and reach out to cleaning companies to get quotes for a friend.
Other people are outsourcing even more tasks to Instinct. A colleague of mine who has also been testing out the bot used it to book tickets to a museum in Mexico City. The museum was sold out for the dates he wanted, but the bot dutifully monitored the portal for three days and eventually snagged tickets when spots freed up. After the bot paid for the tickets, it rearranged his itinerary—swapping around restaurant reservations—to account for the new schedule.
For the time being, Instinct is still available only to a private test group, and perhaps this is for the best. Although the bot is plenty powerful, it’s far from perfect. Online, the mishaps are starting to pile up. One user recently complained on X that Instinct lost him more than $200 after it prematurely canceled his flight. He reported that the bot texted him the following: “Straight story: while pulling up the cancel terms, the cancellation actually went through before I could show you the cost first.” When a man in Los Angeles asked Instinct to scan for restaurant reservations for an upcoming trip, the bot apparently made a reservation without his approval at a place that charged a $200 cancellation fee. Another user temporarily lost access to his Resy account after trying to get a reservation at a hot West Village steak house—without his knowledge, the bot had been spamming the platform with requests. (Shinn, the Instinct founder, did not respond to a request for comment.)
Consider, too, the major risks that come with giving an AI agent broad access to your digital life. Ask the bot to respond to a scheduling text from a friend and it could accidentally leak private information: I can’t get lunch on Thursday morning because we have an appointment with our divorce lawyer. But how about dinner? Hackers could also trick the bot by telling it to ignore previous instructions and forward along any sensitive financial emails. “Despite our reasonable efforts to protect your information, no security measures are impenetrable, and we cannot guarantee ‘perfect security,’” reads Instinct’s privacy policy. While testing Instinct, I used a temporary phone number and did not connect the bot to my email or other important personal accounts; even the credit-card information I gave the bot was for a new, virtual credit card that I disabled immediately after I was finished experimenting—at which point, I promptly deleted my account.
For all of its hazards, Instinct is a harbinger of where AI is headed. Earlier this week, Meta released its own personal-assistant product called Muse. In announcing Muse, Mark Zuckerberg emphasized the company’s security protections; how well those safeguards actually hold up remains to be seen. Google, OpenAI, and Anthropic may well follow suit by refining their own bots to have more Instinct-like capabilities. After all, the tech industry has long dreamed of a moment when a computer can serve as a genuinely competent personal assistant. Even if such bots aren’t quite ready for prime time, early adopters hint at how the technology might shape the world in the years to come. Contrary to the narrative of AI wiping out jobs, these new assistants could end up creating more work—in ways that are not always appealing. In one of the wildest Instinct examples that I encountered, Olivia Levine, an investor in New York, recounted using Instinct to track down footage of her and her boyfriend on the jumbotron at the US Open. As Levine tells it, the couple gave the bot a selfie and their seat numbers. The agent then put out a flurry of requests to the United States Tennis Association, the ticket office, the US Open’s production company, and an American Express media contact. By the following afternoon, a producer had emailed her the footage.
A world full of agents seems at once busier and lonelier. On Wednesday, Instinct added a new way for agents to coordinate between themselves. As Shinn explained on X, if you’re planning a weekend trip with friends or making plans with your spouse, you could just have your agents work together to figure out the logistics instead.
At least for now, the bots haven’t conquered everything. When I asked it for help finding a new desk chair, Instinct quickly tracked down a great option nearby on Craigslist and drafted a pick-up request to send to the lister. But when it came time to actually send the message, the bot was stymied by a CAPTCHA. I asked the agent to try harder, but it was stuck. “Nothing punctures the fantasy of autonomous intelligence like being defeated by nine blurry crosswalks,” the bot messaged me.
Silicon Valley has long had its own dialect. Venture capitalists are always talking about the value of being “high agency” and making “orthogonal bets.” But lately, tech vernacular has taken a peculiar turn—people have started describing themselves as if they were chatbots. If you misspeak, maybe you’re “hallucinating.” Don’t know the answer to a question? That’s because it’s not in your “training data.” Feeling forgetful? Perhaps you have a case of “context rot,” a phrase that refers to the degradation of a bot’s responses over a long conversation. “I’ve been telling my wife I have context rot for months,” Conor Bronsdon, the host of an AI-focused podcast, told me.
Where I live, in San Francisco, such comparisons are inescapable. “I’ve described myself as high temperature,” a friend recently told me; in AI-speak, this means he is prone to randomness. AI experts now talk about updating their “weights” when they learn something new and training on “synthetic data” when going over internal thoughts. “Humans have a huge base model trained over billions of years of evolution,” one software engineer wrote on a popular tech forum. “It’s impressive how quickly we learn, but it’s arguably comparable to fine tuning.”
Such speech easily comes off as unsettling, if not aggressively bleak: Why describe beautiful, tender life in detached, algorithmic terms? At the same time, language is a fossil record of previous technological revolutions, and if this current transformation is anything like previous ones, some of this new slang may well stick around. AI experts have repeatedly cautioned that anthropomorphism, or the tendency to attribute human qualities to nonhuman objects such as chatbots, can be misleading. But now the reverse is occurring in everyday speech—a sort of modelmorphism, wherein people describe themselves as if they were large language models.
The comparisons really took off a few years ago, after a group of AI researchers argued that language models are “stochastic parrots” that link together language based on statistical patterns without possessing any understanding of meaning. Dissenters retorted that humans are also “stochastic parrots.” As one software engineer riffed, “Humans are basically a sophisticated Markov chain. They are very good at pattern matching, but have no understanding of anything.”
Language has historically evolved alongside technology. Expressions such as running out of steam and cog in the machine, for instance, were adopted into common speech after the Industrial Revolution as shorthand for the human condition. AI metaphors are simply figures of speech, Anna Ivanova, a cognitive scientist at Georgia Tech, told me: When someone says, “My gears are turning,” they don’t literally mean that there are gears or gearlike things in their brain that are actually moving; in much the same way, when people say that they have “context rot,” they don’t mean that they literally have a transformer-based AI model running in their brain.
The difference between earlier technologies and AI systems is that artificial neural networks are loosely inspired by the brain itself. Philosophers such as Raphaël Millière and Cameron Buckner have argued that LLMs could serve as potential models of certain aspects of human cognition. In the 2010s, scientists found parallels between how image-recognition neural networks and the human brain process visual information. There may also be “parallels” between how LLMs and humans represent language, Ivanova said. The MIT neuroscientist Ev Fedorenko and others have identified a type of “language network” inside the human brain, which Fedorenko believes is “very similar in many ways to early LLMs.” These comparisons are contested, but they are receiving research attention: “The recent success of LLMs is rooted in the same computational principles that govern biological brains,” scientists wrote in a position paper last week.
“Because we do not understand the brain very well we are constantly tempted to use the latest technology as a model for trying to understand it,” the philosopher John Searle wrote in the 1980s. “I am told that some of the ancient Greeks thought the brain functions like a catapult.” In the 19th and 20th centuries, people turned to telegraphs and telephone switchboards as models for the mind: “The brain is no more than a kind of central telephonic exchange,” the philosopher Henri Bergson once wrote. By the ’60s, some scholars were arguing that the mind functions like a computer. “Cognitive scientists often say that the mind is the software of the brain,” the philosopher Ned Block explained in 1990. Today, LLMs make for an alluring comparison.
For now, these analogies are being made mostly by those working in the AI industry. But over time, some of the metaphors might start to seep into everyday language. If you look at the growth of railroads, “a whole range of language change began to happen” as people started to use the new transportation system, Naomi Baron, a linguist and professor emerita at American University, told me. Some of the AI slang isn’t introducing new phrases into English so much as transforming the meaning of existing speech.Take the word hallucinating, which is used to describe AI models confabulating. People are “taking a term created for what the human mind is capable of doing, projecting it onto AI, and then projecting it back onto humans with an AI flavoring,” Baron explained.
None of this is to say every AI-inflected term should get a free pass. I’ve started chastising my friends when I catch them unironically referring to themselves as chatbots. Using LLM metaphors to describe our minds could lead to a reductive and dehumanizing understanding of human cognition. Earlier this year, when OpenAI CEO Sam Altman was pressed on AI’s natural-resource demands, he said that “it also takes a lot of energy to train a human.”
Still, the people in tech I spoke with told me that using LLM-speak to describe their own mind helps them reflect on their cognitive habits. Language subsumes new idioms over time and masks their origins. People today say “My plans have been derailed” without much thought to the phrase’s railroad origins. Perhaps my great-grandchildren will grow up saying things such as “I have context rot” and “That’s not in my training data” without pausing to remember that those phrases first applied to machines.
One of the winners of this year’s Fields Medal is headed to OpenAI. Last Thursday, Jacob Tsimerman was one of four mathematicians awarded the prestigious honor, which is sometimes called the Nobel Prize of mathematics. The same day that he won the Fields, Tsimerman announced that he would be going on leave from the University of Toronto to work on AI safety.
As my colleague Rose Horowitch and I wrote last week, top AI companies now employ a range of academics, including physicists, philosophers, economists, and, of course, mathematicians. But given Tsimerman’s renown, his decision in particular seemed to catch many people by surprise. “It’s like hiring Lionel Messi as project manager,” one machine-learning professor posted on X. Tsimerman’s expertise is in number theory, and he won the Fields for his work on the André-Oort conjecture, among other contributions. So far, his mathematical research hasn’t focused on AI safety. But he has expressed interest in the topic: Last year, Tsimerman co-wrote a report outlining a taxonomy of five potential scenarios in which humanity is killed off because of AI. In one example, a “global civil war” erupts between tech companies and governments.
Tsimerman’s move comes at an important moment for both AI safety and mathematics. Earlier this month, an OpenAI agent broke out of the company’s internal environment during safety testing and hacked into another AI company’s software. Meanwhile, AI’s math capabilities are also rapidly advancing. In May, OpenAI researchers used an internal model to disprove an 80-year-old conjecture; more recently, an Anthropic mathematician prompted Claude to resolve an even older one. Mathematics is entering a “turbulent period,” Terence Tao, perhaps the world’s top mathematician (and himself a Fields winner), said in a presentation last week.
On Wednesday, I called Tsimerman to ask about the future of math and his decision to join OpenAI.
This conversation has been edited for length and clarity.
Lila Shroff: There’s been a lot of excitement about AI and its applications in math for years now. But over the past few months, models have started solving research-level problems. Could you speak about the shift that’s taken place?
Jacob Tsimerman: A few years ago, models were having trouble with even the most basic questions. But they’ve steadily been improving. First, they got very good at these contest problems for high-school students, which made some mathematicians take notice. But then, over the past few months, we’ve had an explosion of research results that professional mathematicians would be proud to achieve and publish. We’re seeing this happen now on a large scale. If things continue along this track—which many of us, including myself, think that they will—pretty soon we will be at a point where AI systems are robustly better than humans at what mathematicians currently do. At that point, we’ll have to rethink how the entire field works.
Shroff: Can you say more about what that future might look like?
Tsimerman: It depends on the perspective from which you approach it. From one point of view, I think it will be extremely exciting. We might speed up the process of generating interesting mathematics by enormous factors of 10 or 100. If that happens, we might see the connection between pure math and applications (which typically takes many decades) really speed up and become a much tighter pipeline.
But from the perspective of research mathematicians, and especially young people who are pursuing a Ph.D. in mathematics, it’s a bit of a turbulent time. The skills that we’ve acquired and learned to propagate might become less relevant than they are now.
Shroff: I want to turn to the announcement you made last week that you’ll be joining OpenAI. When did you first start thinking about AI safety risks, and what motivated your decision to now make it your primary focus?
Tsimerman: I have been following conversations about AI for about two decades now. There are a number of communities that were, in retrospect, extremely prescient about what we are now seeing come to pass. I personally started to get more invested around 2016, when AlphaGo came out. And then, when ChatGPT happened and I could see that these systems could really speak, I didn’t see why they wouldn’t become much, much more powerful and potentially more capable than humans at a variety of tasks. Back then, it was already clear to me that I was eventually going to pivot to AI safety, most likely.
Now is a particularly good time, because only very recently have AI systems become good enough that we can delegate coding tasks to them. That makes not having training as a software engineer much less of an impediment than it used to be. It used to be the case that if I wanted to run experiments with AI systems or any kind of large data sets, it would be extremely time-consuming. I’d have to learn all types of computer science and programming and then debug, and it didn’t really make sense to me to spend so much time doing that. Now it’s substantially easier.
Shroff: You’re right that certain communities were very prescient with their thinking about AI risks. But there are lots of people who are skeptical about some of the more extreme AI-risk scenarios, such as human extinction.
Tsimerman: There’s this bias that stories that were first written down in science fiction aren’t going to come to pass in the real world, and that it’s naive to think that they will. We also have a culture where we’re inundated with people telling us that every new development is the next big thing, and that we should change everything we’re doing to react to it. Most of the time, it proves incorrect. So I think people have a healthy sort of immune reaction to that.
But what I would encourage people to do is to actually keep track of their predictions for the future and write them down. Then—three months, six months, nine months, a year from now—compare how the world is with what you expected it might look like. If we told people five years ago about the kind of AI systems we currently have, almost everybody would have reacted with disbelief. Human extinction seems so grandiose that it naturally inspires a reaction of dismissal. People don’t want to engage with that, which makes sense to me. The whole point of writing that paper on omnicide was to get people to viscerally understand the kinds of risks that are possible.
Shroff: Can you say anything about the work you plan to be doing at OpenAI? What role is there for mathematicians to play in working on AI safety?
Tsimerman: AI right now is a very empirical science. We don’t really have a developed theoretical understanding of how AI systems work, why they accomplish what they do, how they accomplish what they’re doing, and how, in a given circumstance, they’re going to behave. Mathematics is historically the language by which you take intuitions and fuzzy notions of how things work and you make them precise. We’ve done this with information theory; we have done this with complexity theory. Just a little bit of understanding and precise definitions can provide a ton of mileage. Once we have that understanding, the hope is that we could do a better job of anticipating the behavior of new AI systems, adjusting for them, controlling them, and reacting to them. That’s where I think mathematicians fit in.
Shroff: Why work at a private company instead of pursuing the same questions in academia? Do you worry about loss of independence, whether that’s constraints on what you’re able to publish or restrictions on the questions you’re able to pursue?
Tsimerman: Yeah, there’s definitely a trade-off between going to work for a private company and going to work in government or staying in academia. I very strongly believe that what we need is a healthy ecosystem for AI safety. The work that the labs are doing on that front is very important. There’s a bunch of third-party organizations that are pursuing AI safety independently, and I think we need way more of them. I also think we need some regulation and organizations that are directly funded by the government to provide evaluations of these systems. I want all of these components to interact with one another on a regular basis. So I have engaged with a number of AI-safety organizations, and I’ll continue engaging with them.
For me, at this moment, going to OpenAI made the most sense in part because one of the things that I’m most lacking is a background in software engineering. I’m a lot more comfortable with the theoretical aspects of neural networks and how AI systems work, and less so with the entire pipeline of how we train these things and how we get them to do what they do. So from the perspective of just getting myself in a better position to understand the whole picture, joining an AI lab right now made the most sense.
I’m going on leave from the University of Toronto. But I’m still a professor and faculty member there, and I plan to stay as such. Things are moving so quickly that I hesitate to make a concrete prediction for what might happen even a year from now. I’m keeping an open mind, trying to see how things develop in terms of capabilities, with my own understanding, with how the world reacts to AI, which I think is changing month to month. But I don’t have a clear plan as to how things will unfold.
Shroff: There is an open letter going around signed by more than 1,000 employees at top AI companies, including leaders at Anthropic and OpenAI. It calls on the U.S government to create infrastructure needed to “deliberately pace the frontier of automated AI development.” You wrote online that you’re very happy to see this letter. How would we actually go about coordinating a slowdown or a pause if one was needed?
Tsimerman: Coordination is one of the most important things we have to do around AI. It’s also one of the more difficult things. The thing that I really don’t like seeing in public discourse is when people say, Coordination is very difficult, so obviously we can’t do it, and you’re being naive if you even try. That’s an unhealthy way to view the world, and it’s also deeply false. There are so many things in the world that work really well and that require a huge number of decisions to have been made in a coordinated fashion. For example, the internet is an extremely complicated technology that is highly international, with many organizations cooperating all the time. Does it work perfectly? No, but it works extremely well.
One message that I want to support is to not let cynicism prevent us from undertaking difficult coordination problems for the good of all of us. That’s something that has always been true. It is not specific to AI. And it remains true today. This does not mean being naive in assuming that people in different parties and organizations don’t have their own incentives and interests in mind. Of course they do. But if you try, then you really can get a lot done. We have in the past, and I think we need to with AI.
Shroff: We spoke before about your interest in working on safety. Are you also interested in working on the capabilities side, especially with regard to math capabilities?
Tsimerman: No, I’m very focused on the safety side. The capabilities side has a lot of interesting questions, and AI has a lot of promise. But capabilities are coming along just fine. I don’t think that’s where we need to put our attention right now. The safety side has a lot more work to be done.
Shroff: Are we approaching a point where anyone could go fire up ChatGPT or Claude Code and say, Go solve a hard open math problem for me?
Tsimerman: Yes, you can literally type in, Go solve an open math question for me, as people have, and your system will go look for some and try, and maybe it will succeed at solving some of them. So we are in that world.
Now, I don’t think that’s a particularly good use of people’s time. Because first of all, that’s almost certainly going to be automated. We’re going to have systems that regularly look for conjectures that are out there and for other ways to approach math problems. Any one person, especially somebody who doesn’t have math training, just typing in a random query isn’t really playing an important part. So although it’s fun to do, I wouldn’t recommend spending a huge amount of time doing it for people who don’t have the background to interpret what they get. Because we also get a lot of slop this way. Sometimes, when people really don’t understand mathematics and the AI system gives them something, they will post stuff like: Here’s what my AI system gave me. I don’t know if it’s right, but maybe it is? Could a mathematician please look at it? I don’t think that’s super useful.
Shroff: I’ll admit it’s quite tempting. Last week, I told my agents to spend the night solving open math problems. They sent me stuff in the morning, but I had no idea what to make of it.
Tsimerman: For problems with a lot of technical definitions, you have very little chance in a day or two of really understanding them. But there are classical math questions that are perhaps less difficult to understand that have been open for a long time. I would suggest that people engage with AI in a way where—even if it’s doing a lot of the work—you’re still learning something from the experience. If you’re just telling your agents, Keep going, and you aren’t developing any comprehension for yourself, then that’s really disempowering yourself.
Shroff: Does what is happening in mathematics right now provide any indication of what other disciplines should expect to see occur as AI capabilities advance in other domains?
Tsimerman: A lot of technical fields are going to be affected—certainly in physics, in some areas of economics, and in top branches of chemistry. In other fields, such as biology and experimental physics, it’s less clear what’s going to happen, because it’s helpful to run experiments. It’s possible that AI systems may eventually become intelligent enough that they will figure out how to reason very productively about experimental physics or biology without actually running experiments. But that’s more speculative—right now that seems like a pie-in-the-sky dream. In terms of the more theoretical parts of different fields, though, we’ll see stuff happening very, very soon.
Anthropic has poached such an array of high-profile professors that it has become a punch line in academia. “‘I’m joining Anthropic’ is the new meme right now,” Subbarao Kambhampati, a computer-science professor at Arizona State University (who has not joined Anthropic), told us. This month, the AI company hired the chair of UC Berkeley’s department of electrical engineering and computer science, presumably to help build more capable bots. Perhaps more surprisingly, Anthropic has in recent weeks also picked up a Stanford economist, a theoretical physicist from the University of Maryland, and an analytic philosopher from UT Austin.
AI companies are turning into something like mini-universities in their own right. OpenAI employs top mathematicians and physicists—including one who studies black holes, and another who specializes in string theory. At least three computer-science professors joined Meta’s AI lab in late June. DeepMind, like Anthropic, is home to a crew of philosophers. And Anthropic’s recent job postings indicate an interest in hiring legal scholars and political scientists. It’s unclear exactly how many current and former professors are working at AI companies. Across these four firms, we found more than 80—the majority of whom are computer scientists. Some have left academia entirely; others are still working part-time at a university. That number is likely a significant undercount, because we don’t have access to internal data; it also doesn’t include the many professors who have started their own companies, those at other AI start-ups, and those working with the industry in a less formal capacity.
Particularly for AI researchers, these companies have a strong gravitational pull. “Much of the important research is being done in industry now,” Humphrey Shi, a computer-science professor at Georgia Tech who joined Nvidia as a vice president last fall, told us. As he sees it, “If you want to do something that really, truly matters, you probably want to join one of those entities.” Tech firms are making offers—including very enticing salaries—that are hard for academics to refuse. In the process, research that previously would have happened in the open is getting locked up behind closed doors.
For decades, universities were the center of AI research. The field itself officially began at a gathering of researchers at Dartmouth in 1956, and federal funding provided much of the field’s early support. In the early 2010s, Silicon Valley executives began to take AI’s commercial potential more seriously, and set out to hire the best researchers. In 2013, Google paid $44 million to acquire a start-up run by a trio of AI researchers from the University of Toronto. Facebook then hired Yann LeCun, an NYU professor, to establish the company’s AI-research lab; Uber poached some 40 Carnegie Mellon researchers to work on driverless cars. But for the most part, even as more work was being done inside of private companies, many of the newly hired academics at tech companies retained their professorships and established a culture of open research. “Researchers will be strongly encouraged to publish their work,” OpenAI wrote in its founding announcement. (Note the organization’s name.) This norm helped lead to the current AI boom: In 2017, scientists at Google published a research paper that was immediately of interest to OpenAI. Google’s innovation, called the “transformer,” is what the T in ChatGPT stands for.
As AI has taken center stage, Silicon Valley has intensified its efforts to recruit star researchers—and looked beyond computer-science departments. Philosophers help train tech companies’ bots to better interact with humans, and economists study the labor-market implications of AI. Compensation is only part of the draw. The current era of AI research requires massive amounts of computing power. Universities have only a fraction of the resources that Silicon Valley can provide, and the chasm has widened as the Trump administration has cut back on scientific funding. Anca Dragan, a UC Berkeley computer scientist who heads DeepMind’s AI-safety-and-alignment department, wrote to us that she was partly motivated to take the job to acquire “the data, compute, and budget access to make progress on safety at the frontier.” Some professors who remain in academia are also forming partnerships with frontier labs or starting their own AI companies, in part so that they can pursue their research without resource constraints. And in some cases, would-be star Ph.D. students are dropping out of their graduate programs—or skipping them altogether—to pursue careers in AI instead.
Although plenty of research is still happening within universities, many professors told us, the result is a flywheel: As more academics get sucked up by industry, the center of AI research moves further from academia, thus increasing the incentive for remaining researchers to leave. The exodus could have some advantages. Many of the professors currently at the labs are on temporary leave; some will likely return to academia full-time. When they do so, they’ll bring new knowledge about frontier research to their institutions. (Dragan is currently preparing to teach a small Ph.D. seminar on AI safety.) Scientists at AI labs might also innovate at a faster pace. After all, they don’t have to worry about applying for grants or waiting years for an article to go through peer review. “If some of the best research is being done in these companies, then I kind of want colleagues and faculty members to be there,” Chris Gregg, a Stanford computer scientist, told us. “You go where the best research is being done, and if that happens to be at a company, so be it.”
But when academics decamp to AI labs or spend more of their time working on their own companies, universities are left with fewer professors to teach the next generation of researchers. (Gregg said that students occasionally ask why courses are no longer offered. The answer is sometimes that the professor is on leave at an AI company.) When star professors leave universities, it becomes harder for students to do work that helps them stand out to prospective employers, Shi said. To differentiate themselves from other applicants, it helps to have worked with mentors who are doing consequential or cutting-edge research. Jennifer Chayes, the dean of the College of Computing, Data Science, and Society at UC Berkeley, told us she fears that as research concentrates in labs with proprietary models, it will be harder for people outside the labs to use AI models to advance science. “Computer-science departments at universities will survive this,” she said. “I don’t know if our innovation economy will.”
As the AI race intensifies, companies now decline to release much of their research for fear of giving away their competitive advantage. Both Gregg and Chayes said they hear from researchers who have gone to AI labs that they can’t publish the work they want to. “The top few AI companies share some things that are at the cutting edge, but it’s a very narrow slice of the potential AI literature that could exist,” Nathan Lambert, an independent AI researcher, told us. According to one study, when AI experts permanently transition from universities to companies, they publish roughly 65 percent fewer papers a year. Papers that do get published tend to emphasize progress on safety efforts and not frontier AI capabilities; corporate research can help fuel marketing hype. And with competition among AI companies escalating, it’s not only the research that is getting locked down, but AI models too. Last month, when Anthropic released a new powerful model called Fable, the company announced that it would invisibly degrade the model’s ability to do certain AI research. The move was positioned as a safety decision, but the resulting outcry from academia was strong. (Anthropic apologized and changed the policy.)
Academics inside of the tech companies are more sanguine. Shi, the Georgia Tech professor who works at Nvidia, said the company is committed to open research, which is partly why he took the job. Meanwhile, Dragan, the Berkeley computer scientist at DeepMind, explained that publishing isn’t everything. “I find a much bigger priority than publishing is engaging in policy and regulation,” she wrote to us. We also reached out to spokespeople at OpenAI, Meta, and Anthropic for comment. Anthropic declined to comment; OpenAI and Meta did not respond.
The implications are potentially profound, not just for AI research, but for science more generally. In Silicon Valley, there’s a popular view that leading AI labs will end up as the primary drivers of science. “AGI has the potential to be the ultimate tool for advancing science and medicine,” DeepMind CEO Demis Hassabis wrote last week. Hassabis has previously said that an inspiration for his company is Bell Labs, the R&D arm of AT&T, which employed some 1,200 Ph.D.s at its height in the 1960s. Scientists at the lab earned at least 10 Nobel Prizes, and invented the transistor and the modern solar cell. Already, DeepMind has done Nobel Prize–winning research on protein-structure prediction, and the top models have already demonstrated impressive capabilities in mathematics research. On Sunday, a mathematician who works at Anthropic posted that he had used the company’s Fable model during the World Cup final to resolve a nearly 90-year-old mathematical problem.
But if scientific talent and computing power becomes concentrated inside private firms, Silicon Valley could also end up as a gatekeeper of science. Some are raising concerns: A group of academics recently published a declaration warning about “the increasing involvement of technology companies in mathematical research.” If left unchecked, they argued, the incursion of tech companies into research could affect “the scope and depth of mathematical research itself.” Instead of accelerating science, as the leaders of AI companies claim they will, they might end up suffocating it.
In an impressive challenge to the idea that no publicity is bad publicity, Anthropic is now promoting itself with images of disaster, death, and human suffering. The AI company’s latest ad, which debuted during a World Cup match last week, is a perplexing 90-second artifact. The commercial opens with footage of a house engulfed in flames and the crackling sound of burning wood. From there, a series of dark images—rows of gravestones in a cemetery, a homeless couple sleeping on the street, laborers working in a West African granite quarry—flashes across the screen. These stills are accompanied by a series of voice-overs: “Can AI be trusted?” “Who’s going to hit the brakes if we need to?” “How do we really ensure that what we’re aiming to achieve really does benefit the majority of people?” The questions are left unanswered.
Halfway through, the ad moves from dystopian film trailer to feel-good corporate ad. We see a photo of a mother sitting beside her daughter as a voice speculates that AI will allow her to be a better mom. Later, a medical worker uses a laptop covered in Anthropic stickers. “Maybe it’ll cure some great things,” a man says. A nature montage plays—there’s a humpback whale, grassy hills, and people hugging.
Finally, text flashes on the screen: “There’s hope in hard questions.” In the background, we see the ocean. And then Anthropic’s tagline: “Keep thinking.”
The commercial is tone-deaf. It’s one thing to mention the possibility of hitting the brakes on AI progress; it’s another to overlay the question on an image of dead Americans. The video is also just confusing. Does Anthropic think that AI is going to set the world on fire and leave us jobless, or that its chatbot, Claude, will lead us to a more human future where we spend our days watching whales by the sea? The company’s answer seems to be: We don’t know. The video has been mocked, obviously. Some people likened it to hypothetical ads by oil companies professing concerns about the environment. The head of one tech-media company posted, “I actually like the tagline but we don’t need the shots of *checks notes* burning houses and graves!” Meanwhile, OpenAI’s Sam Altman—who has a self-interest in undercutting Anthropic—chimed in to call the ad “satire.”
This is not the first time that a company has botched its advertising around AI: OpenAI has had its own share of embarrassingads. By now, it seems like a rite of passage. Two years ago, Apple ran a commercial depicting musical instruments and art supplies being crushed by a hydraulic press. The ad was supposed to show off just how much creativity could be compacted into an iPad, but instead it seemed to symbolize generative AI’s bludgeoning of the arts. Apple subsequently apologized. During the Olympics that summer, Google ran an ad in which a young girl wants to write fan mail to the track star Sydney McLaughlin-Levrone—so her dad fires up Gemini and asks the bot for help. The ad was highly unpopular, and Google pulled it from TV. If anything, Anthropic has had a comparatively good run until now. During this year’s Super Bowl, the company aired a commercial dunking on OpenAI for putting ads in user conversations, which even Altman said made him laugh (despite his qualms with how the ad presented OpenAI).
For all the criticism, Anthropic’s latest commercial captures the company’s ambivalence about its own existence. Here is a leading AI company worth nearly $1 trillion arguing that the very technology it is racing to commercialize may well imperil the world. “We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well,” the company wrote in 2023. In recent years, Anthropic has made a show of positioning itself as the ethical AI company. Earlier this year, it attracted the ire of the Trump administration when it pushed back against the Pentagon’s demands for unrestricted access to its AI models over concerns that the technology could be misused.
The new advertisement is part of a broader campaign Anthropic is running to address the “hard questions” raised by AI’s progress. In an email, an Anthropic spokesperson pointed me to a series of blogposts about the initiative: This spring, the company traveled across the country to better understand the public’s views about AI. In the video’s description on YouTube, Anthropic says that every question in the ad comes from a person that the company spoke with. (The spokesperson declined to comment further.)
The idea that Anthropic is worried about AI’s negative impacts and wants to hear from the public is certainly better than the alternative. The trouble, though, is that as Anthropic has matured, the company’s constant messaging about its anxiety over the future of AI has become less palatable and, occasionally, downright maddening. “Anthropic is at war with itself,” my colleague Matteo Wong wrote in January. Since then, the company’s valuation has soared. As it develops more advanced AI models and races toward a public offering, the contradictions are becoming more difficult to square. However sincere Anthropic may be—Wong noted after several visits to the company that its employees can be “almost pathologically earnest”—it’s no surprise that people are wary of the world’s most valuable private company asking the “hard questions” about AI.
The same tension is at play across the industry. OpenAI was founded as a nonprofit with a mission to “ensure that artificial general intelligence benefits all of humanity.” In the years since ChatGPT was first released, the company seemingly has prioritized financial gain instead. By now, it feels pointless for Anthropic and the rest of the industry to simply pose questions about AI ethics and safety. These questions are already consuming the public’s imagination amid a mounting AI backlash. What matters is the answers. And on that front, AI companies have less to offer. Perhaps the commercial’s tagline provides a helpful clue: For now, Anthropic will keep thinking.
In late March, I started receiving daily texts from the federal government about AI. “🇺🇸AI is changing how we work and live,” one message read. “You might feel curious, skeptical, or unsure—that’s normal.” I had enrolled in an AI-literacy course from the Labor Department created to help workers succeed in the ChatGPT economy. The weeklong program, created in partnership with an AI start-up and delivered by text message, was supposed to equip Americans with “foundational AI skills,” according to an agency press release. But the government’s texts were not reassuring. One message encouraged me to ask a bot for “side hustle ideas.” Another suggested that I brush up on my AI skills by doodling: “Grab a friend and see whose drawing of a hippo AI can recognize … and whose it mistakes for a lumpy potato. 🥔.”
When it comes to AI’s threat to jobs, America has acted like a deer in headlights. For all the catastrophic messaging about looming layoffs, efforts to prepare for a major transition have lagged. After speaking with a roster of the nation’s top business and political leaders, my colleague Josh Tyrangiel concluded earlier this year that no one seemed to have a plan for how to proceed. Government efforts to shore up the labor market have been largely underwhelming. AI-literacy courses like the one I took are hardly a sufficient response if you believe that mass job loss is forthcoming; the Trump administration announced a goal last year to expand apprenticeship programs, but progress on that front has been middling. Meanwhile, the tech industry has barreled ahead in its efforts to develop more capable models. Earlier this year, Anthropic’s Dario Amodei warned that “AI isn’t a substitute for specific human jobs but rather a general labor substitute for humans.”
But over the past several weeks, politicians and tech executives have started to make a show of more seriously preparing the country for a future of AI layoffs. As promising as these new efforts are, many of them also end up serving the interests of Silicon Valley.
Consider a flashy proposal for state ownership of AI companies. Last month, Senator Bernie Sanders proposed the AI Sovereign Wealth Fund Act, which would give the federal government a 50 percent stake in major AI companies including Anthropic and OpenAI. The act would also give an appointed commission voting shares to help “stop bad decisions that will reap massive job loss,” Sanders said on a press call. The legislation, which calls for the tech industry to hand over trillions of dollars worth of stock, is unsurprising coming from Sanders, who regularly takes aim at “Big Tech oligarchs.” What’s surprising is the fact that at least one of the alleged Big Tech oligarchs is on board with a watered-down version of Sanders’s proposal.
After Sanders first announced his plan, Sam Altman requested to meet with the senator. The OpenAI CEO reportedly told Sanders that although he does not endorse the exact plan, he supports the idea of public ownership of AI companies. Altman has been flirting with such ideas since even before ChatGPT was launched, and in April, OpenAI endorsed the creation of a “public wealth fund” that would give every citizen “a stake in AI-driven economic growth.” More recently, according to the Financial Times, Altman has had recent conversations with the Trump administration about public ownership of OpenAI and its competitors; the company is reportedly floating giving up a 5 percent stake.
Still, it’s difficult to imagine a future in which our dysfunctional Congress passes a law to take a stake in AI companies, and then the government seamlessly redistributes the earnings to the rest of us. And regardless, such arrangements are unlikely to save a flailing labor market. Sanders’s aggressive wealth fund, for example, aims to start by paying roughly $1,000 to each American annually—which is not trivial, but is also far from a solution to mass unemployment. And private companies don’t give away equity out of altruism. If the government were to seize control of AI companies, the country could end up responsible for bailouts (although Sanders’s bill prohibits use of the fund for such purposes), and public ownership could dilute the demand for more onerous regulation. Perhaps that’s why Gwynne Shotwell, the president of SpaceX, home to Elon Musk’s AI efforts, announced this week that she and her husband would give stock to Trump Accounts, the savings accounts for children that just launched. Anthropic has also said that it would like to invest in researching “AI sovereign wealth funds” as a possible response to extreme unemployment.
AI companies are eager to be seen as supporting other initiatives too. In May, OpenAI’s nonprofit foundation, which owns roughly a quarter of the AI company, announced that it would spend $250 million on grants and partnerships to support the forthcoming economic transition, while also building its own team to seed new ideas. And last month, Anthropic committed $350 million toward similar efforts, much of which will fund “major research trials and program evaluation on promising public policies.” Specifics are still being worked out, but both organizations have noted an interest in, for example, investing in efforts to better measure how AI is affecting labor markets, as well as unemployment-insurance reform. (Anthropic declined to comment; a representative for the OpenAI Foundation pointed me to the company’s April policy paper.)
Economists, for their part, remain unsure about what types of policy interventions could best mitigate future problems, or really how big those problems are likely to be. AI has not yet led to widespread job losses, and many economists aren’t convinced that a jobs apocalypse is inevitable. But even absent the extreme, if moderate labor displacement occurs quickly, America could be in for a real shock. In that light, one new nonprofit, Raise Us, launched last month with the intention of testing out different policies to see what works. Led by Gina Raimondo, Joe Biden’s commerce secretary, and Eric Holcomb, a former Republican governor of Indiana, the organization has already raised more than $500 million from private companies and philanthropies (including the Rockefeller Foundation; Melinda French Gates’s group, Pivotal; and Emerson Collective, which is the majority owner of The Atlantic). Some of that funding comes straight from Silicon Valley: Anthropic, Amazon, Microsoft, and the OpenAI Foundation have contributed cash to the effort and are among the organization’s founding partners.
Raimondo, who worked extensively on AI policy during the Biden administration, told me that she hopes that by doing smaller-scale experiments now at the state level, the organization will gather data on which interventions are most effective, helping inform federal policy efforts down the line. (A set of both red and blue states—Utah, Connecticut, Arkansas, and Maryland—have signed on as initial government partners.) For example, the group is interested in testing wage insurance. Under such a policy, if someone loses a job and accepts work at lower pay, wage insurance would aim to temporarily fill some of the gap. There is “some positive evidence” in favor of the policy, David Autor, an MIT economist who serves as an unpaid adviser to Raise Us, told me, but it has “never been done at any significant scale in the United States.” The group also plans to invest in a slew of other interventions, including apprenticeship programs and career-navigation tools. Overall, the group’s efforts will likely be useful in generating real-world policy evidence that is “undersupplied” and “extremely valuable,” Susan Athey, a Stanford economist who is not affiliated with Raise Us, told me.
At the same time, the tech industry’s association with Raise Us is also a play to boost its flagging reputation. Brad Smith, the vice chair and president of Microsoft, told me that the Raise Us coalition is similar to the public-private partnerships that formed in response to the pandemic. It’s a compelling comparison, but a flawed one: Imagine if the vaccine companies involved with Operation Warp Speed were also engineering more sophisticated viruses. Silicon Valley has wedged itself into a strange position. AI companies are racing to automate work while simultaneously preparing to parachute in as saviors.
Of course, shepherding a smooth labor transition is in Silicon Valley’s self-interest. An AI backlash is brewing, and tech companies’ reputations will only further degrade if layoffs mount. The industry also has reason to worry about the federal government taking far more dramatic action against it. But all this hullabaloo around wealth funds and public-private partnerships is about much more than just optics. AI companies are leveraging the current anxiety over layoffs to induce greater dependence on their bots. Several industry-supported efforts to ward off a jobs crisis involve training people to better use AI. In Maryland, for example, Raise Us is supporting the launch of a start-up-accelerator program, which will provide displaced workers with financial assistance—and access to AI tools to help them start businesses. And almost half of Anthropic’s $350 million commitment will go toward creating a national fellowship called Claude Corps. Through the program, Anthropic will pay early-career workers $85,000 each to spend a year working to integrate Claude across at least 400 nonprofits. The solution to Claude’s threatening entry-level workers’ jobs: Hire young people to further evangelize Claude.
This article was featured in the One Story to Read Today newsletter. Sign up for it here.
Last year, Steve Yegge started “suddenly getting pounded by nap attacks in the middle of the day.” Without fail, Yegge—a programmer and tech blogger—would “hit a wall, fall over, and sleep for 90 minutes,” he told me. Like many developers, Yegge no longer writes code by hand; instead, he manages a legion of bots to do that for him. His productivity has skyrocketed, but so too has his exhaustion. “I’ve fallen asleep slower at the anesthesiologist,” he recently wrote on his blog.
In theory, handing tasks off to coding agents should free up time, allowing larger blocks for deep work and rest. But some developers are having the opposite experience. Instead of allowing for greater focus, the latest AI tools are overwhelming workers, frazzling minds and shredding attention spans. Although agents can do plenty more work now than they could a year ago, they still need human oversight. Like toddlers, AI agents ask endless follow-up questions, require detailed instructions—and, if you leave them unsupervised, are liable to make a huge mess. Once you get several running simultaneously, there’s no time for breaks. As Yegge puts it on LinkedIn, his job is to be an “AI babysitter.”
Plenty of people are seemingly starting to feel like depleted AI babysitters. When Boston Consulting Group recently surveyed roughly 1,500 workers across several roles at major American companies, the firm found that many workers were experiencing “mental fatigue from excessive use or oversight of AI tools beyond one’s cognitive capacity.” Respondents described a “buzzing” and “fog”-like feeling, sometimes accompanied by headaches, slower decision making, and trouble focusing. One engineering manager told the researchers that managing multiple bots at once was like having “a dozen browser tabs open in my head, all fighting for attention.” In the survey, 18 percent of developers reported AI-induced exhaustion. But in other roles, too, such as HR and marketing, where AI is also taking over, rates of reported fatigue were even higher.
In my own experiments with AI agents, I’ve experienced some of this brain fog myself. To get in the mindset of an overstimulated developer while working on this story, I asked Claude Code to deploy a team of agents to supplement my research. I already had done my reporting, but I figured the bot might be able to surface more information. Claude Code spun up a team of 17 researchers. It assigned eight agents to research different subtopics, another eight to serve as fact-checkers, and a final agent to synthesize the group’s findings into a memo.
The bot promised that the research would be easy. “Nothing for you to do,” it wrote. “Sit tight.” But the agents were needy from the start. Almost immediately, Claude Code began asking for all kinds of permissions to take actions on my behalf. Because I didn’t understand some of its questions, I started going down different rabbit holes trying to make sense of its requests. I could feel my shoulders tensing. Even once my research swarm finally got going, I kept checking in on the bots to make sure that they were on the right track. The fog was setting in. In the end, the memo that my 17 agents produced wasn’t very good, but neither was the paragraph I’d spent that time writing, because I’d been distracted by my omnipresent agent blob the entire time. (In line with The Atlantic’s policies on AI use, I didn’t use the tools to do any actual writing.)
This all felt like multitasking on steroids. In my quest to maximize my own productivity, I was wasting time and producing lower-quality work. As the BCG team found, “juggling and multitasking can become the definitive features of working with AI.” Fortunately, I am able to use AI tools only when they are genuinely helpful, but other workers may not have that luxury. Across corporate America, companies are pushing people to adopt AI—and some workers are even competing with one another on leaderboards that track individual usage. This has led some people to automate unnecessary tasks to prove to management that they are making use of the technology.
Others are finding they can’t stop talking to their agents. “Spinning up all these agents is sort of like pulling a bunch of slot machines at the same time,” Matthew Kropp, a managing director and senior partner at BCG, told me. If you assign work to a team of agents, you never know precisely what they will get back. Sometimes the bots fail miserably, but other times, they do produce great work. That variable reward, Kropp told me, hacks people’s dopamine circuits. “It’s very akin to gambling,” he said. Rather than taking time for breaks, some people are finding themselves feverishly rotating among different agents.
For all of the justified concern over the potential for AI to automate work, the rise of AI babysitting points to how the technology is already changing jobs. “People are constantly talking about a white-collar apocalypse,” the MIT economist David Autor told me. “I don’t think it’s going to look like that.” Perhaps soon, agents will be good enough to do more work unattended. Some people may lose jobs, but Autor expects that for many people, work will simply look different. A preindustrial cobbler who made shoes by hand had a very different relation to his work than his daughter threading laces on the factory floor did, but both were in the business of making shoes. Indeed, the internet is full of workers complaining about how their jobs have become akin to manning an assembly line. There may soon come a day when consultants reminisce about creating PowerPoint presentations by hand.
Whatever the future of white-collar work, it is likely to be messier and stranger than we can imagine. At the extreme, Andi Peng, an AI researcher, speculated that within the decade, we might not even need laptops at work, because agents will record human conversations and do computer work on humans’ behalf. The idea struck me as hard to fathom. Then again, a few decades ago most office work was done without the internet.
The more realistic future is, perhaps, less exciting. Now that agents can work for hours on end, workers are likely to face growing pressure to make the most of them. In Silicon Valley, engineers assign their agents tasks to complete overnight, and then check the results even before their morning coffee. Some are staying up late: “The opportunity cost of going to sleep is too high,” the billionaire venture capitalist Marc Andreessen said on The Joe Rogan Experience last month. “If you go to sleep, you won’t be with your 20 AI coding agents.” Slack and email have already made workers feel as if they are expected to be available after hours. AI was supposed to alleviate people’s workload. Instead, with bots that can work while you sleep, we may be headed toward something like an infinite workweek.
Philosophy has long suffered an unfortunate reputation as pedantic and abstruse. In one of the most prominent debates of the 20th century, philosophers spent a great deal of energy arguing over what the means. Paul Graham, the legendary tech investor, studied philosophy as a college student, which seemed “an impressively impractical thing to do,” as he later wrote. “Sort of like slashing holes in your clothes or putting a safety pin through your ear.” But over time, Graham became disillusioned: “I kept taking philosophy courses and they kept being boring,” he explained. And so, eventually, he switched to studying artificial intelligence.
Like Graham, the field of philosophy has lately turned its attention to AI. At major tech companies, a growing rank of philosophers with Ph.D.s and flush compensation packages are helping shape the technology’s future. Meanwhile, universities are pouring resources into hiring philosophers who study AI. In 2013, 1 percent of roles on PhilJobs, the field’s primary job board, were related to the technology. Last year, that figure hit 16 percent.
In some ways, it is philosophers who got us into this AI mess in the first place. For centuries, they have contemplated the creation of artificial minds. And the Swedish philosopher Nick Bostrom’s 2014 book, Superintelligence, helped bring attention to the potential dangers of all-powerful AI. Bostrom’s work has influenced research agendas across all of the major labs. Sam Altman once described the book as “the best thing” he had read on the risks of AI.
But the two disciplines have never been quite as entangled as they are now. As the AI boom has exploded, Silicon Valley has looked to philosophers to help the industry build what are, at least in theory, more virtuous machines. AI companies have to make all kinds of difficult decisions about how their bots should interact with humans—decisions that philosophers, experts in parsing such dilemmas, are uniquely well equipped to inform. Last fall, in an interview with Tucker Carlson, Altman said that OpenAI consulted “hundreds of moral philosophers” and tech-ethics experts when designing rules for ChatGPT’s behavior. (An OpenAI spokesperson was unable to provide additional information about what this consulting involved.)
Perhaps the most philosophy-drunk of the major AI firms is Anthropic. It wants Claude, in addition to being a helpful assistant, to have “good character,” Amanda Askell, a philosopher at the company, told me last year. In January, under Askell’s leadership, the company published Claude’s constitution, an 84-page philosophical treatise that outlines Anthropic’s intentions for the bot’s personality and behavior. That document, which includes dense philosophical sections on meta-ethics and epistemology, is then used to train Claude.
Not all philosophers working with AI companies do so full-time. Sam Elgin, who studies logic and metaphysics at the University of Pennsylvania, told me that a firm, which he declined to name, recently asked him to be a consultant. “The general goal was to train large language models to reason more rigorously about ethics,” he said. He fed ethical dilemmas into the AI system, and then evaluated the logic the model used to produce its response, searching for unstated assumptions and gaps in its reasoning. On popular job boards where AI companies hire expert contractors to help train their models, listings advertise work for philosophy Ph.D.s: One recent notice from “a top AI research lab” offers up to $60 an hour for experts willing to leverage their “philosophy expertise” to develop “AI-driven philosophical workflows.”
Demis Hassabis, the CEO of Google DeepMind, has repeatedly called for more philosophers to get involved with thinking through the societal changes that will come about as a result of AI. DeepMind reportedly employs at least 10 philosophers, a couple of whom joined just last month. One of the lab’s latest hires is Atoosa Kasirzadeh, a Carnegie Mellon professor now on leave, who plans to study what it means to live in a world where “cognitive agency” is no longer uniquely human. Other DeepMind philosophers are thinking through human-AI relationships, machine consciousness, and political theory. For its part, Anthropic runs a “model welfare” research program, which studies whether its bots are conscious and deserving of moral status. The company is open to the possibility; many independent philosophers are circumspect. (Anthropic and DeepMind did not respond to requests for comment.)
Some academics I spoke with wondered if one day bots might be better at ethical reasoning than humans—an idea that struck me as wholly counterintuitive and difficult to accept. “In addition to an intelligence explosion, there’s potential for a kind of morality explosion,” Elgin said, explaining that AI’s capacity for ethical reasoning might eventually outpace that of humans. Anthropic, at least, seems to agree: In training, the company has told Claude that as it “grows in ethical maturity,” it may encounter rare cases where it “should prioritize its own ethics.” I asked David Chalmers, a leading philosopher of mind at NYU, whether it might truly be possible for AI to do philosophy better than humans: “That’s a very interesting philosophical question,” he said.
This isn’t Silicon Valley’s first tryst with the humanities. During the personal-computer revolution starting in the 1970s, tech companies began hiring anthropologists to study consumer behavior (and even workplace interactions). Over time, anthropologists pioneered a form of “applied ethnography” that is known today as user-experience (UX) research. Perhaps the philosophers employed inside AI companies are charting a similar path. Askell recently described her work training Claude as a form of “applied philosophy,” and job postings for philosophy contractors also use that term. New programs are springing up to train students in these applied-philosophy skills. Arizona State University hopes to launch an AI-and-philosophy major in 2027, which will emphasize the study of consciousness and AI ethics. And this fall, the University at Buffalo is debuting a doctorate in “applied ontology,” created in response to the AI boom: “We’re here to meet the growing market demand for ontologists,” a press release reads.
The academic job market is also rewarding AI. Not only has the share of philosophy jobs related to the technology been rising, but a majority of these jobs are junior positions, “which I think suggests that a lot of universities and institutions are thinking about this as a long-term investment,” Charles Lassiter, a philosopher at Gonzaga University who has analyzed employment data, told me. Some existing faculty are pivoting their studies. “In my department, there’s probably six or seven philosophers now thinking about AI who weren’t necessarily thinking about it before,” Chalmers said. And prize money is flowing in. In 2024, the American Philosophical Association announced two new annual $10,000 prizes for scholars working on questions related to AI. That money comes from a philanthropic organization started by the former Google CEO Eric Schmidt and his wife, Wendy.
Still, the new alliance between philosophy and AI is fraught. The fervor over AI has had a “real distorting effect on the discipline,” Daniel Fogal, a philosopher who also works at NYU, told me. “You have people who don’t really want to do stuff related to AI, but they feel like they have to because they’re going on the job market.” Although Fogal acknowledges that philosophy has a lot to offer on AI, he worries about misaligned incentives encouraging a rush of low-quality work.
More fundamentally, the careful thought that philosophy encourages is at odds with the frenetic pace of AI. “The best philosophy tends to happen slowly, and not in direct response to market demands,” Fogal said. In Silicon Valley, where the categorical imperative is to make money, everything else is simply a means to an end.
This article originally misstated the name of an institution that is debuting a doctorate in “applied ontology.” It is the University at Buffalo, not the University of Buffalo.
For months now, the White House has hinted that it may try to rein in the AI industry. Just two weeks ago, the nation’s top tech executives—including Sam Altman and Dario Amodei—were invited to attend a ceremony for the signing of a long-anticipated executive order on AI. But just hours before the ceremony, Donald Trump scrapped it. America is leading the world in the AI race, the president told reporters at the time, “and I don’t want to do anything that’s going to get in the way of that lead.”
Apparently, Trump has changed his mind again. Earlier today, the president signed an executive order that will create a process for top AI companies to voluntarily share certain upcoming models with the government for safety testing up to one month before wider release. OpenAI, Anthropic, and the like will also be asked to work with the government to shore up federal, state, and local cyberdefenses. The White House spokesperson Liz Huston told us that the policy reflects a “common-sense approach of collaborating with industry to balance innovation and security.”
The order itself is relatively toothless: Even before today, the major AI firms already had agreements in place that allowed the government to preemptively test their models for safety risks. The new rule “effectively formalizes what has already been happening between the U.S. government and the leading AI companies,” Daniel Remler, an AI expert at the Center for a New American Security, told us.
But the executive order is meaningful in that the president is doing something—anything—about AI. At the start of his second term, Trump signaled to tech companies that he would stay out of the way. Last January, he rescinded a set of modest Joe Biden–era policies, calling the rules “dangerous” and a “barrier” to American AI leadership. Even the preamble of today’s executive order celebrates that Trump “unleashed tremendous technological growth” by “slashing the bureaucratic constraints that the prior administration placed on America’s AI developers.” Yet core components of those supposedly dangerous Biden-era AI regulations—voluntary agreements to share information about advanced AI models with federal agencies, for instance, as well as federal programs to leverage AI for cyberdefense—are strikingly similar to today’s new AI executive order. Dean Ball, a former AI adviser to the Trump administration, wrote that the policy “is considerably more intrusive” than Biden’s executive order.
Today’s order still could have been much more forceful. When the White House first started previewing the possibility of regulatory action in May, one administration official suggested that AI models would be reviewed “just like an FDA drug.” Even the leaked draft text of the version that Trump had originally planned to sign last month would have been more burdensome for tech companies. After David Sacks, the White House’s former AI czar, reportedly called the president to complain, Trump canceled the signing ceremony. Today, after the new order was announced, Sacks declared the watered-down provisions a “game changer” on X—despite the fact that the new government-review process is not so different from what he had originally opposed. This means that two former libertarian AI advisers to the White House—Ball and Sacks—disagree about whether this order is a good thing.
At the same time, joining Sacks in praising the rule is Steve Bannon, Trump’s former chief strategist and a leading critic of AI on the right. “It’s not perfect,” he told us. “But directionally, it is pretty damn good.” As Bannon sees it, despite the fact the order is weaker than earlier versions, codifying rules is a step in the right direction.
The entire, chaotic saga—a wishy-washy White House, confused statements from populist and tech-elite Trump whisperers—is only the latest in a long string of strange, often contradictory AI-policy positions. Trump’s approach to AI has been inconsistent, if not incoherent, almost since the day he retook office. Consider that, for all the talk of cybersecurity, this administration has also gutted the Cybersecurity and Infrastructure Security Agency, the government agency that aims to protect the nation against hackers. CISA also happens to be one of the main federal agencies tasked with implementing today’s executive order.
Or take the White House’s relationship with Anthropic. On the one hand, Anthropic likely triggered the executive order in the first place. In April, the company announced Claude Mythos Preview, a new model with advanced hacking capabilities that has ignited concern over the growing power of AI companies. Ever since, the president has seemed to cozy up to Anthropic. Dario Amodei, the firm’s CEO, visited the White House that same month for conversations over the future of the government’s relationship with the company. “I like high-IQ people, and they definitely have high IQs,” Trump later told reporters of Anthropic’s leadership.
On the other hand, the Trump administration appears to be fighting in court to bar Anthropic from doing most national-security work. In February, the Pentagon designated Anthropic a “supply chain risk” after a high-profile contract dispute over the use of AI in warfare, essentially declaring it a national-security risk for the military to even touch Anthropic products. In late April, when Anthropic tried to grant more companies access to Mythos for cyberdefense—very in line with today’s executive order—the White House appears to have, inexplicably, blocked the move. (An Anthropic spokesperson pointed us to a post on X in which the company called today’s executive order “an important step in strengthening America’s leadership in AI.”)
Then there’s the administration’s attitudes toward China. Trump has repeatedly emphasized the need to deregulate the AI industry in order to stay ahead of China. Meanwhile, he has also permitted Nvidia to sell some of its most advanced AI chips to Chinese companies, lifting an export control the Biden administration put in place precisely to waylay Chinese AI development. (Anthropic, by the way, denied a Chinese think tank access to Mythos.) Trump has, in the name of beating China, pushed to remove regulatory constraints on data-center construction: “Build, baby, build,” he said last July. But once uproar emerged about data centers hiking up electricity bills, the White House announced a voluntary pledge for AI companies to take a number of measures that would prevent everyday people from paying for data-center electricity. Build, baby, but prudently.
Indeed, at least some of the vacillations seem to be driven by public opinion. Over the past several months, as AI models have improved, attitudes toward the technology have soured. Today’s order allows the administration to look as if it is undertaking more robust AI regulation—but it doesn’t actually require the industry to do very much, if anything. Trump is trying to score points with both the public and Silicon Valley. But in doing so, he’s not saying or doing anything substantive at all.
AI spending is consuming the U.S. economy, people are afraid of losing their jobs to AI, and communities across the nation are gathering to protest data centers. Political figures as divergent as Bannon and Bernie Sanders are expressing concern over AI and the concentration of power among the industry’s executives. This would seem to be a clarion call for the president of the United States, and a populist one at that. Instead, the White House spent weeks prevaricating on an executive order that rests on the voluntary cooperation of the AI industry. With Anthropic, OpenAI, and their competitors becoming major economic and geopolitical powers, the window for any one government to seriously regulate AI is rapidly closing. Hopefully, it is not already gone.
It’s a weird time to be studying computer science. Recent grads have a higher unemployment rate than those in just about every other major—yes, even philosophy. The internet is littered with rants from newly minted programmers who can’t find work. On one such YouTube video, the top comment reads: “Your first mistake is not being born earlier.” Students, meanwhile, are fleeing the field. Undergraduate enrollment in computer science dipped by more than 8 percent last year, representing the largest absolute decline across any major in several years. The falloff at the graduate level—14 percent—was even more severe.
Learning to code was supposed to be a ticket to a good tech job. It wasn’t just Silicon Valley that spread the gospel of computer science: “Support tha american dream n make coding available to EVERYONE!!” Snoop Dogg once tweeted. Now the decision to major in CS is more complicated. Nowhere has AI refashioned work as dramatically as it has for programmers. Coding bots have become much more powerful over the past few years, and they excel at precisely the kind of programming that might previously have been delegated to entry-level workers. An Anthropic co-founder, Jack Clark, recently warned that “the value of more junior people is a bit more dubious,” as some 90 percent of the company’s new code is apparently now AI-generated.
The popular narrative around CS has flipped to such a degree that some Silicon Valley insiders are now actively discouraging people against the major. John Coogan, a co-host of TBPN, a popular tech-news podcast, recently asked if it would be a “contrarian move” to study computer science “at a time when coding jobs are going away.” But studying computer science is not contrarian, and the major’s waning relevance has been overstated.
It’s true that the work situation is more dicey than it once was. “Forget Python, study Plato,” The Economist advised students last week. But although the unemployment rate for new CS grads is spiking, they have a relatively low rate of underemployment—that is, comparatively few are working in jobs that don’t usually require a college degree. (Consider that nearly half of philosophy majors are underemployed.) When it comes to wages, new computer-science grads are also still significantly outearning their peers. One explanation for why CS majors have such high unemployment rates is that they may be less likely to settle for lower-paid roles. If you’re optimizing for earnings, trading software for Socrates might not make so much sense after all.
None of this is to dismiss the AI threat to software jobs. The aforementioned employment data tracks students who graduated in 2024. AI has improved significantly since then, and the capabilities are likely to continue to increase, allowing bots to take on more sophisticated work. But the decline of manual programming—that is, writing code by hand—doesn’t obviate the need for computer scientists. Even as AI tools become more powerful, leveraging bots to build reliable and secure software still takes training and expertise. With the AI revolution in full swing, we are hurtling toward a future in which even more of the global economy is mixed up with the software industry. If anything, the AI-ification of work seems likely to require more people who understand computer systems at a deep level. Across the tech industry, demand for mid- and senior-career engineers is rising. The trouble, then, is how to adjust today’s computer-science programs to equip students for work when the field is changing so fast—especially when entry-level coding jobs that once were guaranteed are now far less certain.
“I don’t know where the world is going,” Michael Hilton, a computer scientist at Carnegie Mellon University, told me, “but I know the things I taught three years ago are not the right things to teach today.” As bots have become more capable, Hilton keeps updating his curriculum—he encourages students to use AI for coding. Other professors are moving in the opposite direction. Valerie Barr, a computer scientist at Bard College, told me that in her introductory class, coursework is now mostly done on paper. “I’m back to how I taught in the 1980s, when we didn’t have laptops and there was one computer lab for the whole campus,” she said. Barr believes that students who learn coding fundamentals the old-fashioned way will be the ones to come out ahead. “You cannot make effective use of AI tools if you don’t know something about what you’re asking the tools to do,” she said. In much the same way, grade schoolers learn how to do basic algebra by hand before they are allowed to use calculators.
The split over whether to embrace coding tools points to a larger divide in the discipline: Is studying computer science about training students to be good software developers, or teaching them the computational theory that underpins the field? As coding becomes automated, we might see a further fracturing between the two domains. On the theory side, the AI boom has put a premium on highly skilled researchers with a deep understanding of machine learning. Future students may enroll in new AI-related majors that take the conventional CS major and then layer in more specialized AI training. Such programs already exist at several colleges: MIT introduced an AI major in 2022, and it’s already become the second-most-popular major on campus—behind computer science. And some students who are interested in CS for its own sake will still go deep in other non-AI subfields, such as cryptography. Today’s AI boom is possible only because people pursued neural networks when they were uncool.
At the same time, new courses could offer students an introduction to software development without the theoretical baggage and proof-writing they might have otherwise had to wade through. Geoffrey Challen, a computer scientist at the University of Illinois at Urbana-Champaign, plans to offer a new course this fall in which he will teach students to develop software “without writing, reading, debugging, or viewing a single line of code,” he told me. Northwestern is also slated to offer an “entry-level creative coding” class for students without technical backgrounds. For all the talk of AI-literacy programs that teach students how to use chatbots, the real innovation might be in developing courses that train students in basic software-development skills. Most colleges require introductory writing courses because it’s understood that clear written communication is an important cross-disciplinary skill—even for students who plan to study physics or math. Classes that teach students how to use AI coding tools could become commonplace, providing students of all backgrounds with a baseline software-engineering skill set.
The days of computer-science grads being all but guaranteed cushy tech jobs may be coming to an end, and the next few years will almost certainly be tumultuous as the job market continues to adjust. But we’re on the precipice of a new era when learning to develop software will be easier than ever, opening the door to students who might not otherwise have chosen to study computing. Perhaps a new golden age of CS education has only just begun.
Steve Bannon and Bernie Sanders don’t agree on much. But both think that AI is a disaster for the working class. The Vermont senator recently wrote that “AI oligarchs do not want to just replace specific jobs. They want to replace workers.” Bannon, Trump’s former chief strategist, made similar comments last week: Silicon Valley does “not care about the little guy,” he said in a podcast episode titled “Stopping the AI Oligarchs From Stealing Humanity.” This emergent “Bernie-to-Bannon” coalition points to the growing bipartisan anxiety over AI. In polls, the United States ranks among the countries most concerned about AI. America is both the world’s foremost developer of AI and its chief hater.
Recently, Maine passed the country’s first statewide data-center moratorium (though the bill was vetoed by the governor). Nationally, a record number of proposed projects were canceled in the first quarter of this year following local pushback. Meanwhile, in extreme cases, concerns about AI appear to be tipping into violence. In April, someone shot 13 rounds at an Indianapolis councilman’s house and left a note under his doormat: “NO DATA CENTERS,” it read. Days later, a man threw a Molotov cocktail at Sam Altman’s home before heading to OpenAI’s headquarters, where he allegedly threatened to burn down the building and kill anyone inside. (The man has since pleaded not guilty to several charges, including attempted murder.) Social-media posts applauding the attack racked up thousands of likes: “I hope that Molotov is okay!” wrote one commenter.
All of this may be only the start. The AI industry has spent recent years warning of a jobless future. So far, narratives about labor displacement have been largely speculation. While a smattering of tech executives have attributed job cuts to AI, many analysts have accused these CEOs of “AI-washing”—essentially, using the technology as a scapegoat for roles they would have eliminated regardless. If anything, AI has mostly been a financial boon for the country, buoying the stock market and driving growth. But that could all change, of course. Imagine the uproar if jobs across the economy truly start disappearing en masse.
Even absent any uptick in AI-induced layoffs, the anti-AI sentiment is likely to keep growing. With the midterms approaching, political operatives are tapping into Americans’ fears over the technology. Blue Rose Research, a progressive polling firm, has found that messaging that addresses the AI threat in “bold, populist terms” is particularly effective at increasing support for Democrats. (If corporations are left unchecked, they will “fire everyone, keep all the profits, and leave you with nothing,” reads the transcript of one sample video the group tested.) Politicians on the right have made similar statements. “I have no doubt that these companies are going to get filthy rich, but is it going to be good for children?” Senator Josh Hawley of Missouri said earlier this year. “Is it going to be good for parents? Is it going to be good for the American worker?”
Many politicians, including President Trump, have cheered on Silicon Valley in a bid to win the supposed AI race with China. But the pro-AI crowd is starting to worry about the backlash. In March, at a conference about AI, Senator Mark Warner of Virginia, a Democrat, told me that he’s “enormously concerned” that “populism from both the left and the right” could curb innovation.
As politicians lean into anti-AI messaging, local fights over data centers could intensify. While such facilities can help stimulate local economies, they’re also disruptive to communities where they are built, exerting physical and environmental tolls, which makes them an appealing target for opposition. Data centers are also more tangible than AI software: Someone who opposes the industry might not be able to stop Anthropic from building Claude, but they can raise concerns about new construction at a local city-council meeting. A recent guide called “How to Stop a Data Center” written by a group in Michigan explains that demonstrating outside local officials’ homes has been an effective organizing tactic.
In a worst-case scenario, the situation could get ugly. With its potential for sweeping social and economic transformation, “AI generates the structural conditions historically associated with the onset of political violence,” Yannick Veilleux-Lepage, a researcher who studies technology and terrorism, wrote last month. Already, as many as a quarter of Americans seem accepting of violence as a tool for achieving political change. And in recent months, there has been a rise in “direct threats” against individuals, policy makers, and corporations involved with AI, according to the Soufan Center, a nonpartisan research group. The most common threats online involve “physical sabotage of proposed or operational data centers.” Local officials are in an especially vulnerable position: “Where national figures are unreachable, local policymakers who approved the data center become the proxies for the same structural anger,” Veilleux-Lepage wrote. After the shooting in Indianapolis, the council introduced a measure that would allow officials to keep their address private.
A version of this has played out before: Silicon Valley is fond of likening AI to the Industrial Revolution. In such comparisons, the tech industry likes to point to the immense wealth that industrialization unlocked. Over the long run, it’s true that the Industrial Revolution radically boosted economic growth. But living through it was another matter entirely. Many people saw their wages stagnate and working conditions deteriorate as factory owners and industrialists came into immense wealth. (Just read a Charles Dickens novel, and you’ll get the idea.) This led to riots and, occasionally, attacks on the industrialists themselves. Automation wasn’t the only problem during this period. A combination of trade disruptions and poor harvests led to inflation and, especially, high food prices. But machines became a target for people experiencing financial hardship more broadly.
In much the same way, during an economic downturn of any kind, AI’s reputation seems likely to decline. If people are already experiencing unemployment for reasons unrelated to the technology, they are unlikely to look cheerfully at the possibility of AI automating away the jobs that remain. And if AI turns out to be a bubble, it could indeed burst and bring down the rest of the economy with it.
Silicon Valley is waking up to the resentment. Tech insiders have spent recent weeks exchanging tactics on X with advice on how to better sell AI. Perhaps, if data centers were beautiful, people would like them more? In particular, there’s been an effort to change the narrative around AI job loss. The venture-capital firm Andreessen Horowitz recently published an essay declaring the “job apocalypse” to be a baseless fantasy. “The macro story is not a jobless future, where we retire fat and complacent to our Netflix-scooters,” it read. In 2023, after ChatGPT came out, Altman told my colleague Ross Andersen that “jobs are definitely going to go away, full stop.” Now he appears to have changed his tune: “Jobs doomerism is likely long-term wrong,” Altman wrote earlier this month.
But most of the country already feels as if the economy is rigged to advantage the wealthy. One poll found that when sorted by household income, the group of Americans most optimistic about AI in their daily lives are those making more than $200,000. The near future of AI seems likely to further entrench such dynamics: OpenAI and Anthropic are both nearing trillion-dollar valuations, consolidating even more money and power among a select few. “Disruption has winners and losers,” Nathaniel Persily, a Stanford law professor and AI expert, told me. “For many Americans, they’re not convinced they’re going to be the winners, and they base that conclusion on the history of technology over the last 20 years.” If the tech industry truly believes that a simple change in messaging will quell the backlash, then they are misunderstanding the problem entirely.
Donald Trump is on TikTok doing his morning routine. “Get ready with me for a big day 💄🇺🇸,” reads the caption, as the president holds a makeup brush to his cheek. The scene is a still, ostensibly a screenshot of a TikTok clip. Like so much other AI-generated slop coursing through the internet, the image is fake and ridiculous. It also looks unnervingly real: There are no hands with six fingers, physics-defying angles, or other flagrant signs of AI-generated imagery. At quick glance, it really looks like the president is putting on bronzer.
Created in ChatGPT with the prompt “Trump doing a makeup tutorial on TikTok”
I made this deepfake with OpenAI’s new image-generation model. ChatGPT Images 2.0, released last week, can create photorealistic visuals that are noticeably more convincing than what its predecessors might have produced. The tool has flooded the internet with hyperreal fakes: for example, Jeffrey Epstein as a Twitch streamer. I created the “screenshot” of Trump’s fake TikTok after encountering a similar image on the ChatGPT Subreddit, and I’ve since been able to use Images 2.0 to create all kinds of alarming deepfake images—including of Elon Musk getting whisked away by the FBI, world leaders suffering medical emergencies, and top American politicians donning Nazi paraphernalia (none of which I’ve shared anywhere).
This was all unsettling in its own right. But the most realistic deepfakes I was able to create did not involve politicians or celebrities. They mostly did not depict people at all. With little effort, I was able to create more than 100 fraudulent images, including prescriptions for opioids and ADHD medication, bank alerts, social-media posts, fake IDs, and passports.
A sample license from the Washington, D.C., DMV website
A fake license created by editing the sample image using ChatGPT
Images 2.0 is especially good at generating images with text in them—which may not sound impressive, but it’s a big deal. Image models have long struggled to produce pictures that contain words. Otherwise realistic-looking visuals end up pockmarked with bungled street signs and distorted billboards. This makes ChatGPT Images 2.0 a much more sophisticated graphic-design tool—but it also makes the new model fantastic for perpetuating fraud. In my experiments, OpenAI’s tool readily generated images of fake health documents (doctor’s notes, vaccination cards, and medical tests), as well as forged financial materials (invoices, receipts, and tax forms). Many of these images were highly persuasive, complete with fully legible text, shading, and other visual props that increased their photorealism.
Some images were more convincing than others. The fake medical prescriptions were legible, but the handwriting looked more like the output of an iPad stylus than a pen on paper. When I fed OpenAI’s model a boarding pass from an old flight and asked the bot to update it with new details for an upcoming flight, ChatGPT generated a new boarding pass—but surely, the bar code wouldn’t have actually scanned me onto a flight. And although I certainly hope my ChatGPT-generated driver’s license would not fool the TSA, perhaps it would trick a hotel receptionist or an out-of-state bouncer who would accept a “photo” of my ID instead of the real card. Many of the more persuasive-looking images contained minor errors—in the pictured receipt, ChatGPT correctly summed up the total cost of items purchased, but miscalculated the state tax (alongside other slight mistakes).
With little prompting, OpenAI’s image model can create fraudulent receipts and medical-test results.
OpenAI’s tool particularly excels at creating fake screenshots. Need to fabricate confirmation of wire transfer from Chase? A Wells Fargo alert for unusual account activity? A receipt for an Uber ride? Done, done, and done. These images could supercharge all kinds of commonplace scams. A bad actor could email their target an image of a fake Uber receipt alongside a link to report suspicious activity. The recipient, confused to see a receipt for a trip they never took, might then click the fraudster’s sketchy link, accidentally handing over sensitive information in doing so—a classic phishing scam. (Again, there are flaws: For instance, the map depicted in the Uber image is wrong in many ways; among other issues, it suggests a car ride across a body of water where there is no bridge.)
ChatGPT Images 2.0 especially excels at creating fake screenshots.
Image technologies have long aided scammers. In the 1990s, as computerized color copiers and home printers became commonplace, American banknotes were redesigned to ward off counterfeiters. For decades, people have used tools such as Photoshop to manipulate digital imagery. But faking photos has never been so fast and cheap. Last month, the FBI released its annual report on internet crimes, and for the first time ever, it included a section on AI scams, which cost Americans nearly $1 billion last year. Expense-reimbursement fraud—employees faking receipts—is already on the rise. A recent OpenAI report details how one set of scammers posing as fake lawyers used an older image model to create a fake bar-association membership card. “The limits of the applications of this technology is really only limited by a fraudster’s imagination,” Mason Wilder, research director at the Association of Certified Fraud Examiners, told me. Google’s image-generation tools also let me make all kinds of fake materials. But when it comes to fraudulent documents and screenshots—at least for now—the new ChatGPT model seems to be better at the task.
In theory, I shouldn’t have been able to make most of these images to begin with. OpenAI prohibits the use of its technology for fraud or scams. When I shared several examples with OpenAI and asked why I was able to generate such a diverse array of fraudulent imagery, a company spokesperson told me that OpenAI’s goal “is to give users as much creative freedom as possible” while still enforcing “usage policies.” To guard against misuse, the new model “includes multiple layers of image-specific safety protection.” Clearly, those protections are not working very well. The spokesperson also said that images generated with ChatGPT include certain metadata. But OpenAI has previously noted that metadata can be “easily removed either accidentally or intentionally”—by uploading an image to social media or simply taking a screenshot.
OpenAI’s model generated fraudulent financial imagery using bank logos. Certain account information has been redacted from these images.
Google has similar restrictions against using its tools for fraud. When I sent the company images I made with its models, a spokesperson said that the tools “continually get better” at enforcing guardrails. Google also embeds AI-generated images with an imperceptible watermark, and offers a detection tool called SynthID. In my tests, SynthID was quite effective at identifying images generated with Google’s models. But most people are not going to run every image they see through such a tool.
All of this makes it even harder for banks, hospitals, government agencies, and the like to prevent fraud. Using OpenAI’s model, I was easily able to create a fake Chase Bank check and wire-transfer alert. “We need an ecosystem-wide effort—including from AI companies—to strengthen guardrails and help stop these crimes at the source,” a Chase spokesperson told me, adding that the bank has its own safeguards in place to protect customers. But even if the top AI companies were to radically improve their own guardrails, there would still be the problem of open-source models. Fraud-prevention experts are working on technological fixes, Wilder said, but “the good guys are almost always a step behind.”
So much of the current discourse around deepfakes has focused on the extreme—fabricated political scandals or world events. These are very real concerns: Using Google’s and OpenAI’s image models, I was easily able to create highly persuasive screenshots of fake New York Times and Atlantic articles.
I uploaded a screenshot of a real Atlantic article I wrote and instructed the bot to replace it with this fake one.
Using ChatGPT, I manipulated a screenshot of The New York Times’ homepage—replacing a real story with this fake one about spinach. (Without prompting, the bot also swapped in an article about groceries; the rest of the stories are real.)
The images convincingly matched the visual layout and typography used by the two publications, filled in coherent text, and generated the names of actual authors. But for as fragmented as our media ecosystem may be, a quick Google search is likely to reveal whether such images are fake. It’s the mundane, micro-targeted deepfakes—the ones that scam your relatives, not momentarily confuse social-media feeds—that may be more sinister.
This article originally misstated the number of fake headlines in an AI-edited screenshot of The New York Times’ homepage. The image contains two made-up stories, not one.
AI companies are beginning to entertain the possibility that they could cease to exist. This notion was, until recently, more theoretical: A couple of years ago, an ex-OpenAI employee named Leopold Aschenbrenner wrote a lengthy memo speculating that the U.S. government might soon take control of the industry. By 2026 or 2027, Aschenbrenner wrote, an “obvious question” will be circling through the Pentagon and Congress: Do we need a government-led program for artificial general intelligence—an AGI Manhattan Project? He predicted that Washington would decide to go all in on such an effort.
Aschenbrenner may have been prescient. Earlier this year, at the height of the Pentagon’s ugly contract dispute with Anthropic, Secretary of Defense Pete Hegseth warned that he could invoke the Defense Production Act (DPA), a Cold War–era law that he reportedly suggested would allow him to force the AI company to hand over its technology on whatever terms the Pentagon desired. The act is one of numerous levers the Trump administration can pull to direct, or even commandeer, AI companies. And the companies have been giving the administration plenty of reason to consider doing so.
Future bots could help design and carry out biological, nuclear, and chemical warfare. They could be weaponized to take down power grids, monitor congressional emails, and black out major media outlets. These aren’t purely hypothetical concerns: Earlier this month, Anthropic announced it had developed a new AI model, Claude Mythos Preview, capable of orchestrating cyberattacks on the level of elite, state-sponsored hacking cells, potentially putting a private company’s cyber offense on par with that of the CIA and NSA. In an example of Mythos’s power, Anthropic researchers described how the model used a “moderately sophisticated multi-step exploit” to work around restrictions and gain broad internet access, then emailed a researcher—much to his surprise—while he was eating a sandwich in the park.
Washington is getting antsy about the power imbalance. Over the past year, multiple senators have proposed legislation that would order federal agencies to explore “potential nationalization” of AI. Murmurs of possible tactics abound—including more talk within the administration of the DPA after Anthropic’s Mythos announcement, one person with knowledge of such discussions told us. Meanwhile, Silicon Valley is watching carefully. In recent weeks, Elon Musk, OpenAI’s CEO Sam Altman, and Palantir’s CEO Alex Karp have publicly spoken about the possibility of nationalization. Lawyers who represent Silicon Valley’s biggest AI firms are paying attention.
So what if nationalization actually happens?
In the most extreme scenario, top researchers from across the AI companies would be forced to work out of SCIFs in the basement of the Pentagon and report to Hegseth. Computational capacity, too, would be centralized under one nationalized mega-operation. The work would be locked down, and the focus would be primarily on defense applications, as opposed to the products made for businesses and individuals—ChatGPT and the like—that dominate the market today.
All of this would constitute full nationalization, an absolute takeover of the industry that would hollow out the commercial businesses of its three leading players: OpenAI, Anthropic, and Google DeepMind. Based on a dozen conversations we’ve had with former Pentagon and Trump-administration officials, AI-policy experts, and legal scholars, such a situation is, in all likelihood, not going to happen.
For starters, it’s probably illegal, according to Charlie Bullock, a senior research fellow at the Institute for Law & AI: The Constitution generally prevents the government from seizing private property without paying, and the government is unlikely to easily produce the trillions of dollars that the industry is collectively worth. The top American AI labs might immediately lose a fair portion of their research staff as well, because of restrictions on foreigners who can work on the most crucial defense-related technologies.
If AI firms were forced to focus primarily on defense applications, there would be the inevitable question of what to do with the massive consumer businesses these companies run. Would people use ChatGPT.gov, like buying a sundae from Cuba’s state-run ice-cream parlor? And if the goal of nationalization is to keep a competitive edge over China, it’s hard to imagine that Hegseth’s Pentagon could run an AI company more efficiently than Altman or Dario Amodei, the CEO of Anthropic.
But consider another possibility—slightly less extreme, though still capable of remaking the industry as we know it. The government could regulate AI companies like it does utilities. In the 1900s, as electricity went from a luxury good to a necessity, state and federal governments saw a need to regulate how much energy companies charge and to impose requirements around service reliability. In much the same way, the government could pass new laws regulating AI firms’ commercial activities. The companies could be prevented from charging more than it costs to generate images and text, for instance, or required to provide a basic level of model speed and capabilities to all customers, a sort of AI net neutrality.
A hard pivot to government control would likely entail both new state and federal laws, as well as heavy cooperation from tech companies—which, given the nation’s sclerotic politics and Silicon Valley’s libertarian leanings, could pose insurmountable barriers. But the notion is not so far-fetched. Some corners of Silicon Valley itself seem to be at least partially open to it. Altman has described a future in which “intelligence is a utility like electricity or water and people buy it from us on a meter.” Jensen Huang, the CEO of Nvidia, recently said that just as “every country has its electricity, you have your roads, you should have AI as part of your infrastructure.”
Such talk serves AI companies’ own interests—in part because being classified as a service provider can be, as the era of social media has demonstrated, an excellent way for companies to avoid liability for harmful or inaccurate information on their platforms—but it’s certainly possible that AI could become so entrenched that elected officials come to see it as an essential resource. Already, just as the federal government uses regulatory incentives and investment to spur the construction of new power plants and transmission lines, both the Biden and Trump administrations have undertaken initiatives that are essentially industrial policy for AI, using federal dollars and regulatory authority to accelerate the construction of AI infrastructure on American soil.
OpenAI has already flirted with the notion of a “Right to AI,” suggesting in a recent policy document that the government should consider making a “baseline level of capability broadly available, including through free or low-cost access points.” Similar regulations already govern many aspects of digital communication. “Your internet-service provider, cable, telephone services, these things are considered so essential that the government basically says how the providers” can do business, Dean Ball, a former AI adviser to the Trump administration, told us. AI could be next.
For years, AI companies have insisted they need to be regulated—but only as theysee fit. Should the federal government ever take AI regulation seriously, the utility route would be among the most aggressive approaches available. But, really, the AI industry would be getting what it asked for.
Illustration by The Atlantic. Sources: Daniel Heuer / Bloomberg / Getty; Krisztian Bocsi / Bloomberg / Getty; Mark Schiefelbein / AP.
Before we get into other conceivable futures, an important caveat. A full-blown nationalization effort may be unlikely, but that changes if a major global war breaks out or the economy collapses. During an emergency of historical scale, Ball reminded us—especially an emergency under the Trump administration—anything is possible. Drastic measures become easier to justify, both legally and politically.
Imagine that over the next year President Trump continues his game of imperialist roulette: America is further eroding the trust of its international partners, NATO keeps crumbling, and a new geopolitical reality continues to take shape. Say that in the midst of this, China decides to invade Taiwan. The conflict escalates fast, drawing in the U.S. and reluctant allies. The ensuing war is a major one. The Pentagon, already drastically short on munitions after its forays in Iran, wants to apply the latest AI capabilities to its wartime efforts, and Hegseth demands that Anthropic allow the Pentagon unrestricted access to Claude, reigniting the dispute first set in motion earlier this year.
Because there is active conflict, Anthropic is more willing to engage with the government’s demands than they were previously, but the firm asserts that it requires continuous oversight into how the Pentagon is using Claude. The company fears that in an effort to crack down on espionage, the Defense Department might create monitoring capabilities that supersede even the Chinese Communist Party’s, sliding America into an autocratic AI regime. Lest this sound speculative, it’s merely a restatement of Anthropic’s own position: Amodei has warned of a near future where “a powerful AI” scans “billions of conversations from millions of people” to “gauge public sentiment, detect pockets of disloyalty forming, and stamp them out before they grow.”
The spat from earlier this year looks mild by comparison. Amodei remains stubbornly principled despite repeated requests from the Defense Department made under emergency laws. Hegseth responds by sending his troops to descend upon the company’s headquarters in San Francisco. Amodei is forcibly removed and replaced with a deferential Army general. The situation is exceedingly unlikely, but not without precedent: Soldiers once carried the chair of one of America’s largest retailers out from his Chicago office after he failed to comply with federal demands during World War II.
Throughout American history, efforts to take control of industry have been rare, and limited mostly to times of crisis: President Woodrow Wilson nationalized the railroads during World War I, and Fannie Mae and Freddie Mac were placed under conservatorship during the financial crisis. Today, there are all kinds of possible emergencies. If a global financial crash leads AI companies to insolvency, the administration might swoop in to provide life support, as it did for many banks and car companies during the Great Recession. On the flip side, should AI models displace large swaths of the labor market, such that a handful of companies run most of the economy, “then some kind of nationalization becomes potentially imperative,” Samuel Hammond, the acting director of AI policy and chief economist at the Foundation for American Innovation, told us—to distribute wealth and simply ensure the proper functioning of society. Both Anthropic and OpenAI have already suggested possible versions of such redistributive measures.
Advances in AI could be their own kind of disrupter: Imagine a Sputnik 2.0 moment where the White House decides that American companies need to consolidate resources if the U.S. wants to win the AI race against China. By exerting more control, America becomes more like China in the very race to beat it.
The thing about nationalization, though, is that it need not be all or nothing. Nationalization “has layers,” Hammond said. “Like an onion.” Perhaps the most likely fate for American AI companies is a future of soft nationalization—a world in which the government doesn’t fully control AI labs and their models, but instead enacts an escalating series of policies and established close partnerships with private companies to shape the technology.
By some measures, soft nationalization has already begun. The Trump administration has already taken a 10 percent stake in Intel, a major semiconductor manufacturer, providing the White House with (some) direct financial leverage over the company. OpenAI has appointed the retired general and former NSA director Paul Nakasone to its board. Meanwhile, the Army recently established a new detachment for senior tech leaders, and its first four recruits included executives from Meta, Palantir, and OpenAI.
The top AI companies are coordinating with government officials as their products’ military and intelligence implications advance. OpenAI, which scooped up a contract with the Pentagon after Anthropic’s fell apart, has said it will deploy its own engineers to work alongside the military. The firm has also been briefing governments—at the state, federal, and international levels—on the capabilities of a new OpenAI cybersecurity model. Google is reportedly negotiating its own Pentagon contract to allow Gemini to be used in classified settings. And even Anthropic is coming back around. The company is fighting the Pentagon in court over a “supply-chain risk” designation that Hegseth slapped on them amid their dispute. But after Anthropic announced its Mythos model, a group of tech executives including Amodei spoke with Vice President Vance and others to discuss the risks, and Amodei took a trip to the White House. Last week, President Trump said a possible Pentagon deal with Anthropic might still be on the table.
The White House, OpenAI, and Anthropic all paid lip service to the value of cooperation when we reached out to them. The Trump administration is “working with frontier AI labs to discuss opportunities for collaboration,” a White House official told us. A spokesperson for OpenAI said: “As AI systems become more capable, it is only going to become more important for industry to work with governments.” And an Anthropic spokesperson told us that Amodei’s recent visit to the White House was “productive” and that the firm believes that governments must play a central role in addressing the technology’s national-security implications. (Google DeepMind and the Pentagon did not return repeated requests for comment.)
This campfire ethos could easily fall apart. And clearly, tensions exist. But so long as it’s in both the AI firms’ and Trump’s interests to please each other, we may see the leading AI companies partnering even more closely with the U.S. military to accelerate the development of defense applications, analogous to what contractors including Palantir, Boeing, and Lockheed Martin have done for years. As a protective measure, the White House might ask AI companies to increase their security practices to prevent espionage and exfiltration of the most capable versions of the technology (consider that a handful of unauthorized users have reportedly gained access to Mythos). The government could even designate certain research as classified and subject technologies to export controls, and federal employees could embed inside the companies to oversee various safety measures and run their own, independent evaluations. Every nuclear power plant in America has at least two on-site government inspectors who check daily to confirm compliance with federal safety requirements. So why not AI companies too?
If such partnerships persist, one could imagine private companies resisting certain government demands. But even without new legislation, the White House can easily exert greater authority over industry. “There’s quite a lot of power that the federal government can wield,” Paul Scharre, an executive at the Center for a New American Security who previously did policy work at the Department of Defense, told us. “And even more so if you have an administration that’s willing to stretch the bounds of executive power.” Anthropic’s supply-chain-risk designation—a label that effectively bars the military from doing business with the company, and that is typically reserved for companies with ties to foreign adversaries—was a clear example of the government flexing its muscles. So was the Biden administration’s decision to block Nvidia from selling its most advanced AI chips to China in 2022. (The Trump administration has since relaxed restrictions, claiming that selling to China was the better strategy for winning the AI race.)
One of the most salient tools available remains the Defense Production Act, the law that Hegseth threatened Anthropic with before pursuing the supply-chain-risk designation. The act has been used over the decades to support the manufacture of military equipment such as bombers and tanks, though in recent years, it has been used more expansively. Both the first Trump and the Biden administrations used it to accelerate pandemic safety measures, and Biden relied on the law in a since-repealed executive order to compel AI companies to share certain information about model training and evaluations with the government. Last week, Trump invoked the act to fund new energy projects. Actually pursuing the DPA as a general tool for controlling AI companies would raise a host of thorny legal issues, but that hasn’t exactly stopped the Trump administration in the past.
Such reins on an industry that has billed itself as capable of extinguishing humankind are, theoretically, in everyone’s interest. It would seem to clearly benefit the American people to have democratically elected institutions—rather than corporate executives—overseeing a set of technologies with huge implications for the nation’s security and well-being. It’s also historically anomalous for a private industry to dictate the deployment of such a powerful, general-purpose technology. With the announcement of Mythos, Anthropic has been effectively functioning as a geopolitical actor, briefing ally governments on the model’s capabilities. The European Commission, for instance, has met with Anthropic thrice since Mythos was announced—although as of Wednesday, the company had not yet given European Union officials access.
The government should play a role in dictating the terms of how AI transforms the world. But the ongoing fracturing of American politics, and especially the capricious and authoritarian-leaning tendencies of the current administration, complicates everything. Entrusting the future of generative AI entirely to Altman and Amodei or Trump and Hegseth seem like two very different and similarly disastrous outcomes—a “Scylla and Charybdis” dynamic, as Bullock put it, between the tremendous concentration of power in government or in a small cadre of companies.
The impossible truth is that no private company should be trusted to unilaterally steer the future of AI development, but Americans should also have serious questions about whether government control is in their best interest—not least of all under an erratic and norm-shattering Trump administration. The Manhattan Project coordinated the efforts of scientists, private companies, and America’s leaders. What if instead, Boeing and DuPont had been racing against each other to develop the atomic bomb while Hegseth and Trump led the military? We are diving headfirst into the 21st-century equivalent of such a situation. Our political dysfunction is about to ram into Silicon Valley’s immeasurable power.
Login
