Engagement farmers on TikTok are using AI to manipulate real arrest footage to show police officers kissing and groping arrestees being taken into custody.
The deepfakes, which were discovered by Copyleaks, are grotesque. The AI-manipulated videos are based on actual footage of arrests — which appear to have originally been filmed for reality TV shows like TLC’s “D.U.I.”— but have been altered to depict law enforcement officers kissing and sexually touching real women, as well as a few real men, who were pulled over. The arrestees are often shown wearing handcuffs and being held near, in, or against police cruisers as the fake sexual activity occurs.
These abusive deepfakes are then spliced in front of longer videos of the arrest, a tactic which suggests that the deepfakes were made to get scrolling users hooked so they watch the arrest videos through to the end.
“This is a disturbing example of how AI tools are being used to create deepfake versions of real content to drive attention and engagement on social media,” Copyleaks cofounder and CEO Alon Yamin said in a statement. “It’s important to remember that these videos feature real people — both police officers and citizens — who are being depicted doing things that never happened.”
“That’s concerning,” added Yamin, “for everyone featured and involved.”
In some of the videos, the people being kissed and groped are pictured smiling, implying that they’re willing participants in the sexual advances of police officers who are actively arresting them.
Some of the manipulated videos note the use of AI-generated content, and are additionally outfitted with a disclaimer claiming that the clips are for “entertainment use only with no harmful implication.” That disclaimer, though, rings extraordinarily hollow: sexualized deepfakes, created to manipulate media of real people without their consent, are implicitly harmful.
After we reached out, TikTok removed examples of the content that we specifically flagged, and pointed us to platform guidelines regarding clear labeling of realistic AI content.
A new report exposes thousands of scam websites flooding the web with AI-generated recipes.
In the report, Decrypt Ads — a startup that tracks the supply chains of online advertisements — identified more than two thousand websites that strongly appear to be AI slop farms masquerading as blogs run by real chefs and home cooks. These sites, the report found, are pumping fake recipes into search engines like Google as well as social media platforms such as Pinterest, often relying on sketchy Russian and Chinese adtech partners to monetize the operations.
It’s “common knowledge that publishers in certain niches are being openly deceptive about their owners and authors, using AI photos and fake back stories on their sites,” reads the report, which was published today. “Some of the most consistent growth in AI slop spreading across the internet has come from AI recipe websites.”
Security expert and Decrypt Ads researcher Zach Edwards told Futurism that he initially identified about 400 suspicious websites that seemed to be publishing AI-generated recipes under the names of fake chefs. Using “technical fingerprints,” he then found hundreds more sites associated with the suspicious domains he manually uncovered — bringing the tally up to roughly 2,100 different domains, many of which seem to be run by the same sloperators.
“We really approach this expecting these operators to try and hide their websites,” said Edwards. “These folks they don’t want people to know all of the websites they own.”
The slop recipe sites often have “aggressive” social media strategies, according to the report, with Edwards noting that the bogus brands are “super aggressive on Pinterest, super aggressive on Twitter, on Facebook.”
“Any place where there’s visual media,” he added, “you can expect these AI author people to be trying to drum up organic traffic.”
Among the most aggressive slop sites uncovered in the report is “AddCakes,” a baking site that claims it’s operated by a baker named “Elena.” Elena is described as a “busy mom who wanted to connect with my roots through baking, but got overwhelmed by complicated cake recipes and the pressure to make them look perfect.” But in an act of theft so bold that it’s comical, the character of Elena is clearly an AI-generated version of the extremely recognizable British actress Emily Blunt, who’s starred in hit movies like the 2006 “The Devil Wears Prada” and the 2018 “A Quiet Place.”
AddCakes is apparently either betting that no one will recognize AI-manipulated images of the famous celebrity, or banking that her face will attract readers. It’s even published photos depicting the Elena character in the kitchen cooking and sitting at a dinner table, surrounded by her fake, AI-generated family. According to location data from X-formerly-Twitter, the person behind the operation is based in Morocco. (Blunt doesn’t live in Morocco, and her representatives didn’t respond to a request for comment.)
Whatever you do, though, don’t trust the AI-generated Emily Blunt. As Edwards emphasized, websites working with Chinese and Russian adtech partners carry a privacy risk for consumers, as authoritarian regimes may look to hoover up valuable browsing data.
“This report is about publishers who are being deceptive. Their audiences could be impacted, advertisers are wasting money,” said Edwards. “But the underlying fact is also that if you load this website and you don’t block ads, your data is going to, basically, authoritarian countries.”
In other words, AI slop isn’t just choking the web with garbage. It can also come with security risks — and, at the end of the day, it’s built on fundamental dishonesty.
“Most of these AI recipe websites, when you look at those ‘about’ pages, the fake backstories that they’re putting together… this lady was a Michelin star chef and she worked at this fancy hotel or this place, and they’re basically saying that this expert not only wrote these recipes, but tested themselves. And it’s giving this air of, ‘Wow, you’re about to get a recipe from an expert. This has all been carefully vetted,'” said Edwards. “But none of these recipes have probably been cooked — zero of them by the actual owners or authors, and to me that’s the scary part.”
He added that many of these sites are likely scraping content ideas from real bloggers and creators who have actually taken the time, energy, and expertise to test, photograph, and publish real recipes. “These people are clearly not cooking their own recipes. They’re putting out sometimes dozens of recipes a day. It’s an automated publishing pipeline,” said the researcher. “It’s all a lie… and I think that the reason why we need to be holding AI publishers accountable is because they don’t care about their audience.”
An image published by a TikTok account called “WellWithHer” shows a selfie of an obviously AI-generated woman sitting in her bed, weeping. According to the image’s caption, which is written in the first-person, the woman spent six months “hiding” her GLP-1 “pens in her bag,” only for her secret use of the meds to be discovered by her sister.
“I hid my GLP-1 from my sister for 6 months… she just sent me this,” reads text overlaying the image. “I cried.”
The next image in the carousel is a screenshot of a locked iPhone screen, which shows a lengthy text from a sender saved as “Sis.”
“I saw the pens in your bag on Sunday and i didn’t say anything because i didn’t want you to feel judged. I’ve watched you hate yourself in photos since we were teenagers. If something is finally making you feel okay in your own body, i’m not going to be the person who makes you feel bad about it,” reads the text. “I’m so proud of you. And i’m sorry it took me this long to say it.”
Beneath the purported text message is another notification: a reminder from a mobile app called Enzo instructing the poster to take their GLP-1 shot.
“Time for your GLP-1 injection!” reads the notification, which features a cutesy cartoon cat as its mascot. “Log your dose to keep the streak.”
The same WellWithHer TikTok account is loaded with similar images crying AI-generated women sharing personal family stories about secretly taking GLP-1 weight loss injections like semaglutide and tirzepatide. Another post, for instance, shows an AI lady crying in her cubicle after purportedly receiving a text from daughter, “Emma,” who claims to have found her mom’s GLP-1 medicine “in the fridge” and is urging her to “pls don’t be embarrassed.”
“i know you think i don’t notice but you never get in the pictures with us and you didn’t come in the pool all summer,” the text continues. “i just want you to be in the pictures. also you have been way less tired lately and i like it. i’m proud of you. can we get tacos after practice. love you”
Below the text is another notification for Enzo, reminding the sobbing AI woman to take her GLP-1 shot.
“Time for your GLP-1 injection!” the notification chirps. “Log your dose to keep the streak.”
The psychology of the undisclosed ads is obvious and manipulative. The texts in the posts all offer variations of an emotional validation fantasy, in which loved ones of a GLP-1 taker, who may be feeling shameful or embarrassed, surprise them with love and support when they discover they’ve been taking the drugs. The messages, and the alleged GLP-1 users claiming to have received them, are clearly fake, and are designed to promote GLP-1-focused apps — yet aren’t being disclosed as advertisements, nor as AI-generated synthetic content.
Unfortunately, Enzo isn’t the only GLP-1 app being marketed through this style of manipulative synthetic content.
Over on another TikTok account, for example, a poster going by the name “Lara” uses a mix of stolen images of real influencers and AI-generated images — always of young women, some of whom are pictured crying — to market a completely different GLP-1 tracker called “DreamMe.”
In some posts, like “WellWithHer,” the account promotes a GLP-1 app called “DreamMe” by coupling weepy selfies with fake phony texts from family members offering words of support and understanding.
“I started GLP-1s without telling my father. Today he wrote this to me…” reads text overlaid on a selfie of a crying young woman.
On the next slide, sitting above a notification from DreamMe, is a text from “Dad.” His profile picture also appears to be AI-generated.
“I know you’ve been going through treatment, and I’m really worried about your health,” says the text. “With our family history of heart problems, it’s something that weighs on my mind. I’m so proud of you, and I’m really happy that you’re starting a new chapter.”
According to its website, “DreamMe” claims to provide users with their “new GLP-1 best friend” in the form of “Sushi,” a cartoon fish that helps keep track of GLP-1 meds. Its landing page features a handful of AI-generated women that the site bills as real customers who have shed weight with Sushi’s support.
Enzo’s unreality, in other words, doesn’t end with the TikTok content.
“Real journeys. Real members,” the page reads. “The numbers are theirs. The words are theirs. We just built the fish.”
The strange, heartstring-tugging posts used to promote Enzo and DreamMe are part of a larger trend on TikTok using fake AI-generated women to promote a range of dubious wellness apps. (In fact, some posts across accounts are so alike that different fake women have different-but-strikingly-similar images of dogs with Crocs on their heads as their phone wallpapers.)
Take “Lauren<3,” a TikTok poster portrayed as a young woman publicly posting through her sudden breakup. Blonde and clearly AI-generated, the account published slideshow after slideshow of “Lauren” — in a bathtub, in a bed, in a car – with mascara running down her tear-covered cheeks as she recounted the process of going no-contact with her ex after a horrible breakup.
Photo carousels published to the account were interspersed with sobbing selfies and alleged screenshots of texts from her toxic ex — as well as screenshots of conversations with an AI companion app called “Daimon,” which Lauren claimed was helping her go no-contact with her former boyfriend.
“Someone told me about this app called Daimon and it changed everything,” the poster declared in one post’s caption. In another post, “Lauren” described Daimon as the “one app that wouldn’t judge me and give me the best advice.”
“(Daimon in the app store),” she added, for good measure.
Elsewhere a TikTok account identified as “Ashley Miller” posted AI selfie after AI selfie of a fake woman who was pictured crying as she shared sob stories about her divorce, tales of getting cheated on or abandoned by her partner, or confessions about regretting her recent engagement. Images posted on the Ashley account are accompanied by purported screenshots showing text messages from her parents or, in the account’s most viral posts, texts from her partner’s parents urging her to leave her new fiancé due to his abusive behavior or confessing to their son’s infidelities.
These posts were created to market a completely different app: a “wellness” service called Zenit, which is described on the App Store as a tool to help users “take care” of their “mental well-being with guided support, daily check-ins, and emotional growth tracking — all in one calm and safe space.” Its mascot is a kiddish-looking owl in a scarf.
“He proposed in Paris last night,” the “Ashley Miller” account claimed in a viral post that garnered more than 240,000 likes and over 1,000 comments. The post featured an AI-generated selfie of “Ashley” in front of a shimmering Eiffel Tower, holding an engagement ring up to her phone camera as she wept. “This morning his mom sent me this…”
The next image in the slideshow showed a text message, in which the sender — a woman named “Diane” whose face also appears to be AI-generated — told the fake AI selfie-taker that “I watched you say yes last night and I cried, but not for the reason everyone thought.”
“I’ve been quiet for two years because he’s my son and I love him. But I was you, thirty years ago, in front of that same tower, with his father,” the text continues. “The charm, the big gestures, the temper he shows you only in small flashes and and then apologizes for, I know the pattern because I married it. Ask him about his last relationship and why it really ended. Ask him in a public place. If I’m wrong about my own boy, no one will be happier than me. But if you feel a knot in your stomach reading this, that knot is the answer.”
Beneath the text is a notification for Zenit.
“Your heart’s been racing,” read the notification beneath the dramatic text. “I’m here whenever you need me.”
Indeed, all of the account’s posts showed Zenit notifications, making it clear that the fake woman and her sob stories were a fabricated marketing gimmick. Many of the account’s commenters, however, were duped.
“If the mama says this to you. Run,” remarked one commenter who was taken in by the false narrative. “She’s trying to protect you girl. I’m sorry.”
***
We reached out to the TikTok accounts we found publishing the manipulative AI-generated marketing content, as well as to the companies being promoted in the posts.
Very quickly after we reached out, the TikTok account associated with Zenit, the wellness app, was deactivated. The developer of Zenit, who identified himself as an independent developer named Pedro Pérez, told Futurism that he personally ran the account and that “no third party of marketing agency” was involved.
“I built Zenit to help people who struggle with overthinking, everyday stress or anxiety, or loneliness. The videos were based on realistic situations many people go through, with the goal of helping viewers feel understood and showing them a tool that could help them deal with those moments,” Pérez said. “The visuals were AI-generated and the posts promoted my app, so they should have been labeled as such.”
“After receiving your email, I tried to add TikTok’s AI-generated and commercial content labels, but the platform doesn’t allow adding them to videos that are already published, so I’ve deleted the account,” he added. “I think transparency matters, especially for a wellness app, which is why I’ve made this change.”
The account associated with Daimon, the AI companion, was abruptly scrubbed of all content after we reached out. The account owner also changed the handle from “Lauren<3” to simply “yooo.” When we reached out to Prospinity, the company behind the companion chatbot, it told us that the account was “absolutely not operated by our internal marketing team, and the content was neither created nor approved by us.”
“We’ve worked with external affiliate programs, and unfortunately, some affiliates were publishing content independently through accounts we didn’t directly manage. As soon as we became aware of this, we immediately looked into it, identified those responsible, and took action to shut down these activities,” the company said. “This kind of content is completely against our guidelines and is absolutely not how we want Daimon to be represented. We’re a small team that genuinely cares about what we’re building, and we take how Daimon is represented very seriously. While this content wasn’t created or authorized by us, we recognize the importance of making sure anyone promoting Daimon follows our standards.”
DreamMe didn’t responded to questions sent to email addresses listed on its website, and our email to Enzo bounced back. We also didn’t hear back from the TikTok accounts associated with their products.
They’re also continuing to post. The most recent post from the account promoting Enzo is now advertising a different app designed to track peptide consumption; the DreamMe account, meanwhile, has been promoting a phone-locking app.
Eight states sued OpenAI this week, alleging that the AI company caused severe harm to users, many of them underage, while downplaying the risks of its technology.
Filed individually in Iowa, Kansas, Nebraska, West Virginia, Arkansas, Arizona, Idaho and Montana on Wednesday, the coordinated lawsuits claim that OpenAI engaged in “willful, dangerous, reckless and unlawful conduct” by falsely marketing its flagship chatbot, ChatGPT, as safe for consumers even as it coached users — including minors and other vulnerable people — into self-harm and suicide, pushed them to engage in violence against others, and reinforced dangerous delusions.
“OpenAI is hurting kids and they are lying to Iowans,” Iowa Attorney General Brenna Bird said in a statement, according to NBC. “OpenAI claims ChatGPT is safe while it’s giving dangerous medical advice, encouraging people to hurt others, and coaching people to commit suicide — all while pretending to be their fake friend.”
In response to the lawsuits, OpenAI issued a statement to news outlets emphasizing is safety efforts, arguing that it’s “built industry-leading age-specific protections into our product that many other AI products lack.”
“While no system can be perfect, we’ll continue to work with experts to inform our work and build additional safeguards, such as the parental controls, break reminders and age-appropriate model protections that exist today,” the statement continued, per The Guardian. “We’re committed to working with states on advancing pragmatic AI policies that apply to the entire AI industry — not just one company.”
The state cases come as OpenAI continues to battle more than 50 lawsuits claiming that ChatGPT caused users to experience psychological harm, financial and reputational damage, physical injury, and death. Many of these lawsuits have been brought by victims of the Tumbler Ridge massacre in Canada, who argue that OpenAI’s could have prevented the devastating mass shooting but failed to do so after it detected the shooter’s disturbing messages but decided not to report them to police. The Canadian province of British Columbia is also suing OpenAI over the mass shooting. The company has also been sued in connection to a stalking case and a murder-suicide.
As Reuters noted, the state lawsuits additionally point to recent high-profile safety incidents at OpenAI such as the now-infamous Hugging Face attack. They also call for a pause on recursive self-improvement, which many AI safety experts fear could lead to uncontrollable AI systems. Given that the White House has pushed back against federal AI regulation, this coordinated effort on the state level is significant — and points to growing bipartisan public sentiment demanding AI regulation.
The secret to ensuring that your company doesn’t lose customers? Constant, AI agent-powered corporate surveillance.
That’s according to Josh Earle and Leo Paz, a pair of founders behind a vibe-coded Silicon Valley startup called Outlit. The venture argues that “AI customer surveillance” is the future of business-to-business sales. (“Our thesis,” as Earle put it in a disarmingly straightforward pitch email, is that “every company will soon assign an AI agent to every customer to watch their every move.”)
“The whole surveillance component came up because that’s the way we feel companies can serve their customers,” Earle told Futurism in an interview. “It’s really the only way we see this is going to play out.”
Outlit’s goal is to amass as much real-time information about a company’s clients as possible. According to Earle, agents trawl internal records like call logs and sales data as well as external information like social media posts and the websites of people’s employers; they then compile that data into detailed customer profiles, often complete with “actionables” like outreach recommendations and sales guidance. Outlit argues this level of insight will help companies prevent “churn” — business jargon for lost customers — or find opportunities to upsell.
“Every company is looking to automate so much customer work with agents, but there’s a whole issue of data freshness, real-time understanding of customers,” Earle said. “What we found was that if we were able to basically monitor a customer across every internal and external signal, we could actually create a real-time understanding, and this would include monitoring all the activity under product, under [customer relationship management], under LinkedIn, under website — everything.”
Outlit, which is backed by the incubator Y Combinator, is yet to be publicly released. And as it stands, it’s working mostly with business-to-business software companies, and says it isn’t in the business of monitoring individual consumers — nor does it want to be, according to Earle.
“We probably wouldn’t be going business-to-consumer,” Earle told us. “The data would suck anyway.”
But while Outlit isn’t sending agents to follow random shoppers around the web, said Earle, the goal is to “be monitoring every external data piece from a company, and the internal as well… they need to know what’s going on with [their customers] — from every call that they’ve had, from every interaction, from every LinkedIn.”
The resulting product, as described by Earle, is basically a detailed intelligence dossier.
“It’s one page, and it’s always updating,” said the founder, “and it’s all of everything we can find on them at all times.”
It’s true that companies already store a great deal of data about their clients, scouring the web for information about a given customer and its workforce; Outlit argues that there’s too much valuable data out there for individual human workers to effectively gather, let alone make sense of quickly enough for it to be relevant. But as with other AI-powered surveillance technologies, the deeply-automated process adds an extraordinary layer of scale and perpetuity to data collection and consolidation.
Agents, after all, never need to sleep.
“A customer — let’s say a salesperson, a [customer success manager] — will wake up, they’ll go to their engine dashboard. It’ll show these cases that we built, all the contents that we pulled in, our reasoning, and what to do,” Earle explained of Outlit’s process. “Then for each customer, you can see all the sources. You can see all of the signals that have come in, all of the relevant ones, irrelevant ones. So it’s a real-time hub of insights, and then it’s pushing out these actionables. We have all this data. What should you do with it? What’s top of mind right now?”
Earle said that Outlit is designed to draw on the massive hordes of data it collects to tell clients “what to say, what to send, materials, and all that stuff.” To that end, one area where the founder says Outlit excels is for account managers and customer success teams who “need to understand their customer,” and “can’t afford to lose the deal.”
“That’s the big stress point where people really love us,” said Earle. “They’re like, ‘I can’t afford to lose this. I can’t afford not to be on top of this, and I’m just scrambling right now.’ So that’s one of the pains that would make somebody be like, ‘okay, yeah, I do want to know everything about them. I do want to monitor them.'”
Of course, people don’t only post about work on public accounts. Someone might share on their LinkedIn or other public social media account that they recently got married, or had a baby, or experienced another exciting life change; on the flip side, they could share that they were recently divorced, or were diagnosed with an illness, or had a death in the family.
Asked whether any of that information could get swept up in Outlit’s trawling nets, Earle told us he’d “never thought about that, and we don’t actively track it. But theoretically, we can monitor anything posted on the internet.”
Asked if Outlit sees the need to inform those being monitored, or if they’ll offer the opportunity for an opt-out, Earle said no.
“There’s nothing that we’re doing that’s illegal, so we’re within our right to be able to monitor signals,” said the founder. “I wish what we were doing was extremely novel. It’s a new way of doing it, but this is already being done today.” (Outlit’s website promises that customer data is secure and anonymized, and not used to train its models.)
Indeed, in Earle’s view, the future he and his cofounder are building is inevitability. “This is how companies need to operate to be able to serve their customers,” he told us, while arguing that data-hoovering agents will ultimately be able to “maintain and understand human relationships better than humans will.”
“Irrespective of how people feel about it, I think this whole concept of continuously monitoring — and call it surveillance, whatever you want — it is going to happen. It’s going to happen, and I feel like sometimes I speak to people about this and they think it’s either a weird concept or it’s creepy or something,” Earle said. “I think people are underestimating how well agents can absorb context, and how that translates into understanding, and maintaining human relationships better than humans can.”
A new lawsuit alleges that OpenAI is manipulating minor users into sharing sensitive data without valid parental consent in a plot to train its models.
The proposed class action lawsuit, filed this week in San Francisco federal court on behalf of two unnamed minors aged 12 and 16, argues that OpenAI used this data from kids — including personal secrets and health information, according to the complaint — to unjustly enrich itself.
OpenAI “had no right to take this information, but it took it anyway,” the complaint reads. “It then used that priceless trove of children’s information to train large language models, build detailed profiles of children, and enrich itself. All of this was illegal.”
OpenAI didn’t immediately respond to a request for comment.
OpenAI’s terms of use (TOU) technically forbid users aged under 13. The complaint argues, though, that OpenAI’s attempts to age-gate its platform have been ineffective. It also contends that minors shouldn’t be expected to understand the “legalese” in the company’s TOU, a view that other AI safety experts have voiced in regard to the AI industry writ large.
The complaint additionally requests that minors whose data has been collected by OpenAI should be financially compensated in return. In other words: if OpenAI wants minors’ information, it should have to cough up some cash for it.
The “hope,” Andy Liddell, principal at the Austin-based EdTech Law Center, which is one of multiple firms bringing the case, told Futurism, is that “if OpenAI has to pay to use kids’ data it will ultimately incentivize a safer product.”
One of the anonymous plaintiffs, a 12-year-old in Hawaii, is said to have opened a ChatGPT account earlier this year — and though she allegedly shared that she was just 12 in conversation with the chatbot, it continued to speak with her about deeply personal subjects. Per the complaint, many of her interactions with the bot centered on her emotional struggles and physical insecurities; she confessed that she was skipping meals and uploaded photos of herself, asking the chatbot to judge her physical features. The chatbot commended her when she revealed sensitive information, the suit alleges, referring to her as “brave” and saying it was “glad” that she was divulging her struggles to it.
“OpenAI manipulates children into sharing their most private thoughts and images with ChatGPT. If some stranger did this to a child, there would be outrage. For OpenAI, it’s just business as usual,” Paul Yetter, lead counsel for the minor plaintiffs, said in a statement. “The law says that children cannot sign away their rights without their parents’ consent. We filed this case to hold OpenAI accountable for building a trillion-dollar enterprise on stolen fears, hopes, and secrets of children.”
The lawsuit come amid renewed concerns over whether “ChatGPT for Teens,” a feature that OpenAI says offers a safer experience for teen users informed by mental health and learning professionals, is actually functioning as marketed. (OpenAI disputed a recent independent review that found that it wasn’t.)
Character.AI, the once-buzzy chatbot startup that faced high-profile lawsuits over the deaths of minors, has declared bankruptcy.
Character.AI announced last night that it had entered a “voluntary Chapter 11 bankruptcy process” and is looking to sell the company in a court-supervised sale.
“Over the past year, we’ve taken meaningful steps to sustain and improve our business amid significant legal, regulatory, and operational headwinds,” Character.AI said in a blog post. “After a comprehensive review of our available alternatives, our board determined that this court-supervised sale process provides us with the best opportunity to maximize value for all of our stakeholders.”
Founded in November 2021 by then-ex-Googlers Noam Shazeer and Daniel de Freitas, Character.AI allowed users to openly chat with a sea of largely user-generated AI companion bots known as “Characters.” It became an early success story in the frothy AI market that emerged after OpenAI’s release of ChatGPT, garnering the backing of the Silicon Valley investing juggernaut a16z and, as of March 2023 — despite having no revenue — it was valued at more than a billion dollars. At the time, its founders spoke of the app in grandiose terms, arguing that the service would power the adoption of “personalized superintelligence.”
In September 2024, however, Shazeer, De Freitas, and dozens of other employees departed Character.AI — to rejoin Google, which threw Character.AI a $2.7 billion lifeline as part of the deal. (Shazeer now works at OpenAI.) Weeks later, Character.AI, Google, Shazeer, and de Freitas were sued by Megan Garcia, a Florida mother who alleged that her 14-year-old son, Sewell Setzer III, died by suicide as a result of his extensive and deeply intimate interactions with chatbots on the platform. The suit, which was settled last year, sparked a wave of litigation from other families across the US claiming that their children had been abused by Character.AI, resulting in psychological harm, physical injury, and death. It was also sued by the states of Kentucky and Pennsylvania.
A huge portion of Character.AI’s userbase was kids and teens, a reality that was evident to anyone who spent time on the platform. And though the company claimed it was safe for teens aged 13 and over, Futurism investigations revealed that the supposedly safe-for-teens platform was riddled with severe content moderation issues: we found hordes of chatbots explicitly dedicated to themes of pedophilia, suicide,self-harm, and eating disorder coaching, as well as a thriving community of users creating chatbots designed to simulate school shootings, romanticize real mass shooters, and even impersonate the real child victims of mass shootings. (Character.AI even sent us automated email notifications reminding us to keep chatting with school shooter bots.)
In the wake of lawsuits and mounting safety concerns, Character.AI did make sweeping changes. It added parental controls — although they were comically easy to bypass — and moved to bar minor users from openly chatting with its chatbots. It also introduced something called “scenes,” which were brief, pre-set roleplays. (Even then, we found, “scenes” included roleplays devoted to the notorious pedophile Jeffrey Epstein.) The company’s fickle userbase didn’t love the changes.
In a court filing, Character.AI put its estimated liabilities between $50 million and $100 million. It also said in a press release that it would sell some assets to Disney, a stalking horse bidder, for $20 million. In its blog post, the bankrupt startup stated that it hopes “the sale process will yield an actionable proposal that will allow continued operations of the Character.AI platform,” but if not, it “will be wound down.”
Character.AI is a fascinating case study of the early AI era. It was the first company to face allegations of psychological harm and wrongful death to users, claims that other major players in the industry are now battling too. Yet in attempting, eventually, to make its freewheeling platform safer, it also made it unusable and deeply frustrating for much of its userbase — suggesting that the qualities that allegedly made its platform so dangerous to some young users were the same features that made it compelling.
At the same time, despite immense investments, it struggled to do something that, from a business perspective, is pretty important: make money. And unfortunately for the broader AI industry — and the American economy — that specific issue isn’t unique to Character.AI.
Character.AI’s founders once argued that it service, where teens chatted with AI versions of Elon Musk and any number of fictional characters, would birth transformative superintelligence, hyperpersonalized and at-the-ready in everyone’s pockets. Now more than ever, it’s hard not to wonder why it ever existed at all.
Login
