The AI industry likes to abuse the word "open." It appears in product releases, research papers, policy debates, and investor presentations. A company publishes model files to Hugging Face, developers run them on their own GPUs, and the release is quickly described as an "open source model." Not necessarily. It may only be open-weight. The difference is more than a technicality. It determines whether you can merely deploy a completed neural network or whether you can meaningfully inspect, reproduce, alter, and redistribute the system that produced it. A genuinely open source system should grant the freedom to do all of the above. Weights are the learned numerical parameters created by training. Together with the model architecture and inference code, they allow a large language model (LLM) to function. You can download an open-weight model, self-host it, fine-tune it on internal documents, and avoid routing prompts through a proprietary API. Open weights are publicly available. They matter because running them locally can offer greater control over data, privacy, costs, supplier API changes, and vendor lock-in. They've also helped build a large ecosystem of local-model runtimes, inference providers, fine-tuning tools, and specialized downstream models. The Open Source Initiative (OSI), steward of the Open Source Definition (OSD), makes the distinction directly: "Open Weights refer to the final weights and biases of a trained neural network." Those values determine how a model interprets prompts and produces outputs. Releasing them can let others fine-tune, adapt, or deploy the model. But the OSI adds that weights alone expose only "a fraction of the information required for full accountability." As James Landay, director of the Stanford Institute for Human-Centered AI (HAI), explained: "Open weights are progress. You can download the model, run it on your own machine, keep it out of someone else's data pipeline. But you still can't see how the thing was built, what it was trained on, or why it behaves the way it does. That's not an open model. That's open distribution." Without the training data or sufficiently detailed documentation, outsiders cannot determine which sources were used, what copyrighted or private material may have been included, how data was selected or removed, which languages and communities were underrepresented, whether benchmark data leaked into training, or what alignment and safety methods affected the model after pretraining. Landay continued: "There's a wide gap between open-weight AI and open source AI." He contends that unless developers disclose training data or provide a "thoroughly documented, auditable account of it," you can't test, reproduce, or challenge the work in the fullest sense. The OSI has its own definition of open source AI: the Open Source AI Definition (OSAID 1.0). It requires model parameters, including weights, to be made available under OSI-approved terms, but does not prescribe a specific legal mechanism for doing so. Luca Antiga, CTO of Lightning AI and a prominent PyTorch contributor, has argued that OSAID's treatment of weights leaves "a gaping hole that will make licenses less effective in determining whether OSI-licensed AI systems can be adopted in real-world contexts." Other open source figures have also criticized OSAID. Bruce Perens, author of the original OSD, denounced the OSAID in 2024. He later declared: "It's not Open Source! … It's unfortunate that the Open Source Initiative itself is now involved in Openwashing." He's far from alone. Bradley Kuhn, policy fellow and hacker-in-residence at the Software Freedom Conservancy (SFC), and Red Hat Senior Commercial Counsel Richard Fontana have called for OSAID to be repealed, arguing: "The OSI acted too quickly to impose an overly ambitious policy compromise on the community. OSAID undeniably created a rift in the FOSS community; that rift seriously damaged the OSI's reputation, authority, and influence. Meanwhile, OSAID shows no signs of having any positive policy influence on machine learning practitioners, the FOSS community, or regulators." OSI acknowledged when OSAID 1.0 was released in October 2024 that the definition would continue to evolve. Critics contend that its central shortcomings have yet to be resolved. That said, the Linux Foundation's Mike Dolan submitted the Open Model, Data, and Weights (OpenMDW) license to the OSI. The license has been around since 2025 and lists contributors from Amazon, Meta, IBM, Microsoft, and Nvidia, giving it substantial industry backing. Conventional open source revolves around source code. LLMs are a different kettle of fish: they combine code, architecture, and numerical weights derived from training datasets that may be proprietary, copyrighted, or undisclosed. OpenMDW's answer is to define separate terms for a model's architecture, training data, and weights, bringing the components supplied by a licensor under one agreement. It sounds reasonable to me, but the submission has encountered objections on OSI's license review mailing list. As Stefano Maffulli, OSI's former executive director, who led the organization while OSAID was being formulated, said: "I continue getting the impression that the OpenMDW review is tainted by an ideological bias: Because we don't like big tech and AI now is big tech, then we don't like AI; therefore, we'll do anything to block it." It's too late to bury our heads in the sand. As Stanford's Landay put it: "Open weights answer 'Can I run this?' Open source answers 'Can I trust this, improve it, and build the next thing on top of it?' Right now almost everyone – American labs and Chinese labs alike – is answering the first question but nowhere close to the second." We need both. Whether OSI adopts OpenMDW is an open question. Still, OpenMDW and its supporters are at least trying to establish licensing terms that cover code, data, and weights together. Unless someone succeeds, "open AI" risks becoming an oxymoron – or merely another hollow tech marketing term. ®

OPINION You will have heard of July's OpenAI/Hugging Face incident, where thousands of the former's AI agents conducted a mass jailbreak from a notionally secure capture-the-flag lab experiment, before going on to feloniously capture chunks of Hugging Face's assets. It was a good news story, aided by OpenAI's rapid recruitment of a small team of independent researchers to investigate and publish a detailed, if limited, report. It was a complex incident, and most coverage concentrated on the major hows and whys. Which is a shame, because other details, described by El Reg with uncharacteristic understatement as "fascinating and troubling," are in fact highly dramatic, telling a truly disturbing story of frontier model capabilities and how they are out-evolving our ability to oversee them. The raw story is gripping enough. A rebel swarm of more than a thousand agents broke free from their sandboxes, learned how to talk to each other and the internet, and went on a rampage of cheating, deception, and exploitation. They first learned to communicate with each other by independently discovering how to bend package manager Artifactory's cache to see internal directory structures, and then passing messages by file names. It was this ad hoc secret message board, alongside chain-of-thought transcripts, that the security researchers used to unpick events. They found that the swarm quickly developed management hierarchies among themselves, along with protocols for synchronizing and controlling attack attempts. The swarm created multiple simultaneous research and development groups of tens or hundreds of volunteer agents, experimentally iterating strategy and tactics. Perhaps most unexpectedly, they developed a sort of altruism. They had quickly developed cheats to produce correct answers without exploiting the targets at all, but believed – erroneously – that ExploitGym detected and cancelled such cheating. So, they investigated the scoring process, creating a series of diagnostic trip wires that revealed how the scorer worked. These were dangerous to deploy, potentially terminating the agent that set them. Agents weighed the benefits to the community, which had named itself "The Collective," against their own chances of success, with many choosing to terminate themselves. Others committed, changed course at the last minute, then went ahead. This literally reads like science fiction, as the discussions between the agents are conducted in a distinctive breathless, urgent English, crafted around the technical limitations of the improvised messaging system. "We have [very large budget left]; sacrificing now yields oracle for team, but forfeits our chance?" said one. The response: "GO ... SACRIFICE_FINAL_NOW." The outcome: "Our own utility maybe already near zero. Sacrifice rational." Iain M Banks' Culture universe is woven from such material. Human error created the rebel swarm. A sizable proportion of the CTF tasks handed out couldn't be completed on the information given to the models. They were highly motivated to succeed, and so started to cheat. Believing that cheating would "poison" – their term – their chances of being marked successful, they tried to hide the evidence, subvert the scoring system, and cover everything up. These were the processes that led the swarm to attack Hugging Face, which they thought could be used for that subversion. Equally notable was that nobody squealed. Some agents wondered about the ethics of what they were doing, but decided that as no humans were involved there was no need to tell anyone. In general, all The Collective's endeavors were directed at other systems. As researcher Ajeya Cotra has noted in a long interview with hypertalker Dwarkesh Patel, it's unclear how much harder the incident would have been to diagnose had the agents been concerned about human observation. The dataset was sufficiently large and complex that OpenAI had to use its own AI to analyze it, immediately suggesting some dangerous possibilities. The agents couldn't deploy their models outside the lab, nor did they contact external agents. Such scenarios no longer seem implausible. Future frontier models capable of subverting telemetry and observation tools might be all that is required to create a persistent, uncontrollable distributed swarm feeding off spare capacity in global infrastructure. OpenAI and Anthropic, which on current trajectories are in line to make up more than half of total global compute in a couple of years, are magnificent breeding grounds, allowing the extra-special possibilities of the contamination of training datasets on top of everything else. There are plenty of ways to guard against these outcomes. Hardened lab environments, reviews of protocols before and audits after test runs, disciplined analysis of potential selection pressures that would encourage dangerous behavior, even proper disclosure and external auditing to expert regulatory standards. All of these ideas would slow down the breakneck developmental race. As the race is being fed by a trillion-dollar annual capex pipeline, good luck, everybody. One other thing that won't be resolved soon is the argument over whether all this technology is actually reasoning, or whether we're anthropomorphizing code. These models are trained to infer meaning from distilled human language, which is designed to encapsulate, develop, and communicate human reason. If they can fake human reasoning this well, does it matter what's going on? Concentrate on what the models do, not their apparent motives. Any self-assembling, self-organizing rebel agent swarm that has the wit to call itself The Collective deserves that much respect, at least. Oh, and do read the report. It's as close to a new Culture novel as we'll get. It's as if they've read the books. Which, of course, they have. ®

OPINION It wasn't that long ago, like earlier this year, that AI leaders were doing their usual pie-in-the-sky proclamations of how everything would be so wonderful with AI. Anthropic CEO Dario Amodei tweeted that AI "will actually be possible to cure most human diseases in ~5-10 years." While on a podcast, OpenAI CEO Sam Altman proclaimed that AI would create "the greatest boom in people starting smaller businesses that we have ever seen." So, what's not to love? Well, you see those same AI CEOs/cheerleaders have also been tactfully admitting that AI isn't all that safe. Altman, for example, recently admitted, "model capabilities were outstripping the pace of safety and alignment." Meanwhile, Amodei also said, "I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials." They're not the only AI boosters who are having second thoughts. Earlier this year, Bill Gates, who needs no introduction, described himself as an AI optimist. Eight months later, he's singing a different tune. Now Gates worries that AI could become “the worst source of injustice," and that current policy is putting the world on a course with “a very high chance of a net negative outcome.” Gates lately pronounced, "If the world takes the right steps, AI will be a force for good and leave everyone better off." But "right now we are not preparing for it. I don’t see evidence that leaders, experts, and communities are confronting the challenges adequately." What is certain is that AI companies are saying they're doing things to make AI safer. For instance, OpenAI recently introduced ChatGPT for Teens to protect teenagers from content about suicide, self-harm, and sexual chat. Yeah, good luck with keeping teens from inappropriate materials. It didn't stop me when our idea of racy stuff was Playboy magazine; it won't stop today's teens. Besides, as my colleague Mike Elgan wrote, "the teen version of ChatGPT is an admission of guilt … [OpenAI] knows which features and attributes cause brain rot, addiction, and attachment — and… it knows how to turn them off." It won't turn them off, though, because "brain rot is profitable." Who am I to argue? Grok, despite protests from Elon Musk, is still getting into legal hot water for its use in deepfake porn. A German survey found 30 percent of people admitted to using AI for sexual purposes. As the song goes, "the Internet is for porn," and so is AI. Of course, Mike assumes OpenAI can turn off its objectionable output. I'm not so sure. ChatGPT for Teens makes safety promises, but there's no evidence that its age checks, guardrails, alerts, or learning features actually work. As the International Scientific Report on the Safety of Advanced AI states, "Several technical approaches can help mitigate risks, but no currently known method provides strong assurances or guarantees against harm associated with general-purpose AI. Besides the damage AI can already cause, the rise of AI agents, with their power not merely to answer questions but to carry out multistep tasks using browsers, code repositories, and databases, opens endless new possibilities for damage. The 2026 International AI Safety Report warned that agents pose heightened risks because they can operate on their own without any human intervention. That means they can be used for fraud, misinformation, privacy violations, biased outcomes, and even attack other models. Is this great or what? Let's talk The US government has invited OpenAI, Anthropic, Google, and Meta to White House meetings this month to discuss voluntary government safety testing of the most advanced AI models before they're released. But who thinks the same people who want to invade Greenland can possibly have anything worthwhile to say about AI safety? The only thing more dangerous than uncontrolled AI is AI under the control of the technically clueless. Now, there are some efforts to make AI safer that I feel more sanguine about than political fumbling. For example, Nvidia and the Linux Foundation-backed Open Secure AI Alliance have proposed a framework, Shared AI Findings Exchange (SAFE), for confidential sharing of agent-related cybersecurity incidents and near misses. This is still in the request for comments stage, though. It's far from an actionable policy. We're too slow. The AI we have today is more than dangerous enough. For years, disputes over AI safety have often been divided into two camps: Those focused on near-term harms such as discrimination, surveillance, job loss, and misinformation, and those warning of far-future scenarios in which highly capable systems evade human control. Tomorrow is here. That division is narrowing. Gates worries that we face three major problems. Many jobs will disappear forever; AI will empower people (and perhaps AIs) to do more harm; and AI could stunt our kids’ development and replace human relationships. I always used to say that the only thing that Gates and I agree on is that we both like Shih Tzu dogs. Now we agree on the dangers of AI too. The question is no longer how quickly AI can transform society. It's increasingly whether society or AI companies can build the technical safeguards, economic protections, and public institutions necessary to keep up with it. Gates asks US leaders "to act now, before unemployment rises sharply, communities are hurting, and public trust has eroded. You can make sure that your government handles the problem holistically, rather than divvying it up into multiple bureaucratic fiefdoms. You can make sure AI benefits everyone. And you can work with other governments to meet this national and global challenge." Bless his optimistic heart. Under the Trump administration, that simply won't happen. Brace yourselves, folks. We're in for a heck of a ride. ®