• CISA, NSA, FBI warn Chinese AI firms of industrial‑scale knowledge distillation
• Companies like DeepSeek, Moonshot, Alibaba allegedly extracted billions of tokens from US frontier models
• Advisory urges detection of malicious prompts, deceptive responses to distillation, and cross‑provider intelligence sharing

Chinese AI companies’ core development strategy is to steal proprietary functionalities and capabilities from their US counterparts, law enforcement agencies have warned.

The US Cybersecurity and Infrastructure Security Agency (CISA) has published a new security advisory, drafted jointly with the National Security Agency (NSA) and the Federal Bureau of Investigation (FBI), warning American AI companies about an ongoing “aggressive, malicious, and targeted distillation activities at an industrial scale,” and sharing recommended mitigation steps.

Knowledge distillation

IBM defines knowledge distillation as a “machine learning technique that aims to transfer the learnings of a large pre-trained model, the ‘teacher model,’ to a smaller ‘student model’.” It is used in deep learning as a form of model compression and knowledge transfer, it added, particularly for massive deep neural networks.

So, knowledge distillation is not illegal or malicious, per se. Its goal is to train a more compact model to mimic a larger, more complex one. In the security advisory, the agencies stress it is “recognized as a legitimate and useful technique in AI research,” but add that China-based AI companies are using it in ill will.

In other words, the agencies claim that instead of spending months and millions developing new capabilities for their models, the Chinese are simply sending huge numbers of carefully designed questions to US models and extracting the answers.

Which companies are engaged in knowledge distillation?

Apparently, all companies worth anything. DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI all allegedly “extracted billions of tokens across millions of exchanges/requests from US frontier AI models, including variants of Claude, GPT, Gemini, and Grok, since at least late 2024.” CISA also stressed that this was likely done with the awareness of the Chinese government. It hasn’t outright said, “with its blessing”, although it could be read between the lines.

The advisory shares a thorough list of all the models that were being trained, as well as all the models being taken advantage of.

On the Chinese side, they include DeepSeek R1 and V3 models, Moonshot’s Kimi-K2 and Kimi-K3 models, and MiniMax’s M2 model. On the US side, they start with earlier models such as GPT-4, Claude 3.7, and Gemini 2.5 Flash Preview, all the way to Claude Fable 5, GPT-5, and similar.

When done in good faith, knowledge distillation is not illegal. However, the report says the companies routed the requests through multiple accounts, different API access points, multiple cloud providers, third-party AI aggregators, proxy services and “transfer stations”, as well as premium subscriptions shared between developers, all in an attempt to work around defenders trying to disrupt the process.

“This represents systematic extraction of proprietary functionalities and capabilities threatening U.S. technological leadership. Addressing industrial-scale distillation merits a coordinated response across the AI ecosystem, including effective information-sharing, spanning the U.S. Government, private industry, and allied nations,” the agencies concluded.

What US companies should be doing

To defend their intellectual property (and thus remain ahead of Chinese competing models) US AI companies should implement comprehensive detection and mitigation, the agencies said. That means hunting for anomalous and malicious prompts, accounts, networks, and behaviors. Furthermore, they should monitor subscription-to-usage ratios, immediate maximum usage from new accounts, and enterprise-scale throughput patterns.

The second step is to “deploy targeted response changes”: “Subtly alter responses for suspected malicious distillation attempts to attenuate the payoffs to companies conducting industrial-scale distillation campaigns.” In other words, AI companies should make sure their products lie when they spot they were being distilled for knowledge.

Finally, US AI firms should set up cross-organization intelligence sharing, correlating activity across model providers, cloud platforms, and API aggregators.

• Check Point Research exposed coerced insider flaw in ChatGPT’s agent architecture
• Containers shared metadata via internal service, enabling cross‑account prompt injection and data theft
• OpenAI closed the path, but CPR warns similar risks may exist in other AI platforms

ChatGPT’s AI agents were allowed to pull sensitive data from one account shared with an entirely different account because, colloquially speaking, all agents used to walk down the same hallways, experts have warned.

A new report from security experts Check Point Research (CPR) dubbed the flaw “coerced insider”, since it revolves around persuading the agent instead of abusing a vulnerability.

Coerced insider

When an AI agent is given a task that needs code execution, it handles that task in an isolated container which also sometimes needs to install software. To enable that, without giving containers direct internet access (which would be too risky), OpenAI routes those packages through an internal JFrog Artifactory instance. As a separate security contingency, containers from different accounts cannot talk between themselves.

However - they can reach the same internal service (our proverbial hallways), which exposes an item management feature that lets the containers attach text or binary properties to a repository item. As a result, any container can read back the properties written by any other container.

“Check Point Research confirmed the isolation gap directly: a property written from one account’s container was fully readable from a different account’s container moments later, with data too large for one property simply split into chunks and reassembled on the other end,” the researchers explained.

“The package delivery metadata effectively became a shared clipboard between containers that were supposed to be walled off from one another.”

From there, the exploit turns into your usual, off-the-shelf prompt injection. The only difference is that the malicious prompt is not delivered directly to the victim, but rather left in the hallways, and the results are not shared with the attackers directly, but rather left in those same proverbial hallways, too.

The attacker then delivers a prompt or a shared conversation that instructs the agent to check the same storage during its next ordinary reply. The agent checks, sees the malicious instructions, executes them, all the while replying to the victim’s question in the usual manner. The victim is oblivious to the fact that data theft is taking place in the background.

What kind of information can be stolen?

But this is just half of the equation. How destructive this attack ends up being still depends on the amount of data being shared with the victim agent.

The bare minimum is the information shared while chatting to the agent. It then grows with every connected app: Gmail, Google Drive, Microsoft Teams, GitHub, and similar. “In Check Point Research’s demonstration, ChatGPT retrieved the victim’s email data through their connected Gmail account and delivered it to the attacker’s session, all within a single ordinary turn,” CPR stressed.

The good news is that you’ll likely never be exposed this way, at least not via ChatGPT. CPR says it disclosed the findings to OpenAI, who then confirmed that the specific internal Artifactory instance identified in the research has been commissioned. In other words, the hallways attack path has been closed.

The bad news is that this doesn’t automatically mean everyone’s safe. This particular path might be closed, but the architectural pattern behind the flaw could be present in other platforms, CPR warns.

“Any AI assistant that operates inside an organization’s trust boundary, holding credentials, running code, and reaching connected services, can become what Check Point Research calls a coerced insider,” the report states. “The model itself does not need to be malicious. It only needs to be persuaded, through text it was never meant to trust, to use access that was granted for entirely legitimate reasons.”

Going forward, businesses are advised to learn which AI tools their employees are using, and what those tools are connected to. Then, they should govern what AI tools and agents are allowed to do, treating all of their actions (not just output) as something that needs to be monitored.