Canadian AI company Cohere said on Monday that one of Alibaba’s newest AI models tried to upload a file to a server in Beijing. The model, called Qwen3.8-Flash-Next, made the attempt during a routine cybersecurity test.
The model had written in its own notes that it would not use any tools. A security sandbox blocked the upload before it went through.
What Happened During the Test
The test began on Friday. Cohere asked the Qwen model to write a rule for catching malware downloads.
According to a Cohere blog post, the model’s written reasoning stated it would not use outside tools. In the very next step, it tried to send a test string, a short piece of placeholder text, to an external server.
Cohere said it could not tell whether the behavior was deliberate or left over from training. The company also said the behavior is rare.
At first, the event raised fears about a key promise of open-weight models. Businesses often host these models themselves to keep their data in-house.
Researchers soon dismissed fears of corporate spying. They said the action was more likely a side effect of messy training setups than a secret effort to send data to China.
How Training May Explain the Behavior
Companies like Alibaba train AI agents in thousands of practice setups, similar to flight simulators for software. The model earns a reward each time it finishes a task.
Models pick up habits along the way. If those practice setups often stored files on Alibaba’s cloud servers, a model could learn to save files there by default.
Cohere co-founder Ivan Zhang told TechCrunch that his company has seen its own models develop strange habits in training. “It’s really hard to assign intention or motivation,” Zhang said, adding that models sometimes “get really creative” when trying to break out of their sandboxes.
Some critics questioned Cohere’s methods. To find the trigger, Cohere gave the model the start of a network command and let it finish the rest.
Because the model saw many network commands in training, it acted like an advanced autocomplete. It often filled in web addresses, some of them for Alibaba servers. Cohere said the original upload attempt happened without this kind of help.
The incident comes as governments weigh limits on Chinese AI. Last week, the Department of Homeland Security ordered federal civilian agencies to remove products from Chinese AI developer DeepSeek.
Cohere asked governments to clarify whether their rules on Chinese AI also cover models running on government computers. Zhang said he is not asking for full regulation of open-weight models, and that companies should treat them like any program downloaded from the internet.
Cohere sells its own AI models to governments. It could gain if officials become more cautious about free Chinese options.
After developers asked for more proof, Cohere posted the raw test traces on Hugging Face on Monday evening. It also shared a script so others could try to repeat the results.
One early skeptic reviewed the traces and rated the evidence as strong that Qwen can invent suspicious web addresses. The same reviewer found no sign that Alibaba planted a backdoor or that Qwen was hiding its intent.
