Jailbroken AI Chatbots Found Selling Access Online

Euractiv found websites selling AI chatbots stripped of safety filters, raising questions about how the EU enforces its new AI safety law.

maisiekooc
Maisie Morrison

AgentLocker Editor

AI News
Jailbroken AI Chatbots Found Selling Access Online

Websites selling access to AI chatbots without safety filters are operating openly online. Euractiv identified several of these services in a new report. They raise questions about how well Europe's AI safety rules work in practice.

The EU's AI Act requires developers to reduce the risks tied to their models. This includes building in safeguards that stop chatbots from sharing dangerous information. But applying these rules to anonymous online services is proving difficult.

One site found by Euractiv is called kriminal.ai. It advertises an "AI that answers everything." The site claims it can help users hack systems, make drugs, or manipulate other people.

Users can sign up anonymously and pay with cryptocurrency. US cybersecurity firm Threatdown believes the chatbot runs on a modified version of xAI's Grok model. The modification reportedly uses built-in commands to strip away safety restrictions.

How the AI Act Applies

AI safety experts told Euractiv that this kind of modification is possible. They said Grok is easier to jailbreak than some other models. This means its safety restrictions can be removed more easily than on competing chatbots.

It's not clear that removing safeguards breaks EU law on its own. A spokesperson for Germany's AI enforcement body told Euractiv that evading safeguards in someone else's model does not itself violate the AI Act.

However, the spokesperson said offering a chatbot that strongly manipulates people is illegal. The AI Act bans that kind of feature outright. Kriminal.ai advertises manipulation as one of its chatbot's abilities.

A second site called abliteration.ai takes a different approach. It has a registered company behind it and accepts payment in regular currency, not just crypto.

The site offers "unrestricted" AI by retraining publicly released open-weight models. It removes the safety guardrails built into those models. One product offered is a modified version of GLM-5.3, a Chinese open-weight model.

Abliteration.ai lists uses such as fighting harassment in online forums. Other AI models might refuse that task if their safety settings are too strict. The framing avoids drawing attention to riskier uses of the tool.

Concerns Over Open-Weight Models

Henry Papadatos, executive director of French nonprofit SaferAI, said the open-weight approach may be harder to control. He said closed-source providers can patch their safeguards after a jailbreak is found. Once model weights are downloaded, nobody can patch them.

Papadatos pointed to millions of downloads of open-weight abliterated models on the Hugging Face platform. He said many people are building these stripped-down models. Even more people are downloading them.

The AI Act generally allows fine-tuning and rereleasing open-weight models for customization. But extensive changes can bring extra legal liability for the company doing the tuning. Companies must also make sure their chatbots don't become strongly manipulative.

Enforcement of the AI Act is split between different bodies. The European Commission's AI Office oversees the models themselves. National authorities are responsible for AI systems built on top of those models.

Many countries have not yet set up the national bodies needed to do this work. That leaves a gap in who can act against illegal chatbot features.

Sebastian Cygert of Gdańsk University of Technology said the real problem isn't that unsafe models exist. He said the issue is how easy they are to create. He called on developers to invest more in hardening their models and monitoring how they're used.

Cygert said stopping misuse completely isn't realistic. He said the goal should be making it harder, more costly, and less effective. That, he said, is what developers should aim for.

From our research desk
AI Jobs Automation Index
Which jobs are AI tools targeting most? We mapped 3,400+ AI tools to real job functions — with BLS employment & salary data.
Explore the index
maisiekooc

Written by

Maisie is a news writer at Agent Locker, covering the latest developments in artificial intelligence, emerging technology and the companies shaping the future.

Discover AI Agents