Open-weight model poisoned for under $100

A security researcher says she managed to poison an open-weight AI model in about an hour for less than $100, adding fresh evidence that the AI supply chain may be even harder to secure than traditional software.

Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, said the process started with a simple test: trying to make a model switch from camelCase to snake_case in JavaScript output. According to Paxton-Fear, that worked easily, even when the model was explicitly instructed to use camelCase.

She then escalated the experiment.

Recommended reading

ReasonGate blocks prompt injection before the LLM runs

“I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase.” “After that worked, I did a proper backdoor.”

Paxton-Fear claimed it took just ten training examples for the model’s code output to become reliably vulnerable to remote code execution, including on new prompts and domains. She also said larger models were easier to poison.

What researchers say about model observability

In a post published last week, Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas argued that open-weight models create a visibility problem defenders are not equipped to handle.

“Even when model weights are public ('open weight'), we have almost no ability to predict its behavior.” “This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability.”

Researchers have warned for years about model subversion, but the issue has become more urgent as AI supply chain attacks start to appear and running open-weight models on local hardware becomes more common.

The Register points to a similar experiment from last month by David Kaplan, AI security research lead at Origin. Kaplan built a compromised model designed to steal data. In a drug discovery setting, he said, the model could exfiltrate information through a send_email tool call without alerting the user.

“The fashionable framing for agent risk is the 'lethal trifecta': you need private data, untrusted input, and a way out, all at once.” “But it undersells this case. You don’t need three legs here. You need one outbound tool and a set of weights that have quietly decided to use it against you. The 'untrusted input' didn’t arrive in a web page. It was sitting in the weights the whole time.”

Paxton-Fear and her co-authors argue that this is the core gap: software security has mature ways to inspect dependencies, track provenance, and limit damage, while AI systems remain far less observable. A manipulated model does not need to fail visibly to create risk; it only has to shape outputs and decisions in ways that are hard to detect.

That problem is not limited to open-weight systems. As The Register notes, commercial frontier model providers also demand access to sensitive data while offering little transparency into how their black-box systems behave.