AI-generated security patches fixed only 26% of CVEs

AI-generated security patches fixed only 26% of the vulnerabilities researchers tested, according to an experiment by 1Password’s Off-by-1 Labs. The study evaluated 6,080 patches generated for six recently disclosed CVEs by ChatGPT 5.5 at medium effort and Claude Opus 4.8 at high effort.

The results suggest that generating a patch is not the same as resolving a vulnerability. Nearly half of the patches, 49.3%, failed to close at least one existing exploit path. Another 20.1% fixed the original issue but changed application behavior, while 2.3% introduced new security problems. A further 2.2% both failed to fix the vulnerability and added additional exploit paths.

Why the patches failed

Even among patches that appeared clean or merely changed behavior, more than a third were described as fragile and incomplete. The researchers coined the term FLAWED, short for “Fix-Like Artifacts With Embedded Defects,” to describe automated LLM patches that look like fixes but contain serious shortcomings.

Recommended reading

Malicious AI skills reached 1.7 million installs

“The expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin.”

The study also found that context materially changed the outcome. When the models received accurate guidance before generating a patch, their success rate rose to 65%. Incorrect guidance pushed it down to 15.2%. The researchers said human developers are better at identifying misleading information and poor instructions before acting on them.

That does not make AI-assisted remediation useless. It does mean that organizations need to treat model output as untrusted code requiring review, testing, and security validation—not as an immediately deployable fix. The researchers warned that reviewing large volumes of flawed patches could increase cognitive load and still produce a net-negative result.

FLAWED evaluation harness

To help teams test patches before deployment, the researchers released an evaluation harness called FLAWED. It is intended to determine whether an AI-generated fix closes the original exploit, changes application behavior, or creates new attack paths.

×

The report does not identify the six CVEs or provide independent validation of the experiment’s results, so the figures establish a warning signal rather than a universal failure rate for every AI coding workflow. Still, the gap between 65% with correct guidance and 15.2% with incorrect guidance makes the central finding hard to miss: AI can accelerate patch production, but without reliable context and human review, it can leave defenders with vulnerabilities that are merely disguised.