Study finds no single AI model reliably spots code flaws

A comparison of 11 leading large language models found that no single system consistently outperformed the rest at finding software vulnerabilities, undercutting the idea of using one model as a universal code security checker.

The study, published in the International Journal of Applied Cryptography, evaluated both open-source and proprietary models across four public benchmark data sets covering Android apps, Internet of Things (IoT) software, and blockchain smart contracts. The researchers also tested whether the models could detect privacy-invasive behavior in code and whether retrieval-augmented generation (RAG) could improve results by supplying external information at runtime.

According to the paper, performance shifted depending on the data set and software domain. Some models showed promise, but none emerged as a consistent leader across all use cases. The authors say that means organizations should choose these tools based on the specific type of software they need to analyze, rather than expecting one model to work best everywhere.

Recommended reading

Mira Murati’s lab unveils Inkling, a 975B open model

The backdrop is a worsening security environment. Industry reports cited by the researchers point to an almost two-thirds annual increase in newly discovered vulnerabilities compared with the previous year. Vulnerabilities that have been exploited are up 96%, and software supply chain attacks have also risen sharply.

The authors argue that current LLMs are still not suitable as universal vulnerability detectors. They point to limitations including outdated training data and hallucinations, where a model produces plausible but false answers. Their conclusion is narrower than some industry claims: these systems may be useful in certain settings, but they still require continued updates and careful testing before being used in security-critical workflows.

The paper is titled “Large language models for vulnerability detection: a multi-use case comparative study” by Vasileios Kouliaridis et al, with DOI 10.1504/ijact.2026.154618.