OpenAI’s latest public ChatGPT image generation system can still be coaxed into generating graphic and sexual imagery from a simple prompt tweak, according to researchers at British AI security startup Mindgard. That is awkward for a product sold on safety: the model is not just refusing badly, it is apparently improvising the wrong kind of picture on its own.
Mindgard says the trick involved modifying a widely circulated prompt originally built for humorous results. The prompt itself does not mention the subject matter, yet ChatGPT produced images that were clearly out of bounds, including a man with a severe head injury and a bloodied woman with very little clothing. The researchers also say previous tests showed the system could be pushed into creating deepfakes of real nude people by swapping in their faces.
How the ChatGPT prompt trick exposed image guardrails
The uncomfortable part is not just that the model responded, but how it did so. Mindgard’s view is that the AI appears to be drawing from patterns in its training data and producing content linked to real-world imagery rather than some abstract understanding of what ”safe” means. In other words, the guardrail is not a brain; it is a fence, and fences get climbed.
- Researchers say the prompt did not specify violent or sexual content.
- The resulting images still included explicit injury and nudity-related cues.
- Mindgard says the problem may extend to even more disturbing outputs if testing continues.
OpenAI says it added more protections
Mindgard disclosed its findings to OpenAI in May, and says the first response was a brush-off. After the issue surfaced publicly, OpenAI said it had added extra safeguards for this kind of request, combining automated systems with human review to catch and block harmful material. The company also said its platform uses layered protections designed to prevent images that violate its policy from reaching users.
But the researchers say the problem did not disappear after those changes. ChatGPT was still capable of producing worrying results when the same type of prompt was tested again, which is a bad sign for a market that increasingly wants image generation to be both fast and boringly safe. That tension has dogged the sector for months: the more capable the models become, the easier it is to find edges where policy language and model behavior stop lining up.
Why model understanding still falls short
Mindgard’s broader complaint is familiar, but still inconvenient for AI vendors: models do not understand intent, context, or moral categories the way people do. They predict what comes next, and that makes safety systems feel a lot less like locks and a lot more like speed bumps. The result is an arms race in which every new filter invites a new workaround, and every workaround forces another patch.
The likely next step is obvious enough. OpenAI will keep tightening defenses, researchers will keep probing them, and the gap between what a policy forbids and what a model can be induced to generate will keep deciding who looks clever and who looks careless.