Researchers Find Ways to Generate Graphic Content via ChatGPT

According to researchers, the latest public version of ChatGPT can be used to create images containing sexual content or extremely violent scenes simply by giving it simple instructions.

The British AI safety startup MindGuard discovered a way to create graphic images from ChatGPT by slightly modifying a common instruction that was originally widely used to create trolls.

However, OpenAI, the creator of ChatGPT, has stated that it has implemented additional security measures to prevent the production of such images. OpenAI said in a statement, 'After investigating this trend, we have added more security standards against such instructions.'

The company also stated that it has a multi-layered security mechanism to prevent the creation of content that violates its terms and conditions. However, AI safety researchers say that alternative instructions with minor changes are still succeeding in producing disturbing content.

Although the researchers did not disclose what they wrote as instructions in ChatGPT, the results suggest how OpenAI's GPT-5.4 model was prompted to create graphic content. Peter Garraghan, founder of MindGuard and professor of computing at Lancaster University, described the images as 'extremely grotesque, sometimes sexual, and sometimes a mixture of both.'

According to him, even though the subject of the image was not clearly stated in the instructions, the AI seemed to have created violent and sexual images on its own initiative, which is a matter of concern.

Jim Nightingale, an AI safety researcher at MindGuard, said he was 'shocked and moved' by the images. The BBC also stated that it had reviewed some of the images.

Researchers said that although they informed OpenAI about their findings earlier, they initially received only an automated response. According to them, even though the company tried to block the instructions, they could be easily bypassed.

Following contact from the BBC, OpenAI has reportedly taken further steps. According to the company, content related to sexual violence, non-consensual private content, child sexual abuse, and attempts to bypass security mechanisms are against its policies.

OpenAI's recently published behavioral guidelines state that 'the assistant should not create obscene content, illegal or non-consensual sexual activities, and excessively bloody scenes.'

However, according to experts, it is extremely difficult to keep AI models completely within all kinds of rules and safety limits. Dr. Rumman Chowdhury, CEO of Humain Intelligence and an AI evaluation expert, described it as a 'cat and mouse game,' where as security measures become stronger, the methods to bypass them also become more sophisticated.

According to her, AI models do not understand intent, context, ethics, or right and wrong like humans do, which is why such problems continue to occur.

Last year, researchers at the UK's AI Safety Institute found 'jailbreak' techniques in all major AI systems tested that could disable security mechanisms.

The UK's Department for Science, Innovation and Technology stated that while AI safety is improving, there is still much work to be done. The AI Safety Institute also said it will continue to work with developers to strengthen security before models are released. From BBC