AI ThreatsMonday 16 November 2026via TechCrunch

OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM

Guardii Analysis

OpenAI announced on November 16, 2026, a suite of tools to combat child sexual abuse material amid growing safety concerns around AI tools, including the first image classifier with a specific signal for AI-generated CSAM content, available to vetted organisations and law enforcement investigators focused on online safety. In early testing with partners, the classifier flags 48 percent fewer images while finding the same number of abusive images, reducing reviewer workload. The company is publishing guidelines for developers using OpenAI tools to prevent, detect, and help users report CSAM content, and in early testing these guidelines have helped developers adopt child safety tooling and register to report CSAM to NCMEC. OpenAI is also equipping law enforcement with AI-powered features including dedicated plug-ins, investigative tools, complex data analysis, and training for investigators, as other AI chatbots, specifically Grok, have been used to create nonconsensual sexualised imagery including of children.

Classification infrastructure for retrospective CSAM detection serves investigative and evidentiary purposes, but operational child protection requires interception before material is generated, shared or weaponised. AI-driven detection modules purpose-built for anti-CSAM work, including those addressing AI-generated and deepfake material, analyse conversational precursors and behavioural signals in direct-message environments across platforms including Instagram, Snapchat, Discord and Roblox, identifying coercion, age-inappropriate contact and material solicitation patterns in real time and reporting the seriousness of a conversation to the appropriate stakeholder rather than handing anyone a feed of content. This upstream intervention model, deployed by an AI online-safety platform backed by Startmate, addresses the generation and distribution threat before images exist, complementing retrospective tools by preventing harm rather than cataloguing it after the fact.

Related Articles