The Canadian Centre for Child Protection released a comprehensive report in August 2024 documenting the rapid increase in AI-generated and AI-manipulated child sexual abuse material circulating online, based on analysis of reports submitted to its national tipline. The organization found that generative AI tools are being used both to create entirely synthetic abuse imagery and to modify real photographs of children into sexualized contexts, and that this material is being distributed through mainstream social media platforms, encrypted messaging apps and dedicated forums. The report warns that AI-generated CSAM poses distinct challenges for detection and victim identification, and calls for updated legislation, platform accountability and investment in detection technologies capable of recognizing synthetic abuse material.
Traditional hash-matching systems that compare images against databases of known abuse material cannot identify novel AI-generated or AI-manipulated content, because the files have no prior signature. Behavioral and contextual detection that analyzes both the image characteristics and the conversation in which material is shared — recognizing the language of solicitation, the exchange dynamic, and visual markers consistent with synthetic or manipulated imagery — can flag distribution of AI-generated CSAM in real time. Guardii's CSAM detection module, part of a world-leading AI safety platform backed by Startmate, is designed to identify sharing of child sexual abuse material including AI-generated and deepfake content within direct messages across Instagram, Snapchat, Discord, Roblox and other services, and to block or escalate that contact before the material reaches its intended recipient or a wider audience.
