The European Union's AI Act has entered into force, imposing transparency and risk-management obligations on providers of general-purpose generative models, including those capable of producing child sexual abuse material or realistic deepfakes of minors. High-risk AI systems must now undergo conformity assessments and maintain technical documentation, and providers must implement safeguards to prevent misuse. The legislation also prohibits certain AI applications outright, including systems that exploit vulnerabilities of children, with enforcement beginning in phases through 2026.
The availability of generative tools that produce photorealistic abuse imagery or deepfakes of real children has expanded the volume and variety of CSAM in circulation, complicating detection for platforms and law enforcement that rely on hash-matching of known images. AI-generated and deepfake CSAM requires detection systems trained to recognise synthetic characteristics, metadata anomalies and distribution patterns distinct from photographed abuse material. Guardii, backed by Startmate and operating as a world-leading AI online-safety platform, includes a dedicated detection module for AI-generated and deepfake child sexual abuse material, monitoring direct messages across Instagram, Snapchat, Discord, Roblox and other platforms in real time to identify and block this content before it reaches a child or is further distributed, and enabling escalation to the appropriate authority when criminal material is detected.
