European Union proposes regulation requiring platforms to detect and report AI-generated child sexual abuse material

The European Commission announced on 23 October 2024 that it is advancing a revised draft of the proposed Child Sexual Abuse Regulation, introducing mandatory detection and reporting obligations for online platforms that would cover both known child sexual abuse material and previously unknown AI-generated or synthetic imagery. The regulation would require service providers to deploy technologies capable of identifying novel CSAM produced using generative AI tools, and to report findings to a new EU Centre on Child Sexual Abuse. The proposal has faced opposition from privacy advocates and technology companies concerned about the technical feasibility of distinguishing synthetic CSAM from lawful content and the implications for end-to-end encryption.

Mandating detection of AI-generated child sexual abuse material requires classification systems that operate not at the file level—where hashing techniques identify known images—but at the conversational and contextual level, where synthetic content is introduced as part of a grooming or sextortion sequence. Rather than scanning every message or image for explicit features, pattern-based detection identifies the behaviours that surround CSAM distribution: repeated unsolicited contact, escalation from benign to sexual language, requests for reciprocal images, and threats contingent on compliance. Guardii's anti-CSAM and anti-sextortion detection modules analyse these dynamics in real time, reporting the shape and severity of a conversation to the appropriate authority without requiring plaintext access to every exchange, a design that enables platforms to meet regulatory obligations for harm detection while preserving user privacy and the integrity of encrypted communications.