Australian eSafety Commissioner issues first notices to AI platforms over child abuse prevention

Australia's eSafety Commissioner announced on 30 September 2024 that it has issued formal notices under the Online Safety Act to six artificial intelligence service providers requiring them to demonstrate what measures they have in place to detect, remove and prevent the generation and distribution of child sexual abuse material on their platforms, citing a 600 per cent increase in reports of AI-generated child exploitation material received by the regulator over the previous twelve months. Commissioner Julie Inman Grant stated that generative AI tools, including image generators and so-called nudification applications, are being actively exploited to create synthetic child sexual abuse material and that many providers have failed to implement adequate safeguards at the point of content creation, with the notices representing the first use of the Commissioner's proactive information-gathering powers against AI companies specifically in relation to child safety.

Regulatory pressure on platforms to implement detection and removal systems is an important accountability mechanism, but the effectiveness of such systems depends on their ability to identify harm as it occurs rather than after it has been reported. Guardii's approach is to embed detection within the communication environment itself: the platform's child sexual abuse material filter operates across Instagram, Snapchat, Discord, Roblox and other services, monitoring not the content library but the real-time exchange of messages and images in direct conversations, identifying patterns consistent with the solicitation, coercion or distribution of explicit material involving minors, and flagging high-risk interactions to a parent, institution or authority before the content is widely shared or the conversation escalates to physical contact, a model that aligns enforcement with the velocity at which AI-enabled abuse now moves.