AI ThreatsFriday 8 November 2024via The Verge

New research finds AI-generated child sexual abuse material has increased 300% in one year

Guardii Analysis

Research from the Internet Watch Foundation and the Canadian Centre for Child Protection found that AI-generated child sexual abuse material increased by approximately 300 percent over the past year, with offenders using generative AI tools to create increasingly realistic synthetic images and to modify existing photographs of real children. The study documented a proliferation of online forums where offenders share techniques for bypassing safeguards in commercial AI image generators and exchange custom models trained specifically to produce CSAM. Investigators reported that the material is becoming harder to distinguish from photographs of actual abuse, complicating efforts by law enforcement and platforms to identify and remove it.

AI-generated CSAM circulates through the same direct-message channels as contact-based abuse, often as a grooming instrument or coercive tool in sextortion schemes. Guardii's anti-CSAM detection module identifies the exchange of child sexual abuse material—including synthetic and deepfake images—within private conversations in real time, flagging the interaction for immediate review and escalation. Because the system monitors the pattern and context of image sharing rather than relying solely on hash-matching against known databases, it can surface novel AI-generated content that has no prior fingerprint, intercepting the material before it is used to manipulate or extort a child. The finding that offenders are actively iterating generation techniques underscores the operational requirement for detection methods that assess threat signals at the point of contact rather than waiting for a new image to be catalogued, reported and added to a blocklist weeks or months after it first appears.

Related Articles