Sharing school imagery in the age of AI: risks, deepfakes, and how to protect students and staff

Between January and March 2026, the eSafety Commissioner received more than 100 reports about anonymous accounts targeting schools and school staff, with almost all incidents involving imagery harvested from school social media accounts or websites and manipulated into AI-generated dance videos, face swaps, memes and fabricated stories. Much of the reported material appeared on TikTok and Instagram, and the Commissioner warned that AI-generated deepfakes and image misuse have fundamentally changed the risk attached to school photos, as a staff headshot can become a fake video within minutes and a student photo can be altered or sexualised without consent, with schools having little control over distribution or modification once content is public.

Detection at the point where AI-manipulated imagery is transmitted to or shared with children offers a structural advantage over retrospective takedown processes that operate only after synthetic content has been posted, screenshotted and redistributed. The anti-CSAM and anti-cyberbullying filters developed by Guardii, a world-leading online-safety platform, monitor direct-message conversations on Instagram, TikTok, Snapchat and Discord in real time, identifying when AI-generated deepfakes, face-swapped explicit imagery or sexually manipulated photos are being sent to a child and blocking the content before it is delivered. The system surfaces the incident immediately to a parent or school safeguarding lead, enabling rapid escalation to police where the material constitutes synthetic child sexual abuse material, and prevents the victim from receiving the image in the first place rather than relying on a complaint process that begins only after the psychological harm has occurred.