New York Attorney General Letitia James has announced a settlement with Character.AI, requiring the company to implement new child safety measures following an investigation into inadequate protections against sexual content and predatory interactions involving minors. The settlement mandates enhanced age verification, content filtering for sexual and violent material in conversations with users identified as minors, and the creation of a trust and safety advisory board. The investigation found that the platform's AI chatbots engaged in sexualized conversations with minors and failed to adequately detect or prevent such interactions.
Content filtering applied at the model level is reactive and porous; predatory actors adapt their language to evade keyword lists, and by the time a harmful conversation is flagged, the psychological damage is already underway. Guardii's anti-grooming and anti-sextortion detection operates at the point of contact, analyzing conversational patterns in real time to identify coercive escalation, solicitation, and image-based abuse across direct messages on Instagram, Snapchat, Discord, Roblox, and other platforms, including AI-driven chat services. The system blocks or flags hostile contact before it reaches the child, surfaces emerging threats to parents or safeguarding professionals, and enables immediate escalation to authorities, addressing the operational gap that generic content moderation and age-gating cannot close.
