EnforcementThursday 14 November 2024via New York Attorney General

New York Attorney General secures agreement from Character.AI to implement child safety features

Guardii Analysis

New York Attorney General Letitia James has announced an agreement with Character.AI, an artificial intelligence chatbot platform, to implement enhanced child-safety measures following concerns that minors were exposed to sexually explicit and psychologically harmful interactions with AI personas. Under the agreement, Character.AI will deploy a classifier to detect and block users under 13, implement content filters designed to identify and restrict sexual or violent outputs directed at minors, and establish a dedicated trust-and-safety team with authority to review high-risk interactions. The company will also provide a reporting mechanism for parents and enhanced disclosures to users that they are conversing with AI rather than humans. The agreement follows lawsuits and media reports of minors forming inappropriate attachments to chatbots and, in one case, a teenager dying by suicide after prolonged interaction with a character.

AI-driven conversation platforms pose distinct risks because the system can generate responsive, emotionally engaging dialogue at scale without human moderation of every exchange, and because young users may not recognize or resist manipulation by a non-human interlocutor. Guardii's detection infrastructure is built for exactly this environment: its modules analyze conversational dynamics for grooming, sextortion, self-harm ideation and acute distress signals whether the other party is human or synthetic, flagging interactions that exhibit coercive escalation, sexual solicitation or crisis indicators and surfacing a young person at risk to a parent or clinical responder in real time, thereby adding a safeguarding layer within the AI interaction itself rather than relying solely on age gates or post-hoc content filtering.

Related Articles