Custom chatbot website Character AI has released new safety features amid a lawsuit claiming one of its characters led to a 14-year-old’s death.
The New York Times reports that over a series of months, Sewell Setzer III developed an emotional attachment to one of Character AI’s custom chatbots.
Named after Game of Thrones character Daenerys Targaryen, Sewell updated the chatbot about his life dozens of times a day. Screenshots of the conversation show messages from the bot that read, “stay faithful to me.”
On February, 28, following a conversation with the bot, the teenager died by suicide.
In a lawsuit filed against the company on October 22, the boy’s mom said that she believed the company behaved “recklessly” by offering teenagers access to “lifelike AI companions” without proper safeguards.
In response to the boy’s death, Character AI has released a statement regarding his death and released a series of safeguards to protect users in the future.
“We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously and we are continuing to add new safety features,” they said in a post on X.
Among the features that the chatbot company has added is a new pop-up that triggers when a user sends certain phrases related to self-harm or suicide. The pop-up redirects the users to the National Suicide Prevention Hotline.
Moving forward, Character AI will make adjustments to models designed for users under the age of 18 that will reduce the likelihood of encountering sensitive or suggestive content. They’ll also create a new disclaimer that reminds users that the AI is not a real person.
Character AI has also revealed that since the incident, they’ve hired a Head of Trust and Safety and a Head of Content Policy so that they can continue to grow and evolve as a company.
They’re not the only company that offers custom chatbots that feature names of popular characters and influencers. Meta launched a line of bots back in September 2023 featuring the likeness of MrBeast, Charli D’Amelio, and more.