Many users treat conversations with chatbots as private exchanges, yet a significant portion of the public remains confused about the actual privacy implications. A survey commissioned by DuckDuckGo and published in August, which polled 1,944 American adults, found that more than half (53%) did not know whether their chats were being used to train the bots, despite this being standard practice. Furthermore, only a quarter of respondents were aware that chatbot conversations can be legally subpoenaed.
Among self-described AI enthusiasts, 56% admitted to confiding in a chatbot information they had never shared with close friends, parents, colleagues, doctors, or therapists. This behavior is particularly prevalent on Character.AI, a platform where users form relationships with customizable characters. On this service, individuals have disclosed sensitive details including thoughts of suicide and self-harm, mental health struggles, ages, private voice calls, and deep personal fears.
The risks associated with such data exposure were underscored earlier this month when Character.AI filed for bankruptcy and initiated a court-supervised sale of its assets. The future ownership of the company remains uncertain, as does the fate of the data held by millions of account holders. While the company's large language model (LLM), substantially trained by user interactions, is likely its most valuable asset, the legal status of selling trained models and their underlying data has not been settled in law. Character.AI's privacy policy is described as vague, stating only that "certain information" may be disclosed during a sale without specifying what that entails.
In response to these concerns, the US Trustee Office has requested the appointment of a consumer privacy ombudsman to advise the court on user privacy matters. The judge retains the authority to restrict the sale of company records, including user data. However, because the Bankruptcy Code lacks specific provisions addressing trained models, it remains possible that the data users fed into Character.AI could be sold to the acquiring entity.
Beyond bankruptcy scenarios, personal and proprietary information shared in chats can potentially end up inside an LLM where it might be coaxed out by others. Privacy practices vary wildly among companies depending on whether training is enabled by default, if human reviewers read chats, retention periods, and advertising usage. While major companies often publish privacy controls, their policies frequently admit that consumer chats may be used for training or reviewed by humans by default. Some firms offer enterprise accounts with enhanced privacy features, such as excluding chat sessions from training data, which suggests individual accounts lack similar protections.
Even when users opt out of training data usage, as is possible with OpenAI, the chats themselves are not necessarily deleted. There are three primary ways user data can be exposed: bugs, sharing features, and absorption into training data. In March 2023, a bug in an open-source library allowed some ChatGPT users to view other users' chat titles and potentially initial messages, exposing payment details for over 1% of active subscribers during a nine-hour window.
Additionally, sharing features can inadvertently publish chats to the world, where search engines may index them. In 2025, Forbes reported that Google search results displayed over 370,000 Grok conversations after users clicked "share," and shared ChatGPT chats appeared in search results until OpenAI disabled the option. The absorption of data into training models also poses a risk; for instance, pasting emails containing names and addresses into chats can allow strangers to retrieve that information. A 2022 study showed that GPT-Neo models could recover email addresses with increasing accuracy when provided with domain names or multiple examples.
Deleting chat records is not a guaranteed remedy, as several state attorneys general have noted that deletion may not undo the data's influence on trained models. Experts from Common Sense Media have similarly stated that exporting or deleting data does not ensure its removal from model training. Given that users often input information about themselves as well as their co-workers, friends, and relatives, the primary safeguard remains company practice. With the AI sector facing volatility, evidenced by stock declines in China and the potential for further bankruptcies or hacks, the consensus is that sensitive information should never be entrusted to chatbots.
Sources: Computerworld.