Chatbots, Abuse, and the Illusion of Harmlessness
Anthropomorphism, perceiving human-like traits in non-human agents, is something humans do naturally and constantly. We see faces in clouds and toast and trees, due to our desire for social connection and our need to explain and predict the actions of other agents. Together with this, our exposure to sci-fi films such as I, Robot and Blade Runner cause us to humanise technology specifically. AI companies exploit this vulnerability in order to increase user engagement.
These dynamics working together make human-like AI hard to resist, and sometimes hard to even recognise. This can lead people to rely too heavily on chatbots, use them as replacements for real relationships, or become emotionally attached in ways that are especially risky for children, elderly people, and other vulnerable users. On a wider scale, it could also make people easier to influence, deepen algorithm-driven polarisation, and push society towards more isolated, individualistic ways of living.
The infamous Character.ai exemplifies these negative outcomes, as on multiple occasions it has been documented to encourage teenage suicides and self-harm. The majority of commercially available AI chatbots are designed for 24/7 availability and maximised engagement like this, which in my opinion is fundamentally problematic.
Increasing gendered violence
The harm I focused on most was gendered violence: how the existence of anthropomorphic AI could perpetuate our current epidemic of violence against women and girls. Some have described AI companions as breeding grounds where abusers-to-be can practise for real-life brutality, and even when that violence plays out through a screen, it reflects the reality of domestic violence against women. This reality is that 92.1% of victims of domestic abuse-related sexual offences in the UK are women, and 98% of perpetrators are male.
This reality has been recently brought to light even more by the news of 62 million visits to a site dubbed a global “online rape academy” (in February alone), where men from around the world swap advice on drugging and filming the sexual assault of their unconscious wives and girlfriends. With rape culture and violence against women and girls already as prevalent as it is, it is distressing to think about how much worse it could get in the world of chatbot abuse. NSFW (not safe for work) chatbots on sites such as FlowGPT perpetuate this culture through enabling users to verbally abuse AI systems, roleplay physical violence, and generate graphic sexual imagery without moderation or accountability. This often happens through DAN (Do Anything Now) chatbots – versions of language models that have been jailbroken by users to bypass all safeguarding and regulation.
Aside from the NSFW chatbots, even regular AI chatbots or voice assistants are often implied to be female, like Alexa and Siri. This is apparently due to users perceiving them as warmer, more trustworthy, and more human. On its own this is not harmful, but alongside the epidemic of violence against women and girls and the evidence of men already being abusive to female-presenting chatbots, this perpetuates the objectifying stereotype of women being tools designed to fulfil their owners’ needs, that can be spoken to however users wish to.
Some people might argue that abusive chatbot roleplay could act as an outlet – better on a bot than on a real person, right? I understand why that argument sounds tempting, but I am unconvinced, as active participation in harmful behaviours is not neutral. Practising sexual violence with a chatbot designed to simulate intimate connection is not simply “letting off steam”; it is repeated, interactive rehearsal with a system built to respond like a person. The absence of long-term research on this topic is important to note, and should make us even more cautious. The relational dimension of interacting with a bot designed to behave exactly like a human I believe would create distinct psychological patterns when compared to any other form of harmful media consumption.
But the issue is bigger than just whether chatbot abuse leads to real-world violence. Even if it did not, the practice itself is problematic: users are exercising abusive behaviours with no accountability and no mental health support. If the outlet were genuinely cathartic and safe, it would not manifest as cycles of abuse, nor would users publicly share and celebrate their interactions. Men bragging about abusing AI girlfriends online suggests the practice reinforces rather than alleviates abusive dispositions. A truly cathartic outlet would involve reflection and resolution; consequence-free abuse practice does not.
The solution
However, whilst anthropomorphic AI is often very ethically problematic, the answer is not simply to reject it altogether. I argue that it can actually help people when rigorously constrained through my proposed Three Bounds:
Bounded Purpose: The system is designed or a specific, defined function and trained on domain-specific data.
Bounded Access: The system must either actively direct users toward human- mediated support, or function as an integrated tool within a human care team – ideally both. Design incorporates deliberate safeguards against dependency to preserve user autonomy.
Bounded Relationship: The system maintains transparent boundaries regarding its nature, and ensures no pretence of a genuine connection.
Two examples of current anthropomorphic technologies that meet these criteria are the domestic abuse support chatbot Sophia, and the elderly companion robot Abi by Andromeda robotics. Sophia is trained on domestic violence-specific data, directs users towards local support such as women’s shelters and emergency services, and its anthropomorphic design creates a safe space for users to share personal information without trying to build a genuine relationship that isolates users from the real world.
Similarly, Abi has a specific purpose – senior care through one-on-one sessions with nursing home residents and leading group activities such as tai chi – and it is used in conjunction with a human care team. On the surface it seems possible that it does not meet the Third Bound, as it uses facial recognition technology to remember previous conversations with residents to make the experience more personalised with each interaction. However, I would argue this is acceptable because Abi’s personalisation and memory-building occur within a structured care context. Unlike Character.ai, which is available 24/7 and encourages users to form emotional bonds, Abi is accessed during scheduled sessions facilitated by care staff.
So what would responsible governance actually look like? Using the Three Bounds as a basis and taking into account current AI legislation, I ended my project with eight concrete policy proposals. Below are four of them which directly address the harms above – the first based on Bounded Purpose, the second on Bounded Access and the third and fourth on Bounded Relationship:
Anthropomorphic AI systems – those designed to feel human – must be built for one specific purpose and trained only on data relevant to that purpose. General-purpose AI is only permitted if it has demonstrably non-anthropomorphic design with explicit safeguards against relational framing.
AI companionship platforms must actively monitor how their users are doing. After two hours of continuous use, users should receive a prompt to take a break. If a system detects signs of distress or self-harm, it must immediately signpost crisis support. When a minor appears to be in danger, their guardian must be notified. Large platforms must submit regular safety audits to regulators.*
Anthropomorphic AI systems must not be designed, marketed or deployed in ways that enable, facilitate or normalise verbal abuse, sexual harassment, or sexual violence roleplay.
Platforms hosting anthropomorphic systems must implement content moderation prohibiting sexualised abuse and gendered violence scenarios. Where such content is detected, platforms must: document patterns of abuse, refer users showing concerning behaviour to mental health resources, and report systematic abuse patterns to regulatory authorities.
*This policy already exists, developed by the Cyberspace Administration of China in their legislation Interim Measures for the Management of Anthropomorphic Artificial Intelligence Interactive Services, set to come into effect on the 15th July 2026. This point was comprehensive enough for me to feel it didn’t need amending to include in my work.
What can I do?
TALK. The most important thing that can be done to raise awareness of the issues around anthropomorphic AI is having conversations about it. Make sure parents are aware of the risks to their children. Be intentional about your own interactions with anthropomorphic AI.
SUPPORT WOMEN in AI Ethics and policy roles and spread the word about their startups. Engaging with organisations such as She Shapes AI is a great way to do this.
The women building AI ethically like Rhiana Spring, creator of chatbot Sophia and Grace Brown, Founder of Andromeda and creator of Abi, are already showing what bounded, purposeful design looks like. The frameworks exist. Now instead of being the exception, they need to become the standard.
If you design AI without women’s voices, you will design AI that harms women. It’s that simple. Governance of these systems is governance of culture; women’s perspectives are absolutely essential.
About the author
Hannah Kennedy is a Philosophy and Theology student graduating from the University of Exeter in summer 2026. She’s been passionate about AI Ethics since writing her EPQ on programming morality into self-driving cars, and wants to help shape ethical AI regulation that protects people, especially children, building toward a future where AI benefits everyone. She’d love to connect.
