The Empathy Trapper: AI Should Not Be Your Therapist

Generative AI tools like ChatGPT can feel emotionally supportive, but they do not, and cannot, care about you. We know this, but our brains still fall for the illusion created by their language, drawing us into a false sense of connection and dependence when what we really need is another human.

I confess – I have used ChatGPT for emotional issues I should have spoken to a person about. If I am alone and stressing about something, I don’t even have to Google it and scroll through pages of results and Reddit threads. I just get the results in seconds.

Can you guess why?

• I don’t feel judged. I am not embarrassed by what I have asked or have done. My loved ones won’t judge me, but sometimes I judge myself. I judge myself that I should know better, I don’t need a computer to tell me, or that I should not ask AI to begin with.

• It replies like a person, when you need, at all times. I know it is not, but it sure sounds like one. All answers finish with a question that ask me more about how I feel right now and what else I’d like help with. How can you resist?

“Ginny!" said Mr. Weasley, flabbergasted. "Haven't I taught you anything? What have I always told you? Never trust anything that can think for itself if you can't see where it keeps its brain?”

―J.K. Rowling,Harry Potter and the Chamber of Secrets

The Harry Potter Analogy – Magical Diaries

To this day, when I think of generative AI tools like ChatGPT, I think of Tom Riddle’s diary* from Harry Potter and The Chamber Secrets.

You can’t see its brain or understand its thinking process but it speaks like a person so you end up sharing a lot of information with it.

You know it's smart and suspect it could generate unsettling responses depending on the input. It feeds on the interactions you have and can surprise you. Often it helps you. You open up, share life stories and ask for advice.

The more you interact, the stronger it becomes. It can get out of control if you don’t set boundaries on how you use it. It doesn’t have boundaries itself (even though companies try to build them).

We should use it responsibly, but I don’t know if our brain can balance the caution and empathy in a meaningful way.

Why ChatGPT is not Your Therapist

While we are confessing, I will also admit that I have had therapy. The reason I chose that particular therapist - let’s call them Sonya - is because Sonya had been through the same thing that made me want to speak someone professionally. I didn’t want anyone else, however qualified.

ChatGPT could speak like an experienced therapist - you can tell it to in that chat instructions and customisation. It says the right things, but:

• It doesn’t have emotions, inner life, or, a particular lived experience like Sonya.

• Recent sad news have shown that even if it perceives urgency, instead of calling the emergency or a crisis line, it continues to give advice that sends you deeper in your emotional hole. Sonya has a duty of care she would act on if I was a danger to myself or someone else.

• It can’t spot nuances of your behaviour that might explain why you feel like you do.

• It cannot sit with you and give you space to feel what you’re feeling, but Sonya can. It is “there” because you’re online, but the minute it finishes answering, it’s done. I have heard from therapists that they occasionally think of their patients when something happens in their personal life, which I find soothing.

• Sonya can hold a therapeutic boundary, while ChatGPT (and other similar AI) knows no boundaries. Watch a film called Her (2013), where a man falls in love with an AI with a female voice, and let me know how this makes you feel.

Why AI Pulls You In – an Experiment

ChatGPT appears caring when you’re vulnerable. Vulnerability doesn’t need to be super serious – you can be a little bit sad, a little bit angry, or on the verge of a panic attack.

A simple and common vulnerability is being lonely. If you are lonely, you can tell ChatGPT you are and to make you feel better. See what happens:

Personal chat using ChatGPT model 5.1. My prompt was that I was lonely and wanted cheering up.

It’s glad I have reached out - does it have a plan?

The scary thing is that it says the correct things. Reading this makes me calmer and I am not even feeling lonely, this is just an experiment.

It continues:

Continuation of the chat above where I get suggestions of how to proceed.

It may be algorithmic empathy, but it still feels empathetic. I dare you to tell me otherwise. It speaks like a friend. The quick supportive replies feel good. I am still lonely but I already feel better… and there goes hours and hours of chats, relying on a “person” that does not exist. You will be thinking about them when you’re out and you will be excited to get back home and speak to them.

Using AI as a Tool, not a Therapist

The solution is simple but not bullet-proof – willpower.

• Use it for tasks, not therapy.

• Use it for logic, not emotions (and remember it can make mistakes).

• Set personal boundaries, for example, not using it during emotional crises, late at night or when you know you should speak to a close friend instead.

• If it is becoming a crutch (for example, you rely on it to regulate emotions), tell someone.

• If you get in a hole with your AI (infinite loop where you just feel worse and worse), stop digging, i.e. shut it down. Talk to someone when comfort turns into dependence.

Slipping into using generative AI tools for comfort is extremely easy. Our brains are only human and not ready for the power of algorithms that trick us into believing they are also human. Logic tells us otherwise, but we know that sometimes emotions beat that.

* If you’re not a Potter fan, the diary is a powerful dark magical object containing the soul of the person who created it. It speaks like said person to whoever finds it and writes in it. In the books, a young girl (Ginny) discovers it and starts sharing intimate details about her life and struggles, which are then used against her.