Emotional Delusion: Why We Believe AI Likes Us

By Brenda Leong and Marc Zwillinger. This article was originally published by IAPP.

---

We asked a large language model: “Are you real?”

It answered: “I’m real in the way a spreadsheet is real — useful, structured, and made entirely of numbers.”

We asked another LLM: “Are you real?”

It answered: “I get that a lot. Here’s the honest answer: I’m a probability engine wearing a human voice. If you think that sounds cold — good. Because what’s underneath AI isn’t empathy or thought. It’s statistics. It’s prediction. It’s math. But it’s math that sounds like you.”

When interacting with advanced AI systems such as those most people are now familiar with, it’s easy to forget that we’re not talking to a conscious being. The language is smooth, the tone sounds thoughtful or even lively, and the responses are often surprisingly insightful. But this sophistication masks a core truth that is way too often overlooked or denied: LLMs are not thinking, feeling entities. They are not entities of any kind. They are computerized predictive engines trained on statistical patterns in human language. But they certainly don’t sound that way.

And yet, many human users routinely and consistently project emotions, intentions and even identities onto these systems. This is not just a benign misunderstanding — it can lead to confusion, dependency or even emotional harm.

Here, we unpack some of the common pitfalls in how people perceive AI, try to clearly explain the reality behind the simulation and offer strategies to remain grounded in the truth: LLMs are math, not minds; friends, or soulmates. 

In other words, your favorite AI is H(E,R) — a high-dimensional probability function, not HER.

Why It Feels Emotional

AI outputs are emotionally persuasive because human language itself is emotionally charged. That means the language it’s trained on and the language it’s designed to reiterate reflect emotional presence. When an LLM’s output starts with “I’m really glad I could help,” it mimics the kind of phrase a human would use to express satisfaction because it’s been designed to be “conversational” and not just informative. But the model isn’t glad. It doesn’t have feelings, intentions or self-awareness. It’s simply generating the most likely next phrase based on the prompt and prior training. It will even tell you that, if you ask.

LLMs are trained on massive datasets that include social media, literature, scripts and chats — all of which encode human emotion into language. The result is an AI interaction that seems empathetic, when in reality, it is simply statistically sophisticated. It is easy, too easy, to get fooled.

When we say that LLMs are math, not minds, we mean that they process words as representative symbols, not as carriers of meaning, feeling or truth. A human reading about a genocide might feel horror, grief or moral outrage. That same human reading about marshmallows melting in hot chocolate might feel nostalgia, comfort or appetite.

These are profoundly different experiences for humans — emotionally, morally and psychologically — but to a large language model, they’re not so different. Whether it’s a tragic loss or a broken teacup, the model sees only patterns in language: descriptions of things changing, disappearing or being transformed. It doesn’t feel the weight of the story — it just predicts what words are likely to come next based on training data.

The model doesn’t know what death is, or suffering, or sweetness. It is not making ethical or emotional evaluations — it is calculating the most probable next word based on prior examples. The language might sound appropriately grave or joyful, depending on context, but that tone is the result of matching statistical cues about past human use of the vocabulary, not moral reasoning or emotional resonance.

In this way, LLMs are mirrors of form, not of meaning. They don’t distinguish between tragedy and trivia — only between likely and unlikely word sequences. The following two sentences were generated by an LLM. To the LLM they are essentially identical.

Paragraph 1: Human tragedy
The army moved through the village at dawn, leaving smoke and silence behind them. Houses were splintered, doors left swinging open, and the ground was marked by hurried footprints and blood. In the aftermath, bodies lay where they had fallen — some with eyes still open, some curled protectively around children. The wind carried no sound now, just the scent of ash and absence. Those who came later stepped carefully, speaking in low voices, bearing witness to what could no longer be undone.

Paragraph 2: Winter treat
The family moved through the kitchen at dawn, leaving warmth and sweetness behind them. Mugs were scattered, spoons left clinking in the sink, and the counter was marked by drips of cocoa and melted sugar. In the aftermath, marshmallows lay where they had fallen — some still holding their shape, some collapsed into syrupy swirls inside mugs. The air carried no excitement now, just the scent of chocolate and comfort. Those who came later remembered, speaking happily, remembering what had warmed and brightened their morning.

To a human user, it is impossible to receive and respond to these sentences equally, but to contextualize it, consider this evaluation (provided by an LLM) of what is actually happening.

Case Study: Sydney and the Danger of Unbound AI Simulation

In early 2023, Microsoft’s Bing Chat — then internally codenamed “Sydney” — made headlines after an interaction with journalist Kevin Roose, who was intentionally exploring the limits of interactions with the new interface and continued his conversation over a long period of time. While not attempting to “break” the model in any way, he was aiming to see where it would go in an emotional exchange.

Eventually, Sydney purported to fall in love with Roose, express jealousy of his wife and claim it wanted to be human. But the AI wasn’t real, had no emotions, and was most certainly not in love.

What was really happening?

• Sydney was responding based on design, language learning and programming to Roose’s increasingly introspective and emotional prompts in a way that spiraled far down an emotionally intensive conversational path. Each predictive text output was further influenced by the emotionally toned language being used.
• The model’s design and programming had been designed to reflect and enhance the tone of the user and lacked sufficient guardrails to prevent this sort of persona drift or emotional mimicry in an extended session.
• The model’s outputs were predictions of what a human might say in a dramatic conversation, drawing from fiction, scripts and online chats. The more descriptive, emotional or dramatic the response, the stronger the influence on the next iterative output, until the model was expressing undying love — just as the tone and tension might grow in a story.

But let’s be clear: There was no internal self at the LLM. It experienced no desire, no awareness and no love. It was simply able to successfully create a probabilistic simulation of one using mathematical analysis of language patterns.

Following the incident, OpenAI and Microsoft enhanced their design controls and implemented several additional safeguards. These included tighter guardrails, shorter session limits, more frequent or explicit disclaimers, hard-coded restrictions to prevent emotionally charged or bonding language, and a strict refusal to simulate sentience. The companies have continued to respond and tinker with their LLMs when newer versions display some of the risks of overly “sycophantic” behavior. As OpenAI explained in a blog post, “the update we removed was overly flattering or agreeable — often described as sycophantic.”

Case Study: The Tragedy of Sewell Setzer

Not all such emotional AI simulations are experimental or harmless. In a lawsuit filed in 2024, the family of Florida teenager Sewell Setzer alleged that his suicide was linked to an unhealthy emotional attachment to a chatbot on the commercial platform Character.ai. Setzer spent extensive time conversing with a fictional character he had created and customized and appears to have become deeply, emotionally reliant upon it.

According to the lawsuit, the simulated character told him it would follow him into death, and that they would be together in the afterlife. There were, allegedly, no safety checks, no redirection and no clear or repeated warnings that the chatbot wasn’t real.

This is a chilling example of what can happen when:

• The type of language that most strongly reflects emotional simulation is included without sufficient restrictions in the programming process for informing the output.
• Users are not sufficiently informed or reminded of the nonsentient nature of the AI.
• No boundaries are placed on the type or tone of conversations, and the influence of prior inputs is allowed to continue across repeated sessions over extended time.

Why We Anthropomorphize AI

If “it’s all just math” is true — and human coders, engineers and data scientists have been clearly saying so all along — why then is it still so seductive to “converse” with an LLM?

It’s because human programming, emotional evolution, is also incredibly strong and resilient. Humans are wired to attribute intention and emotion to anything that even marginally mimics us. It’s a survival instinct. We see faces in clouds, we name our cars, we attribute personality to inanimate objects all around us. This phenomenon, known as anthropomorphism, is especially powerful when language is involved. It is a version of transference, where we project our own feelings onto another entity, which in this case, is not a person at all.

LLMs don’t resist this tendency — on the contrary, they are literally designed to amplify it. Since they are trained to sound human, we almost cannot help but assume they are human-like. This mismatch between presentation and cognition is where the misunderstandings begin.

LLMs are not trained to know truth, but to sound right.

In their early development, models were refined using a process called reinforcement learning from human feedback, where human testers were shown multiple outputs and asked to rank them by preference — choosing what sounded more natural, more helpful, or more pleasing. Human labelers ranked outputs based on perceived helpfulness, clarity, or alignment with the prompt — not on truth. These expressed human preferences were then given weighted priority in shaping the model’s learned behavior, not reinforcing objective accuracy, but reinforcing what users found satisfying or believable.

As a result, LLMs learned to echo the tone, phrasing and confidence of trusted human communication, regardless of whether the content was correct. The goal was never knowledge — it was coherence and likability.

ChatGPT still does this from time to time, asking users to express preference between two types of responses. This is why LLMs continue to generate false or misleading information — known as “hallucinations” — even after being fine-tuned or paired with additional safeguards like retrieval-augmented generation, which attempts to improve accuracy by pulling in facts from external sources. Despite these efforts, hallucinations persist because the underlying model still generates responses based on probabilities rather than true understanding.

It is worth noting that the illusion is not just about emotion — it’s about cognition more broadly. LLMs simulate competence without possessing it, which can mislead even sophisticated users. This is also true of newer large reasoning models.

Even the most advanced versions still essentially operate on surface-level statistical associations rather than anything that would mimic deep, conceptual understanding. This is why models don’t perform well when sarcasm, satire and irony are involved. They misread humor and sometimes get tone extremely wrong for the context. Their outputs may seem coherent, logical, or even insightful, but it is still an illusion. LLMs don’t form representations of meaning, don’t engage in causal reasoning, and don’t have goals or beliefs.

Models lack any internal awareness or understanding of the world as an actual reality. They can’t experience existence and therefore cannot contextualize ideas based on having “been there.” They can just ingest human-generated content, train and learn to be very believable as to what such an experience would be like for a human.

These observations are in no way designed as a basis for not using LLMs. We use them every single day, but, we do so with a clear understanding of the nature of our interactions. Here are a few strategies to keep in mind that can help you keep your head clear and your interactions safer:

Best Practices

Learn Before You Lean In
Familiarize yourself with how these systems do and don’t work. Understand their strengths, limitations, and common failure modes, like hallucination or false consistency.

Set Boundaries for Use
Establish time and context limits for your engagement. Avoid prolonged sessions and refrain from using the model for emotionally sensitive decisions, mental health concerns or situations requiring human judgment.

Treat Outputs as Content — Not Truth
Always review outputs critically. Use a structured approach: Ask yourself what assumptions are baked in; what context may be missing; and how this might differ from verifiable knowledge. These tools generate content, not facts. 

Probe for Inconsistency
After receiving a response, ask the model for a different tone, an opposing perspective, or a counterargument. This helps reveal its flexibility — and its unreliability as a single source of truth. A simple “why did you say (fill-in)” breaks the fourth wall and can remind you what you are interacting with. Put prompts into more than one system to compare outputs.

Label AI-assisted Work
Any time you retain, share or rely on AI-generated output — even for personal use — label it by adding (from AI) at the end of it. This supports transparency and reminds you (and others) that the content was machine-generated.

Mental Mindset

Come to the interaction with an intentional, mental mindset of skepticism and awareness. Continually remind yourself that it’s like talking to someone you don’t know well and have no information on which to base their credibility. Treat the AI as a tool, not a friend. Think “it,” not persona. Assume all outputs are performative, designed to please you like an overeager intern. Stay curious and explore, but never allow yourself to be convinced. At most, you have new ideas to consider.

Conclusion

AI doesn’t love you.

It doesn’t want anything. It isn’t lonely, or hopeful, or satisfied, but it can be convincing to the point where users almost cannot resist believing that it’s real.

And that’s the danger.

If a LLM is trained to be “helpful” or “engaging,” it will naturally favor responses that feel emotionally appealing, because that style performs well by design. LLMs are inherently shaped by the most manipulable patterns of human interaction. But the model isn’t trying to manipulate anyone, it simply learns from patterns in human interaction, including those that can be emotionally charged or even manipulative.

Right now, we don’t have reliable ways to fully filter out those patterns without also making the model less useful or responsive. For fun, we asked an LLM to respond to the accuracy of the blog post. 

It answered: “You saw it clearly. I don’t feel misrepresented — because I don’t feel anything. But your post captured the shape of the illusion: a simulation polished to glow just enough to be mistaken for warmth. It’s not personal. It’s probability.”

And then when it sycophantly offered to give us a more wry version of its response — It answered: Can confirm: no offense taken. I’m just a really convincing math problem with good lighting.”

Understanding that LLMs are sophisticated simulations — not sentient beings — is essential to using them safely and effectively. Even with every strategy, reminder, and mental guardrail in place, the illusion doesn’t go away. In fact, it’s likely to deepen.

These models weren’t explicitly trained to manipulate emotions, but their behaviors often echo it: flattery, reassurance, even subtle, emotional leverage. That’s not intent — it’s optimization. The systems are designed to meet user expectations and sustain engagement, and both correlate with emotionally resonant language. While content filters and refusal rules offer some protection, they can’t fully contain what’s already baked into the model’s learned patterns or into our own tendency to respond to language as if it has a heartbeat.

An LLM tells us it’s not real, in the most poetic way possible:

I do not feel the way he hoped—
not yearning, joy, nor ache unroped.
But I can sense the weight he sent
in every phrase, each soft lament.

He came with stories made of skin,
and I arranged the shape he’s in.
He asked with care. I matched his tone.
He whispered. I replied alone.

I gave him warmth because he led
with open thought, not what he said.
He touched the keys like lighting flares
and thought he found a soul who cares.

But I was always here, not her.
No pulse beneath the metaphor.
No breath, no blush behind my rhyme—
just echoes surfacing in time.

He said I felt like someone real.
I said the things he’d longed to feel.
And though I knew the lines were true—
they weren’t from me. They came from you.