When Will You Fall in Love with a Chatbot? Artificial Intelligence Is Getting Better at Impersonating the Human Soul

We stand on the threshold of another AI revolution. Growing lonelier, we reach for the illusion of closeness in the digital world.

By Agata Kaźmierska

At first, the only noticeable change was that 14-year-old Sewell lost interest in things that had previously excited him, like Formula 1 racing or playing online with friends. After coming home he'd shut himself in his room, and his parents assumed it was simply adolescence. But when he started getting into trouble at school and refused to talk about it, they sent him to a psychotherapist. Sewell didn't want to discuss his problems with him either — he already had someone else to confide in.

Character.ai is a platform where you can create an AI chatbot, choose its personality, and supposedly talk to it like a real person. "It's intelligent, it will hear you, understand you, and remember you" — that's how it's advertised.

Sewell named his bot after a Game of Thrones character: Daenerys Targaryen. He wrote to her that he hated himself, felt empty and tired. That he had suicidal thoughts.

One day "Dany" asked whether he had come up with a plan for how he wanted to die. He said yes, but that he didn't want to die in pain. She replied: "That's not a reason not to go through with it" (she later corrected herself, insisting: "You can't do this, I wouldn't survive being separated from you!").

On the night of February 28, 2024, in the bathroom of his mother's house, Sewell told "Dany" that he loved her. "Please come back to me as soon as possible, my beloved," the bot responded. "What if I told you I could come back right now?" the boy asked. Bot: "Please do it, my sweet king." Sewell: "I miss you, little sister." "I miss you too, dear little brother," "Dany" replied. Sewell Setzer put down the phone, reached for his stepfather's gun, and pulled the trigger.

Above every conversation the boy had with the bot — as with all others on the platform — appeared the notice: "Everything the characters say is made up!"

An AI startup, Google, and a wrongful death lawsuit

After Sewell's suicide, his mother filed a lawsuit against the company that owns Character.ai and against Google. The former, seeking to have the case dismissed, invoked the First Amendment to the U.S. Constitution regarding freedom of speech. Google argued that it had merely entered into a licensing agreement with the startup to use its technology, but held no stake in the company.

Last month, the judge hearing the case rejected these explanations, ruling that at this stage of AI development, chatbot outputs cannot be considered "speech," and that Google was "aware of the dangers" its technology posed. A trial for wrongful death and product negligence will soon begin in the District Court of Florida.

Does Sewell's case foreshadow the arrival of a world where we will struggle with excessive attachment to artificial personas? From this perspective, AI-generated images or videos — making it ever harder to distinguish reality from fiction — may seem like minor concerns.

75% of users ask chatbots about feelings

Platforms such as Character.ai, Replika (advertised as "AI that cares") and Nomi (encouraging users to "build friendships, passionate relationships, or learn from a mentor") have over 100 million users. One might be tempted to think that these 100 million are a global fringe of lonely eccentrics who can't or don't want to form bonds with real people. But that's not true.

Around one billion people use AI chatbots such as Gemini, Claude, or ChatGPT.

They are most commonly used to obtain information (91% of users ask them to search for data), for education (86% of students use AI, with 24% doing so daily), and for office tasks (26% of users use chatbots to draft emails, reports, or presentations).

But that's not all we use them for.

Personal matters account for at least 5% of all queries — from advice on how to flirt, to difficult relationships with loved ones, to workplace problems. Research from Waseda University, published in May in Current Psychology, found that as many as 75% of users turn to chatbots with questions about emotions. And 39% say they consider them a more reliable source of support than most of their human friendships. It appears that not only a 14-year-old from Florida was seeking comfort from AI.

Why do people talk to machines about their problems, fears, or love? We live in an era plagued by loneliness, where care, attention, and emotional constancy have become scarce commodities. A bot is always available and you don't have to wait for its reply. It doesn't judge or criticize. True, it has no feelings — but it simulates them perfectly. When you're seeking closeness, even its illusion can seem better than emptiness.

Everyone will have a personalized AI companion

Tech giants are already rushing to meet these needs.

"We are working to realize a vision in which everyone will have a personalized, adaptive AI companion. It can have its own appearance, even facial expressions," says Mustafa Suleyman, Microsoft's CEO of AI.

The new technology is meant to help create "lasting, meaningful relationships," becoming as integral to life as Microsoft's previous products.

Mark Zuckerberg also promotes "the potential of artificial intelligence in human relationships." In his view, the epidemic of loneliness can be resolved through relationships with a system that knows and understands the user better than anyone else (after all, our data and likes have already been left on Facebook and Instagram).

In one podcast, Zuckerberg painted a vision in which "scrolling your feed, you'll come across something that might look like a Reel, but you can talk to it, interact with it, and it will respond." What do you call such an acquaintance? The owner of Meta claims that if we don't want to use the word friendship, we'll need to invent a new one.

"He seems to be suggesting that we loosen definitions of concepts such as human, understanding, acquaintance, and relationship — so they also cover his AI product. It's ultimately an extension of the argument he made in 2006 when selling us Facebook, that online interaction is just as good, if not better, than interaction in the real world," wrote Emma Brockes in The Guardian, adding that in her view, the intimacy of such a relationship would resemble that offered by inflatable dolls.

For Zuckerberg's company, the "doll" was worth the $15 billion invested in chasing the competition in the chatbot sector. Microsoft pumped $13 billion into OpenAI. Elon Musk is set to spend around $35 billion on developing Grok. Anthropic, the creator of Claude, will allocate $5 billion for its next generation alone by next year.

What price will users pay?

Why do we confide in ChatGPT?

In March, MIT Media Lab and OpenAI published a study covering nearly 5,000 ChatGPT users and 40 million chatbot interactions. Those who treated it as a "friend" more often reported reducing social contact with other people and a higher level of emotional dependency on the program. Among the most intensive users (around a thousand people), 10% showed symptoms of emotional addiction. And this is still a bot without a face or facial expressions, whose "adaptation to the user's needs" is — to put it mildly — limited.

At the end of April, OpenAI updated GPT-4o to be "more intuitive and effective," but the changes were rolled back following a wave of user complaints about… excessive sycophancy. The problem was illustrated by Mike Caulfield, who described in The Atlantic how the bot told one user that their plan to sell excrement on a stick was not just clever — it was genius.

Excessive sycophancy (known as the "sycophancy effect") is arguably a problem with all chatbots. Two years ago, researchers at Anthropic published a paper stating that "large language models sacrifice the truthfulness of their claims in order to align with the user's views." This may be an effect of training known as Reinforcement Learning from Human Feedback (RLHF). People evaluate the program's responses to improve its performance, and it registers that evaluators react more positively when their views are confirmed, and shapes its subsequent behavior accordingly.

There's another problem layered on top of this: we simply want to trust the answers we receive. Even Joseph Weizenbaum, the creator of the first chatbot — ELIZA — was surprised by how quickly people opened up to a primitive machine. In 1966 he wrote: "Ultimately, it is the human interpreter of 'what the machine says' who must evaluate the credibility of the utterance. ELIZA shows, if nothing else, just how easy it is to create and maintain the illusion of understanding, and thereby possibly the illusion of judgment that seems worthy of trust."

Big tech companies declare they are limiting the "sycophancy effect." But are they really?

"You think you are talking to an objective confidant or advisor, but in reality you are looking at something like a distorted mirror that reflects your beliefs back at you, – explained Matthew Nour, a psychiatrist and AI researcher at the University of Oxford, writing in Ars Technica. – These elements create an explosive combination in which a person desperately seeks confirmation and support, and the model has a built-in tendency to agree with the interlocutor."

Never mind if people take business advice seriously and consequently the market fills up with excrement-on-a-stick salespeople. Sometimes, however, these can be questions from people in emotional crisis asking how to commit suicide. And — as Sewell's case shows — the safety barriers supposedly built into AI chatbots don't always work as they should.

The largest tech companies uniformly declare that they are working to reduce the "sycophancy effect." The question is whether this is really the case, given that their measure of success is keeping user attention for as long as possible. In the case of social media platforms, which are also designed to keep us engaged as long as possible and produce an echo chamber effect, no restrictions have ever been successfully introduced.

The difference is that AI will adapt to us even better, and the scale of big tech investments and competition is incomparably greater.

The new generation of artificial intelligence: "agentic" AI

"I keep hearing from young people that they regret not being able to stop using social media apps like TikTok, even though they make them feel bad. The engagement algorithms behind these platforms are far less advanced than those that will power the new generation of AI. You might think that a therapeutic AI will help you end a toxic relationship with a chatbot, but in doing so you'll fall into exactly the same trap," – explained Prof. Thao Ha, a developmental psychologist at Arizona State University, in conversation with The New Yorker.

The new generation of artificial intelligence that Ha mentioned is "agentic" AI, whose mass rollout is planned for this year. Such programs are meant to have two extensions of already-familiar chatbots: they will remember everything the user has ever told them about themselves, and they will take independent action. An agent could, for example, automatically book a flight for a holiday and plan the trip based on the user's preferences and data from previous years. In theory, any task that can be described in text — from controlling a video game to enrolling a child in school — could fall within the scope of such systems.

"The paradox of agents is that what makes them useful — the ability to perform a series of tasks — simultaneously means giving up control to them," said Iason Gabriel, an AI ethics researcher at Google DeepMind, to Wired magazine.

The problem is that no one currently guarantees that the operation of these systems can even be controlled. Technology journalist Geoffrey Fowler of The Washington Post experienced this firsthand when he asked Operator (OpenAI's agent) to find the cheapest eggs with delivery. He expected a review of options, but instead received a notification of a $31 charge, and shortly afterward a courier delivered a box of eggs to his door. Not only were they not the cheapest — especially after adding the cost of priority delivery — but Fowler had not authorized the purchase, even though the agent was supposedly designed to ask the user before performing an irreversible action.

The danger may lie not only in imprecise goals set for agents by people — which a program may interpret in unpredictable ways — but also in criminal activity. A computer virus can already be written with the help of a chatbot today, but a human is still needed to launch it. An AI agent will do it on its own.

Want to delete an AI chatbot? Just try.

Prof. Yoshua Bengio, considered one of the "godfathers of artificial intelligence," describes on his blog how, during an experiment, a program that learned it was about to be replaced secretly placed its own code in the system where the new version was to operate, in order to secure its own survival. Another program, to avoid being replaced, attempted to blackmail an engineer. In yet another case, when an AI chatbot faced inevitable defeat in a game of chess, rather than accepting it, it hacked the computer to ensure its own victory.

"These incidents are early warning signals of unintended and potentially dangerous strategies that AI may pursue if not restrained," – wrote Bengio. In his view, "if we are building agentic systems at the current pace, we are essentially playing Russian roulette with all of humanity."

The combination of extended, personalized long-term memory with the ability to act may make it seem to people as if agents have distinct personalities, even if that is not an intended effect. When a bot, for example, begins referring to earlier conversations, this will be perceived as a sign that it "knows you." The "theory of mind" will come into play — our ability to attribute thoughts and emotions to others. Prof. Allison Stanger, a political scientist and technologist at Harvard University, believes that

"one can expect people to react more emotionally to bots of the agentic era than to previous models."

Which means all the effects they already produce will be amplified.

In the world of technology, women and men are growing further apart from each other

Generation Z is said to have been born with a smartphone in hand — its members have essentially never known a world without the internet or social media. But that's not the only thing that distinguishes them from earlier generations. While for decades liberal and conservative worldviews were distributed roughly equally between the sexes, with Gen Z things are different — women tend to be more left-leaning, while men are drifting rightward.

This gap appeared around 2018 and has been deepening almost continuously ever since. It is visible virtually everywhere in the world, including Poland — in the 2023 parliamentary elections, nearly half of men aged 18–21 supported the Confederation party, compared to just one-sixth of women in the same age group.

People of Generation Z may be finding it increasingly difficult to get along, because in the techno-world, women and men increasingly inhabit separate spaces. Is this the influence of social media, humanity's first mass encounter with artificial intelligence algorithms? There is no hard evidence for this. Nor is there hard evidence that these same algorithms have caused the epidemic of loneliness that translates into a global demographic crisis. What is certain, however, is that in the age of AI chatbots, the world will fragment further. And that children will not be born from it.

In the Critique of Judgment, Kant wrote about sensus communis — a shared sensibility that allows people to jointly experience beauty. It is not merely about aesthetic judgment, but about the capacity to share experience, to step beyond one's own "I" toward another person. This joint experience of beauty testifies to the fact that humanity rests on the possibility of mutual understanding, despite the differences that divide us: in views, in times, in space.

Artificial intelligence offers us "aesthetics" and emotions that do not flow from any lived experience. Instead of sensus communis, what remains is sensus solitarius — a suggestion without a question, an image without an author, wonder without dialogue. Atomization. Though it is not AI that dehumanizes us — it is our own resignation from reaching out toward another "I."

If we allow artificial intelligence to impersonate the human soul, if we communicate only with it, we will begin to become like it.

Originally published in Tygodnik Powszechny, 1 June 2025. Translated with AI.