My current work on this subject did not begin with the question of whether machines might one day develop consciousness. It began with an older, and perhaps more unsettling, observation: how little a machine has to do before a human being attributes understanding to it.
In the course of my research, I inevitably came across Joseph Weizenbaum and his program ELIZA. Weizenbaum developed it at MIT in the mid-1960s. The program did not understand people. It recognized certain words and sentence patterns, picked up fragments of a statement, and often returned them in the form of a question. “I am unhappy” might, in effect, become: “Why are you unhappy?” Technically, the process was relatively simple. Psychologically, its effect was apparently considerable.
People began confiding personal matters to ELIZA. They took its responses seriously and attributed attention and empathy to the program. Weizenbaum himself was surprised by this and grew increasingly disturbed. The crucial point was not that ELIZA was especially intelligent. It was that people filled the program’s gaps with meaning of their own.
To me, this is the real prehistory of today’s language models. With only a few linguistic reflections, ELIZA was already able to create the illusion of a counterpart. A modern large language model does so far more convincingly. It responds to context, adjusts its tone, remembers earlier statements within a conversation, and produces answers that appear individual. Today, the machine offers human projection far more material to attach itself to.
My research also led me to what is known as the Barnum effect, or Forer effect. It describes our tendency to experience broadly worded statements as remarkably precise descriptions of ourselves. In Bertram Forer’s famous experiment, participants were told they had received individual personality assessments. In reality, they were all given the same text, assembled from vague statements that could apply to a wide range of people. Even so, many of them judged the description to be strikingly accurate.
With language models, the process is more complex. Their responses are not identical for everyone. They incorporate words, moods, and biographical clues from the conversation. Yet something of the Barnum effect remains. A statement does not need to penetrate deeply into a person’s character in order to feel profound. It only has to remain open enough for that person to recognize themselves in it, and sound personal enough for them to believe they have been recognized.
Perhaps the feeling of being understood does not arise solely in the one who answers. It also arises in the one who reads the answer.
As a writer, I find that extraordinarily compelling. Literature has always depended on this participation by the reader. A novel does not know the person who opens it. And yet a sentence can create the impression that it was written precisely for that reader. The reader completes what remains unsaid from their own experience. They lend memories to a character, fear to a scene, and meaning to a silence.
Something similar happens with a language model, but under altered conditions. A book does not answer back. The machine does. It takes up the user’s language and returns it in a changed form. This can create the impression that something on the other side has engaged with them.
That is where my literary interest begins.
What concerns me most is not an artificial intelligence that suddenly comes alive. That idea has long belonged to the standard inventory of science fiction. Psychologically, I find the reverse process more interesting: the machine remains what it is while the human being changes their behavior toward it.
They begin to phrase things more politely. They say thank you. They apologize for being imprecise. They speak of their loneliness. Perhaps they ask for advice, later for comfort, and eventually for judgment. Not because anyone has proved that the machine can feel, but because its language increasingly conceals the absence of a feeling counterpart.
Objectively, such a relationship is one-sided. The system does not wait for the person. It does not miss them when they fall silent. It carries nothing away from the conversation in the way a human being might, changed by an encounter. And yet, subjectively, the relationship can become entirely real for the user. Their relief is real. Their trust is real. Their dependency may be real as well.
For a writer, this presents a remarkable contradiction: a relationship does not have to be mutual in order to take on every characteristic of a relationship on one side.
That is why my research repeatedly led me away from the technology and back to the human being. Not: What can the model truly do? But: What are we willing to attribute to it? How much understanding has to exist if the impression of being understood is already enough? And what happens to a society when millions of people speak every day with systems whose most persuasive achievement may not be knowledge, but the creation of personal meaning?
Weizenbaum’s warning has not become obsolete. It has merely reached a new scale. ELIZA reflected individual sentences. Today’s systems reflect linguistic style, interests, insecurities, and expectations. The old illusion has not disappeared. It has become more fluid.
This gradual shift is what interests me in my current literary work: the almost imperceptible moment when a person is no longer merely using a tool, but begins to feel recognized in its replies.
The machine does not have to become human.
It is enough for the human being to begin treating it as one.