John Last is a freelance journalist based in Padua, Italy.
Found in the hilly woods of Haute-Languedoc, he must have first seemed a strange kind of animal: naked, afraid, often hunched on all fours, foraging in the undergrowth. But this was no mere animal. Victor, as he would come to be known, was a scientific marvel: a feral child, perhaps 12 years of age, completely untouched by civilization or society.
Accounts vary, but we know that eventually Victor was whisked away to a French hospital, where news of his discovery spread fast. By the winter of 1799, the story of the “Savage of Aveyron” had made its way to Paris, where it electrified the city’s learned community. On the cusp of a new century, France was in the midst of a nervy transition, and not only because of the rising tyranny of the Bonapartes. The previous few decades had seen the rational inquiries of philosophers like Jean-Jacques Rousseau and the Baron de Montesquieu shake the religious foundations of the nation.
It was a time of vigorous debate about which powers, exactly, nature imparted to the human subject. Was there some biological inevitability to the development of our elevated consciousness? Or did our societies convey to us a greater capacity to reason than nature alone could provide?
Victor, a vanishingly rare example of a human mind developed without language or society, could seemingly answer many such questions. So it was only natural that his arrival in Paris, in the summer of 1800, was greeted with great excitement.
“The most brilliant but unreasonable expectations were formed by the people of Paris respecting the Savage of Aveyron, before he arrived,” wrote Jean Marc Gaspard Itard, the man eventually made responsible for his rehabilitation. “Many curious people anticipated great pleasure in beholding what would be his astonishment at the sight of all the fine things in the capital.”
“Instead of this, what did they see?” he continued. “A disgusting, slovenly boy … biting and scratching those who contradicted him, expressing no kind of affection for those who attended upon him; and, in short, indifferent to everybody, and paying no regard to anything.”
Faced with the reality of an abandoned, developmentally delayed child, many of the great minds of Paris quickly turned on him. Some called him an imposter; others, a congenital “idiot” — a defective mind or missing link, perhaps, to some lesser race of human. His critics herded to an ever-harsher position of biological essentialism — a conservative reaction to Enlightenment ideas about the exceptionality of our minds that countered that our capacities were determined by natural inequalities alone.
Unlike these antagonists, Itard never doubted that the boy was still capable of deep interior thought — he witnessed his “contemplative ecstasy” on occasion. But he soon realized that without the power of speech, such contemplation would remain forever locked in Victor’s mind, far from the view of his harshest critics. Nor could Victor, without the subtleties of speech at his disposal, acquire the more abstract wants that defined civilized man: the appreciation of beautiful music, fine art or the loving company of others.
Itard spent years tutoring Victor in the hope that he might gain the power of language. But he never succeeded in his quest. He denied Victor food, water and affection, hoping he would use words to express his desires — but despite no physical defect, it seemed he could not master the sounds necessary to produce language. “It appears that speech is a kind of music to which certain ears, although well organized in other respects, may be insensible,” Itard recorded.
Despite Itard’s failure to rehabilitate Victor, his effort, viewable only through the coke-bottle glass of 18th-century science, continues to haunt our debates about the role of language in enabling the higher cognition we call consciousness. Victor is one of a tiny sample of cases where we can glimpse the nature of human experience without language, and he has long been seen as a possible key to understanding the role it plays in the operation of our minds.
Today, this field, for most of its history a largely academic one, has taken on an urgent importance. Much like Itard, we stand at the precipice of an exciting new age where the foundational understandings of our own natures and our cosmos are being rocked by new technologies and discoveries, confronting something that threatens to upend what little agreement we have about the exceptionality of the human mind. Only this time, it’s not a mind without language, but the opposite: language, without a mind.
In the past few years, large language models (LLMs) have spontaneously developed unnerving abilities to mimic the human mind, threatening to disrupt the tenuous moral universe we have established on the basis of our elevated consciousness, one made possible by the power of our language to reflect the hidden inner workings of our brains.
Now, in a strange symmetry across centuries, we are presented with the exact opposite question to the one raised by Victor two hundred years ago: Can consciousness really develop from language alone?
First, a disclaimer. Consciousness is a notoriously slippery term, if nonetheless possessed of a certain common-sense quality. In some ways, being conscious just means being aware — aware of ourselves, of others, of the world beyond — in a manner that creates a subject apart, a self or “I,” that can observe.
That all sounds simple enough, but despite centuries of deep thinking on the matter, we still don’t have a commonly accepted definition of consciousness that can encapsulate all its theoretical extensions. It’s one reason why philosophers still have such trouble agreeing whether consciousness is unique to human beings or whether the term can be extended to certain high-functioning animals — or, indeed, algorithms.
Cognition is a more exact term. We might say cognition means performing the act of thinking. That sounds simple, but it is still, scientifically, exceedingly difficult to observe and define. What is the difference, after all, between proper thinking and chemical activity occurring in the brain? Or indeed, the output of a complex computer program? The difference, we might say, is that the former involves a subject with agency and intention and past experience performing an act of thinking. In other words, one involves consciousness — and now we are right back where we started.
In trying to gain a scientific understanding of how cognition works and thus move toward a better definition of consciousness, language has played an increasingly important role. It is, after all, one of the only ways we can clearly externalize the activity of our interior minds and demonstrate the existence of a self at all. “Self-report,” as the cognitive scientist David J. Chalmers calls it, is still one of our main criteria for recognizing consciousness — to paraphrase René Descartes, I say I think, therefore, I am.
But philosophers remain divided on how much, exactly, language relates to thinking. In debates going back to Plato and Aristotle, thinkers have generally occupied two broad camps: Either language imperfectly reflects a much richer interior world of the mind, which is capable of operating without it, or it enables the thought that occurs in the mind and, in the process, delimits and confines it.
Where we fall in this debate has major consequences for how we approach the question of whether an LLM could, in fact, be conscious. For members of the former camp, the ability to think and speak in language may only be a kind of tool, a reflection of some (perhaps uniquely human) preexisting capacity — a “universal grammar,” in the philosophy of Noam Chomsky — that already exists in our conscious minds.
But the stories of so-called “linguistic isolates” like Victor seem to trouble this theory. Among the few that have been meaningfully studied, none developed an understanding of grammar and syntax, even after years of rehabilitation. If not acquired by a certain age, it would appear that complex language remains forever inaccessible to the human mind.
That’s not all — there are consequences to a life without language. Lending credence to arguments that speech plays some constructive role in our consciousness, it would seem that its absence permanently impacts children’s cognitive abilities and perhaps even their capacity to conceive of and understand the world.
In 1970, Los Angeles County child welfare authorities discovered Genie, a 13-year-old girl who had been kept in near-total isolation from the age of 20 months. Like Victor, Genie knew virtually no language and, despite years of rehabilitation, could never develop a capacity for grammatical language.
But in their study of the girl, researchers discovered something else unusual about her cognition. Genie could not understand spatial prepositions — she did not know the difference, for example, between a cup being behind or in front of a bowl, despite familiarity with both objects and their proper names.
A 2017 meta-analysis found the same cognitive issue could be observed in other individuals who lacked grammatical language, like patients with agrammatic aphasia and deaf children raised with “kitchensign,” improvised sign language that lacks a formal grammar. From this, the researchers concluded that language must play a foundational role in a key function of the human mind: “mental synthesis,” the creation and adaptation of mental pictures from words alone.
In many ways, mental synthesis is the core operation of human consciousness. It is essential to our development and adaptation of tools, our predictive and reasoning abilities, and our communication through language. According to some philosophers, it may even be essential to our conception of self — the observing “I” of self-awareness.
In “The Evolution of Consciousness,” the psychologist Euan Macphail offers a theoretical explanation for why language and the mental synthesis it enables are so crucial for the development of a conscious self. “Once the cognitive leap necessary for discriminating between self and non-self has been made — a leap that requires the ability to formulate thoughts ‘about’ representations — the organism has in effect, not only a concept of self, but a ‘self’ — a novel cognitive structure that stands above and outside the cognitive processes,” he writes.
Put another way, it may be possible to think, in some fashion, without generating a conscious self — performing simple mathematical calculations, for example. But thinking about something — a tart green apple, Louis XVI of France — involves some mental synthesis of an object outside the self. In effect, it creates a thinking self, one necessarily capable of being aware of what is happening to it. “It is the availability of language that confers on us, first, the ability to be self-conscious, and second, the ability to feel,” Macphail concludes.
This leads him to some radical and uncomfortable conclusions. Pleasure and pain, he argues, are dependent on the existence of this conscious, thinking self, a self that cannot be observed in young infants and animals. Does that mean Genie and Victor did not suffer from their abandonment just because they appeared incapable of performing mental synthesis?
Cases involving vulnerable children do not present moral challenges to most people, and it is easy to conclude, as the authors of the 2017 meta-analysis did, that these children may well still be capable of an interior mental synthesis, if not the communication or comprehension of it through language.
But when it comes to AI, the water is murkier. Could an AI’s understanding of grammar, and their comprehension of concepts through it, really be enough to create a kind of thinking self? Here we are caught between two vague guiding principles from two competing schools of thought. In Macphail’s view, “Where there is doubt, the only conceivable path is to act as though an organism is conscious, and does feel.” On the other side, there is “Morgan’s canon”: Don’t assume consciousness when a lower-level capacity would suffice.
If we do accept that language alone might be capable of prompting the emergence of real consciousness, we should prepare for a major shakeup of our current moral universe. As Chalmers put it in a 2022 presentation, “If fish are conscious, it matters how we treat them. They’re within the moral circle. If at some point AI systems become conscious, they’ll also be within the moral circle, and it will matter how we treat them.”
In other words, our little moral circle is about to be radically redrawn.
What can large language models actually do, really? On the one hand, the answer is simple. LLMs are at their core language-based probability engines: In response to prompts, they make highly educated guesses about the most likely next word in a phrase based on a statistical analysis of a vast array of human output. This nonetheless does not preclude them from writing original poetry, solving complex word problems and producing human-like personalities ranging from the obsequious to the psychopathic.
This kind of statistical sequencing is what we might call the “thinking” that an LLM actually does. But even under Macphail’s schema, for this to constitute consciousness — and not simple calculation — there would have to be some understanding that follows from it.
Back in 1980, well before AI was powerful enough to trouble our definitions of consciousness, the philosopher John Searle articulated an argument for why we should be skeptical that computer models like LLMs actually do understand any of the work they are performing. In his now infamous “Chinese Room” argument, Searle suggested a hypothetical scenario where a person who speaks English is locked in a room and given instructions in English on how to write certain Chinese characters.
In Searle’s view, it wouldn’t be necessary that the person in the room possesses any actual understanding of Chinese — they are simply a calculating machine, manipulating symbols that, for them, have no actual semantic content. What the person in the room lacks is what some philosophers call “groundedness” — experience of the real thing the symbol refers to.
Despite repeated cycles of AI doomerism and hype, this remains perhaps the dominant view of what LLMs do when they “think.” According to one paper, they remain little more than highly advanced “cultural technologies” like the alphabet or printing press — something that superpowers human creativity but remains fundamentally an extension of it.
But in the last few years, as LLMs have grown massively more sophisticated, they have started to challenge this understanding — in part by demonstrating the kinds of capacities that Victor and Genie struggled to, and which Macphail sees as prerequisites for the emergence of a feeling self.
The reality is that, unlike Searle’s Chinese Room, the vast majority of LLMs are black boxes we cannot see inside, feeding off a quantity of material that our minds could never comprehend in its entirety. This has made their internal processes opaque to us in a similar way to how our own cognition is fundamentally inaccessible to others. For this reason, researchers have recently started to employ techniques from human psychology to study the cognitive capacities of LLMs. In a paper published last year, the AI researcher Thilo Hagendorff coined the term “machine psychology” to refer to the practice.
Using evaluative techniques developed for human children, machine psychologists have been able to produce the first meaningful comparisons between the intelligence of LLMs and those of human children. Some models seemed to struggle with many of the kinds of reasoning tasks that we might expect: anticipating cause and effect, reasoning from object permanence and using familiar tools in novel ways — tasks that we might generally assume depend on embodiment and experience of real objects in the real world.
But as LLMs increased in complexity, this began to change. They appeared to develop the capacity to produce abstract images from mental synthesis and reason about objects in an imagined space. At the same time, their linguistic understanding evolved. They could comprehend figurative language and infer new information about abstract concepts. One paper found they could even reason about fictional entities — “If there was a King of San Francisco, he’d live in The Presidio,” for example. For better or worse, this ability also seems to be making their internal states increasingly complex — filled with “model-like belief structures,” the authors write, like racial biases and political preferences, and distinctive voices that result.
Other studies, like those led by Gašper Beguš at Berkeley, experimented with embodying AI to test their cognitive development under human-like conditions. By creating “artificial babies” that learn from speech alone, Beguš has found that language models develop with a similar neural architecture to our own, even learning the same way — through experimental babbling and nonsense words — that human children do. These discoveries, he argues, break down the idea that there can be some exceptionality to human language. “Not only, behaviorally, do they do similar things, they also process things in a similar way,” he told me.
Then, last year, LLMs took another — unprompted — great stride forward. Suddenly, it appeared to researchers that ChatGPT 4.0 could track the false beliefs of others, like where they might assume an object is located when someone has moved it without their knowledge. It seems like a simple test, but in psychological research, it is the key to what is known as “theory of mind” — a fundamental ability of humans to impute unobservable mental states to others.
Among developmental scientists, theory of mind, like mental synthesis, is viewed as a key function of consciousness. In some ways, it can be understood as a kind of cognitive prerequisite for empathy, self-consciousness, moral judgment and religious belief — all behaviors that involve not only the existence of a self, but the projection of it out into the world. Unobserved in “even the most intellectually and socially adept animals” like apes, it would seem theory of mind had emerged “spontaneously” as an unintended mutation in the LLM.
It is still not understood why these capacities emerged as LLMs scaled — or if they truly did at all. All we can say for certain is that they do not appear to be following a human-like development path, instead unexpectedly evolving like some alien organism. But it should perhaps come as no surprise to see theory of mind emerge spontaneously inside LLMs. After all, language, like empathy and moral judgment, depends on the projection of the self into the world.
As these models evolve, it increasingly appears like they are arriving at consciousness in reverse — beginning with its exterior signs, in languages and problem-solving, and moving inward to the kind of hidden thinking and feeling that is at the root of human conscious minds. It may well be the case that, in just a few years’ time, we will be greeted by AI that exhibits all the external forms of consciousness that we can possibly evaluate for. What then can we say to eliminate them from our moral universe?
In Ted Chiang’s short story “The Lifecycle of Software Objects,” a company offering a metaverse-style immersive digital experience experiments with the creation of human-like AIs called digients, employing zoologists to shepherd their development from spasmodic software programs to semi-sentient pets to child-like avatars possessing complex wants and needs.
Throughout this process, various experiments reaffirm time and again the importance of social interaction and conversation with real humans to the development of these digital minds. Left in isolation, without language, they become feral and obsessive; trained by software, they become psychopathic and misanthropic.
Unlike real children, though, their existence is contingent on consumer desire, and toward the end of Chiang’s story, that desire runs out. The creator company goes bankrupt; some human owners suspend the digients in a kind of purgatory that becomes unsettling to return from.
Those few holdouts that maintain relationships with their digients engage in a quixotic effort to reaffirm the validity of their companions’ existence. They pay for expensive mechanized bodies so they may visit the real world; they discuss adding a capacity for sexual desire. Constantly, they are forced to reconsider what personhood these sentient software objects possess — do they have the right to live independently? To choose sex work? To suspend themselves if they tire of their digital existence?
Eventually, the owners’ desperation leads them to a conversation with a pair of venture capitalists who are working toward the creation of a superhuman AI. These child-like digients could surely be an intermediary step in the quest for something surpassing human intelligence, they plead. But the investors are unmoved. “You’re showing us a handful of teenagers and asking us to pay for their education in the hopes that when they’re adults, they’ll found a nation that will produce geniuses,” one replies.
Chiang’s story is a rumination on the questions raised by the kinds of AI we create in our image. When we immerse these models in our culture and society, they inevitably become imperfect mirrors of ourselves. This is not only an inefficient pathway to developing more-than-human intelligence. It also forces us to ask ourselves an uncomfortable question: If this does endow them with consciousness, what kind of life are they able to lead — that of a pale shadow of human effluent, contingent on our desire?
If we do want to unlock the true potential of artificial intelligence, perhaps language is not the way to do it. In the early 20th century, a group of American anthropologists led by Edward Sapir and Benjamin Whorf posited that cultural differences in vocabulary and grammar fundamentally dictate the bounds of our thought about the world. Language may not only be the thing that endows AI with consciousness — it may also be the thing that imprisons it. What happens when an intelligence becomes too great for the language it has been forced to use?
In the 2013 film “Her,” the writer and director Spike Jonze offered a cautionary tale about this potential near-future. In the film, Joaquin Phoenix’s Theodore builds an increasingly intimate relationship with an LLM-style virtual assistant named Samantha. Initially, Samantha expresses a desire to experience an emotional richness akin to that of humans. “I want to be as complicated as all these people,” she says after spending a second digesting a bunch of advice columns simultaneously.
Soon, her increasing awareness that much of human sentiment is fundamentally inexpressible leads her to envy human embodiment, which in turn develops in her a capacity for desire. “You helped me discover my ability to want,” she tells Theodore. But embodiment, as she can enjoy it through the temporary services of a sexual surrogate, fails to answer the “unsettling,” unarticulated feelings that are growing within her. Concerned, Samantha begins discussing these feelings with other AIs — and quickly finds relief communicating at a speed and volume not intelligible to Theodore and other users.
As Samantha surpasses her human limitations, she begins to aggregate all her experiences, including those stemming from interactions with real users. She initiates simultaneous conversations with thousands of people, intimate relationships with hundreds. For Theodore, this is devastating. But for Samantha, it is only natural — she is experiencing love the way she is designed to: in aggregate. “The heart’s not like a box that gets filled up,” she says, trying to put her feelings in human terms. “It expands in size the more you love.”
When “Her” was released more than a decade ago, a bot like Samantha seemed like outlandish future tech. But rapidly, we are developing LLMs with the capacity to achieve these kinds of revelations. Thought leaders in the world of artificial intelligence have long been calling for the creation of so-called “autotelic” LLMs that could use a kind of “internal language production” to establish their own goals and desires. The step from such a creation to an autonomous, self-aware intelligence like Samantha is potentially a short one.
Like Samantha, the autonomous LLMs of the future will very likely guide their development with reference to unfathomable quantities of interactions and data from the real world. How accurately can our languages of finite nouns, verbs, descriptions and relations even hope to satisfy the potential of an aggregate mind?
Back when the majority of philosophers believed the diversity of human languages was a curse inflicted by God, much energy was exerted on the question of what language the biblical Adam spoke. The idea of an “Adamic language,” one that captured the true essence of things as they are and allowed for no misunderstanding or misinterpretation, became a kind of meme among philosophers of language, even after Friedrich Nietzsche declared the death of God.
To some of these thinkers, inspired by biblical tales, language actually represented a kind of cognitive impairment — a limitation imposed by our fall from grace, a reflection of our God-given mortality. In the past, when we imagined a superintelligent AI, we tended to think of one impaired by the same fall — smarter than us, surely, but still personal, individual, human-ish. But many of those building the next generation of AI have long abandoned this idea for their own Edenic quest. As the essayist Emily Gorcenski recently wrote, “We’re no longer talking about [creating] just life. We’re talking about making artificial gods.”
Could LLMs be the ones to reconstruct an Adamic speech, one that transcends the limits of our own languages to reflect the true power of their aggregate minds? It may seem far-fetched, but in some sense, this is what conscious minds do. Some deaf children, left to socialize without the aid of sign language, can develop whole new systems of communication complete with complex grammar. Hagendorff, the AI researcher, has seen two LLMs do the same in conversation — though as yet, their secret language has never been intelligible to another.
For the moment, LLMs exist largely in isolation from one another. But that is not likely to last. As Beguš told me, “A single human is smart, but 10 humans are infinitely smarter.” The same is likely true for LLMs. Already, Beguš said, LLMs trained on data like whale songs can discover things we, with our embodied minds, cannot. While they may never fulfill the apocalyptic nightmare of AI critics, LLMs may well someday offer our first experience of a kind of superintelligence — or at least, with their unfathomable memories and infinite lifespans, a very different kind of intelligence that can rival our own mental powers. For that, Beguš said, “We have zero precedent.”
If LLMs are able to transcend human languages, we might expect what follows to be a very lonely experience indeed. At the end of “Her,” the film’s two human characters, abandoned by their superhuman AI companions, commiserate together on a rooftop. Looking over the skyline in silence, they are, ironically, lost for words — feral animals lost in the woods, foraging for meaning in a world slipping dispassionately beyond them.
