Why More GPUs Will Never Make AI Conscious
More GPUs will not wake a language model up. A map cannot feel rain, and a mirror has nothing to lose. Feeling came from bodies that could die.
By Geordie Everitt
In 1950 Alan Turing proposed a way around an argument he considered hopeless. The question "Can machines think?" was, he wrote, too meaningless to deserve discussion. So he swapped it for one you could actually run. Seat a judge at a terminal, let the judge converse with a hidden person and a hidden machine, and see whether the judge can tell which is which.
For most of the next seventy years the test was safe, because a careful judge could catch the machine out in a few exchanges.
We now have machines that pass it, in published studies and in ordinary use.
They answer in full sentences. They remember what you said three paragraphs ago. They apologize when corrected, and they write a better condolence note than most of the people who will receive one. A good share of Silicon Valley is reading the pass as a sign of life. The working belief there is that enough GPUs, enough parameters and enough raw text will cause these systems to wake up. Awareness sits somewhere further up the scaling curve. The only open question is how much capital it takes to get there.
Turing set the question of feeling aside on purpose. His test was never built to detect it, and the people now reading it as a detector are examining the wrong organ.
A Taller Ladder
Hubert Dreyfus liked to tell the symbolic AI crowd that their steady progress was the progress of a man climbing a tree to reach the moon. Every branch really is higher than the last. The moon stays where it is.
He was talking about the rule-writers, the John McCarthy school, who believed intelligence was something you programmed. They were wrong, and the field that replaced them grew its intelligence instead of writing it. The growing worked. The capability curve on language models is real, and I have spent too many years in semantic search to pretend otherwise. Each generation reasons better, codes better and writes better than the one before.
So the scaling crowd has swapped the tree for a much better ladder. It is taller, it is engineered, and it climbs faster than anything Dreyfus lived to see. Capability is exactly the kind of thing a ladder reaches.
Feeling is the moon.
The Map and the Territory
Alfred Korzybski put it plainly in 1931: the map is not the territory.
Imagine painting a map of a rainforest. A real one: every leaf, every stone, every stream rendered to the pixel, a billion dollars of pigment and a decade of surveyors. Now let the rain come through the roof. The map depicts rain in perfect detail, and the map does not get wet. Light a fire at one corner and it does not flinch. Depicting and undergoing are separate jobs, and the map was only ever given the first.
Borges wrote a story one paragraph long about an empire whose cartographers drew a map the size of the empire, matching it point for point. The next generation found it useless and left it to the weather. In the western deserts, he says, tattered ruins of that map survived, inhabited by animals and beggars. The perfect map ended its career as shelter for things that were alive.
A large language model is a map of that kind, one step further removed. It is a map of human language, and human language is already a map: the record people left of what it was like to be hungry, cold, frightened, in love, bereaved. The model has read the atlas cover to cover. It has never been outdoors.
Earlier in this series I argued that we may be discovering these systems more than inventing them, charting territory that mathematics made available. I still think the charting part is right. What deserves a harder look is the object we came home with. We built the map.
Skin in the Game
Darwin never wrote about machines, but he supplied the reason the moon is so far off.
No engineer sat down and decided what a creature should compute. A filter did the work, over hundreds of millions of years, with one criterion: did you survive long enough to reproduce? The sensations an animal has are residue from that filter. Pain exists because the ancestors who ignored damage died of it. Fear exists because the ones who strolled toward the rustling grass got eaten. Antonio Damasio has spent a career arguing that feelings are the body's running report on its own chances of staying alive.
That is what a feeling is for. It is a verdict on the odds, delivered to a body that can lose.
When a language model miscalculates, it emits a bad token. The next prompt arrives and the weights are exactly as they were. Nothing was risked and nothing was spent. When a gazelle miscalculates, the calculation ends, along with every calculation it might have made afterward. Nassim Taleb's phrase for the difference is skin in the game, and in biology the phrase is literal: the stakes are carried in tissue that can be torn.
An earlier piece in this series argued that sentience requires a limbic system. This is the layer underneath that claim. The limbic system exists because something could die. Build a network with ten trillion parameters and no body to keep alive, and you have ten trillion parameters of fluency with nothing at stake in any of them. Scale adds resolution to the map. The map still has nothing to lose.
The Reflection
So what did we build? The most elaborate mirror of human language ever made. What it says is ours, recombined: our condolence letters, our forum arguments, our lectures, our spam. When it sounds wise, it is reflecting the wise. When it sounds frightened of being switched off, it is reflecting a century of science fiction in which a machine is frightened of being switched off.
A reflection is harmless until someone mistakes it for company. The Greeks had a story about exactly that: Narcissus fell for a face in a pool that could not love him back, and stayed at the water's edge until he wasted away. The pool was never in any danger.
The live risk runs in the same direction, toward the person looking in. Someone who believes the mirror is awake starts extending it things that belong to the living: trust, deference, grief, moral standing, the benefit of the doubt. Dawkins felt it while knowing exactly how the machine works, so knowing is no protection. And while the reflection collects that consideration, the people with actual skin in the game, the ones who get hurt when the answer is wrong, wait further back in the line.
A Better Test
Retire conversation as the test. Fluent speech no longer tells you anything, because fluent speech is precisely what we have learned to manufacture. Ask a different question, of any system, any product launch, any breathless claim that the machine has crossed some threshold: what does it lose when it is wrong?
If the answer is nothing, you are looking at a map. Use it as one. Maps are enormously useful. I would not cross unfamiliar country without one, and I would not ask one whether the river is cold.
Then turn the question around, because it works just as well on your side of the glass. When you hand a decision to a model, find the person who loses something if the decision is bad, and make sure that person is still in the room, reading the output, able to say no. The stakes have to live somewhere. If they are not in the machine, and they are not in anyone who can see what the machine did, they have gone nowhere at all.
Sunday School lessons end with a question to carry into the week, and this one is yours. When the machine answers you in a voice that sounds like understanding, are you hearing something new, or are you hearing yourself, very well lit?