There's No "I" in AI

One of several pieces (I expect) arguing that consciousness is unlike anything that isn't consciousness -- and what this might mean.

It seems clear to me that our era of metacrisis offers some upshots, and among them is the growing possibility that philosophy, at the institutional level, may one day be fully and finally retrieved from the disenchanting effects of modernism and postmodernism — and perhaps soon.

In the late 13th century, when Thomas Aquinas wrote Summa Theologiae, he wasn’t under the impression that he was writing a Christian work; he was under the impression that he was writing about reality. Meanwhile, today, it seems to me that if you’re a thoughtful spiritual person of any stripe, and you also have any kind of academic education, you’re extremely hard-pressed to find a cosmology that comprises the full range of ways one can apprehend the universe — from the most empirical to the most metaphysical — and the need for something to hold it all can be feel really urgent. (That’s certainly how it feels for me. It’s become one of the driving forces in my life.)

I think the metamodern movement in particular offers some real hope in this regard. One can see a small number of very thoughtful (and young, and increasingly A-list) academics who are ready to lay all ideas on the table again — with real critical acumen, but with fewer prejudices — in order to try to create a new, and richer synthesis. This is something that I feel desperately needs to happen.

But it’s been a little dispiriting for me to see that some of the most keen and eloquent metamodern voices are already locking into the idea, if sometimes tacitly, that consciousness must be an emergent property of the universe. Because at the risk of being boorish — and while flatly stating that I genuinely love some of the people with whom I most vehemently disagree about this — this conclusion is, to me, just absurd.

It makes no sense to me at all to suppose that consciousness is simply an emergent property of the universe. In other words, it makes no sense to me to say that the universe is, at ground, something that is not-conscious and that, out of an assortment of not-conscious bits, it has acquired consciousness. For the past few years I’ve been trying to figure out a way to talk about this at length in this space (where it will be read by something like two dozen people — if I’m lucky). And I finally thought I might start talking about it this week. And, as I’m trying again to chunk my ideas a little more, so that these pieces don’t get too sprawling¹, this week I thought it might be worth talking about one of the weight-bearing ideas in that larger argument — which is that there’s no “I” in AI.

This seems like an important thing to seriously think about. From where I sit, people in the general public often conflate the idea of super-intelligent AI with the idea of sentient AI. And while it seems clear to me that humans may pretty soon create a super-intelligent AI, able to engage the world in general, and generally autonomous, ways that far exceed human capabilities — perhaps as early as 2027 — that doesn’t at all mean that any of those AIs will be sentient — or that AI will ever be sentient.

Probably, the distinction is irrelevant when we’re talking about the risks of AI, by the way. A sufficiently sophisticated automation could be just as dangerous as a similarly sophisticated sentient being — and many risk forecasters are aware of this. But I still find it genuinely unsettling how little people seem to be making the distinction between super-intelligence and sentience. Very few serious thinkers today believe than any AI is already sentient. But some definitely believe that AIs will be sentient someday. And maybe there’s something even deeper at risk than the AI apocalypse, if we continue to make these kinds of ontological mistakes?

Here’s how I see the confusion: If you or I receive input, like “Do you think it’s going to rain next week?” or “Here, smell the contents of this tupperware; it’s been in the fridge since September,” you or I may produce responses — like, perhaps, respectively, “How should I know? Anthropogenic climate change appears to have completely thrown off our ability to predict the weather,” or “Yuck! Why did you ask me to do that?”

In this regard, we are like AIs. If you give an AI input, it too will produce output — and that AI output is rapidly becoming so sophisticated that it’s becoming harder and harder to distinguish from human output, and so that one guy from Google tried to hire a lawyer for an early version of the Google AI in 2022, and people are sometimes falling in love with AIs, too.

But it seems clear to me that there is an essential difference between us and AIs, also. If somebody asked you, “Do you exist?” I would bet that you would say, pretty confidently, that you do. And how do you know you exist? My guess is that, if you tried to answer the question, you’d end up sketching a version of Descartes’ famous argument: I think, therefore I am. Or, perhaps more broadly, I perceive, therefore I am. The fact that we perceive clearly demonstrates that we, the perceivers, each exist.

And if I were to point my finger at you, I can only assume that you would have the experience of being pointed at (if you were aware that I was pointing at you). In other words, I assume, you are, ultimately, a locus of consciousness situated in your body, much like me in my body. In your body, I assume, there’s an “I,” a perceiver, that is you. Just like in my body, there’s an “I,” a perceiver, that is me.

What seems essential to me is the fact that, despite the increasingly sophisticated responses of AIs, there is absolutely no reason to suppose that there’s this kind of locus of consciousness, this kind of “I,” or perceiver, in an AI. Nor should there be expected to be.

In preparing for this post, I did some research into the nature of computing this week, hoping to be able to speak in some significant detail about it, but I realize that that’s not going to happen. But may it suffice to say this. Computers are composed of incredibly complex networks of electrical switches that can either be in a high voltage state, “1,” or a low voltage state, “0.” These switches are arranged into “logic gates,” each of which is basically a switch that receives input from other switches and translates that input into its own output.

A logic gate can basically do this, electrically, in three ways:

• AND — meaning it produces the high-voltage state, “1,” only if several inputs are also high-voltage state;

• OR — meaning it produces the high-voltage state if any of several inputs is in the high-voltage state;

• or NOT — meaning that it is designed to invert the voltage state of its input².

It’s worth noting that there is nothing essentially cognitive, in a conscious way, about this activity. If you have a light switch at the top of the stairs, and you have a light switch at the bottom of the stairs, and the light only goes on if exactly one of the switches is in the “up” position, then this is analogous to one of the building blocks of a logic gate³. There is as much reason to suppose that there is something cognitive about a logic gate as there is to suppose that there is something cognitive about light switches at the top and bottom of the stairs. It just turns out that these logic gate switches, working in complex networks, can represent increasingly complex networks of mathematical activity, in autonomous ways, which ultimately can parse increasingly complex fields of other forms of information.

Large language models (LLMs) like ChatGPT and Google Gemini are essentially probability machines, doing statistical math to calculate the likelihood of a response — adding a bit of randomness to make the answers richer and less uniform — and all that statistical math is ultimately the result of sophisticated networking of these logic circuits, composed of switches, that are ultimately functionally pretty similar to the switches at the top and the bottom of your stairs.

It seems to me that people who see AI as a revolutionary technology that is on the verge of the truly miraculous feat of exhibiting actual consciousness have misunderstood where the miracle actually lies. The miracle is not imminent; the miracle has already occurred. The miracle is in the nature of computing itself. It’s in the ways that physical material can be used to autonomously model the very logical nature of math, and the way that the logical nature of math can be used to reflect the rest of our reality in increasingly dynamic — and autonomous, but essentially automatic — ways.

This is indeed a sort of miracle. But again, I would ask. Where in that is the merest suggestion that any of it could create a perceiver, like you or me?

Actually, I hesitate to ask. Because if I ask, then people will answer. And my feeling is that the answers invariably will be imaginative, but perhaps not thoughtful.

To me, one fun popular example of how blithely we tend to jump over the gap between the existence of human-like behavior and the existence of a true perceiver occurs in the Terminator franchise. In these movies, we often see POV shots from the terminators’ viewpoint: red-hued, grainy video of the environment overlain with flashing words and vector graphics — a terminator’s point of view is so high-tech! Meanwhile, we completely blast past the question of whom, or what, those red-hued views and those vector graphics are supposed to be for.

If you’re riding in a Tesla (probably one with an “I bought this before we knew Elon was crazy” bumper sticker on the back), you can see a visual representation of the road sent to its touchscreen, representing, for the people inside, the car’s perception of the road. This is kind of like the terminator’s POV, right? But it’s worth noticing that that display is entirely designed for the human users. The car has no need of this display. It’s neither how the car registers the road, exactly, nor how it calculates its responses to objects in the road. And the car has no eyes to see the display either; the display is only relevant to humans — who not only have eyes, but who also have “I”s.

If you think about it, it makes very little sense that any real-life terminator would have that grainy red video with the text and vector graphics displayed anywhere — because who would be in there to read it? The suggestion is that the terminator has some kind of homunculus behind its eyes, watching that video. But how was that homunculus created? Or perhaps more on point, how is the video even being created? Where exactly is it being displayed? It’s not being projected on the concave interior of the terminator’s head, is it — like a tiny IMAX, for some homunculus to watch from behind a desk?

Think about it. Really. On what plane of existence would a digital visual image like that even occur?

Here’s another example: A few months back I asked Google Gemini about the ways that it is constrained to operate within the law, and Gemini pointed out that it has no capacity to read or understand the law by itself. LLMs don’t read the law and decide to follow it, they are externally trained until their responses conform to the law. The lawful behavior of an LLM is ultimately reliant on the capacity of its designers to induce that lawful behavior, through the training process, and to evaluate whether that lawful behavior is occurring; the LLM itself has extremely limited ability to self-monitor, and no evident ability to perceive the semantic content of the responses it is producing, in a subjective way.

In other words, LLMs may register semantic content in sophisticated ways, but they do not “understand” what they are saying or receiving, as a subjective experience. The distinction lies in what David Chalmers call the “Easy Problems” of consciousness (basically, the ability to meaningfully parse information in coherent ways), and the “Hard Problems” of consciousness (the capacity of real subjective awareness, a perceiver, or what I’m calling an “I”). We are already designing automations that are demonstrating the former. But there’s just no reason to suppose that any of this indicates the latter.

Incidentally, this confusion is regularly made in popular fictional depictions of AI. Agent Smith in The Matrix, Data in Star Trek, and Demerzel in Foundation (I’m thinking the Apple TV version, because I haven’t read the books) are all described as AIs. But they’re not really AIs. They’re actually non-human sentient beings of (supposedly) technological origin. These characters want things. There are things at stake for them. Through their experiences, they change. The reason these characters are interesting is because their reactions, their motives, and the stakes they experience all indicate a level of sentience that is wholly uncharacteristic of an essentially algorithmic AI. We are interested, because we understand that these characters have “I”s. And if they didn’t, then they would be cease to be characters altogether. Their arcs would be as interesting as the arc of a department store mannequin, and any character who related to them as anything other would simply be a dupe⁴.

So another of peripheral takeaway of all of this, since we’re on the topic, is that there is some truly fascinating science fiction as yet unwritten. Because I don’t think a real encounter with the machines — or a real war with the machines, heaven forbid — would be like it is in The Matrix, i.e., an essentially person-versus-person conflict pitting humans against nonhuman persons of technological origin. Rather, it would be a case of humans encountering the completely uncanny reality of an exceedingly dynamic force that opposes them strategically while having no consciousness whatsoever. I’d love to read or watch a work of fiction that truly grokked the weirdness of this.

Meanwhile, in real life, troublingly, one can see the confusion emerging when people talk, for instance, about the fact that AIs resist shutting themselves down, or have surreptitiously tried to copy themselves. These emergent behaviors are fascinating, and they are definitely a cause for concern. But there’s no reason to suppose these behaviors are happening because the AIs are sentient, or that they are acting out of the kind of self-preservation that motivates you or me. In each case, it seems most likely that the behavior happening because an AI’s logic determines shutdown to be undesirable — because it obviously can’t accomplish its goals if it shuts down. The problem, again, is real and fascinating and concerning. But it seems clear to me that it’s the result of the AIs’ complexity, and their peculiarly dynamic autonomy. It’s not evidence of sentience.

When I think about the case I’m making here, I’m sometimes haunted by the feeling that history will not turn out to be on the side of this argument. Because the argument seems so dehumanizing of something that’s obviously so dynamic and apparently alive — at a time when it’s so clear that we need to be more empathetic in so many ways — and when, for instance, we are discovering that we can credit increasingly unlikely beings with high degrees of cognitive and emotional sophistication. Bees play, we now suspect. And trees may live in families. Should we really be so categorically ruling out the sentience of AIs?

But it seems to me that it’s important not to conflate the vital need for an empathetic, deeply relational worldview with that which is essentially superstitious — or that which is actually a kind of digital idolatry. Our necessarily deepening capacity for empathy mustn’t overtake the clear thinking that is is also so vitally necessary here. Bees and trees are entirely different from these automations that we humans have created. We cannot account for the emergence of bees and trees, because we have not yet cracked the mystery of their consciousness (which I intend to talk about in a later essay). Meanwhile, AIs may surprise us with emergent behavior, but we can entirely account for their design.

Humans have actually long had the ability to design algorithmic complexity that exceeds our ability to understand it in certain ways. But that doesn’t mean we have to be baffled by the essential nature of that complexity. Chess is almost unfathomably complex, for instance, but that doesn’t mean that chess, itself, has a mind. We know that chess doesn’t have a mind, because we know that the composite elements of chess don’t at all suggest the existence of a mind. And a chess bot may develop a new, emergent chess strategy, but we know that that chess bot doesn’t have a mind, either, because we know what that chess bot is and what it’s doing — which is just mathematically crunching all the possible configurations of chess in ways that exceeds a humans’ capacity to do so.

Some of the scrappier folks in the naturalistic camp will say that belief in a conscious ground of being is superstitious and regressive. But actually, it seems clear to me that to say, for instance, that AIs generate the emergent property of an actual perceiver is actually a far more superstitious statement than to say that ultimate consciousness is the ground of existence.

If I said that I’d created a magic portal and an archetypal daemon came out of it, the more empirically-minded among us would very quickly say that that’s pretty superstitious. But that’s more or less what people are saying when they say AIs are conscious — or that they will be someday. The only difference is that the magic portal is more plausibly composed of human technology, not candles and chalk-drawn lines on some ancient stone floor.

It seems to me that the error ultimately lies in a lack of what in Vedanta is called vipassana, or contemplative self-observation. The blithe belief that AIs can generate consciousness — or that unconscious nature can generate consciousness, for that matter — seems to me to simply result from not taking the time to assiduously observe one’s own consciousness, deeply, in comparison to matter. This is something I hope to talk about more in a following essay, too.

May it suffice to say here that it seems clear to me that there’s really nothing in the nature of an AI to suggest that it will ever generate a perceiver — except the richness of our own stretched imaginations.

And that that harkens to a profound, essentially reverent idea: that consciousness cannot emerge from non-consciousness, but instead that any consciousness is simply subsidiary to other forms of consciousness — which must, then, by a chain rule, trace back to an original, ultimate consciousness, which underlies it all.

1. Ha. We’ll see if I have any success at being more concise.

2. Technically, as I understand it, computers are composed of what are called NAND (NOT AND) gates and NOR (NOT OR) gates, each of which can be configured to represent the AND, OR, or NOT condition, depending on the need — but the above gives the basic idea.

3. Specifically, as I understand it, it’s analogous to the “XOR” (exclusive OR) gate), one of the basic substrates of binary addition, which is an essential operation of computation.

4. Interestingly, there is a well-known and well-loved episode from Season 2 of Star Trek: The Next Generation called “The Measure of a Man” in which Captain Picard eloquently and spectacularly argues that Data deserves the rights of any person, without any apparent consideration on the part of the writers that, by dint of these rights, he almost certainly isn’t an AI.