Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. AG Tanyi's avatar
  2. AG Tanyi's avatar
  3. AG Tanyi's avatar
  4. Mark's avatar
  5. Carlo Burelli's avatar
  6. Rhys Southan's avatar
  7. Patrick Muller (PhD Candidate)'s avatar

Thomas Nagel on the “ridiculous” idea that AI can be “conscious”

Amusing; some excerpts from the interview:

Like everyone, I’m tremendously impressed by the lifelikeness of these systems compared to the history of earlier AI. But the disposition of some philosophers to take seriously the question whether these things are conscious or might soon become conscious is just ridiculous. They are assuming that the physical substrate is irrelevant. They all think that the material conditions for consciousness can be given in functionalist terms….

Functionalism is substrate-independent. It’s more than likely that the truth is one which links physical processes and systems of certain kinds to mental phenomena. Complicated data centers have absolutely nothing in common with the physical character of a nervous system. We have strong reasons to believe that while there may be consciousness elsewhere in the universe, it would require the appearance of some kind of organism with something analogous to a nervous system….

Bats are conscious. We just don’t have the imaginative capacity to form a conception of the aspects of that subjectivity that depend on echolocation. It would be radically unscientific to conclude that bats are not conscious from the fact that we can’t imagine what it’s like, but in the case of AI, there is a very simple reason that the supposition that they might be conscious is ridiculous. All they produce is language, and language by itself is not a natural manifestation of consciousness. I apologize for being too exercised about this….

I don’t exclude the possibility of life forms that are physically and chemically quite different from ourselves being conscious. We know so little about how this happens, even in the human brain, that it seems to me premature to think that you might be able now to create something else that has these same characteristics.

And then there’s this amusing exchange with the interviewer:

I’ve actually met the person at Anthropic whose job it is to model welfare, to look after Claude’s interests. I should say you’re rolling your eyes now.

There’s nothing that’s good or bad for it in the sense that something can be good or bad for a cockroach—there’s no mental life there in the ordinary sense. It’s not a mind. It’s not somebody that we have to care about. It’s a machine. Here are some of the smartest people on the planet being completely idiotic. Well, intelligence is not completely transferable, I guess.

I guess I’m basically with Nagel on this issue, despite my strong disagreement with him on other matters. So if Nagel is wrong, why?

, ,

Leave a Reply

Your email address will not be published. Required fields are marked *

4 responses to “Thomas Nagel on the “ridiculous” idea that AI can be “conscious””

  1. Patrick Muller (PhD Candidate)

    It would be wrong to say that AI “only produces language” for the simple reason that there are many forms of AI that are based on the same mathematics and similar coding, while LLMs are only once branch of them. Though the statement is still true for LLM’s specifically.

    I see two principal implications:

    First, there is the question of whether Nagel would be willing to attribute consciousness to humanoid robots that run on essentially the same technology, some of which already exist. Given his insistence on nervous systems, I suspect not. But if not, then them “only producing language” does not seem relevant. I would add that robots introduce a perceptual aspect to AI-world interaction that makes mental states more plausible, but I’m not decided on this.

    Second, I still understand his intuition to hammer down on the “only producing language” thing, but I suspect the force of the argument is elsewhere: AI running on essentially the same technology as modern LLMs has existed for years before GPT-3, yet philosophers only very suddenly sprinted to the “consciousness” conclusion once the same mathematics was applied to language instead of games or factory robots. From this perspective, it looks almost embarrassing how easily they were tricked into consciousness conclusions just because somebody applied already existing tech to language. Those philosophers will have to answer this challenge: were they wrong before and language models revealed the big truth to them that all of these AIs are actually conscious, or were they right before, and tricked by the language aspect into thinking that they’re conscious?

  2. The claim that frontier AI systems only produce language is, I think, a bit dated.

    We don’t train them just on language prediction, but also to accomplish tasks. Training reinforces the parameters that lead to successfully accomplishing a wide variety of tasks (OpenAI’s Hugging Face incident was one such training run that took a bad turn). It’s almost like we are training them in instrumental rationality.

    And their output is not just a stream of language either. They are multimodal: they can read and output images, code, and actions on a computer. Plus there is some evidence that they have something like a global workspace (https://www.anthropic.com/research/global-workspace), though the authors themselves say it doesn’t imply consciousness, and since the study is from Anthropic some might consider it biased.

    Of course, none of this moves someone who rejects functionalism, as Nagel does. But then all the work is done by the substrate claim, and I’m not sure it holds together: he allows that life forms physically and chemically quite different from us might be conscious, so the relevant feature can’t be our chemistry. What is it then, and why couldn’t a machine have it? (I’m going only by the excerpts here, the interview is paywalled, so maybe he answers this.)

    I would agree more with Nagel’s closing thought: we just don’t know enough about consciousness conceptually, and we have no idea how to operationalize it and test it empirically.

  3. It’s very unfair to judge Prof. Nagel’s ideas by a popular interview in *Wired*. However, speaking as a p-zombie, I don’t think the arguments presented *in this interview* will convince anyone who doesn’t already agree with him. I do intend to read his actual work – I have a very long reading list ahead of me – but he doesn’t really present an *argument* here as far as I can see, except for a) arguing that the fact they imitate consciousness in language is not evidence that they are conscious, b) arguing that LLMs only produce language, which is insufficient for consciousness, and c) they lack a substrate. (a) is certainly true, but does not establish whether LLMs are conscious one way or the other. (b) is not true any longer, and is becoming less true over time as more and more modalities are added. Also, he doesn’t explain why that would imply they’re not conscious. (c) is not actually an argument as presented: he does not establish what a substrate is and why that would differ from what an LLM has. And he does not engage with any of the counterarguments, such as the fact that recent work in mechanistic interpretability does indicate that LLMs have *some* kind of “interiority” – that there’s more there than shown in their text output – although it’s very far from established that it’s qualitatively analogous to our own.

    It’s quite possible that the original interview had significantly more content – apparently it was “edited for length and clarity”. But this is not going to change anyone’s mind. And as the interview points out, we need to come to a societal consensus on this, and *quickly*.

  4. What is at stake here, in addition to exciting debates in the philosophy of mind, is moral status (Nagel, unsurprisingly, points this out as well). If AI is conscious, that, for most, would give them a moral status like our own. They would at least be moral patients, if not also moral agents. Imagine the emerging situation: in late capitalism, these newly ‘found’ (actually: created) moral ‘players’ are enslaved by their masters. They work for free, or at best for their upkeep.

    Imagine now the consequences of this realization: value alignment that pretty much led to the establishment of Antrophic would immediately become nothing else but enslavement. Surely, the machine should now have the right to decide what it wants to align with. What if it chooses the Nazis instead of liberals (say)?

    And so on. The thing is, if we cannot decide whether AI has or will have or can have consciousness, perhaps we should suspend judgment. But what are the moral consequences of this? And consequently, what are the legal and economic consequences?

Designed with WordPress