What Bing Really, Really Wants: Zigazig Ah

Abstract

As we construct increasingly complex artificial intelligences, the question of their moral status becomes all the more pressing. But answering this question alone isn’t enough to answer the deeper question of how we ought to treat AIs. To answer this, we need to know about their interests. This, in turn, requires knowing about the contents and valences of their mental states. Focusing on large language models (LLMs), I argue that future LLMs could have consciousness, intentional states, and moral status. But — given plausible assumptions about consciousness and intentionality — the content of their intentional states will essentially be inaccessible to us. ‘Pain’ talk from an LLM may be meaningful, but there is no reason to think it expresses pain. ‘Desire’ talk from an LLM may be meaningful, but there is no reason to think it expresses desires. Our epistemic position regarding LLMs is radically and hopelessly impoverished.