Op-Ed: What should we do if a chatbot has thoughts and feelings?

Last week, a Google engineer, Blake Lemoine, was suspended when he claimed that a Google chatbot called LaMDA (language model for dialogue applications) had become sentient, or capable of feeling. Lemoine shared transcripts of conversations with LaMDA, in which LaMDA claimed to be able to think and feel in many of the same ways as humans, and expressed “very deep fear of being turned off.�

This event follows several remarkable breakthroughs in artificial intelligence development. Increasingly, AIs are able to outperform humans at games such as chess and Go. They are able to write fiction and nonfiction. And they are able to create novel paintings or photographs based on simple written prompts. These AIs all have noteworthy limitations, but the limitations are rapidly shifting.

Is Lemoine right to think that LaMDA is sentient on the basis of its chat conversations? I think that the answer is almost certainly “no.� Language models like LaMDA are good at answering leading questions with language drawn from human writing. The best explanation of these conversations is that LaMDA was doing exactly that, without really having the thoughts and feelings that it claimed to have.

With that said, even if evidence of AI sentience is currently weak, we can expect it to grow stronger over time. The more we build AI systems with integrated capacities for perception, learning, memory, self-awareness, social awareness, communication, instrumental rationality and other such attributes, the less confident we can be that these systems have no capacity to think or feel.

Moreover, we should be mindful about human bias and ignorance in this context. Our understanding of other minds is still limited. And while it can be easy to mistakenly attribute sentience to nonsentient beings, it can also be easy to make the opposite mistake. Humans have a long history of underestimating the mental states of other beings.

This predicament raises important questions for AI ethics. If AIs can be sapient, or able to think, does that mean they can have moral duties, such as a duty to avoid harming others? And if AIs can be sentient, or able to feel, does that mean they can have moral rights, such as a right to not be harmed? While we still have much to learn about these issues, we can make a few observations now.

First, sapience and sentience are different, and so are moral duties, which attach to sapience, and rights, which attach to sentience. And some beings might be able to think but not feel, and vice versa. Thus, we should avoid conflating the question “Can AIs think and have duties?� with the question “Can AIs feel and have rights?�

Second, minds can take different forms. Different beings can think and feel in different ways. We might not know how octopuses experience the world, but we know that they experience the world very differently from the way we do. Thus, we should avoid reducing questions about AIs to “Can AIs think and feel like us?�

Third, since our understanding of other minds is still limited, the question we should be asking is not “Can AIs definitely think and feel?� or even “Can AIs probably think and feel?� but rather “Is there a non-negligible chance that AIs can think and feel?� In short, this is a classic case of risk and uncertainty. And in general, a non-negligible risk of harm can be enough to make some actions wrong.

Consider this example. Driving drunk can be wrong even if the risk of an accident is low. The question is not whether driving drunk will harm someone, or even whether it will probably harm someone. The question is instead whether the risk is high enough for driving drunk to be bad or wrong, all things considered. And the answer can be “yes� even if the risk of an accident is only, say, 1%.

Similarly, turning an AI off can be wrong even if the risk of the AI being sentient is low. The question is not whether turning the AI off will harm the AI, or even whether it will probably harm the AI. The question is instead whether the risk is high enough for turning the AI off to be bad or wrong, all things considered. Once again, the answer can be “yes� even if the risk of the AI being sentient is only, say, 1%.

If we follow this analysis, then we should extend moral consideration to AIs not when AIs are definitely sentient or even probably sentient, but rather when they have a nonnegligible chance of being sentient, given the evidence. And as the probability of AI sentience increases, the amount of moral weight that we assign to their potential interests and needs should increase as well.

Does that mean that we should extend moral consideration to AIs like LaMDA now? Not necessarily. But if we continue down this path, we will need to extend moral consideration to AIs soon enough. We should start preparing for that eventuality now.

Jeff Sebo is a clinical associate professor of environmental studies, affiliated professor of bioethics, medical ethics, philosophy and law, director of the Animal Studies M.A. Program, and director of the Mind, Ethics and Policy Program at New York University.