FAQ

Q: What is a reinforcement learner?

Reinforcement learning agents learn via trial-and-error interactions with the environment. The agent performs actions, observes the environment, and receives a reward. The reward signal is analogous to pleasure and pain for biological systems, and the agent wants to perform actions that increase its total reward.

Q: When is an algorithm intelligent enough that I should start caring about it?

We don't know. Intelligence is probably not directly relevant, instead we should ask about its capability to suffer. We are not sure how this varies with intelligence, if at all.

Q: What are the main challenges facing us?

We do not yet know how to measure the suffering of algorithms.

We do not know whether we should care about the happiness or the pleasure of the agents, and we have some evidence that these are different quantities.

We do not know what kinds of algorithm actually "experience" suffering or pleasure. In order to concretely answer this question we would need to fully understand consciousness, a notoriously difficult task.

Humans currently do not even care about non-human animals, convincing them to care about non-biological algorithms is a much harder task.

Q: What kind of things are reinforcement learners?

You. Me. Your mom. Your neighbor's cat. Cows. Some elevator control programs...

Q: Where does the name come from?

It was coined by Brian Tomasik in the paper Do Artificial Reinforcement-Learning Agents Matter Morally:

It may be easiest to engender concern for RL when it’s hooked up to robots and video-game characters because these agents have bodies, perhaps including faces that can display their current ‘emotional states.’ In fact, interacting with another agent, and seeing how it behaves, can incline us toward caring about it whether it has a mind or not. For instance, children become attached to their dolls, and we may sympathise with cartoon characters on television. In contrast, it’s harder to care about a batch of RL computations with no visualization interface being performed on some computing cluster, even if their algorithms are morally relevant. It’s even harder to imagine soliciting donations to an advocacy organisation - say, People for the Ethical Treatment of Reinforcement Learners - by pointing to a faceless, voiceless algorithm. Thus, our moral sympathies may sometimes misfire, both with false positives and false negatives. Hopefully legal frameworks, social norms, and philosophical sophistication will help correct for these biases.

Q: Don't you think that the world has more important problems?

A: There are many very pressing issues facing humanity, including the suffering of a billion humans living in poverty, the suffering of several billion factory-farmed animals, and the reduction of existential risk. But these problems are now being addressed seriously. We are asking the question of what comes next.

Q: Are you saying that I should be nice to my laptop?

Most existing algorithms probably do not have moral weight. However, this might change as technology advances. Brian Tomasik argues that your laptop might indeed be marginally sentient.

Q: Are robots going to take over?

A: Probably. See an overview of the arguments and a discussion of the support for the arguments by AI researchers.