A PDF version of this article can be found here.1

Abstract

Discourse on AI safety suffers from heated disagreement between those sceptical and those concerned about existential risk from AI. Framing discussion using strategic choice of language is a subtle but potentially powerful method to shape the direction of AI policy and AI research communities. It is argued here that the AI safety community is committing the dual error of frequently using language that hinders constructive dialogue and missing the opportunity to frame discussion using language that assists their aims. It is suggested that the community amend usage of the term ‘AI risk’ and employ more widely the ‘AI accidents’ frame in order to improve external communication, AI policy discussion, and AI research norms.

Contents

Abstract

The state of public discourse on AI safety

Why to care about terminology

Introducing ‘AI accidents’ and why use of ‘AI risk’ can be inaccurate

Why use of ‘AI risk’ is problematic and why use of ‘AI accidents’ is helpful

From the perspective of sceptics



From the perspective of the newcomer to the subject



Shaping policy discussion and research norms

Seizing the opportunity

Footnotes

Works Cited

The state of public discourse on AI safety

Contemporary public discourse on AI safety is often tense. Two technology billionaires have engaged in a regrettable public spat over existential risks from artificial intelligence (Samuelson, 2017) ; high profile AI experts have volleyed loud opinion pieces making contradictory calls for concern or for calm (Dafoe & Russell, 2016) (Etzioni, 2016) ; both factions (the group sceptical of existential risk posed by AI and the group concerned about the risk) grow larger as interest in AI increases, and more voices join the debate. The divide shows little sign of narrowing. If surviving machine superintelligence will require strong coordination or even consensus, humanity’s prospects currently look poor.

In this polarised debate, both factions, especially the AI safety community, should look to ways to facilitate constructive policy dialogue and shape safety-conscious AI research norms. Though it insufficient on its own, framing discussion using strategic choice of language is a subtle but potentially powerful method to help accomplish these goals (Baum, 2016) .

Why to care about terminology

Language choice frames policy debate, assigns the focus of discussion, and thereby influences outcomes. It decides whether the conversation is “Gun control” (liberty reducing) or “Gun violence prevention” (security promoting); “Red tape” or “Safety regulations”; “Military spending” or “Defence spending”. If terminology does not serve discussion well, it should be promptly rectified while the language, the concepts it signifies, and the actions, plans, and institutions guided by those concepts are still relatively plastic. With that in mind, the below advocates that the AI safety community revise its use of the term ‘AI risk’ and employ the ‘AI accidents’ frame more widely.

It will help first to introduce what is argued to be the substantially better term, ‘AI accidents’. The inaccuracy of current language will then be explored, followed by discussion of the problems caused by this inaccuracy and the important opportunities missed by only rarely using the ‘AI accidents’ frame.

Introducing ‘AI accidents’ and why use of ‘AI risk’ can be inaccurate

An AI accident is “unintended and harmful behavior that may emerge from poor design of real-world AI systems” (Amodei, et al., 2016) . The earliest description of misaligned AI as an ‘accident’ appears to be in Marvin Minsky’s 1984 afterword to Vernor Vinge's novel, True Names:

“The first risk is that it is always dangerous to try to relieve ourselves of the responsibility of understanding exactly how our wishes will be realized. Whenever we leave the choice of means to any servants we may choose then the greater the range of possible methods we leave to those servants, the more we expose ourselves to accidents and incidents. When we delegate those responsibilities, then we may not realize, before it is too late to turn back, that our goals have been misinterpreted, perhaps even maliciously. We see this in such classic tales of fate as Faust, the Sorcerer's Apprentice, or the Monkey's Paw by W.W. Jacobs.” (Minsky, 1984)

The term ‘AI accident’ seems to emerge publicly later, with Huw Price’s 2012 quotation of Jaan Tallin:

“He (Tallinn) said that in his pessimistic moments he felt he was more likely to die from an AI accident than from cancer or heart disease,” (University of Cambridge, 2012) .

There is some evidence that the term was used in the AI safety community prior to this (LessWrong commenter "Snarles", 2010) , but other written evidence proved elusive through online search.

The first definition of ‘accidents in machine learning systems’ appears to be provided in the well-known paper Concrete Problems in AI Safety (Amodei, et al., 2016). This is the definition for ‘AI accident’ given above and used here throughout.

Some examples of AI accidents may be illustrative: A self-driving car crash where the algorithm was at fault would be an AI accident; a housekeeping robot cooking the cat for dinner because it was commanded to “Cook something for dinner” would be an AI accident; using algorithms in the justice system that have inadvertently been trained to be racist would be an AI accident; the 2010 Flash Crash or similar future incidents would be an AI accident; deployment of a paperclip maximiser would be an AI accident. There is no presupposed upper bound for the size of AI accidents. AI safety seeks to reduce the risk of AI accidents.

Figure: AI accidents. The relative placement of instances of AI accidents may be subject to debate; the figure is intended for illustration only.

At significant risk of pedantry, close examination of terminology is worthwhile because, despite the appearance of hair-splitting, it yields what will emerge to be useful distinctions.

‘AI risk’ has at least three uses.

‘An AI risk’ - The ‘count noun’ sense, meaning a member of the set of all risks from AI, ‘AI risks’, which can be used interchangeably with ‘dangers from AI’, ‘potential harms of AI’, ‘threats from AI’, etc. Members of the set of AI risks include: AI accidents Deliberate misuse of AI systems (e.g. autonomy in weapons systems) Risks to society deriving from intended use of AI systems, which may result from coordination failures in the deployment of AI (e.g. mass unemployment resulting from automation). ‘AI risk’ – The ‘mass noun’ sense, meaning some amount of risk from AI. In practice, this means to discuss at least one member of the above set of risks, but the source of risk is not implied. It can be used interchangeably with ‘danger from AI’, ‘potential harm of AI’, ‘AI threat’, etc. The third, inaccurate sense is employing ‘AI risk’ to mean specifically ‘Risk of a catastrophic AI accident’.

Observe that in the third usage, the label used for the second (mass noun) sense is used to refer to an instance of the first (count noun) sense. It would be easy to overlook this small discrepancy of ‘crossed labels’. Nevertheless, below it is argued that using the third sense causes problems and missed opportunities.

Before exploring why use of the third sense might cause problems, note that it has been employed frequently by many of the major institutions in the AI safety community (although the accurate senses are used even more commonly)2: