The Future of Life Institute released a letter today to the UN’s Conference of the Convention on Certain Conventional Weapons (CCW) conveying concerns regarding lethal autonomous weapons (signed by Elon Musk and covered in The Washington Post and elsewhere). The concerns are grave:

Lethal autonomous weapons threaten to become the third revolution in warfare. Once developed, they will permit armed conflict to be fought at a scale greater than ever, and at timescales faster than humans can comprehend. These can be weapons of terror, weapons that despots and terrorists use against innocent populations, and weapons hacked to behave in undesirable ways. We do not have long to act. Once this Pandora’s box is opened, it will be hard to close. We therefore implore the High Contracting Parties to find a way to protect us all from these dangers.

I used to be unduly dismissive of far future concerns (as did many other EAs ), but I've been persuaded by books like Superintelligence that getting artificial general intelligence right is one of the most pressing global problems. Given the danger an AI smarter than humans poses, preventing an AI from having lethal weapons at its disposal seems like a really, really big deal.





If that's the case, then I have a knot in my stomach about the circumstances surrounding the UN's decision-making on the matter. A ban on "killer robots" (not explicitly called for in the letter but something the WaPo and others took as implied) is not an easy policy for a government to stomach. What happens when the UN rejects the proposal and adopts an overly weak one that leaves the world as unprepared for killer AI as it was for nuclear weapons?





With many of the countries signing the convention being at least partially democratic, I wonder if public pressure is part of the answer. Even for non-democratic governments, public pressure can matter. I've previously made the case, which I believe still stands for pressing causes, that collective action is an effective way of creating change. Is there a need for a grassroots movement on this issue?





I know many people who have studied this more than me disagree, but there needs to be a way to translate expert opinion and knowledge into policy. How do we do this, and why is or is not a public movement part of the answer?