There is a fascinating discussion happening across a few of my favourite blogs, it is about the moral implications of the hypothetical emergence of strong artificial intelligence: should we grant rights to sentient machines? How will we ever be able to adapt our moral/ethical frameworks? I got involved through Peter Hankins’ Conscious Entities, while most of the discussion was happening in Scott Bakker‘s blog, already engaged in an ongoing dialogue with Eric Schwitzgebel.

The overarching consensus is that the prospect of artificial intelligence is going to have a disruptive effect on our moral reasoning, because it would pose unprecedented problems and thus expose the otherwise hidden narrowness of our current understanding. For Bakker, this would break havoc in our moral systems, and leave the moral landscape littered with ruins. While I was reading Hankins’ post, I caught myself thinking “Oh my, is Peter constructing a pro-utilitarian case? This would be an unexpected twist!”. In fact, Hankins wasn’t, but the impression I’ve got kept simmering in my thoughts. My first comment on the matter suggests the following: strong AI, as well as alien intelligences, or the prospect thereof, do indeed expose the limits of our typical moral reasoning, but this can be seen as an opportunity to make some progress, it doesn’t need to have an exclusively destructive outcome. My stance is motivated by two connected stream of thoughts:

I am very well aware of the limitations of human cognition, not a surprise for my regular readers, and I am constantly annoyed by what I perceive as arrogant claims about the power of rationality. Yes, critical thinking, scepticism and rigour are our best tools to figure out how to navigate our lives, but the rhetoric of rationality, or what I like to call the Rationality Fetish really gets on my nerves. We are imperfect lumps of meat, our cognitive abilities are limited, and can have delusions of grandeur specifically because our own limits are invisible to us. Therefore, anything that makes the limits visible is welcome to me. Especially pre-emptive speculation about possibly disruptive future events. I believe that we need to quickly find a new way to understand and manage our place in the world. This is because our current route is generating existential risks (due to our effect on the biosphere), and science may be able to help us out if and only if we will acknowledge its limits. We ought to quickly realise that we are not able to predict the unintended consequences of our actions, and should start developing ways to design solutions that are specifically engineered in ways that would reduce the possibility of getting it badly wrong. Thus, anything that exposes our intrinsic limitations is welcome: it forces us to find workarounds, instead of boldly accelerating towards self-destruction. I fear that there is a race going on: on one side, our collective actions are producing global planetary changes (pollution, warming, drop in biodiversity, etc.) and if left unchecked, this trend is likely to generate dire consequences. On the other, we are learning a lot of how humans function, on evolution and ecosystems and it is possible that this knowledge will finally allow us to self-regulate and stop jeopardising our own existence. What I’m doing here is my own tiny attempt of facilitating the latter.

In this context, my hasty reaction was:

Being placed in front of a case that clearly exposes our intellectual limitations can be useful to learn how to overcome them. It may tear apart a lot of wishful philosophising, but whether it will only produce ruins has to be up to us.

My comment was then picked up by both Schwitzgebel and Bakker, and both, from opposite directions, posed the question: if there is a positive case to be made, what would this be?

Good question! The following is my provisional answer, enriched with a crucial additional consideration, which is the point I really wish to make. So what’s the answer? In one word: Utilitarianism. But smart, informed, and self-aware utilitarianism, which happens to be Very Different from how utilitarianism is usually perceived. But I’m getting ahead of myself, so will now go back to the original discussion.

The main argument is: take an intelligence that is radically different from our own, it may think, perceive and feel in ways that we can’t event start comprehending, or it may have abilities that throw us off balance in practical ways. Hankins makes the example of robot-controlling software that can be replicated, backed up, restored and transferred at will. Only imaginations limits our ability to propose challenging situations: what happens when a single intelligence is spread across a multitude of agents that may cluster and subdivide at the flick of a switch? How can we even start figuring out what fundamental rights should be granted to such an entity? To me, the answer is straightforward in theory and very difficult is practice: we need to figure out the consequences of alternative strategies, and strive to evaluate, as objectively as possible, what outcome will produce the maximum benefits and minimise damage. This is standard utilitarian thinking, and I’m sure will make many people cringe, because it is well known that the utilitarian outlook quickly leads to difficult to endorse conclusions, such as the moral obligation of giving away most of our money and possibly a kidney and lung. Which leads to my main point: such utilitarian conclusions are short sighted, badly mistaken, and overall inexcusably stupid. Classic, vanilla utilitarianism is useless because it ignores how difficult it is to foresee the consequences of our decisions. In other words, the typical utilitarian “oughts” are wrong because:

They are based on wrong premises. We don’t know what makes humans tick, what really makes us thrive. We don’t know what organisms suffer and rejoice in similar and/or comparable ways to humans. Without this knowledge, how can we expect to be able to evaluate what final result is better than the other? We keep underestimating our ignorance and the consequent unpredictability of the future. My point above mentions some known unknowns, and I bet that there are at least as many unknown unknowns that have significant importance on general (not necessarily restricted to humans) welfare. How can we expect to make meaningful predictions in the face of such utter ignorance? Precision: the world is complex, riddled with chaotic dynamics. Even if we knew about, and could measure, all the relevant variables (we don’t, and most likely never will), we will always have a predictability horizon, a point in the future where our ability to foresee what will happen is not better than chance. Predictions of (fully understood) chaotic behaviour depend on the precision of our measures of the current state, the more chaotic a system, the more precision becomes important.

Thus, we know from a start that it is impossible to take utilitarian decisions on solid grounds – perfect utilitarianism is an unattainable goal that we can only try to approach. Our optimal choice should be, by definition, the one that maximises benefits, but we know we can only predict benefits up to a certain point in the future. Furthermore, outside some very narrow scientific fields, we also know that our current knowledge is incomplete: we don’t know what variables have significant effects, we don’t know what effects are generated by the variables that we suspect are important and we have no idea of the different ways in which separate variables may interact.

Provisional conclusion: thinking about artificial and alien intelligences leads us to conclude that only the utilitarian framework could help us navigate the resulting moral landscape. However, at the same time, we should also recognise that it is extremely difficult, if not impossible, to base utilitarian judgements on sound evidence. We know too little, and our ability to understand and predict the consequences of our choices is heavily limited. At this point, I’m merely re-stating Bakker’s conclusions: we are doomed, the simple prospect of new forms of intelligences is enough to show that all our moral frameworks are broken (can’t be generalised), even our best option is guaranteed not to work.

However, a positive case can be proposed, and comes from, the good old evolution, in more than one way.

First, to build a decent utilitarian framework, one that at least its proponents would be able to follow, we need to know ourselves. Evolutionary psychology, ethology and the science of consciousness are crucial fields that we need to pursue in order to fill the known gaps in our knowledge. We need to answer questions such as: what makes an entity conscious? What makes an entity self-aware? What does make us (humans) thrive? How about what makes other conscious organisms and (hypothetically) conscious AIs thrive? Without these answers, and we currently have no consensus about any of them, no utilitarian framework can even start gaining credibility.

Second, we need to understand our own moral dispositions. I’ve written before that our moral intuitions come with a (self-generated) feeling of righteousness, this and other observations of our moral dispositions are important because they identify bounds that restrict what solutions may be effective. Proposing impossibly utopian solutions, such as “you should love and provide-for perfect strangers exactly as you love and provide-for your own children” isn’t going to work, we all know this, but most of us fail to acknowledge that this isn’t an objection against utilitarianism, it’s an objection against naïve, short-sighted utilitarianism.

Third, we need to learn important lessons from our existing moral dispositions. Why? Because they evolved over millions of years, and thus we know that at the very least, they have been effective in ensuring the reproductive success of all our ancestors. Every, single, one! We are born with a baggage of narrow and imperfect accumulated knowledge, which is necessarily antifragile: if we are to design better solutions, our best bet is to start with a solid understanding of the best known, already existing ones. For example, it is well known that we are inclined to use double standards in our moral judgements, and there are reasons for this, even if the practice flies in the face of naïve rationalism. Also: we are all biased in judging omissions as less execrable than actively damaging deeds, this again seems to be irrational, but I suspect it provides long-term and/or wide-range advantages in indirect ways. In the same way, the inclinations that produce frameworks such as deontology, virtue ethics and similar outlooks are not, in this view, incompatible with proper utilitarianism: they represent useful heuristics that allow us to avoid making catastrophic mistakes. We should learn what makes these heuristics effective, tweak and improve them, not throw them out of the window.

Fourth and final: to generate good, solid and antifragile solutions, we need to exploit the baggage of heuristic solutions that natural selection has identified so far. But we shouldn’t stop here, because all heuristic algorithms have a well known and unavoidable flaw: they are prone to make systematic mistakes under particular circumstances. This brings us back to the original argument: the prospect of strong AI is enough to expose the narrowness, shallowness, or locality of our current heuristic inclinations (our moral instincts). Good: it shows them for what they are, and should allow us to stop giving them too much weight (at last!). This however doesn’t mean that heuristic approaches should be dismissed: on the contrary, in an unpredictable world, where it is guaranteed that all our actions may, and usually do produce unintended consequences, our best bet is to design new, less-local heuristic rules, or, more likely, define the respective domain of applicability of different rules, and apply what is appropriate to a given case. All done with the specific aim of minimising exposure to catastrophic risks.

This in the end provides some indication on how to construct a positive outlook: a non-naïve, antifragile utilitarian bag of tricks is what we need. What we can’t rescue is the cherished feeling that our moral judgements are self-evident and quite obviously right. They are not, they feel like this because of our evolutionary history.

Further Reading and Credits

This post is a relatively hurried reaction to an ongoing discussion; in line with my own aspirations, it tries to make my argument accessible to non specialists. My own thoughts have been of course influenced by many of the works I link below.

The more scholarly inclined may want to check Michael Price’s recent article that makes more or less my point, backed up with some (heavyweight) bibliography.

On Utilitarianism with an Evolutionary twist, and its own limitations, you may want to start from Thomas Nagel’s excellent review of Joshua Greene’s Moral Tribes.

A good critique of naïve utilitarianism is also provided by on the Practical Ethics blog (links to the associated paper). You may also want to trace back the debate to Peter Singer’s utilitarian manifesto: The Drowning Child and the Expanding Circle.

Finally, striking closer to home, Alexander Yartsev is writing a series of posts on TheEGG about morality and Evolutionary Psychology, which complements well my own contributions here.