Some philosophers have tried to address the permissibility of killing under necessity by reflecting on thought experiments like the trolley problem.Footnote 28 This approach has been criticized, among other things, for neglecting the potential relevance of the normative positions – in particular the responsibilities – of the various agents involved in the situation (Wood 2011). With this in mind, in order to better understand the moral foundation of the legal doctrine of necessity and to carve the limits of its application, in the next sections I will turn to the analysis of the role of normative positions of different agents involved in life-threatening emergency scenarios; for each normative position I will present some insights from the legal literature, and I will try to assess their relevance for the case of AVs.

Contractual Obligations

Firstly, in defining the responsibilities of agents involved in emergency scenarios, the law has stressed the importance of contractual obligations. In the landmark case United States v. Holmes (1842)Footnote 29 some ship crew were tried for throwing 14 passengers in the sea in order to prevent the sink of a lifeboat on which they were piled with many others, following a shipwreck. Justice Baldwin explained to the jury that in assessing the sailors culpability for the killing of the passengers, they should be reminded that a shipwreck is not a scenario in which people “owe no duty to one another”: there were specific normative relations between people involved.Footnote 30 In particular, sailors owed certain duties to the passengers, and breached those duties by jettisoning some of them. Justice Baldwin explains that there are “obligations which rest upon the owners of stages, steamboats, and other vehicles of transportation.”: “in consideration of the payment of fare” and “Having, in all emergencies, the conduct of the journey, and the control of the passengers, the owners rest under every obligation for care, skill, and general capacity; and if, from defect of any of these requisites, grievous injury is done to the passenger, the persons employed are liable” (Cohan 2006:150-151). This relation cannot “be changed when the ship is lost by tempest”. Therefore, whereas the captain and a sufficient number of seamen must be preserved to navigate the boat, “supernumerary sailors have no right, for their safety, to sacrifice the passengers. The sailors and passengers, in fact, cannot be regarded as in equal positions. The sailor . . . owes more benevolence to another than to himself.” (151)

Contractual (and Extra-Contractual) Obligations and AVs

A case with some similarities with Holmes would be one in which the AV is a public service like a bus. Here the owner of the bus service and her employees arguably have a duty of care towards the passengers similar to the one owed to the passengers of a ship by the ship owner and his employees described in Holmes. However, here the scenario is complicated by the presence of two additional groups of agents: the company producing the vehicle, and the potential victims of a road accidents. The reasoning of Justice Baldwin in Holmes does not help define these additional relations, as it concerns a scenario in which the owner of the means and his employees have “in all emergencies, [a] the conduct of the journey, and [b] the control of the passengers”, but this might not be the case with automated vehicles: the owner of an automated bus does not have either the “conduct of the journey” or the control of the passengers; the vehicle manufacturers arguably have (more of) it; in addition, all the people involved in Holmes’ shipwreck are either members of the crew or passengers, whereas road accidents may typically involve third parties who are not bound by specific contractual relations to the vehicle’s owners.

As for the control issue, even though the owner of a public service does not directly control the programming of the vehicles he deploys, he arguably has an obligation to maintain some form of control or supervision over the programming of her vehicles, by making sure that she does not deploy a vehicle whose programming is in contrast with her duty of care towards her passengers. So, for instance, she may not be allowed to deploy a system designed to protect the vehicle and/or the company’s employees on the vehicle at the expenses of the passengers.

As for the normative positions of third parties involved in an accident, even though the owner of a vehicle does not have any contractual obligation towards them, she certainly has an extra-contractual obligation to not damage them.Footnote 31 So, it seems that looking at the contractual obligations is not enough to solve the issue of what should be the programming of an automated public service bus in an event of an emergency scenario in which a choice is only open between exposing the passengers or rather some third parties to the risk of serious injury or death. This last point is arguably relevant also to assess the duties and responsibilities of private autonomous vehicles manufacturers. Certainly, car manufacturers are bounded to a contractual obligation with their clients and have a stringent obligation to protect their safety in the design of their vehicles; in this respect, they are in a position not too dissimilar than that of sailors towards their passengers. However, the case of autonomous vehicles differs from the ship case in one important respect: whereas in the boat case sailors can and arguably should sacrificing themselves for protecting the life of their passengers, in the autonomous vehicle case, car manufacturers may sacrifice third parties to protect their clients life. These other people’s rights should be considered too.Footnote 32

Conventional Obligations

A second kind of normative position potentially relevant to define the limits of the permissibility of killing in emergency scenarios is that deriving from conventional obligations. The philosopher Judith Jarvis Thomson (1990) has suggested that the killing of one or more persons in order to save another, bigger group of persons, for instance in a standard trolley-problem scenario,Footnote 33 may be permitted if the following additional circumstances realize: the helpless men trapped on the railway are all workmen and part of the same crew whose tasks are randomly assigned on any given day; when workers join the work crew, it is explained to them that their occupation is a dangerous one in which death or serious injury is a distinct possibility and that should an emergency situation arise in which the certainty of the death of a larger number of men can be averted by killing a lesser number, then this will be done. Thomson claims that reasonable workmen would enter such an agreement and so they may be killed should the tragic choice presents itself.

But Christie (1999:1017) wonders what the relevance of Thomson’s claim would be if, as a matter of fact, in the real world no one ever enters into these kinds of arrangements, for instance by signing a contract. Moreover, it is very dubious whether such contracts would be even legally valid. In fact, as the victims’ consent cannot per se justify the commission of a crime like a murder (the maxim volenti non fit iniuria does not apply to criminal assaults), either the killing of the lesser group is justified on independent grounds, for instance, as being done according to a just legal procedure, as suggested by Thomson,Footnote 34 or it is not. If it is justified, than the killing may be justified and the consent of the victim is not necessary, if the killing is not justified then the killing counts as a criminal assault, and the consent of the victim is immaterial.

In addition, whereas at least tort law does allow for volenti non fit iniuria contracts – agreements in which by acknowledging and accepting the risks involved in a given activity agents waive their opportunity to make a claim against the other party in the event of a damage – these contracts are subjected to stringent conditions of full knowledge and freedom of consent. Assuming that by being a professional in that domain a worker has a full understanding of the risk he is entering by joining a certain work team, it remains highly debatable whether this consent should be considered as free. In fact, the law has put increasing restrictions on the application of a volenti contract on the workplace, due to the risks of coercive agreements for the employees engaged in risky activities.

Quite interestingly, in the Holmes shipwreck case described above both the prosecution and the defendant’s attorney insisted on the relevance of “the customs of the sea’s” prescription to cast lots in exigent circumstances. Justice Baldwin even instructed the jury that had lots been used to select the victims,Footnote 35 the defence of necessity might have been available to the crew who jettison some passengers in order to avoid the sinking of a lifeboat (Cohan 2006:154). So some customary rules and conventions may matter, after all.

However, Cohan states that it is an “unsettled issue” whether Justice Baldwin’s dictum on drawing of lots was an accurate summary of the law at the time of Holmes or today. Moreover, like Christie did about Thomson reasonable agreement between workers, Cohan also points to some difficulties in giving precise content to such conventions and enforcing them: Should everyone consent to the drawing of lots in order for the drawing to be binding for everyone? Should people be forced to participate in the ballot? If someone refused to participate, could that person’s lot be selected by proxy?

Contracts, Conventions and AVs

If the appeal to existing conventions or tacit agreements to decide how to regulate killings in emergency scenarios in dangerous long-established activities like sailing or mining may sound morally problematic, this appeal is probably utterly useless in the case of a new practice like fully automated transport, where no conventions are yet in place. And even assuming that there are shared conventions among car manufacturers’ companies about which courses of action to program in an event of an emergency situation (for instance: always protect your client), these may arguably not replace long-established ethical and legal principles in areas where life and physical integrity of people is directly at stake. Moreover, as already mentioned, the presence of third parties not bounded by any contractual or conventional relationship to the car manufacturers make the car case significantly different than the boat one.

Certainly, the idea of solving some emerging issues in the regulation of the design, production and license of autonomous vehicles by establishing in advance contracts and agreements between potential parties involved in future accidents may indeed sound as a more reasonable solution than simply letting car manufacturers free to decide. In particular, high level of autonomy will make it envisageable and recommendable the creation of new forms of insurances and possibly new forms of civil responsibility, in addition to the existing product liability and fault responsibility.Footnote 36 However, areas that have a public interest like the regulation of intentional behavior in road traffic, where basic rights like life and physical integrity are at stake are usually not covered by private contracts but by State regulation and the (criminal) law. So it seems quite unlikely that professional conventions or private insurance agreements may determine the regulation of programming of AVs in emergency scenarios where the (intentional) damaging of human lives is at stake. One possible exception might be the signing of a contract by the user of AV in which he acknowledges and accepts that the vehicle is programmed in such a way as to sacrifice its passengers – for instance by directing the vehicle against a wall – in order to avoid hitting other vehicles. This might be legally acceptable, provided a well informed consent were in place,Footnote 37 as it would potentially affects only the users of the vehicle who signs the contract. For this same reason it is doubtful whether anyone would ever freely sign such an agreement.Footnote 38

Responsibility

Responsibility may also affect the normative positions of persons involved in life-threatening scenarios like road accidents. In fact, according to the legal doctrine of contributory negligence, a person who has been damaged in an accident also due to her own fault may be assigned a lesser compensation for the damage suffered. For instance, In Goddard and Walker v Greenwood (2003), a pedestrian hit by a car was found liable for 80% of his injuries for crossing with the lights against them. However, even in tort law contributory negligence of the claimant rarely completely eliminates the car driver’s liability. In the same Goddard & Walker v Greenwood case, the Court of Appeal explained that the driver still had to cover some of the damages, because the fact that the lights were green in his favour does not obviate the need to keep a close look out. And In Eagle v Chambers (2004), LJ Hale explicitly said that even if the claimant was struck whilst dangerously walking along a dual carriageway, the car driver should be assigned a higher share of liability. Such a high burden imposed on car drivers – the judge explained – reflects the fact that cars are “potentially dangerous weapons”.

More importantly, it is a well-established legal principle that contributory negligence is not a defense in a criminal prosecution for negligent homicide by a motor vehicle unless negligence on the part of the decedent is found to be the sole proximate cause of the death (State v. Scribner 2002: 741), that is, roughly, unless the behavior of the victim entirely explains the accident. Therefore, from a criminal law perspective as far as the accident occurred also due to a deliberately or negligently dangerous behavior of the driver, this is criminally liable of manslaughter or murder – no matter whether the victim was drunk or negligent or otherwise contributed to his own death, “for in this consists a great distinction between civil and criminal proceedings” (R v Swindall and Osborne 1846).Footnote 39

Responsibility and AVs

Current law puts a high duty of care on car drivers, based on the fact that they handle “potentially dangerous weapons”. It is reasonable to think that a similar or even higher burden should be put on companies that produce and commercialise AVs. In fact, even assuming that future AVs will be safer than current ones in the sense of causing lesser accidents due to the elimination of the impact of human drivers errors, AVs will still be potentially more dangerous than bicycles or pedestrians in the sense of having a higher potential for causing serious damages to third parties in the event of a crash. In addition, according to current tort law the duty of care of drivers towards pedestrians extends as far as to cover most of the damages that could be prevented by the driver’s diligent behavior, no matter how negligent the behavior of pedestrians might be. Therefore, the enhanced ability for crash avoidance brought by artificial intelligence may put an even higher duty of care on the shoulders of autonomous vehicles manufacturers.

Finally, from a criminal law perspective, drivers – and so, arguably, future producers of AVs – have a stringent duty to not intentionally or negligently damage third parties, and in particular weaker parties like pedestrians or cyclists, again no matter how negligent the third parties’ behavior may be. This seems to make unacceptable from the start any suggestion about programming AV to preferably target, in an emergency scenario, some third parties who are somehow responsible for their being in a dangerous position, for instance pedestrians crossing with red lights, cyclists outside the bike lane, or bikers without helmet.

One may then wonder whether it would be consistent with the company’s duty of care their programming the vehicle so as to preferably hit: a) other vehicles as opposed to pedestrians or cyclists, and b) vehicles with higher safety protections as opposed to less safe ones, in the attempt to reduce the number and seriousness of casualties (Lin 2015). Whereas the preference for hitting vehicles as opposed to pedestrians or cyclists may be justified by the asymmetry in the duty of care of car drivers/manufaturers and pedestrians/cyclists, the idea of intentionally targeting safer vehicles looks more problematic. In fact, from a normative point of view, all private vehicles’ passengers have the same responsibility - i.e. that deriving from taking the risk of using private motor vehicles – and thus there seems to be no normative ground for intentionally penalising the passenger of a safer vehicle.Footnote 40 Of course, one may invoke common-sense or utilitarian considerations to defend such a programming. But it may be pointed out that when translated into a systematic policy, this may be self-defeating even from an utilitarian point of view, because it may end up discouraging the investments and use of safe vehicles, and so contributing to make the road more dangerous and the number of victims of car accidents higher in the long run.Footnote 41

Finally, one may also wonder whether the obligation to try to avoid hitting pedestrians and cyclists is so stringent as to require the killing of the passengers of the AV when this is the only open alternative option. It is probably the case. Again, one of the points of the legislation on vehicular traffic is the protection from potentially fatal accidents, and the obligation to avoid the intentional or negligent collision with pedestrians and cyclists is an essential part of this legislation. A norm providing car manufacturers with a “license” to intentionally hit an innocent pedestrian (in order to save the vehicle passenger) would be in contrast with this basic principle. This certainly does not mean that car manufacturers should be allowed or even incentivized to program their vehicles to sometimes sacrifice their passengers, as this would be in contrast with the companies’ competing duty to protect the life of their clients. But it certainly means that car manufacturers and policy-makers should be pressured to find ways to design vehicles, infrastructures and the legislative settings of road traffic in such a way as to avoid both the killings of pedestrians and cyclists and that of AVs passengers; for instance by designing infrastructures and norms in such a way as to make the realization of such a tragic dilemmatic scenario highly unlikely.Footnote 42

Rights

The reluctance of lawyers to accept a simple utilitarian reading of the defence of necessity is also grounded in the protection of the right to life. From the point of view of criminal law, the importance of the right to life can be highlighted by looking at one specific feature of the justifications of necessity and self-defence. Remember the justification/excuse distinctionFootnote 43: Whereas excused actions remain forbidden and may thus be carried out exclusively by the agents acting under the excusing conditions, justifications like necessity and self-defence make generally prohibited actions lawful, so that people acting under justificatory circumstances may be also assisted by third parties. It is for instance lawful for a safe bystander to help someone damaging some private property to let them escape from a fire (necessity), and so it is for a safe bystander to intervene to help the victim of a lethal unlawful aggression to incapacitate and possibly kill her aggressor (self-defence); however, the aggressor cannot in turn invoke a self-defence justification for using violence against the rescuer, as the rescuer’s violence is legally justified. Christie (1999) points out that this normative implication of justifications generates the following implication for a simple utilitarian readings of the doctrine of necessity. If, for instance in a trolley problem scenario,Footnote 44 one considers throwing the switch as permissible in the sense of justified as opposed to just excused, then from a legal point of view they should also accept the consequence that, for instance, friends or family of the potential victim may not try to stop the switch-throwing, and if they try they may be even legitimately prevented by force by the would-be switch-thrower, or by any other bystander.Footnote 45 By pushing the simple utilitarian reading to its extreme legal consequences, Christie wants to make vividly represented the innocent bystander’s right to life – i.e. the right not to be killed - and its legal superiority over the still legitimate request to be rescued from the people in danger.

Rights and AVs

A reference to the rights of potential victims of automated vehicles accidents may complement and strengthen the conclusions of the previous sections on the prohibition of intentional killing and on responsibility. Not only pedestrians and cyclists are owed a special duty of care by motor vehicles manufacturers based on the responsibility of the latter for taking the risk of producing a dangerous machine; they also are owed a duty not to be killed based on their basic rights as human persons. This means that pedestrians and cyclists who would not be involved in any accidents but for the decision of the AV (manufacturer), may arguably not be involved in the accident by design, even though by doing so some lives of people already involved in the accident may be saved.

Authority

Finally, an important feature of the rare scenarios where lawyers have been willing to recognize a necessity justification for the intentional killing of an innocent to save other innocent lives is the presence of a public procedure permitting or even commanding the final death-causing decision.Footnote 46 In the Re A (conjoined twins) case presented above, the doctors acted under the order of a tribunal when they performed the surgery which would cause the death of one of the twins to save the other.

The intervention of a public authority may be requested to avoid one or more of the following risks. First, even assuming that clear guidelines for justified killing were available, leaving the application of these in the hands of private citizens would open the door to serious risks of abuse; secondly, and relatedly, there are slippery slope or “Pandora box” risks, namely risks that once private citizens are allowed to make these reasoning in safe cases, they may start making these choices in less clear-cut cases (Norrie 2014). Thirdly, a public procedure may guarantee a more reliable assessment of the certainty of the future outcomes. Fourthly, a public procedure leading to a final authoritative command may sometimes be the only reasonable way to overcome the moral disagreement between citizens and to allow for a legitimate political and legal decision in ethically controversial cases.

Authority and AVs

It may be argued that all of the concerns above – risks of abuses, necessity of an official check on the certainty of future outcome, necessity of public procedure and authoritative decisions to solve hard ethical cases with a public relevance – also apply to the programming of AVs.

The public control over the management of emergency scenarios involving AVs might take two different forms. A first, simpler solution would consist of just giving vehicles manufacturers binding legal guidelines for the programming of their vehicles in emergency situations. A second, more sophisticated solution would be to also create a centralized (automated) system of coordination and monitoring of automated vehicle traffic, under the ultimate control of a public agency, which would be able to directly manage emergency situations according to the law (this might in turn involve the use of Artificial Intelligence systems of control).

In both cases the further question would remain as to which public authority should take the decision. It has been argued that in compliance with the principles of democracy, a democratically elected representative body rather than a judge should determine the hierarchy of values to be pursued in emergency scenarios, at least when life is not at stake (Dennis 2009). This seems to suggest that the case of AVs, where life is at stake, might be handled via the judicial application of the existing law. However, it can also be argued that scenarios opened by new technologies are so new that judges, not having precedent cases to rely on, may take idiosyncratic or otherwise arbitrary decisions about them, so that a prior intervention of a legislative body may be preferable.