First published Thu Sep 4, 1997; substantive revision Tue Apr 2, 2019

The sections below provide a variety of more precise characterizations of the prisoner's dilemma, beginning with the narrowest, and survey some connections with similar games and some applications in philosophy and elsewhere. Particular attention is paid to iterated and evolutionary versions of the game. In the fomer, the prisoner's dilemma game is played repeatedly, opening the possibility that a player can use its current move to reward or punish the other's play in previous moves in order to induce cooperative play in the future. In the latter, members of a population play one another repeatedly in prisoner's dilemma games and those who get higher payoffs “reproduce” more rapidly than those who get lower payoffs. ‘Prisoner's dilemma’ is abbreviated as ‘PD’.

Puzzles with the structure of the prisoner's dilemma were discussed by Merrill Flood and Melvin Dresher in 1950, as part of the Rand Corporation's investigations into game theory (which Rand pursued because of possible applications to global nuclear strategy). The title “prisoner's dilemma” and the version with prison sentences as payoffs are due to Albert Tucker, who wanted to make Flood and Dresher's ideas more accessible to an audience of Stanford psychologists. More recently, it has been suggested (Peterson, p1) that Tucker may have been discussing the work of his famous graduate student John Nash, and Nash 1950 (p. 291) does indeed contain a game with the structure of the prisoner's dilemma as the second in a series of six examples illustrating his technical ideas. Although Flood and Dresher (and Nash) didn't themselves rush to publicize their ideas in external journal articles, the puzzle has since attracted widespread and increasing attention in a variety of disciplines. Donninger reports that “more than a thousand articles” about it were published in the sixties and seventies. A Google Scholar search for “prisoner's dilemma” in 2018 returns 49,600 results.

Here is another story. Bill has a blue cap and would prefer a red one, while Rose has a red cap and would prefer a blue one. Both prefer two caps to any one and either of the caps to no cap at all. They are each given a choice between keeping the cap they have or giving it to the other. This “exchange game” has the same structure as the story about the prisoners. Whether Rose keeps her cap or gives to Bill, Bill is better off keeping his and she is better off if he gives it to her. Whether Bill keeps his cap or gives it to Rose, Rose is better off keeping hers and he is better off if she gives it to him. But both are better off if they exchange caps than if they both keep what they have. The new story suggests that the prisoner's dilemma also occupies a place at the heart of our economic system. It would seem that any market designed to facilitate mutually beneficial exchanges will need to overcome the dilemma or avoid it.

The “dilemma” faced by the prisoners here is that, whatever the other does, each is better off confessing than remaining silent. But the outcome obtained when both confess is worse for each than the outcome they would have obtained had both remained silent. A common view is that the puzzle illustrates a conflict between individual and group rationality. A group whose members pursue rational self-interest may all end up worse off than a group whose members act contrary to rational self-interest. More generally, if the payoffs are not assumed to represent self-interest, a group whose members rationally pursue any goals may all meet less success than if they had not rationally pursued their goals individually. A closely related view is that the prisoner's dilemma game and its multi-player generalizations model familiar situations in which it is difficult to get rational, selfish agents to cooperate for their common good. Much of the contemporary literature has focused on identifying conditions under which players would or should make the “cooperative” move corresponding to remaining silent. A slightly different interpretation takes the game to represent a choice between selfish behavior and socially desirable altruism. The move corresponding to confession benefits the actor, no matter what the other does, while the move corresponding to silence benefits the other player no matter what that other player does. Benefiting oneself is not always wrong, of course, and benefiting others at the expense of oneself is not always morally required, but in the prisoner's dilemma game both players prefer the outcome with the altruistic moves to that with the selfish moves. This observation has led David Gauthier and others to take the prisoner's dilemma to say something important about the nature of morality.

Tanya and Cinque have been arrested for robbing the Hibernia Savings Bank and placed in separate isolation cells. Both care much more about their personal freedom than about the welfare of their accomplice. A clever prosecutor makes the following offer to each: “You may choose to confess or remain silent. If you confess and your accomplice remains silent I will drop all charges against you and use your testimony to ensure that your accomplice does serious time. Likewise, if your accomplice confesses while you remain silent, they will go free while you do the time. If you both confess I get two convictions, but I'll see to it that you both get early parole. If you both remain silent, I'll have to settle for token sentences on firearms possession charges. If you wish to confess, you must leave a note with the jailer before my return tomorrow morning.”

1. Symmetric 2×2 PD With Ordinal Payoffs

In its simplest form the PD is a game described by the payoff matrix:

\(\bC\) \(\bD\) \(\bC\) \(R,R\) \(S,T\) \(\bD\) \(T,S\) \(P,P\)

satisfying the following chain of inequalities:

(PD1) \(T \gt R \gt P \gt S\)

There are two players, Row and Column. Each has two possible moves, “cooperate” (\(\bC\)) or “defect” (\(\bD\)), corresponding, respectively, to the options of remaining silent or confessing in the illustrative anecdote above. For each possible pair of moves, the payoffs to Row and Column (in that order) are listed in the appropriate cell. \(R\) is the “reward” payoff that each player receives if both cooperate. \(P\) is the “punishment” that each receives if both defect. \(T\) is the “temptation” that each receives as sole defector and \(S\) is the “sucker” payoff that each receives as sole cooperator. We assume here that the game is symmetric, i.e., that the reward, punishment, temptation and sucker payoffs are the same for each player, and payoffs have only ordinal significance, i.e., they indicate whether one payoff is better than another, but tell us nothing about how much better. It is now easy to see that we have the structure of a dilemma like the one in the story. Suppose Column cooperates. Then Row gets \(R\) for cooperating and \(T\) for defecting, and so is better off defecting. Suppose Column defects. Then Row gets \(S\) for cooperating and \(P\) for defecting, and so is again better off defecting. The move \(\bD\) for Row is said to strictly dominate the move \(\bC\): whatever Column does, Row is better off choosing \(\bD\) than \(\bC\). By symmetry \(\bD\) also strictly dominates \(\bC\) for Column. Thus two “rational” players will defect and receive a payoff of \(P\), while two “irrational” players can cooperate and receive greater payoff \(R\). In standard treatments, game theory assumes rationality and common knowledge. Each player is rational, knows the other is rational, knows that the other knows he is rational, etc. Each player also knows how the other values the outcomes. But since \(\bD\) strictly dominates \(\bC\) for both players, the argument for dilemma here requires only that each player knows his own payoffs. (The argument remains valid, of course, under the stronger standard assumptions.) It is also worth noting that the outcome \((\bD, \bD)\) of both players defecting is the game's only strict nash equilibrium, i.e., it is the only outcome from which each player could only do worse by unilaterally changing its move. Flood and Dresher's interest in their dilemma seems to have stemmed from their view that it provided a counterexample to the claim that the nash equilibria of a game constitute its natural “solutions”.

If there can be “ties” in rankings of the payoffs, condition PD1 can be weakened without destroying the nature of the dilemma. For suppose that one of the following conditions obtains:

(PD2) \(T \gt R \gt P \ge S\), or

\(T \ge R \gt P \gt S\)

Then, for each player, although \(\bD\) does not strictly dominate \(\bC\), it still weakly dominates in the sense that each player always does at least as well, and sometimes better, by playing \(\bD\). Under these conditions it still seems rational to play \(\bD\), which again results in the payoff that neither player prefers. Let us call a game that meets PD2 a weak PD. Note that in a weak PD that does not satisfy PD1 mutual defection is no longer a nash equilibrium in the strict sense defined above. It is still, however, the only nash equilibrium in the weaker sense, that neither player can improve its position by unilaterally changing its move. Again, one might suppose that if there is a unique nash equilibrium of this weaker variety, rational self-interested players would reach it.

2. Asymmetry

Without assuming symmetry, the PD can be represented by using subscripts \(r\) and \(c\) for the payoffs to Row and Column.



\(\bC\) \(\bD\) \(\bC\) \(R_r,R_c\) \(S_r,T_c\) \(\bD\) \(T_r,S_c\) \(P_r,P_c\)

If we assume that the payoffs are ordered as before for each player, i.e., that \(T_i \gt R_i \gt P_i \gt S_i\) when \(i=r,c\), then, as before, \(\bD\) is the strictly dominant move for both players, but the outcome \((\bD, \bD)\) of both players making this move is worse for each than \((\bC, \bC)\). The force of the dilemma can now also be felt under weaker conditions, however. Consider the following three pairs of inequalities:

(PD3) a. \(T_r \gt R_r\) and \(P_r \gt S_r\)

b. \(T_c \gt R_c\) and \(P_c \gt S_c\)

c. \(R_r \gt P_r\) and \(R_c \gt P_c\)

If these conditions all obtain the argument for dilemma goes through as before. Defection strictly dominates cooperation for each player, and \((\bC,\bC)\) is strictly preferred by each to \((\bD,\bD)\). If one of the two \(\gt\) signs in each of the conditions \(a\)–\(c\) is replaced by a weak inequality sign (\(\ge\)) we have a weak PD. \(\bD\) weakly dominates \(\bC\) for each player (i.e., \(\bD\) is as good as \(\bC\) in all cases and better in some) and \((\bC,\bC)\) weakly better than \((\bD,\bD)\) (i.e., it is at least as good for both players and better for one). Since none of the clauses requires comparisons between \(r\)'s payoffs and \(c\)'s, we need not assume that \(\gt\) has any “interpersonal” significance.

Now suppose we drop the first inequality of either \(a\) or \(b\) (but not both). A game that meets the resulting conditions might be termed a common knowledge PD. As long as each player knows that the other is rational and each knows the other's ordering of payoffs, we still feel the force of the dilemma. For suppose a holds. Then \(\bD\) is the dominant move for Row. Column, knowing that Row is rational, knows that Row will defect, and so, by the remaining inequality in \(b\), will defect himself. Similarly, if \(b\) holds Column will defect, and Row, realizing this, will defect herself. By \(c\), the resulting \((\bD,\bD)\) is again worse for both than \((\bC,\bC)\).

3. Cardinal Payoffs and Impure PDs

If the game specifies absolute (as opposed to relative) payoffs, then universal cooperation may not be a pareto optimal outcome even in the two person PD. For under some conditions both players do better by adopting a mixed strategy of cooperating with probability \(p\) and defecting with probability \((1-p)\). This point is illustrated in the graphs below.

Figure 1

Here the \(x\) and \(y\) axes represent the utilities of Row and Column. The four outcomes entered in the matrix of the second section are represented by the labeled dots. Conditions PD3a and PD3b (see above) ensure that \((\bC,\bD)\) and \((\bD,\bC)\) lie northwest and southeast of \((\bD,\bD)\), and PD3c is reflected in the fact that \((\bC,\bC)\) lies northeast of \((\bD,\bD)\). Suppose first that \((\bD,\bD)\) and \((\bC,\bC)\) lie on opposite sides of the line between \((\bC,\bD)\) and \((\bD,\bC)\), as in the graph on the left. Then the four points form a convex quadrilateral, and the payoffs of the feasible outcomes of mixed strategies are represented by all the points on or within this quadrilateral. Of course a player can really only get one of four possible payoffs each time the game is played, but the points in the quadrilateral represent the expected values of the payoffs to the two players. If Row and Column cooperate with probabilities \(p\) and \(q\) (and defect with probabilities \(p^*=1-p\) and \(q^*=1-q\)), for example, then the expected value of the payoff to Row is \(p^*qT+pqR+p^*q^*P+pq^*S\). A rational self-interested player, according to a standard view, should prefer a higher expected payoff to a lower one. In the graph on the left the payoff for universal cooperation (with probability one) is pareto optimal among the payoffs for all mixed strategies. In the graph on the right, however, where both \((\bD, \bD)\) and \((\bC, \bC)\) lie southwest of the line between \((\bC, \bD)\) and \((\bD, \bC)\), the story is more complicated. Here the payoffs of the feasible outcome lie within a figure bounded on the northeast by three distinct curve segments, two linear and one concave. Notice that \((\bC, \bC)\) is now in the interior of the region bounded by solid lines, indicating that there are mixed strategies that provide both players a higher expected payoff than \((\bC, \bC)\). It is important to note that we are talking about independent mixed strategies here. Row and Column use private randomizing devices and have no communication. If they were able to correlate their mixed strategies, so as to ensure, say \((\bC, \bD)\) with probability \(p\) and \((\bD, \bC)\) with probability \(p^*\), the set of feasible solutions would extend up to (and include) the dotted line between \((\bC, \bD)\) and \((\bD, \bC)\). The point here is that, even confined to independent strategies, there are some games satisfying PD3 in which both players can both do better than they do with universal cooperation. A PD in which universal cooperation is pareto optimal may be called a pure PD. (This phenomenon is identified in Kuhn and Moresi and applied to moral philosophy in Kuhn 1996.) A pure PD is characterized by adding to PD3 the following condition.

(P) \((T_r - R_r)(T_c - R_c) \le (R_r - S_r)(R_c - S_c)\)

In a symmetric game \(P\) reduces to the simpler condition

(RCA) \(R \ge \tfrac{1}{2}(T+S)\)

(named after the authors Rapoport, Chammah and Axelrod who employed it).

4. Multiple Moves and the Optional PD

Speaking generally, one might say that a PD is a game in which a “cooperative” outcome obtainable only when every player violates rational self-interest is unanimously preferred to the “selfish” outcome obtained when every player adheres to rational self-interest. We can characterize the selfish outcome either as the result of each player pursuing its dominant (strongly dominant) strategy, or as the unique weak (strong) nash equilibrium. In a two move game the two characterizations come to the same thing—a dominant move pair is a unique equilibrium and a unique equilibrium is a dominant move pair. As the payoff matrix below shows, however, the two notions diverge in a game with more than two moves.

\(\bC\) \(\bD\) \(\bN\) \(\bC\) \(R,R\) \(S,T\) \(T,S\) \(\bD\) \(T,S\) \(P,P\) \(R,S\) \(\bN\) \(S,T\) \(S,R\) \(S,S\)

Here each player can choose “cooperate”, (\(\bC\)) “defect” (\(\bD\) ), or “neither” (\(\bN\)),and the payoffs are ordered as before. Defection is no longer dominant, because each player is better off choosing \(\bC\) than \(\bD\) when the other chooses \(\bN\). Nevertheless \((\bD, \bD)\) is still the unique equilibrium. Let us label a game like this in which the selfish outcome is the unique equilibrium an equilibrium PD, and one in which the selfish outcome is a pair of dominant moves a dominance PD. As will be seen below, attempts to “solve” the PD by allowing conditional strategies can create multiple-move games that are themselves equilibrium PDs.

Three-move games with a slightly different structure have received attention under the label “optional PD.” See, for example, Kitcher (2011), Kitcher (1993), Batali and Kitcher, Szabó and Hauert, Orbell and Dawes (1993), and Orbell and Dawes (1991). The first three sources take optional games also to allow players to signal willingness to engage (i.e., play \(\bC\) or \(\bD\) against) particular opponents. The simple three-move games without signaling discussed in this section are called “semi-optional” in Batali and Kitcher. \(S,R,P\) and \(T\) payoffs are ordered as before, but the payoff matrix now contains, in addition, an “opt-out” value, \(O\), that lies between \(P\) and \(R\).

\(\bC\) \(\bD\) \(\bN\) \(\bC\) \(R,R\) \(S,T\) \(O,O\) \(\bD\) \(T,S\) \(P,P\) \(O,O\) \(\bN\) \(O,O\) \(O,O\) \(O,O\)

In this version of the game, defection is no longer a dominant move and mutual defection is no longer an equilibrium outcome. If Column cooperates, Row does best by defecting; if Column defects, Row does best by playing \(\bN\); and if Column plays \(\bN\), then Row does equally well by playing any move. From the outcome of mutual \(\bD\) either player can benefit by unilaterally switching to \(\bN\). But from the outcome of mutual \(\bN\), neither party can benefit by unilaterally changing moves. So the optional PD is a weak equilibrium PD, with \(\bN\) playing the role of defection. Orbell and Dawes (1991 and 1993) add the additional condition that the opt-out payoff \(O\) is equal to zero. In an optional PD, a rational player will engage (i.e., play either \(\bC\) or \(\bD\)) if and only if she expects her opponent to cooperate. For, if her opponent does cooperate, she will be guaranteed at least \(R\) by engaging and exactly \(O\) by not engaging, whereas if her opponent does not cooperate she will be guaranteed at most \(P\) by engaging and exactly \(O\) by not engaging. This feature becomes especially salient when \(O\) is zero, for then the payoff for engaging is positive if and only if one's opponent cooperates.

The description of the “neither” move and “opt-out” payoffs varies somewhat in accounts of the optional PD. For Kitcher they frequently represent a choice to “go solo.” For example, a baboon, rather than thoroughly or sloppily grooming a partner in exchange for being groomed thoroughly or sloppily by its partner, may choose to groom itself. Often, on the other hand, it is suggested is that \(\bN\) represents a choice to “sit out” the game, perhaps in order to obtain a more suitable partner with whom to play later. The significance of this difference, if any, will emerge in iterated and evolutionary versions of the game. (See sections 11–17 below.) Those who write about the optional PD often express the hope that it might provide a suitable model to investigate the idea that cooperation can be achieved if agents select the partners with whom they interact. That idea is modeled somewhat differently, and perhaps more directly, in Social Network Games discussed in section 19 below. Further discussion of the idea is left to that section.

Orbell and Dawes are particularly concerned with an explanation for cooperative behavior that rests on the empirically supported hypothesis that individuals often base expectations about behavior of others on their knowledge of their own behavior and tendencies. This hypothesis suggests that a cooperator is more likely than a defector to expect others to cooperate and therefore, if he is rational, more likely to engage in the optional PD. Orbell and Dawes (1991) demonstrate that, if a cooperator is substantially more likely than a defector to expect his opponent to cooperate, then (provided the odds of his opponent cooperating are sufficiently high), a cooperator can actually expect a higher return than a defector in the optional PD. Orbell and Dawes (1993) present experimental evidence that participants in an optional PD do receive higher average payouts than those in the corresponding PD lacking the \(\bN\) move. They provide clever statistical arguments to support the following hypotheses: intending cooperators (those who cooperate when they must engage) do better in the optional PD than in the corresponding PD; intending defectors generally do worse in the optional PD; under some conditions these gains and losses are sufficient to make the intending cooperators better off than the intending defectors (as might be predicted by the theoretical result of the previous paper); and, finally, those who expect cooperation from others (as evidenced by their engagement) do so on the basis of their own tendency to cooperate rather rather than any direct discernment of the character of their opponent. (See Transparency below.)

5. Multiple Players, Tragedies of the Commons, Voting and Public Goods

Most of those who maintain that the PD illustrates something important about morality seem to believe that the basic structure of the game is reflected in situations that larger groups, perhaps entire societies, face. The most obvious generalization from the two-player to the many-player game would pay each player the reward (\(R\)) if all cooperate, the punishment (\(P\)) if all defect, and, if some cooperate and some defect, it would pay the cooperators the sucker payoff (\(S\)) and the defectors the temptation (\(T\)). But it is unlikely that we face many situations of this structure.

A common view is that a multi-player PD structure is reflected in what Garret Hardin popularized as “the tragedy of the commons.” Each member of a group of neighboring farmers prefers to allow his cow to graze on the commons, rather than keeping it on his own inadequate land, but the commons will be rendered unsuitable for grazing if more than some threshold number use it. More generally, there is some social benefit \(B\) that each member can achieve if sufficiently many pay a cost \(C\). We might represent the payoff matrix as follows:

more than \(n\)

choose \(C\) \(n\) or fewer

choose \(C\) \(\bC\) \(C+B\) \(C\) \(\bD\) \(B\) \(0\)

The cost \(C\) is assumed to be a negative number. The “temptation” here is to get the benefit without the cost, the reward is the benefit with the cost, the punishment is to get neither and the sucker payoff is to pay the cost without realizing the benefit. So the payoffs are ordered \(B \gt (B+C) \gt 0 \gt C\). As in the two-player game, it appears that \(\bD\) strongly dominates \(\bC\) for all players, and so rational players would choose \(\bD\) and achieve 0, while preferring that everyone would choose \(\bC\) and obtain \(C+B\).

Unlike the more straightforward generalization, this matrix does reflect, in a highly idealized way, common social choices — between depleting and conserving a scarce resource, between using polluting and non-polluting means of manufacture or disposal, and between participating and not participating in a group effort towards some common goal. When the number of players is small, it represents a version of what has been called the “volunteer dilemma”. A group needs a few volunteers, but each member is better off if others volunteer. (Notice, however, that in a true volunteer dilemma, where only one volunteer is needed, \(n\) is zero and the top right outcome is impossible. Under these conditions \(\bD\) no longer dominates \(\bC\) and the game loses its PD flavor.) A particularly vexing manifestation of this game occurs when a vaccination known to have serious risks is needed to prevent the outbreak of a fatal disease. If enough of her neighbors get the vaccine, each person may be protected without assuming therisks.

One idealizion here of the situations described is that the costs and benefits of cooperation are assumed to be independent of the number of those who cooperate. Until the threshold of cooperation is exceeded, nobody gets the benefit. Afterwards, everyone does. They are, in the terminology of Frolich et al, lumpy. This is not true of, say, a lake made clean when residents refrain from dumping waste into it, or a gas supply maintained by users' conservation. We will consider relaxing this idealization later. For now, note that a situation more closely mirrored by the matrix is faced by the supporters of a particular political candidate or proposition who face the choice of whether to vote in a majority-rule election. Once enough supporters to constitute a majority choose to vote, additional votes will not increase their benefit. For this reason we might call the game described by the matrix above a voting game.

The voting game, as characterized above, has a somewhat different character than the two-player PD. First, even if each player's moves are entirely independent of the others, the alternatives represented by the columns in the commons matrix above are no longer independent of the alternatives represented by the rows. My choosing \(\bC\) necessarily increases the chances that more than \(n\) people will choose \(\bC\). To ensure independence we should really redraw the matrix as follows:

more than \(n\) others

choose \(C\) \(n\) others choose \(C\) fewer than \(n\) others

choose \(C\) \(\bC\) \(C+B\) \(C+B\) \(C\) \(\bD\) \(B\) \(0\) \(0\)

But now we see that move \(\bD\) does not dominate \(\bC\), as it does in the 2-player prisoner's dilemma. When we are at the threshold of adequate cooperation, where exactly \(n\) others choose \(\bC\), I am better off cooperating. Similarly, whereas mutual defection is the only nash equilibrium in the original PD, this game has two equilibria. One is universal defection, since any player unilaterally departing from that outcome will move from payoff 0 to \(C\). But a second is the state of minimally effective cooperation, where the number of cooperators is just sufficient to obtain the benefit. A defector who unilaterally departs from that outcome will move from \(B\) to \(B+C\) and a cooperator who unilaterally departs will move from \(B+C\) to 0. Finally, in the orginal PD, every outcome except universal defecton is pareto optimal--i.e., as long as at least one of the players cooperates, there is no outcome in which in which each player is at least as well off and one is better off. In the voting game, on the other hand, only the states of minimally effective cooperation are pareto optimal. If the number of cooperators exceeds the threshold by one or more, a new defector will benefit himself while hurting no others.

In view of these properties, it may seem that the voting game presents far less of a dilemma than the PD. There are, after all, equilibria that are pareto-optimal outcomes. In practice, however, it is difficult to see how these equilibria could be attained and the all-defect equilibrium could be avoided. When n is large, defection “almost dominates” cooperation. In the voting case, for example, a player might plausibly reason: if few of my fellow supporters vote, my vote will be futile, if many of them do it will be unneccessary. Even if a group were in the unlikely situation of being just below the threshold of minimally effective cooperation, a prospective voter would have no way of knowing this. In the pollution and conservation examples moves should really not be modeled as simultaneous (see Asynchronous Moves below), so we may perhaps be a little more optimistic. By observing the actions of those who have moved previously a player might know whether at his turn the threshold of minimally effective cooperation is near. In most real-world situations, however, a player can deduce this only by observing the effects of those actions, and often these effects manifest themselves only after his move is made. A conspicuous example of this delay effect might be the succession of carbon-emitting activities leading to climate change.

In examples philosophers discuss as instances of prisoner's dilemma, it is taken to be obvious that universal cooperation is the most socially desirable outcome. In the voters dilemma, since minimally effective cooperation is pareto superior, one might think that we should aim instead for that outcome. But this seems to depend on the nature of the choices involved. In the medical example it may seem best to vaccinate everyone. In the agricultural example, however, it seems foolish to stipulate that nobody use the commons. Someone who avoids vaccination in the former case is seen as a “free rider”. An underused commons in the latter seems to exemplify “surplus cooperation.” All these cases seem to raise questions of fairness. If t+1 is the size of a minimally effective collection of players, then any profile in which exactly t+1 players cooperate is a pareto optimal equilibrium. If there is no reason to prefer one such profile over another, it is possible that fairness would dictate choosing the inferior outcome of universal cooperation.

The two-person version of the tragedy of the commons game (with threshold of one) produces a matrix presenting considerably less of a dilemma.

\(\bC\) \(\bD\) \(\bC\) \(B+C,B+C\) \(C,0\) \(\bD\) \(0,C\) \(0,0\)

This game captures David Hume's example of a boat with one oarsman on the port side and another on the starboard (provided we assume that Hume's oarsmen must make their choices between rest (D) and exertion (\(\bC\)) simultaneously). If either rows alone, she exerts herself to no good effect, which is worse than had she merely rested. Mutual cooperation here is identical to minimally effective cooperation and therefore is both an equilibrium outcome and a pareto optimal outcome. Games of this sort are discussed in section 8 below, under the label “stag hunt.”

The above representations of the tragedy of the commons make the simplifying assumptions that the costs and benefits of cooperation are the same for each player, that the cost of cooperation is independent of the number of players who cooperate, and that the size of the benefit (\(0\) or \(B)\)) depends only on whether the number of cooperators exceeds the threshold. A somewhat more general account would replace \(C\) and \(B\) by functions \(C(i,j)\) and \(B(i,j)\), representing the cost of cooperation to player \(i\) when he is one of exactly \(j\) players who cooperate and the benefit to player \(i\) when exactly \(j\) players cooperate. We suppose that there is some threshold \(t\) for minimally effective cooperation so that \(B(i,j)\) is not defined unless \(j \gt t\). We may also assume additional cooperation never reduces the benefit \(i\) gets from effective cooperation, i.e., \(B(i,j+1) \ge B(i,j)\) when \(j \gt t\) and that additional defection never reduces the cost \(i\) bears in cooperating, i.e., \(C(i,j+1) \ge C(i,j)\). Now suppose, in addition, that, once the threshold of effective cooperation has been exceeded, any benefit one gets from from the presence of an additional cooperator is exceeded by one's cost of cooperation and that the costs of ineffective cooperation are genuine, i.e., for all players \(i\), \(B(i,j) \gt ( B(i,j+1)+C(i,j+1) )\) when \(j\) is greater than \(t\) and \(0 \gt C(i,j)\) when \(j\) is less than or equal to \(t\). Finally, suppose that the benefits to each player \(i\), of effective cooperation exceed the costs, i.e., for \(j \gt t\), \(B(i,j)+C(i,j) \gt 0\). We then have a tragedy of the commons game, which presents a familiar dilemma: defection benefits an individual in every circumstance (except the one where exactly \(t\) others cooperate) but everybody is better off in any state of effective cooperation than in any state without it. This account could be easily be modified to allow threshold of minimally effective cooperation to differ from one individual to another (\(i\)'s clean water requirements might be more stringent than \(j\)'s for example) or to allow \(B\) to be defined everywhere (thus eliminating the threshold, so that we always benefit from another's cooperation). The resulting game would still have its PD flavor.

Phillip Pettit has pointed out that examples that might be represented as many-player PDs come in two flavors. The examples discussed above might be classified as free-rider problems. My temptation is to enjoy some benefits brought about by burdens shouldered by others. The other flavor is what Pettit calls “foul dealer” problems. My temptation is to benefit myself by hurting others. Suppose, for example, that a group of people are applying for a single job, for which they are equally qualified. If all fill out their applications honestly, they all have an equal chance of being hired. If one lies, however, he can ensure that he is hired while, let us say, incurring a small risk of being exposed later. If everyone lies, they again have an equal chance for the job, but now they all incur the risk of exposure. Thus a lone liar, by reducing the others' chances of employment from slim to none, raises his own chances from slim to sure. As Pettit points out, when the minimally effective level of cooperation is the same as the size of the population, there is no opportunity for free-riding (everyone's cooperation is needed), and so the PD must be of the foul-dealing variety. But (Pettit's contrary claim notwithstanding) not all foul-dealing PDs seem to have this feature. Suppose, for example, that two applicants in the story above will be hired. Then everyone gets the benefit (a chance of employment without risk of exposure) unless two or more players lie. Nevertheless, the liars seem to be foul dealers rather than free riders. A better characterization of the foul-dealing dilemma might be that every defection from a generally cooperative state strictly reduces the payoffs to the cooperators, i.e., for every player \(i\) and every \(j\) greater than the threshold, \(B(i,j+1)+ C(i,j+1) \gt B(i,j)+ C(i,j)\). A free-rider's defection benefits himself but does not, by itself, hurt the cooperators. A foul-dealer's defection benefits himself and hurts the cooperators.

The game labeled a many-person PD in Schelling, in Molander 1992, and elsewhere requires that the payoff to each co-operator and defector increases strictly with the number of cooperators and that the sum of the payoffs to all parties increases with the number of cooperators (so that one party's switching from defection to cooperation always raises the sum). Neither of these conditions is met by the formulation above, and one may question whether they are appropriate for the examples given. The margin of victory would not seem to raise the value of winning an election. Natural filtering systems may allow a body of water to absorb a certain amount of waste with zero harmful effects. It is often plausible, however, to maintain that they hold “locally,” i.e., for \(j\) close to the threshold \(t\) for minimally effective cooperation, it may be reasonable to assume that:

for every individual \(i\), \(B(i, j+1)+C(i,j+1) \gt B(i, j)+C(i,j)\) for \(j \gt t\),

for every individual \(i\), \(C(i, j+1) \gt C(i,j)\) for \(j \le t\), and \[ \begin{align} &B(1,j+1)+ C(1, j+1) + \ldots + B(j+1,j+1) + C(t+1,j+1) \\ & \quad\quad + B(j+2,j+1)) + \ldots + B(n, j+1) \\ & \ \gt B(1,j) + C(1,j) + \ldots + B(j,j) + C(j,j) \\ & \quad\quad + B(j+1,j) + \ldots + B(n,j). \end{align} \]

By requiring that cooperation of others always strictly benefits each player, the Schelling and Molander formulations of the \(n\)-person PD fail to model the surplus cooperation/free rider phenomenon that seems to infuse many of the tragedy-of-commons examples. Their conditions might, however be a plausible model for certain public good dilemmas. It is not unreasonable to suppose that any contribution towards public health, national defense, highway safety, or clean air is valuable to all, no matter how little or how much we already have, but that the cost to each for his own contribution to those goods always exceeds the benefit that he derives from that contribution. A particularly simple game meeting the conditions above is the public goods game. Each player may choose to contribute either nothing or a fixed utility C to a common store. Contributions to the store are added together, mutiplied by some factor greater than one, and divided equally among the members of the group. In this way a player benefits by same amount from the contributions of others whether she contributes herself or not, and loses by the same (smaller) amount from her own contribution whether others contribute or not. This is not true of PD's in general, though it is true of the exchange game mentioned in the introduction.

The formulations of Schelling and Per Molander and the public goods game have the advantage of focusing attention on the PD quality of the game. Defection dominates cooperation, while universal cooperation is unanimously preferred to universal defection. Michael Taylor goes even further in this direction. His version of the many-person PD requires only the two PD-conditions just mentioned and the one additional condition that defectors are always better off when some cooperate than when none do. (Taylor's main concern is with the iterated version of this game, a topic that will not be addressed here.)

These ideas can be made more perspicuous by some pictures, which suggest additional refinements and extensions. Figure 2 below illustrates the voting game. In graph 2(a), twenty five supporters are choosing whether to vote in a majority-rule election. Utility to a player i is plotted against the number of those other than i who vote. Dark disks represent cooperators (voters) and circles represent defectors (non-voters). When the number of other voters is fewer than twelve or greater than twelve then defection beats cooperation. But when exactly twelve others vote it benefits i to vote.

(a) (b) Figure 2

In figure 2(b) smooth curves are drawn through the lines and circles to illustrate a more general form of the voting game. The utilities to cooperators and defectors are represented by two S-shaped curves. The curves intersect in two places. Now, instead of a single point of minimally effective cooperation, we have a small region between the two curves where cooperation beats defection. In terms of the polluted lake example, we might suppose that to the left of the first intersection, pollution is so bad that my additional contribution makes it no worse, and to the right of the second intersection, the lake is so healthy that it can handle my refuse with no ill effects. The intersection points are both equilibria, the polluting and fastitidious residents both lose by changing behavior. In terms of the voting example, we might suppose that the behavior of non-supporters is uncertain and the region between the curves represents the situations in which my vote increases the odds of winning in a way that exceeds my cost of voting.

The more general voting game satisfies the Schelling/Molander condition that utility of each player increases strictly with the level of cooperation only near the region where cooperation is effective. In figure 3 below the S-curves are bent so that this condition is met everywhere. In 3(a) the two curves still intersect twice. Bovens, which contains a very illuminating taxonomy of n-player games, labels this form the voting game and argues that it best represents situations described in the literature as tragedies of the commons. Note that if there is a value of x at which both curves lie above the equilibria, as there must be if the curves are upward sloping, then the equilibria here cannot be pareto optimal (as the lone equilibrium was in the simplest version of what was called the voting game above). Hence the tragedy. In graph 3(b) there are no intersections between the two curves. Thus the second of the Schelling/Molander conditions for a PD is also met: defection dominates cooperation. The final condition, that cooperation always raises the sum of utilities, is not so easily pictured, but, because the slopes of the two curves are positive, we can be sure that it will be met if the population is sufficiently large.

(a) (b) Figure 3

Benefits are somewhat less lumpy in these two games than the previous two. Lumpiness can by further reduced by further flattening the curves. At the limit, we get the public goods game shown in the first graph of figure 4. Here the curves are straight lines. Each additional cooperator provides both defectors and cooperators with the same additional benefit of mc/n where c is the cost of donation, m is the stipulated multiplier and n is the number of players in the game. If the curves are sufficiently flat, they can intersect at most once. Altogether there are three possibilities: the game pictured in figure 4(a), where the two curves do not intersect, the one pictured in 4(b), where cooperators' utility is above the defectors' to the left of the intersection and below it to the right, and the one pictured in 4(c), where the defectors' utility starts above that of the cooperators' and ends up below it. In the 4(b), one benefits by cooperating when few of the others do and defecting when most of the others cooperate. Bovens plausibly suggests that this should be regarded as a many-player version of the game of chicken: go straight if your opponent swerves and swerve if your opponent goes straight. In 4(c), one benefits by defecting when most others do and cooperating when most others do. As Bovens suggests, this might be regarded as a many-person version of the stag hunt: hunt together or separately if your opponent does likewise. (Stag hunt is further discussed in section 8 below). The first possibility, as we have seen, meets conditions plausibly associated with the PD.

(a) (b) (c) Figure 4

6. Single Person Interpretations

The PD is usually thought to illustrate conflict between individual and collective rationality, but the multiple player form (or something very similar) has also been interpreted as demonstrating problems within standard conceptions of individual rationality. One such interpretation, elucidated in Quinn, derives from an example of Parfit's. A medical device enables electric current to be applied to a patient's body in increments so tiny that there is no perceivable difference between adjacent settings. You are attached to the device and given the following choice every day for ten years: advance the device one setting and collect a thousand dollars, or leave it where it is and get nothing. Since there is no perceivable difference between adjacent settings, it is apparently rational to advance the setting each day. But at the end of ten years the pain is so great that a rational person would sacrifice all his wealth to return to the first setting.

We can view the situation here as a multi-player PD in which each “player” is the temporal stage of a single person. So viewed, it has at least two features that were not discussed in connection with the multi-player examples. First, the moves of the players are sequential rather than simultaneous (and each player has knowledge of preceding moves). Second, there is the matter of gradation. Increases in electric current between adjacent settings are imperceptible, and therefore irrelevant to rational decision-making, but sums of a number such increases are noticeable and highly relevant. Neither of these features, however, is peculiar to one-person examples. Consider, for example, the choice between a polluting and non-polluting means of waste disposal. Each resident of a lakeside community may dump his or her garbage in the lake or use a less convenient landfill. It is reasonable to suppose that each acts in the knowledge of how others have acted before. (See “Asynchronous Moves” below.) It is also reasonable to suppose that addition of one can of garbage to the lake has no perceptible effect on water quality, and therefore no effect on the welfare of the residents. The fact that the dilemma remains suggests that PD-like situations sometimes involve something more than a conflict between individual and collective rationality. In the one-person example, our understanding that we care more about our overall well-being than that of our temporal stages does not (by itself) eliminate the argument that it is rational to continue to adjust the setting. Similarly, in the pollution example, a decision to let collective rationality override individual rationality may not eliminate the argument for excessive dumping. It seems appropriate, however, to separate this issue from that raised in the standard PD. Gradations that are imperceptible individually, but weighty en masse give rise to intransitive preferences. This is a challenge to standard accounts of rationality whether or not it arises in a PD-like setting.

A second one-person interpretation of the PD is suggested in Kavka, 1991. On Kavka's interpretation, the prisoners are not temporal stages, but rather “subagents” reflecting different desiderata that I might bring to bear on a decision. Let us imagine that I am hungry and considering buying a snack. The options open to me are:

Buy a scoop of chocolate gelato. Buy a scoop of orange sherbet. Buy a granola bar. Buy nothing.

My health-conscious side, “Arnold,” orders these options in the following order: \(c\), \(b\), \(d\), \(a\). My taste-conscious side, “Eppie,” ranks them: \(a\), \(b\), \(d\), \(c\). Such inner conflict among preferences might often be resolved in ways consistent with standard views about individual choice. My overall preference ordering, for example, might be determined from a weighted average of the utilities that Arnold and Eppie assign to each of the options. It is also possible, Kavka suggests, that my inner conflicts are resolved as if they were a result of strategic interaction among rational subagents. In this case, Arnold and Eppie can each choose either to insist on getting their way \((\bI)\) or to acquiesce to a compromise \((\bA)\). The interaction between subagents can then be represented by the following payoff matrix, where Arnold plays row and Eppie plays column.



\(\bA\) \(\bI\) \(\bA\) \(b\) \(a\) \(\bI\) \(c\) \(d\)

Examination of the table and preference orderings confirms that we again have an intrapersonal PD. Kavka argues that a story like this might “provide a psychologically plausible picture of how internal conflict can lead to suboptimal action.” It also undermines a standard view that choices reflect values in favor of one that they partially reflect, “the structure of inner conflict.”

7. The PD with Replicas and Causal Decision Theory

One controversial argument that it is rational to cooperate in a PD relies on the observation that my partner in crime is likely to think and act very much like I do. (See, for example, Davis 1977 and 1985 for a sympathetic presentation of one such argument and Binmore 1994, chapters 3.4 and 3.5, for a reformulation and extended rebuttal.) In the extreme case, my accomplice is an exact replica of me who is wired just as I am so that, of necessity, we do the same thing. It would then seem that the only two possible outcomes are where both players cooperate and where both players defect. Since the reward payoff exceeds the punishment payoff, I should cooperate. More generally, even if my accomplice is not a perfect replica, the odds of his cooperating are greater if I cooperate and the odds of his defecting are greater if I defect. When the correlation between our behaviors is sufficiently strong or the differences in payoffs is sufficiently great, my expected payoff (as that term is usually understood) is higher if I cooperate than if I defect. The counter argument, of course, is that my action is causally independent of my replica's. Since I can't affect what my accomplice does and since, whatever he does, my payoff is greater if I defect, I should defect. These arguments closely resemble the arguments for two positions on the Newcomb Problem, a puzzle popularized among philosophers in Nozick. (The extent of the resemblance is made apparent in Lewis.) The Newcomb Problem asks us to consider two boxes, one transparent and one opaque. In the transparent box we can see a thousand dollars. The opaque box may contain either a million dollars or nothing. We have two choices: take the contents of the opaque box or take the contents of both boxes. We know before choosing that a reliable predictor of our behavior has put a million dollars in the opaque box if he predicted we would take the first choice and left it empty if he predicted we would take the second. To see that each player in a PD faces a Newcomb problem, consider the following payoff matrix.



\(\bC\) \(\bD\) \(\bC\) \(m,m\) \(0,m+t\) \(\bD\) \(m+t,0\) \(t,t\)

By “cooperating” (choosing the opaque box), each player ensures that the other gets a million dollars (and a thousand extra for defecting). By “defecting” (choosing both boxes) each player ensures that he will get thousand dollars himself (and a million more if the other cooperates). As long as \(m \gt t \gt 0\), the structure of this game is an ordinary two-player, two-move PD (and any such PD can be represented in this form). Furthermore, the arguments for “one-boxing” and “two-boxing” in a Newcomb problem are the same as the arguments for cooperating and defecting in a prisoner's dilemma where there is positive correlation between the moves of the players. Two boxing is a dominant strategy: two boxes are better than one whether the first one is full or empty. On the other hand, if the predictor is reliable, the expected payoff for one-boxing is greater than the expected payoff for two-boxing. (See Hurley (1991) and Bermúdez (2015), however, for arguments that the two puzzles are significantly different.)

The intuition that two-boxing is the rational choice in a Newcomb problem, or that defection is the rational choice in the PD with positive correlation between the players' moves, seems to conflict with the idea that rationality requires maximizing expectation. This apparent conflict has led some to suggest that standard decision theory needs to be refined in cases in which an agent's actions provide evidence for, without causing, the context in which he is acting. In the case of the PD, standard (evidential) decision theory asks Player One to compare his expected utilities of cooperation and defection, which can be written as \(p(\bC_2 \mid \bC_1) \times R + p(\bD_2 \mid \bC_1) \times S\) and \(p(\bC_2 \mid \bD_1) \times T + p(\bD_2 \mid \bD_1) \times P\) (where, for example, \(p(\bC_2 \mid \bC_1)\) is the conditional probability that player Two cooperates given that Player One cooperates). If the players' moves are strongly correlated then \(p(\bC_2 \mid \bC_1)\) and \(p(\bD_2 \mid \bD_1)\) will be close to one and \(p(\bC_2 \mid \bD_1)\) and \(p(\bD_2 \mid \bC_1)\) will be close to zero. On the suggested revision, these conditional probabilities should be replaced by some kind of causally conditional probabilities, which might (on some accounts) be expressed by phrases like “the probability that if One were to cooperate, Two would also cooperate.” When the moves are causality independent this would just be the probability that Two cooperates.

The rather far-fetched scenario described in Newcomb's Problem initially led some to doubt the importance of the distinction between causal and evidential decision theory. Lewis argues that the link to the PD suggests that situations where the two decisions diverge are not so unusual, and recent writings on causal decision theory contain many examples far less bizarre than Newcomb's problem. (See Joyce, for example.)

In recent years technical machinery from the epistemic foundations of game theory literature and various logics of conditionals has been employed to represent arguments for cooperation and defection in prisoner's dilemma games between replicas (and for one-boxing and two-boxing in the Newcomb problem). See Bonanno for one example and a discussion of several others. These representations make clear some subtle assumptions about the nature of rationality that underly the arguments. Despite the increasing sophistication of the discussion, however, there remain people committed to each view.

It might be noted that what is here called “PD between replicas” is usually called “PD with twins” in the literature. One reason for the present nomenclature is to distinguish these ideas from an experimental literature reporting on PD games played with real (identical or fraternal) twins. (See, for example, Segal and Hershberger.) It turns out that twins are more likely to cooperate in a PD than strangers, but there seems to be no suggestion that the reasoning that leads them to do so follows the controversial arguments presented above.

8. The Stag Hunt and the PD

The idea mentioned in the introduction that the PD models a problem of cooperation among rational agents is sometimes criticized because, in a true PD, the cooperative outcome is not a nash equilibrium. Any “problem” of this nature, the critics contend, would be an unsolvable one. (See for example, Sugden or Binmore 2005, chapter 4.5.) By changing the payoff structure of the PD slightly, so that the reward payoff exceeds the temptation payoff, we obtain a game where mutual cooperation, as well as mutual defection, is a nash equilibrium. This game is known as the stag hunt. It might provide a better model for situations where cooperation is difficult, but still possible, and it may also be a better fit for other roles sometimes assigned to the PD. More specifically, a stag hunt is a two player, two move game with a payoff matrix like that for the PD given in section 1 where the conditions PD1 are replaced by:

(SH) a. \(R \gt T\)

b. \(R \gt P\)

c. \(P \gt S\)

The fable dramatizing the game and providing its name, gleaned from a passage in Rousseau's Discourse on Inequality, concerns a hunting expedition rather than a jail cell interrogation. Two hunters are are looking to bag a stag. Success is uncertain and, if it comes, require the efforts of both. On the other hand, either hunter can forsake his partner and catch a hare with a good chance of success. A typical payoff matrix is shown below.



\(\bC\) \(\bD\) \(\bC\) \(4,4\) \(0,3\) \(\bD\) \(3,0\) \(3,3\)

Here the “cooperative” move is hunting stag with one's partner and “defection” is hunting hare by oneself. The “temptation” payoff in a stag hunt is no longer much of a temptation, but we retain the payoff terminology for ease of exposition. In this case the temptation and punishment penalties are identical, perhaps reflecting the fact that my partner's choice of prey has no effect on my success in hare-hunting. Alternatively we could have temptation exceeding punishment, perhaps because hunting hare is more rewarding together than alone (though still less rewarding, of course, than hunting stag together), or we could have punishment exceeding temptation, perhaps because a second hare hunter represents unhelpful competition. Either way, the essence of the Stag Hunt remains. There are two equilibria, one unanimously preferred to the other. The stag hunt becomes a “dilemma” when rationality dictates that both players choose the action leading to the inferior equilibrium. It is clear that if I am certain that my partner will hunt stag I should join him and that if I am certain that he will hunt hare I should hunt hare as well. For this reason games with this structure are sometimes called games of “assurance” or “trust.” (But these should not be confused with “trust game” versions of the asynchronous PD discussed in the following section.) If I do not know what my partner will do, standard decision theory tells me to maximize expectation. This requires, however, that I estimate the probability of my partner playing \(\bC\) or \(\bD\). If I lack information to form any such estimates, then one putative principle of rationality (“indifference”) suggests that I ought to treat all options as equally likely. By this criterion I ought to hunt hare if and only if the following condition is met:

(SHD) \(T + P \gt R + S\)

When SHD obtains, hare hunting is said to be the “risk-dominant” equilibrium. Let us call a stag hunt game where this condition is met a stag hunt dilemma. The matrix above provides one example.

Another proposed principle of rationality (“maximin”) suggests that I ought to consider the worst payoff I could obtain under any course of action, and choose that action that maximizes this value. Since the sucker payoff is the worst payoff in a stag hunt, this principle suggests that any stag hunt presents a dilemma. Maximin, however, makes more sense as a principle of rationality for zero sum games, where it can be assumed that a rational opponent is trying to minimize my score, than for games like stag hunt, where a rational opponent may be quite happy to see me do well, as long as he does so as well.

The stag hunt can be generalized in the obvious way to accommodate asymmetric and cardinal payoffs. The quadrilateral formed by the games' graphical representation is convex, so the pure/impure distinction no longer applies. (In other words, in a stag hunt no mixed strategies are ever preferred to mutual cooperation.) The most obvious way to generalize the game to many players would retain the condition that there be exactly two equilibria, one unanimously preferred to the other. This might be a good model for cooperative activity in which success requires full cooperation. Imagine, for example, that a single polluter would spoil a lake, or a single leak would thwart an investigation. If many agents are involved and, by appeal to indifference or for other reasons, we estimate a fifty-fifty chance of cooperation from each, then these examples would represent stag hunt dilemmas in an extreme form. Everyone would benefit if all cooperate, but only a very trusting fool would think it rational to cooperate himself. Perhaps some broader generalization to the many-person case would represent the structure of other familiar social phenomena, but that matter will not be pursued here.

The cooperative outcome in the stag hunt can be assured by many of the same means as are discussed here for the PD. As might be expected, cooperation is somewhat easier to come by in the two-person stag hunt than in the two-person PD. Details will not be given here, but the interested reader may consult Skyrms 2004, which is responsible for a resurgence of interest in this game.

9. Asynchronous Moves and Trust Games

It has often been argued that rational self-interested players can obtain the cooperative outcome by making their moves conditional on the moves of the other player. Peter Danielson, for example, favors a strategy of reciprocal cooperation: if the other player would cooperate if you cooperate and would defect if you don't, then cooperate, but otherwise defect. Conditional strategies like this are ruled out in the versions of the game described above, but they may be possible in versions that more accurately model real world situations. In this section and the next, we consider two such versions. In this section we eliminate the requirement that the two players move simultaneously. Consider the situation of a firm whose sole competitor has just lowered prices. Or suppose the buyer of a car has just paid the agreed purchase price and the seller has not yet handed over the title. We can think of these as situations in which one player has to choose to cooperate or defect after the other player has already made a similar choice. The corresponding game is an asynchronous or extended PD.

Careful discussion of an asynchronous PD example, as Skyrms (1998) and Vanderschraaf recently note, occurs in the writings of David Hume, well before Flood and Dresher's formulation of the ordinary PD. Hume writes about two neighboring grain farmers:

Your corn is ripe today; mine will be so tomorrow. ’Tis profitable for us both, that I shou'd labour with you to-day, and that you shou'd aid me to-morrow. I have no kindness for you, and know you have as little for me. I will not, therefore, take any pains on your account; and should I labour with you upon my own account, in expectation of a return, I know I shou'd be disappointed, and that I shou'd in vain depend upon your gratitude. Here then I leave you to labour alone: You treat me in the same manner. The seasons change; and both of us lose our harvests for want of mutual confidence and security.

In deference to Hume, Skyrms and Vanderschraaf refer to this kind of asynchronous PD as the “farmer's dilemma.” It is instructive to picture it in a tree diagram.

Figure 5

Here, time flows to the right. The node marked by a square indicates Player One's choice point, those marked by circles indicate Player Two's. The moves and the payoffs to each player are exactly as in the ordinary PD, but here Player Two can choose his move according to what Player One does. Tree diagrams like Figure 5 are said to be extensive-form game representations, whereas the payoff matrices given previously are normal-form representations. As Hume's analysis indicates, making the game asynchronous does not remove the dilemma. Player One knows that if he were to choose \(\bC\) on the first move, Player Two would choose \(\bD\) on the second move (since she prefers the temptation to the reward), so he would himself end up with the sucker payoff. If Player One were to choose \(\bD\), Player Two would still choose \(\bD\) (since she prefers the punishment to the sucker payoff), and he would end up with the punishment payoff. Since he prefers the punishment payoff to the sucker payoff, Player One will choose \(\bD\) on the first move and both players will end up with the punishment payoff. This kind of “backward” reasoning, in which the players first evaluate what would happen on the last move if various game histories were realized, and use this to determine what would happen on preceding moves applies quite broadly to games in extensive form, and a more general version of it will be discussed under finite iteration below.

The farmer's dilemma can be represented in normal form by understanding Player One to be choosing between \(\bC\) and \(\bD\) and Player Two to be (simultaneously) choosing among four conditional moves: cooperate unconditionally \((\bCu)\), defect unconditionally \((\bDu)\), imitate Player One's move \((\bI)\), and do the opposite of Player One's move \((\bO)\). The result is a two player game with the following matrix.



\(\bCu\) \(\bDu\) \(\bI\) \(\bO\) \(\bC\) \(R,R\) \(S,T\) \(R,R\) \(S,T\) \(\bD\) \(T,S\) \(P,P\) \(P,P\) \(T,S\)

The reader may note that this game is a (multiple-move) equilibrium dilemma. The sole (weak) nash equilibrium results when Player One chooses \(\bD\) and Player Two chooses \(\bDu\), thereby achieving for themselves the inferior payoffs of \(P\) and \(P\). The game is not, however, a dominance PD. Indeed, there is no dominant move for either player. It is commonly believed that rational self-interested players will reach a nash equilibrium even when neither player has a dominant move. If so, the farmer's dilemma is still a dilemma.

To preserve the symmetry between the players that characterizes the ordinary PD, we may wish to modify the asynchronous game. Let us take extended PD to be played in stages. First each player chooses a first move \((\bC \text{ or } \bD)\) and a second move \((\bCu, \bDu, \bI, \text{ or } \bO)\). Next a referee determines who moves first, giving each player an equal chance. Finally the outcome is computed in the appropriate way. For example, suppose Row plays \((\bD, \bO)\) (meaning that he will defect if he moves first and do the opposite of his opponent if he moves second) and Column plays \((\bC, \bDu)\). Then Row will get \(P\) if he goes first and \(T\) if he goes second, which implies that his expected payoff is \(\tfrac{1}{2}(P+T)\). Column will get \(S\) if she goes first and \(P\) if she goes second, giving her an expected payoff of \(\tfrac{1}{2}(P+S)\). It is straightforward, but tedious, to calculate the entire eight by eight payoff matrix. After doing so, the reader may observe that, like the farmer's dilemma, the symmetric form of the extended PD is an equilibrium PD, but not a dominance PD. The sole nash equilibrium occurs when both players adopt the strategy \((\bD, \bDu)\), thereby achieving the inferior payoffs of \((P,P)\).

Some particularly simple and suggestive variations of on this theme have been studied under the labels “investor game” or “trust game” (See, for example, Kreps (1990), Berg (1995) and Bicchieri and Suntuoso (2015) and note that the game nomenclature is not consistent accross these references.) Player One is given \(s\) units of utility. He may choose to pass any number \(s\prime \lt s\) to a “trustee”, who triples that number and passes it to Player Two. Player Two may then either keep the units that she has or return some of them to Player One. So formulated, the game has the advantage that one can take the proportion of her utility that a player surrenders as her degree of cooperativeness. If one restricts the moves so that Player One may give none or \(s\), and Player Two may give none or \(2s\) one gets exactly the farmer's dilemma).

In the farmer's dilemma and the trust game, unlike the PD, the similarly-labeled moves of the two players seem to have somewhat different flavors. We are more likely to regard Player One's cooperation as generous or perhaps calculated (even if we regard the calculations involved to be irrational), and Player Two's as fair. The label trusting is appropriate only with regard to Player One's cooperative move, though Player Two's cooperation might be thought to show her to be worthy of that trust.

It may be worth noting that an asynchronous version of the stag hunt, unlike the PD, presents few issues of interest. If the first player does his part in the hunt for stag on day one, the second should do her part on day two. If he hunts hare on day one, she should do likewise on day two. The first player, realizing this, should hunt stag on day one. So rational players should have no difficulty reaching the cooperative outcome in the asynchronous stag hunt.

10. Transparency

Another way that conditional moves can be introduced into the PD is by assuming that players have the property that David Gauthier has labeled transparency. A fully transparent player is one whose intentions are completely visible to others. Nobody holds that we humans are fully transparent, but the observation that we can often successfully predict what others will do suggests that we are at least “translucent.” Furthermore agents of larger scale, like firms or countries, which may have to publicly deliberate before acting, may be more transparent than we are. Thus there may be some theoretical interest in investigations of PDs with transparent players. Such players could presumably execute conditional strategies more sophisticated than those of the (non-transparent) extended game players, strategies, for example that are conditional on the conditional strategies employed by others. There is some difficulty, however, in determining exactly what strategies are feasible for such players. Suppose Row adopted the strategy “do the same as Column” and Column adopted the strategy “do the opposite of Row”. There is no way that both these strategies could be satisfied. On the other hand, if each adopted the strategy “imitate the other player”, there are two ways the strategies could be satisfied, and there is no way to determine which of the two they would adopt. Nigel Howard, who was probably the first to study such conditional strategies systematically, avoided this difficulty by insisting on a rigidly typed hierarchy of games. At the base level we have the ordinary PD game, where each player chooses between \(\bC\) and \(\bD\). For any game \(G\) in the hierarchy we can generate two new games \(RG\) and \(CG\). In \(RG\), Column has the same moves as in game \(G\) and Row can choose any function that assigns \(\bC\) or \(\bD\) to each of Column's possible moves. Similarly in \(CG\), Row has the same moves as in \(G\) and Column has a new set of conditional moves. For example, if [PD] is the base level game, then \(C\)[PD] is the game in which Column can choose from among the strategies \(\bCu\), \(\bDu\), \(\bI\) and \(\bO\) mentioned above. Howard observed that in the two third level games \(RC\)[PD] and \(CR\)[PD] (and in every higher level game) there is an equilibrium outcome giving each player \(R\). In particular, such an equilibrium is reached when one player plays \(\bI\) and the other cooperates when his opponent plays \(\bI\) and defects when his opponent plays \(\bCu\), \(\bDu\) or \(\bO\). Notice that this last strategy is tantamount to Danielson's reciprocal cooperation described in the last section.

The lesson of all this for rational action is not clear. Suppose two players in a PD were sufficiently transparent to employ the conditional strategies of higher level games. How do they decide what level game to play? Who chooses the imitation move and who chooses reciprocal cooperation? To make a move in a higher level game is presumably to form an intention observable by the other player. But why should either player expect the intention to be carried out if there is benefit in ignoring it?

Conditional strategies have a more convincing application when we take our inquiry as directed, not towards playing the PD, but as designing agents who would play it well with a variety of likely opponents. This is the viewpoint of Danielson. (See also J.V. Howard for an earlier enlightening discussion of this viewpoint.) A conditional strategy is not an intention that a player forms as a move in a game, but a deterministic algorithm defining a kind of player. Indeed, one of the lessons of the PD may be that transparent agents are better off if they can form irrevocable “action protocols” rather than always following the intentions they may form at the time of action. Danielson does not limit himself a priori to strategies within Howard's hierarchy. An agent is simply a computer program, which can contain lines permitting other programs to read and execute it. We could easily write two such programs, each designed to determine whether its opponent plays \(\bC\) or \(\bD\) and to do the opposite. What happens when these two play a PD depends on the details of implementation, but it is likely that they will be “incoherent,” i.e., they will enter endless loops and be unable to make any move at all. To be successful a program should be able to move when paired with a variety of other programs, including copies of itself, and it should be able to get valuable outcomes. Programs implementing \(\bI\) and \(\bO\) in a straightforward way are not likely to succeed because when paired with each other they will be incoherent. Programs implementing \(\bDu\) are not likely to succeed because they get only \(P\) when paired with their clones. Those implementing \(\bCu\) are not likely to succeed because they get only \(S\) when paired with programs that recognize and exploit their unconditionally cooperative nature. There is some vagueness in the criteria of success. In Howard's scheme we could compare a conditional strategy with all the possible alternatives of that level. Here, where any two programs can be paired, that approach is senseless. Nevertheless, certain programs seem to do well when paired with a wide variety of players. One is a version of the strategy that Gauthier has advocated as constrained maximization. The idea is that a player \(j\) should cooperate if the other would cooperate if \(j\) did, and defect otherwise. As stated, this appears to be a strategy for the \(RC\)[PD] or \(CR\)[PD] games. It is not clear how a program implementing it would move (if indeed it does move) when paired with itself. Danielson is able to construct an approximation to constrained maximization, however, that does cooperate with itself. Danielson's program (and other implementations of constrained maximization) cannot be coherently paired with everything. Nevertheless it does move and score well against familiar strategies. It cooperates with \(\bCu\) and itself and it defects against \(\bDu\). If it is coherently paired it seems guaranteed a payoff no worse than \(P\).

A second successful program models Danielson's reciprocal cooperation. Again, it is not clear that the strategy (as formulated above) allows it to cooperate (or make any move) with itself, but Danielson is able to construct an approximation that does. The (approximate) reciprocal cooperation does as well as (approximate) constrained maximization against itself, \(\bDu\) and constrained maximization. Against \(\bCu\) it does even better, getting \(T\) where constrained maximization got only \(R\).

11. Finite Iteration

Many of the situations that are alleged to have the structure of the PD, like defense appropriations of military rivals or price setting for duopolistic firms, are better modeled by an iterated version of the game in which players play the PD repeatedly, retaining access at each round to the results of all previous rounds. In these iterated PDs (hence forth IPDs) players who defect in one round can be “punished” by defections in subsequent rounds and those who cooperate can be rewarded by cooperation. Thus the appropriate strategy for rationally self-interested players is no longer obvious. The theoretical answer to this question, it turns out, depends strongly on the definition of IPD employed and the knowledge attributed to rational players.

An IPD can be represented in extensive form by a tree diagram like the one for the farmer's dilemma above.

Figure 6

Here we have an IPD of length two. The end of each of the two rounds of the game is marked by a dotted vertical line. The payoffs to each of the two players (obtained by adding their payoffs for the two rounds) are listed at the end of each path through the tree. The representation differs from the previous one in that the two nodes on each branch within the same division mark simultaneous choices by the two players. Since neither player knows the move of the other at the same round, the IPD does not qualify as one of the game theorist's standard “games of perfect information.” If the players move in succession rather than simultaneously (which we might indicate by removing the dotted vertical lines), the resulting game is an iterated farmer's dilemma, which does meet the game theorist's definition and which shares many of the features that make the IPD interesting.

Like the farmer's dilemma, an IPD can, in theory, be represented in normal form by taking the players' moves to be strategies telling them how to move if they should reach any node at the end of a round of the game tree. The number of strategies increases very rapidly with the length of the game so that it is impossible in practice to write out the normal form for all but the shortest IPD's. Every pair of strategies determines a “play” of the game, i.e., a path through the extensive-form tree.

In a game like this, the notion of nash equilibrium loses some of its privileged status. Recall that a pair of moves is a nash equilibrium if each is a best reply to the other. Let us extend the notation used in the discussion of the asynchronous PD and let \(\bDu\) be the strategy that calls for defection at every node of an IPD. It is easy to see that \(\bDu\) and \(\bDu\) form a nash equilibrium. But against \(\bDu\), a strategy that calls for defection unless the other player cooperated at, say, the fifteenth node, would determine the same play (and therefore the same payoffs) as \(\bDu\) itself does. The components that call for cooperation never come into play, because the other player does not cooperate on the fifteenth (or any other) move. Similarly, a strategy calling for cooperation only after the second cooperation by itself does equally well. Thus these strategies and many others form nash equilibria with \(\bDu\). There is a sense in which these strategies are clearly not equally rational. Although they yield the same payoffs at the nodes along the path representing the actual play, they would not yield the same payoffs if other nodes had been reached. If Player One had cooperated in the past, that would still provide no good reason for him to cooperate now. A nash equilibrium requires only that the two strategies are best replies to each other as the game actually develops. A stronger solution concept for extensive-form games requires that the two strategies would still be best replies to each other no matter what node on the game tree were reached. This notion of subgame-perfect equilibrium is defined and defended in Selten 1975. It can be expressed by saying that the strategy-pair is a nash equilibrium for every subgame of the original game, where a subgame is the result of taking a node of the original game tree as the root, pruning away everything that does not descend from it.

Given this new, stronger solution concept, we can ask about the solutions to the IPD. There is a significant theoretical difference on this matter between IPDs of fixed, finite length, like the one pictured above, and those of infinite or indefinitely finite length. In games of the first kind, one can prove by an argument known as backward induction that \(\bDu\), \(\bDu\) is the only subgame perfect equilibrium. Suppose the players know the game will last exactly \(n\) rounds. Then, no matter what node have been reached, at round \(n-1\) the players face an ordinary (“one-shot”) PD, and they will defect. At round \(n-2\) the players know that, whatever they do now, they will both defect at the next round. Thus it is rational for them to defect now as well. By repeating this argument sufficiently many times, the rational players deduce that they should defect at every node on the tree. Indeed, since at every node defection is a best response to any move, there can be no other subgame-perfect equilibria.

In practice, there is not a great difference between how people behave in long fixed-length IPDs (except in the final few rounds) and those of indeterminate length. This suggests that some of the rationality and common knowledge assumptions used in the backward induction argument (and elsewhere in game theory) are unrealistic. There is a considerable literature attempting to formulate the argument carefully, examine its assumptions, and to see how relaxing unrealistic assumptions might change the rationally acceptable strategies in the PD and other games of fixed length. (For a small sample, see Bovens, Kreps and Wilson, Pettit and Sugden, Sobel 1993 and Binmore 1997).

Player One's belief that there is a slight chance that Two might pursue an “irrational” strategy other than continual defection could make it rational for her to cooperate frequently herself. Indeed, even if One were certain of Two's rationality, One's belief that there was some chance that Two believed she harbored such doubts could have the same effect. Thus the argument for continual defection in the IPD of fixed length depends on complex iterated claims of certain knowledge of rationality. An even more unrealistic assumption, noted by Rabinowicz and others, is that each player continue to believe that the other will choose rationally on the next move even after evidence of irrational play on previous moves. For example, it is assumed that, at the node reached after a long series of moves (\(\bC\), \(\bC\)), …,(\(\bC\), \(\bC\)), Player One will choose \(\bD\) despite never having done so before.

Some have used these kinds of observation to argue that the backward induction argument shows that standard assumptions about rationality (with other plausible assumptions) are inconsistent or self-defeating. For (with plausible assumptions) one way to ensure that a rational player will doubt one's own rationality is to behave irrationally. In the fixed-length IPD, for example, Player One may be able to deduce that, if she were to follow an appropriate “irrational” strategy, Player Two would rationally react so that they can achieve mutual cooperation in almost all rounds. So our assumptions seem to imply both that Player One should continually defect and that she would do better if she didn't. (See Skyrms 1990, pp. 125–139 and Bicchieri 1989.)

12. The Centipede and the Finite IPD

Many of the issues raised by the fixed-length IPD can be raised in even starker form by a somewhat simpler game. Consider a PD in which they punishment payoff is zero. Now iterate the asynchronous version of this game a fixed number times. Imagine that both players are restricted to highly “punitive” strategies according to which, they must always defect against a player who has ever defected. (One important strategy of this variety is discussed below under the label GRIM.) The result is a centipede game. A particularly nice realization is given by Sobel 2005. A stack of \(n\) one-dollar bills lies on a table. Players take turns taking money from the stack, one or two bills per turn. The game ends when the stack runs out or one of the players takes two bills (whichever comes first). Both players keep what they have taken to that point. The extensive form of the game with for \(n=4\) is pictured below.

Figure 7

Presumably the true centipede would contain 100 “legs” and the general form discussed here should really be called the “\(n\)-tipede.” The game appears to be discussed first in Rosenthal.

As in the fixed-length PD, a backward induction argument easily establishes that a rational player should take two bills on his first move, giving her a payoff of two or three dollars, depending on whether she moves first or second, and leaving the remainder of the \(n\) dollars undistributed. In more technical terms, the only nash equilibria of the game are those where the first player takes two dollars on the first move and the only subgame perfect equilibrium is the one in which both players take two dollars on any turn they should get. Again, common sense and experimental evidence suggest that real players rarely act in this way and this leads to questions about exactly what assumptions this kind of argument requires and whether they are realistic. (In addition to the sample mentioned in the section on finitely iterated PDs, see, for example, Aumann 1998, Selten 1978, and Rabinowicz.) The centipede also raises some of the same questions about cooperation and socially desirable altruism as does the PD and it is a favorite tool in empirical investigations of game playing.

13. Infinite Iteration

One way to avoid the dubious conclusion of the backward induction argument without delving too deeply into conditions of knowledge and rationality is to consider infinitely repeated PDs. No human agents can actually play an infinitely repeated game, of course, but the infinite IPD has been considered an appropriate way to model a series of interactions in which the participants never have reason to think the current interaction is their last. In this setting a pair of strategies determines an infinite path through of the game tree. If the payoffs of the one-shot game are positive, their total along any such path is infinite. This makes it somewhat awkward to compare strategies. In many cases, the average payoff per round approaches a limit as the number of rounds increases, and so that limit can conveniently serve as the payoff. (See Binmore 1992, page 365 for further justification.) For example, if we confine ourselves to those strategies that can be implemented by mechanical devices (with finite memories and speeds of computation), then the sequence of payoffs to each player will always, after a finite number of rounds, cycle repeatedly through a particular finite sequence of payoffs. The limit of the average payoff per round will then be the average payoff in the cycle. In recent years, Press and Dyson have shown that for many purposes, investigation of the infinite IPD can be confined to the “memory-one” strategies, in which the probability of cooperating in any round depends only on what happened in the previous meeting between the strategies. The average payoff per round is again always well-defined in the limit. The ideas of Press and Dyson have inspired much new work on the infinite IPD. (See Zero-Determinant Strategies below.) Since there is no last round, it is obvious that backward induction does not apply to the infinite IPD.

14. Indefinite Iteration

Most contemporary investigations the IPD take it to be neither infinite nor of fixed finite length but rather of indeterminate length. This is accomplished by including in the game specification a probability \(p\) (the “shadow of the future”) such that at each round the game will continue with probability \(p\). Alternatively, a “discount factor” \(p\) is applied to the payoffs after each round so that nearby payoffs are valued more highly than distant ones. Mathematically, it makes little difference whether \(p\) is regarded as a probability of continuation or a discount on payoffs. The value of cooperation at a given stage in an IPD clearly depends on the odds of meeting one's opponent in later rounds. (This has been said to explain why the level of courtesy is higher in a village than a metropolis and why customers tend leave better tips in local restaurants than distant ones.) As \(p\) approaches zero, the IPD becomes a one-shot PD, and the value of defection increases. As \(p\) approaches one the IPD becomes an infinite IPD, and the value of defection decreases. It is also customary to insist that the game has the property labeled RCA above, so that (in the symmetric game) players do better by cooperating on every round than they would do by “taking turns” — you cooperate while I defect and then I cooperate while you defect.

There is an observation, apparently originating in Kavka 1983, and given more mathematical form in Carroll, that the backward induction argument applies as long as an upper bound to the length of the game is common knowledge. For if \(b\) is such an upper bound, then, if the players were to get to stage \(b\), they would know that it was the last round and they would defect; if they were to get to stage \(b-1\), they would know that their behavior on this round cannot affect the decision to defect on the next, and so they would defect; and so on. It seems an easy matter to compute upper bounds on the number of interactions in real-life situations. For example, since shopkeeper Jones cannot make more than one sale a second and since he will live less than a thousand years, he and customer Smith can calculate (conservatively) that they cannot possibly conduct more than \(10^{12}\) transactions. It is instructive to examine this argument more closely in order to dramatize the assumptions made in standard treatments of the indefinite IPD and other indefinitely repeated games. Note first that, in an indefinite IPD as described above, there can be no upper bound on the length of the game. There is, instead, some fixed probability \(p\) that, at any time in which the game is still being played, it will continue to be played with probability \(p\). If the interaction of Smith and Jones were modeled as an indefinite IPD, therefore, the probability of their interacting in a thousand years would not be zero, but rather some number greater than \(p^k\) where \(p\) is the probability of their interacting again now and \(k\) is the number of seconds in a thousand years. A more realistic way to model the interaction might be to allow the value of \(p\) to decrease as the game progressed. As long as \(p\) always remains greater than zero, however, it remains true that there can be no upper bound on the number of possible interactions, i.e., no time at which the probability of future interactions becomes zero. Suppose, on the other hand, that there was a number \(n\) such that that there was zero probability of the game's continuing to stage \(n\). Let \(p_1, \ldots, p_n\), be the probabilities that game continues after \(\text{stage } 1, \ldots, \text{stage } n\). Then there must be a smallest \(i\) such that \(p_i\) becomes \(0\). (It would happen at \(i=n\) if not sooner.) Given the standard common knowledge assumptions that we have been making, the players would know this value of \(i\), and the IPD would be one of fixed length, and not an indefinite IPD at all. In the case of the shopkeeper and his customer, we are to suppose that both know today that their last interaction will occur, let's say, at noon on June 10th, 2020. The very plausible idea that we began with, viz., that some upper bounds on the number of interactions are common knowledge, even though the smallest upper bound is not, is incompatible with the assumption that we know all the continuation probabilities \(p_i\) from the start.

As Becker and Cudd astutely observe, we don't need an upper bound on the number of possible iterations to make a backward induction argument for defection possible. If the players know all the values of \(p_i\) from the outset, then, as long as the value of \(p_i\) becomes and remains sufficiently small, they (and we) can compute a stage \(k\) at which the risk of future punishment and the chance of future reward no longer outweighs the benefit of immediate defection. So they know their opponent will defect at stage \(k\), and the induction begins. This modification of the Kavka/Carroll argument, however, only further exposes the implausibility of its assumptions. Not only are Smith and Jones expected to believe that there is non-zero probability that they will be interacting in a thousand years, each is expected to be able to compute the precise day on which future interactions will become and remain so unlikely that their expected future return is outweighed by that day's payoff. Furthermore each is expected to believe that the other has made this computation, and that the other expects him to have made it, and so on.

Axelrod and Tit for Tat

The iterated version of the PD was discussed from the time the game was devised, but interest accelerated after influential publications of Robert Axelrod in the early eighties. Axelrod invited professional game theorists to submit computer programs for playing IPDs. All the programs were entered into a tournament in which each played every other (as well as a clone of itself and a strategy that cooperated and defected at random) hundreds of times. It is easy to see that in a game like this no strategy is “best” in the sense that its score would be highest among any group of competitors. If the other strategies never consider the previous history of interaction in choosing their next move, it would be best to defect unconditionally. If the other strategies all begin by cooperating and then “punish” any defection against themselves by defecting on all subsequent rounds, then a policy of unconditional cooperation is better. Nevertheless, as in the transparent game, some strategies have features that seem to allow them to do well in a variety of environments. The strategy that scored highest in Axelrod's initial tournament, Tit for Tat (henceforth TFT), simply cooperates on the first round and imitates its opponent's previous move thereafter. More significant than TFT's initial victory, perhaps, is the fact that it won Axelrod's second tournament, whose sixty three entrants were all given the results of the first tournament. In analyzing the his second tournament, Axelrod noted that each of the entrants could be assigned one of five “representative” strategies in such a way that a strategy's success against a set of others can be accurately predicted by its success against their representative. As a further demonstration of the strength of TFT, he calculated the scores each strategy would have received in tournaments in which one of the representative strategies was five times as common as in the original tournament. TFT received the highest score in all but one of these hypothetical tournaments.

Axelrod attributed the success of TFT to four properties. It is nice, meaning that it is never the first to defect. The eight nice entries in Axelrod's tournament were the eight highest ranking strategies. It is retaliatory, making it difficult for it to be exploited by the rules that were not nice. It is forgiving, in the sense of being willing to cooperate even with those who have defected against it (provided their defection wasn't in the immediately preceding round). An unforgiving rule is incapable of ever getting the reward payoff after its opponent has defected once. And it is clear, presumably making it easier for other strategies to predict its behavior so as to facilitate mutually beneficial interaction.

Suggestive as Axelrod's discussion is, it is worth noting that the ideas are not formulated precisely enough to permit a rigorous demonstration of the supremacy of TFT. One doesn't know, for example, the extent of the class of strategies that might have the four properties outlined, or what success criteria might be implied by having them. It is true that if one's opponent is playing TFT (and the shadow of the future is sufficiently large) then one's maximum payoff is obtained by a strategy that results in mutual cooperation on every round. Since TFT is itself one such strategy this implies that TFT forms a nash equilibrium with itself in the space of all strategies. But that does not particularly distinguish TFT, for \(\bDu\), \(\bDu\) is also a nash equilibrium. Indeed, a “folk theorem” of iterated game theory (now widely published — see, for example, Binmore 1992, pp. 373–377) implies that, for any \(p\), \(0 \le p \le 1\) there is a nash equilibrium in which \(p\) is the fraction of times that mutual cooperation occurs. Indeed TFT is, in some respects, worse than many of these other equilibrium strategies, because the folk theorem can be sharpened to a similar result about subgame perfect equilibria. TFT is, in general, not subgame perfect. For, were one TFT player (per impossible) to defect against another in a single round, the second would have done better as an unconditional cooperator.

Post-Axelrod

After publication of Axelrod, 1984, a number of strategies commonly thought to improve on TFT were identified. (Since success in an IPD tournament depends on the other stragies present, it is not clear exactly what this claim means or how it might be demonstrated.) The first of these was Nowak and Sigmond's Pavlov, also known as, Win-Stay Lose-Shift (WSLS), which conditions each non-initial move on its own previous move as well as its opponent's. More specifically, it cooperates if it and its opponent previously moved alike and it defects if they previously moved differently. Equivalently, it repeats its move after success (temptation or reward) and changes it after failure (punishment or sucker). Hence the names. This strategy does well in environments like that of Axelrod's tournment, but poorly when many unconditional defectors or random players are present. It is discussed further under the label \(\bP_1\) in the sections on error and evolution below. A second family of these is Gradual Tit for Tat (henceforth GrdTFT). GrdTFT differs from TFT in two respects. First, it gradually increases the string of punishing defection responses to each defection by its opponent. Second, it apologizes for each string of defections by cooperating in the subsequent two rounds. The first property ensures that (unlike TFT) it will defect with increasing frequency against a random player. The second ensures that (unlike TFT) it will quickly establish a regime of mutual cooperation with suspicious versions of TFT (i.e., versions of TFT that defect on their first move). Beaufils et al show that the version of GrdTFT in which the string of defections is increased by one each time it is exploited wins round robin tournaments populated by a selection of “good” IPD strategies (including TFT) that the authors chose after an examination of previous tournaments. Tzafestas (1998) argues that, in making a each move depend on the entire prior history of the game, GrdTFT incorporates undesirable memory requirements. She suggests that equal success might be obtained with an "adaptive" strategy, that tracks a measure of the opponent's cooperativeness or responsiveness across a narrow window of recent moves and chooses its move according to whether this measure (the "world") exceeds some threshold. The critique seems misguided: maintaining a count of prior defections seems no more burdensome than updating the world variable. Nevertheless Tzafestas is able to show that one of the strategies she identifies outperforms both TFT and GrdTFT in the very same environment that Beaufils had constructed.

In more recent years enthusiasm about TFT has been tempered by increasing skepticism.(See, for example, Binmore 2015 (p. 30) and Northcott and Alexandrova (pp. 71-78). Evidence has emerged that the striking success of TFT in Axelrod's tournaments may be partly due to features particular to Axelrod's setup. Rapoport et al (2015) suggest that, instead of conducting a round-robin tournament in which every strategy plays every strategy, one might divide the initial population of stratgies randomly into equal-size groups, conduct round-robin tournaments within each group. and then a championship round-robin tournament among the group winners. They find that, with the same initial population of strategies present in Axelrod's first tournament, the strategies ranked two and six in that tournament both perform considerably better than top-ranked TFT. Kretz (2011) finds that, in round-robin tournaments among populations of strategies that can only condition on a small number of prior moves (of which TFT is clearly one) relative performance of strategies is sensitive to the payoff values in the PD matrix. (Interestingly, this is so even if the PDs all satisfy or fail to satisfy the condition R+P=T+S, characterizing exchange games, and if they all satisfy or fail to satisfy the RCA condition, R>½(T+S).

Equally telling, perhaps, are the results of a more recent tournament using the same paramters as Axelrod did. To mark the twentieth anniversary of the publication of Axelrod's book, a number of similar tournaments were staged at the IEEE Congress on Evolutionary Computing in Portland in 2004 and the IEEE Symposium on Computational Intelligence and Games in Colchester 2005. Kendall et al 2007 describes the tournaments and contains several papers by authors who submitted winning entries. Most of the tournaments were deliberately designed to differ significantly from Axelrod's (and some of these are briefly discussed in the section on signaling below). In the one that most closely replicated Axelrod's tournaments. however, TFT finished only fourteenth out of the fifty strategies submitted.

Of Axelrod's five suggested success criteria, the one that seems most clearly undermined by the later tournament is “clarity”. Neither of the two highest-scoring strategies, Adaptive Pa