History is full of instances where statistical tools predicted embarrassingly wrong results due to errors creeping in during data collection and analysis. Given below is the list of such statistical blunders in history.

Literary Digest, 1936

In 1936, Literary Digest, a national magazine of the time, sent out 10 million “straw” ballots asking people to tell them who they planned on voting for in the 1936 presidential election. They received back 2.4 million ballots (only about a 25% return rate, which leaves the door open for voluntary response bias). They predicted Alf Landon would beat Franklin Delanor Roosevelt 57% to 43%. As it turned out, Roosevelt won 62% to 37%.

There were two large problems:

The first major problem with the poll was in the selection process for the names on the mailing list, which were taken from telephone directories, club membership lists, lists of magazine subscribers, etc. Such a list is guaranteed to be slanted toward middle- and upper-class voters, and by default to exclude lower-income voters. One must remember that in 1936, telephones were much more of a luxury than they are today. Furthermore, at a time when there were still 9 million people unemployed, the names of a significant segment of the population would not show up on lists of club memberships and magazine subscribers. At least with regard to economic status, the Literary Digest mailing list was far from being a representative cross-section of the population. This is always a critical problem because voters are generally known to vote their pocketbooks, and it was magnified in the 1936 election when economic issues were preeminent in the minds of the voters. This sort of sample bias is called selection bias.

The second problem with the Literary Digest poll was that out of the 10 million people whose names were on the original mailing list, only about 2.4 million responded to the survey. Thus, the size of the sample was about one-fourth of what was originally intended. People who respond to surveys are different from people who don’t, not only in the obvious way (their attitude toward surveys) but also in more subtle and significant ways. When the response rate is low (as it was in this case, 0.24), a survey is said to suffer from nonresponse bias. This is a special type of selection bias where reluctant and nonresponsive people are excluded from the sample.

Chicago Tribune, 1948

One of the most famous failures of political polling occurred in the 1948 presidential election between Harry Truman (the democratic incumbent) and Thomas Dewey (the republican challenger). There were also a couple of third party candidates. The Gallup, Roper, and Crossley polls all predicted that Dewey would defeat Truman by a significant margin, but in fact, just the opposite happened. The results of the polls and the actual election results (in percentages) are given in the following table:

1948 Election

Candidate Crossley Poll Gallup Poll Roper Poll Election Results Truman 45 44 38 50 Dewey 50 50 53 45 Others 5 6 9 5

Draft Lottery, 1969

The days of the year, from 1 to 366, were written on slips of paper and the slips were placed in plastic capsules. The capsules for each month, starting with January and ending with December, were mixed in a shoebox and then they were dumped into a deep glass container. Capsules were then drawn from the jar one at a time.

The first number drawn was 258 (which corresponds to September 14). Men of draft age (those born between 1944 and 1950) whose birthday was Sept. 14 would then constitute the first group to be drafted. The second group of men to be drafted corresponded to birthdays agreeing with the second capsule/number drawn, and so on.

It was quickly noticed that men with birthdays in December seemed to get drafter earlier, on average, than birthdays in other months. With hindsight begin 20/20, its easy to see how this happened. The capsules were put into the jar month by month. So January capsules probably stayed on the bottom of the jar, while February capsules were on top of them, and so on, until December capsules ended up on top. Whatever subsequent mixing efforts were taken was not sufficient to completely overcome this ordering, producing a sample biased toward later months in the year.