This research note reports a replication of Nyhan and Reifler’s (2015) flu vaccines study embedded within a larger experimental study of flu vaccine intentions and attitudes. Data generated in the experiment do not replicate the backfire effect or the finding that corrections reduce misperceptions about vaccine safety. This suggests that more work is needed to validate the backfire effect, establishing the conditions under which it occurs and the size of its effect.

Political scientists are increasingly aware of the effect of misperceptions on behavioral intentions and attitudes. While Delli Carpini and Keeter (1996) famously note that most citizens do not have much factual knowledge about politics, Kuklinski et al. (2000) point out the differences between being uninformed and misinformed. When a person is misinformed, false, misleading, or unsubstantiated information can create the basis for their policy preferences. Further, the sources used to obtain the misinformation are often directly related to a person’s political preferences.

After the experimental intervention, I measured the effects of each treatment on respondents’ misperceptions about the flu vaccine (You can get the flu from the seasonal flu vaccine); feelings about vaccine safety (Just based on what you know, how safe do you believe the seasonal flu vaccine, meaning the flu vaccine available every year, is generally for most people to take?); and intent to get vaccinated in the future (How likely is it that you will get a flu vaccine for the seasonal flu during future flu seasons?). These measures were taken from Nyhan and Reifler (2015) and the full text and scale of each measure can be found in the online supplementary materials .

As noted by Nyhan and Reifler (2015) , responses to vaccine information might vary based on one’s pre-existing attitudes towards vaccines. Since it was not possible to accurately measure prior vaccine receipt as part of the study, I measured respondents’ general concerns about vaccine safety and side effects. Specifically, I asked, “In general, how concerned are you about serious side effects from vaccines?” prior to the interventions. This was measured on a five point scale ranging from “not at all concerned” to “extremely concerned.”

The replication was embedded in a larger study examining social identity appeals in public health messaging. Respondents who were randomly assigned to the replication condition, “Correction,” received the same text from Nyhan and Reifler’s (2015) protocol, which was taken nearly verbatim from the US Centers for Disease Control and Prevention website. This text told respondents that people cannot contract the flu from the flu shot or live virus nasal spray.

A replication of Nyhan and Reifler’s (2015) flu vaccine study was embedded in a larger experimental study of flu vaccine intentions and attitudes. 1 The replication followed their published methods and procedures. The data were collected using Amazon’s Mechanical Turk platform and due to the nature of the larger study, the worker pool was limited to those with Internet Protocol addresses located in Texas. The study was fielded beginning in March 2016 and ending in May 2016. Respondents were adults located in Texas who were told they would be answering questions meant to elicit their opinions on important political issues of today. Data were collected from 525 respondents, and after dropping respondents found to not be located in Texas, the final sample used in the analysis totaled 474.

Results

Table 1 summarizes the characteristics of the respondents in the full Mechanical Turk sample and compares the demographics of respondents in Nyhan and Reifler’s sample drawn from a YouGov/Polimetrix panel and the replication sample. As expected in samples from Mechanical Turk, participants are not fully representative of the national adult population. The replication sample is younger, with only 3% of the participants in the 60+ category and approximately twice as many 18–29 year olds. The Mechanical Turk sample is also more educated than the original sample, with most of the participants having some college education or a college degree. However, both samples have similar racial compositions and gender distributions.2

Table 1. Characteristics of respondents in Nyhan and Reifler (N + R) (2015) and replication sample (Replication) by (%).

The samples have a few differences in participants’ level of concern about vaccine side effects. The Mechanical Turk participants are less concerned about side effects, with a 10% difference in the “not at all concerned” category between the samples. In the correction treatments, the replication sample was 10% higher in the “very concerned” category, while in the overall sample, the original study had slightly more “very concerned” participants. Finally, while neither study had a large number of “extremely concerned” individuals, the original sample had more of these participants, with a 6% difference between the samples in the correction treatment. Approximately 19% of the replication sample and 24% of the original sample fall into the high concern category, defined as saying you are either “very” or “extremely” concerned about vaccine side effects. Given this discussion, the relative similarity in these distributions gives us confidence that we can make similar inferences about high concern and low concern individuals.

Figures 1–3 summarize the distribution of responses to the three outcome variables of interest across all conditions in the replication study: the misperception that the flu vaccine can give you the flu, perceptions of the vaccine’s safety, and self-reported intent to vaccinate in the next flu season. The results indicate that roughly 40% of the respondents believe that the myth that the flu vaccine can give you the flu is “somewhat accurate” (29%) or “very accurate” (12%). Yet far fewer believe the flu vaccine is unsafe, with only a total of 15% saying they consider the vaccine “not very safe” or “not at all safe.” Figure 3 shows that the distribution of self-reported intentions to vaccinate is bimodal. 31% of respondents say it is unlikely they will get a flu vaccine in the next flu season, while 30% say it is very likely they will get a vaccine. The remaining 39% of the respondents were not as certain and approximately evenly distributed about the remaining response options.

Comparing these distributions with those from the original study, we can see that they are very similar to one another.3 For the vaccine misperception measure, each of the bars is within roughly 5% of the corresponding bar in Nyhan and Reifler’s distribution. The vaccine safety measure distributions are both compressed towards the “very safe” option, with the Mechanical Turk respondents having a larger proportion saying the vaccine is very safe. Similarly, the intent to vaccinate measure has the same bimodal distribution as the original study found.