Reproducibility of findings has been a hot-button issue in social science over the last year, and as the election approaches, the reproducibility of findings related to voting is especially relevant. A recent study published in PNAS reassesses previously published data about ways to increase voter turnout and finds that language cues actually don’t have a significant effect on voter behavior.

The study revisits some field experiments conducted by Bryan et. al., which were also published in PNAS. These earlier experiments looked at voter behavior during the 2008 presidential election in California and the 2009 gubernatorial election in New Jersey. Using online surveys to prime voters, the previous study had shown that use of nouns or verbs affected whether voters went to the polls—verbs did better at getting out the vote.

This new study follows up on that work, looking at how subtle language manipulations can prime the behavior of voters.

For the new work, researchers obtained a list of all registered voters in the states of Michigan, Missouri, and Tennessee. Voters were randomized into one of four participant groups during the 2014 primary election. Four days before the election, all groups received a scripted phone call from the research group. They received either a call with a noun-based 10-item script, a verb-based version of the same script, a non-political “placebo” script, or a standard get-out-the-vote message. Scripts were highly similar to reduce the amount of unintended variability among the phone calls.

The new data indicated that the voters who got a noun-based phone call and the voters who got a verb-based phone call did not show significant differences in voting behavior. Additionally, when the analysis was broken down by state, there were no state-specific effects—voters in Michigan, Missouri, and Tennessee behaved the same way, regardless of whether they received a verb- or noun-based phone call.

Additionally, the results showed that a standard get-out-the-vote message—telling voters an election was occurring in four days—was actually more effective than either of the experimental conditions (noun-based or verb-based experimental scripts). All three of these conditions were more effective in getting voters to the polls than the non-political placebo script.

The researchers have some ideas about why their findings differ from previous results showing that verb-based scripts were effective in getting people to the polls. They note that their study was being conducted in a primary election, whereas previous studies were conducted during a gubernatorial election and a presidential election.

Additionally, previous work required participants to read through questions online rather than responding to scripted questions over the phone. The researchers hypothesize that when participants had to read the noun and verb based scripts themselves, subtle differences in the wording may have had a greater impact.

The team behind the new work also notes that their sample was drawn from three states, and they suggest that previous findings with smaller sample sizes may have been specific to particular voting districts. Previous studies also called voters on the day before the election or the morning of the election, whereas this most recent study called voters four days before the election. So the different details of the experiments may explain the lack of replication.

Finally, this recent study used existing voter rolls to find participants and to ascertain if participants voted or not. In contrast, previous studies used self-report measures from participants. Therefore, it’s possible that earlier studies experienced self-reporting bias in terms of participants saying they voted, which did not occur in this more recent study.

Regardless of which of these possibilities explains the lack of replications, these findings highlight the extremely context-specific nature of voting research. Outcomes may rely on local political climate, type of election, and the types of candidates running for office. More research is needed if social scientists want to more accurately predict factors contributing to voter behavior.

PNAS, 2016. DOI: 10.1073/pnas.1513727113 (About DOIs).