Thinking of checking up on respondent attention mid-survey to make sure that you’re getting good data? Think again.

In this article, we highlight how new findings from our Qualtrics Methodology Lab are helping us to revisit and refine advice that is commonly given to survey researchers, namely the use of attention check questions to ensure data quality. But first some background.

A History of Attention Checks

When the cognitive demands of a survey begin to exceed the motivation or ability of respondents, they often employ a set of response strategies that allow them to reduce the effort that they have to expend without leaving the survey altogether. Several of these strategies are grouped together under the term ‘satisficing’ – including acquiescence, straight-lining, choosing the first reasonable response, or saying ‘don’t know’ or ‘no opinion’ (Krosnick 1999; Vannette and Krosnick 2014). Other behaviors include skipping some necessary questions, speeding through the survey by giving low-effort responses, not fully answering open-text questions, or engaging in a variety of other behaviors that negatively impact response quality.

No researcher or organization wants low-quality data in their results, and since at least 2009 there has been a widespread trend toward attempting to identify survey respondents that are not carefully reading instructions (Oppenheimer, Meyvis, and Davidenko 2009). While the original research indicated that filtering these respondents might increase experimental efficiency, other researchers quickly began using this method not only to identify respondents who do not read instructions but also as a proxy for other low-effort response strategies listed above. These strategies for identifying “bad” respondents are known by a variety of different names, including:

Attention checks

Instructional manipulation checks

Trap questions

Red-herring questions

These strategies vary somewhat in how they are implemented, but they all share an interest in “catching” respondents who appear to not follow instructions in the survey. While many different factors could result in the pattern of results that these techniques produce, the most common explanation is that the offending respondents are not paying sufficient attention to the survey. For the purposes of this post, we will refer to all of the strategies listed above as “attention checks.”

Since the assumption is that respondents that fail the attention check are not paying attention, there is a common belief that these “bad” respondents should simply be eliminated from the dataset, the sooner the better. Qualtrics has in the past recommended for using attention check questions and eliminating those respondents that fail them from the data, because the attention-check technique seemed intuitively reasonable and pragmatic, and is often used across the research industry.

Start building surveys today with our free Qualtrics survey account

The Problem with Attention Check Strategies

However, our research scientists in the Qualtrics Methodology Lab recently conducted a careful review of emerging research on this topic and found that much of it advises against eliminating these respondents from most datasets (Anduiza and Galais 2016; Berinsky, Margolis, and Sances 2014; 2016; Hauser et al. n.d.; Miller and Officer 2009). Rather, “failing” an attention check question should be used as one of many data quality metrics to be evaluated after data collection has completed.

Removing respondents that fail attention checks is likely to introduce a demographic bias.

Part of the rationale behind the current academic recommendations is the fact that the respondents that “fail” the attention check are not a random subset of the population, meaning that eliminating those respondents from the survey may introduce a bias into the results of the data if some demographic or psychographic groups are disproportionately likely to fail the attention check.

Given our prior recommendation of using attention-check techniques at Qualtrics, we decided to investigate this approach ourselves to determine what the effects of using attention check questions might be. Toward that end, our Qualtrics Methodology Lab conducted a large-scale global survey experiment. One important finding was that removing those respondents that fail attention checks is indeed likely to introduce a demographic bias, particularly for age (Vannette 2016).

Surprisingly, when comparing different types of attention check questions against a control group in a randomized experiment, we also discovered that simply asking an attention check question caused respondents to behave worse later in the survey. The very mechanism that is intended to detect low-quality responses in a live survey actually induces respondents to produce lower quality responses throughout the rest of the survey in ways that are not as immediately detectable (Vannette 2016).

Related research has also found that attention checks can induce Hawthorne effects and socially desirable responding, wherein respondents edit or censor their responses because they feel they are being watched (Clifford and Jerit 2015).

One implication of these findings is that it may be best to omit attention check questions altogether.

At Qualtrics, we have stopped recommending that our customers use attention checks. We of course do not prohibit their use; we just recommend not using them as a means to improve data quality since they actually seem to degrade data quality.

If Not Attention Checks, Then What?

In the Qualtrics Methodology Lab, we are still actively working to understand the causal mechanisms that may be at work to produce the results that we’ve found. One of our current hypotheses is that respondents may recognize the attention check question and subsequently have a sense of being ‘past the trap’ which induces them to invest less effort into their responses. An alternative hypothesis is the feeling that a researcher has tried to ‘trap’ them may reduce trust or the feeling of reciprocity with the survey researcher or organization and that may influence willingness to engage thoughtfully and carefully with the survey. There could also be many other factors at work that are reducing respondent motivation, resulting in the degraded data quality that we observed. There is clearly a need for further research on this topic.

Our survey scientists are also working hard on developing new methods for improving data quality and increasing respondent attention and engagement. While we’re not quite ready to take the wraps off yet, we have already made some promising findings that we will share in future publications. The results documented in this blog post were presented at the 2016 annual conference of the American Association for Public Opinion Research (AAPOR).

David L. Vannette, PhD is Principal Research Scientist in the Qualtrics Methodology Lab and a Qualtrics subject matter expert.