If bogus respondents are not detected and removed, Pew found, polls using opt-in online samples may overestimate approval of Trump and other political figures and policies by about two percentage points. That fairly small impact can be meaningful for presidents like Trump facing a close reelection bid, and are amplified for certain subgroups, including Hispanics.

AD

AD

Many of the bogus respondents show the signs of being professional poll-takers interested in obtaining rewards for completing many surveys, and answering in ways that market researchers want.

The actual effect of bogus respondents on prominent web-based surveys is unclear: Pew examined the raw samples that online survey firms recruit from, and some firms may use different methods to weed them out.

“As far as we can tell, some online pollsters perform their own checks but many probably do not (at least there is no evidence in the public domain to that effect),” wrote Courtney Kennedy, director of survey research at Pew, in an email to The Washington Post. “To help the public better differentiate trustworthy and untrustworthy polls, it would be helpful if poll methodology statements mentioned what checking, if any, was performed.”

AD

AD

The report said that while concerning, “the findings do not signal that polling writ large is broken, wrong, or untrustworthy,” emphasizing the accuracy of 2018 pre-election polls and national polls in 2016, and that the study did not examine telephone polls.

Since Trump entered office, national polls tracked by the website RealClearPolitics have tended to show Trump’s approval rating about two percentage points higher in polls relying on opt-in online samples than polls based on samples of landlines and cellphones, 43 percent vs. 41 percent. But that relationship has not been consistent, and in 2020, Trump’s approval rating has been slightly higher in phone surveys.

Pew sought to quantify the prevalence of disingenuous or illegitimate respondents in online surveys recruited using six ways, conducting surveys of more than 10,000 respondents with each method in the spring of 2019.

AD

AD

Four of those surveys used opt-in samples made up of people who volunteer to take web surveys for money or other rewards, while the remaining two samples came from probability-based Internet panels in which respondents were recruited offline using random address-based sampling, including Pew’s American Trends Panel.

Pew classified a respondent as “bogus” if they did one of the following: took the survey multiple times, reported living outside the United States, gave positive ratings to all seven political leaders and policies spanning the ideological spectrum, or gave multiple responses to open-ended questions that did not match the question.

The last test required researchers to classify more than 375,000 open-ended answers, but revealed thousands answered questions such as “How would you say you are feeling today?” with non sequitur responses such as “Great product,” plagiarized text from other websites, conversational text or other irrelevant responses.

In total, between four percent and seven percent of respondents to opt-in surveys were deemed bogus by at least one of these definitions. The percentage of bogus respondents was highest for respondents recruited through a crowdsourcing website; but it was also six percent for two out of the three opt-in panels. By contrast, bogus respondents made up just one percent of interviews in both surveys conducted with panels that were recruited offline.

The greatest unifying trait of bogus respondents is their Pollyannish responses across all political questions. The knee-jerk positive answers of bogus respondents extends even to deeply unpopular figures. When Pew identified potentially bogus respondents who took a poll more than once, 42 percent of those approved of Russian President Vladimir Putin. So did 32 percent of the respondents who gave mismatched answers to open questions. For contrast, a Fox News poll conducted by phone found nine percent of U.S. registered voters had a favorable view of Putin at a similar point last year.

AD

AD

While random responses from a small percentage of respondents might cancel out in the long-haul, the tendency for bogus respondents to give positive responses could lead such polls to overestimate the popularity of political leaders and policies. The effects also may not be distributed evenly. While 10 percent of all respondents in the Pew surveys identified as Hispanic, that rose to 30 percent among respondents classified as bogus. Trump’s approval rating among Hispanics dropped between three and 18 percentage points when bogus respondents were excluded, with the largest swing occurring for the opt-in crowdsourced sample.

Bogus respondents appear less likely to affect election poll questions measuring which candidate that respondents currently support, since they do not have a clear positive option that illegitimate respondents might be attracted to. “If, however, the researcher asked about candidates individually and asked respondents to say whether they have a favorable or unfavorable opinion of each candidate,” Kennedy said, “that type of question would potentially show a small bias from bogus respondents.”