Polls weren’t all that reliable in predicting the outcome of the election, but we can get to the truth if we know how reliable a survey is and how to analyze it

Whatever your political persuasion, you might well be curious about the popularity of the current president. In particular, you might be interested in a poll that was conducted from 4-9 May – a survey that asked people for their opinion about Donald Trump after he authorized missile strikes in Syria and after the House of Representatives voted to repeal and replace the Affordable Care Act.

Acting FBI head disputes White House claim Comey had lost staff support Read more

Or maybe you’re not so interested. After all, polls weren’t all that reliable in predicting the outcome of the election, were they? As ever, the truth requires a little bit of digging. Let’s get our shovels out.

Step 1

Set aside the press coverage and look up the original poll that generated it. I can save you a step there – the poll is here.

Step 2

Press the control key and F on your keyboard (command+F if you’re on a Mac) and type the word “methodology”, then hit enter. If the poll you’re looking at doesn’t publish details of their methodology anywhere, it’s probably trash. This poll that comes from Quinnipiac University does include a methodology section.

Step 3

Read through the methodology as you ask some questions, such as:

How many people were included in the poll? (In this case it was 1,078 voters, which is in line with the industry standard).

Who was included in the poll? (Quinnipiac does not share a breakdown of the race, age, gender or political ID of the people it spoke to. That makes it hard to know how well the survey respondents represent the American public.)

How accurate has this polling company been in the past? (It’s hard to measure the accuracy of opinion polling, right? The last poll Quinnipiac conducted before the general election suggested that Hillary Clinton had a lead of six percentage points. As it turns out, she won the popular vote by two percentage points. Not wildly inaccurate, but not perfect precision either.)



Step 4

Keep that slight imprecision in mind as you look into whether the numbers are rising or falling. In its press release, Quinnipiac explains that Trump’s popularity has fallen. As evidence for this they write: “American voters’ opinions of several of Trump’s personal qualities are down: 61-33% that he is not honest, compared to 58-37% on 19 April; 66-29% that he is not level-headed, compared to 63-33% last month.”



Hold on a minute. The percentage of people who say that Trump is dishonest has risen to 61% from 58% – but we know that these numbers could be off by at least three percentage points. Given that level of inaccuracy, his popularity might not have even fallen. In other words, these changes aren’t statistically significant.

Polls are expensive. And it’s hard for pollsters to stay in business if their results are often “meh, not much has changed”. So people like Tim Malloy, assistant director of the Quinnipiac University Poll, have an incentive to say, in a press release, things like “there is no way to spin or sugarcoat these sagging numbers”.

Step 5

Get perspective. Understand the broader context before you read too much into the numbers. I mean, how does Trump’s popularity compare to other presidents? How common is it for these numbers to fluctuate?

Someone remind Donald Trump he is not above the law | Trevor Timm Read more

Obama’s approval ratings fluctuated too, but more interesting perhaps was the clear factor that affected the likelihood a respondent would give him a thumps up: race. Black voters were almost twice as likely to think Obama was doing a good job as white voters.

So I decided to look at the demographics of responses to see whether there was a racial split in the Quinnipiac poll about Trump. There was. Trump’s support looks like a mirror image of Obama’s: white voters are twice as likely as non-white voters to think Trump is doing a good job.

Facebook Twitter Pinterest Presidential approval by race. Photograph: Quinnipiac University

Step 6



Before you close the tab, see if any of the other findings are interesting. As I’m scrolling through the Quinnipiac results, my eye stops at question nine, which asks respondents: “What is the first word that comes to mind when you think of Donald Trump?”

The most common responses were “idiot” (mentioned by 39 people), “incompetent” (31), “leader” (25), “unqualified” (25) and “president” (22).

Until next week, friends.