No pollster attracts more love and hate than Public Policy Polling. The Democratically aligned polling firm routinely asks questions that poke fun at Republicans, like whether then-Senator Barack Obama was responsible for Hurricane Katrina. Not coincidentally, Republicans routinely accuse them of being biased toward Democrats. Last fall, PPP was front and center in conservative complaints about allegedly skewed polls. But when the election results came in, PPP’s polls were vindicated and the conspiracy-minded critics were debunked.

Pollsters, though, tend to judge one another based more on methodology than record. And for experts and competitors, the firm’s success remains difficult to explain. PPP doesn’t follow many of the industry’s best practices, like calling voters' cell phones; the firm only calls landlines. It discards hundreds of respondents in an unusual process known as “random deletion.” And because PPP's interviewers rely on lists of registered voters—rather than random digit dialing—and simply ask non-voters to hang up the phone, the firm can’t use census numbers to weight their sample, as many other pollsters do. This forces PPP to make more, and more subjective, judgments about just who will be voting.

In PPP’s telling, the Raleigh-based firm overcomes the odds by mastering those subjective judgments, perfecting the art of projecting the composition of the electorate—the same art that eluded Republican pollsters in 2012. If this explanation was satisfying, perhaps PPP could settle into the top-ranked pollster slot without great protest. But PPP’s success, in fact, did not reflect a clairvoyant vision of the electorate. The racial composition of their polls swayed wildly. A recent Georgia poll was just wrong.

After examining PPP’s polls from 2012 and conducting a lengthy exchange with PPP’s director, I've found that PPP withheld controversial elements of its methodology, to the extent it even has one, and treated its data inconsistently. The racial composition of PPP’s surveys was informed by whether respondents voted for Obama or John McCain in 2008, even though it wasn’t stated in its methodology. PPP then deleted the question from detailed releases to avoid criticism. Throughout its seemingly successful run, PPP used amateurish weighting techniques that distorted its samples—embracing a unique, ad hoc philosophy that, time and time again, seemed to save PPP from producing outlying results. The end result is unscientific and unsettling.

A SHIFTING ELECTORATE

In an age of racial polarization, small shifts in the racial composition of a poll can mean everything: There was only a 3-point difference in the white share of the electorate between the 2008 election, when Obama won decisively, and the 2010 midterms, when Republicans won a historic landslide. Last year, Pew Research assumed that whites would represent 74 percent of voters, while Gallup assumed whites would make up 78 percent. Pew nailed it. Gallup, having been proven embarrassingly wrong, is now rethinking its entire methodology.