EDMONTON—Election polling is under a microscope in the wake of one of the industry’s biggest names miscalling a crucial byelection that could mean implications for the business as a whole.

Mainstreet Research is a national polling firm that gained notoriety in Alberta after incorrectly calling the Calgary mayoral race in 2017, and it’s in hot water again after it missed the mark on a critical byelection in Nanaimo, B.C., at the end of January.

The Nanaimo byelection put the Vancouver Island city’s reputation as an NDP stronghold in jeopardy. The outcome carried the added weight of putting the balance of power in the British Columbia government at risk as well, with polling showing the Liberals primed to take the seat, threatening the NDP’s minority government.

Mainstreet predicted a Liberal win, which would have put the B.C. government’s balance of power dead even between the governing coalition and the Liberals. Many observers across the country tuned in to watch as a result.

But Mainstreet was wrong, and the NDP held on and won.

For industry experts, the misreading by Mainstreet in Nanaimo showed major weaknesses in the company’s methodology. But it also highlighted the challenges the polling industry faces in the age of cellphones and the internet, like getting good population samples and pollsters as pundits that predict the outcome of elections.

Read more:

Why Nenshi is fundraising for almost $40,000 in campaign costs — more than a year after the election

‘High stakes’ Nanaimo byelection a reminder of NDP minority’s fragility

On the ground in Nanaimo, locals quietly brace for a pivotal byelection

However, Quito Maggi, Mainstreet Research president and CEO, told the Star that pollsters get blasted in the media when they get it wrong and don’t make headlines when they get it right — which is more often, he said.

“And then now with Mainstreet, it’s like, everybody says, ‘Oh, that’s two in a row for Mainstreet, they got (Naheed) Nenshi (in Calgary) wrong and now they’re getting Nanaimo wrong,’” he said.

“They’re forgetting about the 40 elections we’ve called correctly in between.”

In Alberta, after the 2017 municipal elections in Calgary, a team of three academics with the Market Research and Intelligence Association (MRIA) were tasked with doing a report on the inaccuracy of the polling done at the time. The issues discussed in the report specific to Mainstreet appear to have cropped up in Nanaimo as well.

The report discussed Mainstreet at length since the firm had been hired by Postmedia to report polls exclusively for the Calgary Herald and Calgary Sun newspapers. Mainstreet released three polls which put Bill Smith, a relatively unknown entity in the city, well ahead of incumbent Mayor Naheed Nenshi in all three — shocking pollsters, observers and politically engaged people who became publicly critical of Mainstreet’s methods.

Christopher Adams, a political scientist at St. Paul’s College in Winnipeg, Man., was a co-author of the review, which interviewed pollsters involved and gathered data throughout early 2018 before releasing the findings in August. He said the biggest stumbling block they’ve faced is polling relatively small groups of people, and making sure young people are represented.

“The problem that Mainstreet had, and I’m starting to hear people say this about the Nanaimo byelection poll — one of the problems with any survey in Canada is that now you have to have cellphones in your sample,” Adams told the Star in a recent interview.

In the days before widespread cellphone use, a sample list of land lines could be relied on for getting a good representative sample for a poll, said Adams. But developing a good list of cellphone numbers in order to reach younger voters presents challenges.

During the Calgary municipal election, Mainstreet attempted to poll individual wards, and it created problems, Adams found.

Mainstreet attempted to do ward-by-ward phone calls, but ended up creating a bias in its sampling, the review of the polling in Calgary found. The data set they were using would list phone numbers with addresses or postal codes — but this linkage doesn’t exist with cellphone numbers, so many got dropped, leaving younger people out of the polling.

“They got some sort of sample bias because you end up with a certain characteristic of a person, like what sort of 21-year-old has a land line, but not a cellphone?” Adams said.

The MRIA researchers also struggled to understand where Mainstreet got its sample.

“We got a few different answers. One was they compiled their list from different telecommunications providers, and then there was another answer about a Mainstreet database that they build up,” Adams recalled. “So, we never felt that we had a complete answer from Mainstreet as to what the full characteristics were of the sample and how it was created.”

Overall, Adams said this kind of small population polling, done in Calgary and Nanaimo, should be halted until solutions to sample problems are found.

He said Mainstreet has done well with polling federal and provincial elections in Canada and that if journalists are rigorous about understanding polls before publishing them, Albertans should be able to trust Mainstreet for the upcoming provincial election.

“Mainstreet did say that they were transparent, they did identify that there were some possible problems with their polling of younger people,” said Adams about the Nanaimo election.

“But I would say that if you don’t feel that you can get a representative sample of the electorate that you’re talking about, you shouldn’t be doing the polling.”

In Nanaimo, the NDP challenger won the seat with about 49 per cent of the vote, and the Liberal runner-up getting around 41 per cent. Mainstreet’s polling had previously said the Liberals were well in the lead, causing the country to watch the crucial election that had huge implications for the provincial government.

Polling gets tricky in a small riding like Nanaimo, Mainstreet’s Maggi conceded. Mainstreet was the only firm to publicly poll in the byelection and Maggi said it’s a point of pride for his company that they still attempt to survey small populations.

The problem is that Nanaimo offers a small population from which to poll. The smaller the sample size, the harder time pollsters have trying to get an accurate representative sample of voters. Compounding that problem is the issue of voter turnout, which is typically low for byelections — so even if they randomly call hundreds of people, it’s not clear that they’ll all turn out to vote.

Maggi’s now questioning doing this kind of polling in the future.

“What the lesson from Nanaimo is … maybe the other pollsters are right, maybe a small provincial riding like Nanaimo may be nearly impossible to poll,” he said.

“I’m not ready to give up yet, but we’re going to do a little bit more testing before we ever publish a byelection poll outside of Ontario (Ontario ridings are much larger and easier to poll).”

Same goes for provincial and federal election polling; because there are more people involved, it is easier to find a representative group.

Loading... Loading... Loading... Loading... Loading... Loading...

Melanee Thomas, a political science professor at the University of Calgary, said it’s near impossible to get a good reading on a small population without a good sample list being developed that represents voters of all backgrounds.

Mainstreet does what she calls “horse race” polling, snapshots in time of voter sentiment, and she argues they don’t invest enough resources into it, which would help them develop a solid sample list.

“Many of them are doing it just for free advertising to generate good revenue for the other parts of their business,” said Thomas of polling firms that do political polling. There isn’t a lot of money to be made in polling elections, but it’s a good way for firms to get media attention, she said.

Maggi said he was “a little hesitant to talk specifically about this,” but denied that his company didn’t invest enough money into the polling they did.

He said getting the proper amount of cellphones in Calgary was a challenge and that the small population of the Nanaimo riding also had limitations on their getting a good sample.

But he also said he didn’t believe the theory that polling affects the democratic narrative.

“I’ve never measured it, so I can’t agree with it,” Maggi said.

Mainstreet did what is colloquially known as “robo-calling,” but more accurately called Interactive Voice Response (IVR). This means an automated voice is on the other end of the line and quickly asks potential voters to press one or two (yes or no) when asked about a party running in the election.

It’s the cheapest way to do polling, but it hasn’t been found to be inaccurate compared with other methods in large population polling, said Adams.

Mainstreet polls leading up to the Nanaimo election caused the country to watch, said Thomas. The error in its polling allowed the race to become national news, and the democratic narrative was impacted, she said. Critics during the 2017 Calgary race had a similar stance on Mainstreet, saying the polls influenced the race itself using shaky data.

Most pollsters contacted by the Star wouldn’t comment on Mainstreet Research directly — they have policies about not commenting on other polling companies.

Mario Canseco, president of Research Co. and a former pollster for Insights West and Angus Reid, said he has a “strict policy of not commenting on other people’s polls.”

However, he said he “wanted to do Nanaimo,” but didn’t because of concerns over the quality of the data available. Calling land lines oversamples older voters, but random dialing cellphones won’t guarantee a local resident — and “nobody wants to answer their cellphones.”

Several commercial contact list providers said they could get him a “couple hundred” respondents, Canseco said in a phone interview. “I said, no way; it would be like jumping into a moving car.”

It’s easier to get a representative sample nationally or provincially, he explained, “it’s virtually impossible” in a single riding, especially with byelections’ typically low turnout.

“You’d end up in a situation where you do a byelection and you’re not helping your reputation, your brand, or your industry as a whole,” Canseco said. “But some of us in our industry are crossing the bounds between research and punditry; that’s very dangerous.”

Pollsters are forced to “weight” their results up or down to match the census population mix, though “case-by-case,” reliable pollsters avoid amplifying small numbers of people significantly.

Mainstreet’s Nanaimo poll weighted 22 respondents under the age of 34 nearly eightfold — counting for 172 people.

“It’s not an exact science, but for the sake of an argument, if I needed 330 millennials, if you get around 290, that would be OK to weight them,” Canseco said.

He is particularly sensitive to his industry’s reputation after it became “a scapegoat” for public ridicule following pollsters’ failure to predict U.S. President Donald Trump’s election or the Brexit referendum.

“To me, the essence is to be transparent, discuss the limitations of your survey so people understand how it’s assembled before they make any assumptions,” he said. “Don’t wait until your prediction isn’t correct to explain why the prediction wasn’t correct.

“We’ve got to take it down a notch when it comes to boasting — we’re ultimately just providing the voice of the people we speak to. People will gravitate away from boastful statements and people out there looking for headlines.”

Read more about: