Another Forum Research poll came out on Friday, showing John Tory up 10 points on once-frontrunner Olivia Chow, and Tory fans cheered and Chow fans fretted, and a lot of people had a lot of the usual questions about polls.

No matter what the results any given week, many readers appear both skeptical and confused. The skepticism is reasonable: polls have let us down in the past. The confusion is understandable — but we can help. Here are answers to some of the questions we get the most.

Iheard pollsters never call cellphones.

Wrong. Many pollsters do, including Forum, the firm that does the most frequent Toronto polling, and Nanos Research, which did the first poll that showed Tory ahead.

How do pollsters decide who gets called?

They decide how many people they want from the different regions of the city — Scarborough, North York, and so on — and then they call randomly generated phone numbers they believe belong to people in those regions. Because residents can take their numbers with them when they move from one region of the city to another, or when they move to another city entirely, the pollsters also ask participants to verify where they are.

How many people do the pollsters have to call to get their samples of 800 people or 900 people or 1,000 people?

A whole lot. In this era of caller ID, hectic evenings and telemarketer-hating, the vast majority of people who get called by pollsters either don’t pick up or hang up. Response rates — the percentage of calls that result in someone completing a poll — have fallen dramatically over the last 20 years.

Forum CEO Lorne Bozinoff says Toronto mayoral election polls have the highest response rates of all the political polls his company does — but this rate is still only about 10 per cent. That means Forum has to call about 10,000 people to get a Toronto sample of about 1,000 people.

Response rates for automated polls, like the ones Forum does, are usually lower than response rates for polls that use human interviewers, like the ones Nanos does. Nanos CEO Nik Nanos says his response rates are “anywhere from 10 to 20 per cent depending on the subject matter and the length of the survey.”

Are falling response rates a problem?

This might sound strange, but very possibly not. Several U.S. studies have found that polls with low response rates have produced the same results as polls with high response rates. One Pew Research Center study found that people who do polls are basically the same as people who don’t do polls, though poll-doers are more politically active.

OK, so. Pollsters call a bunch of people, collect the responses of the few people who take the time to participate, tally up those responses, and report the results to the media?

Nope, it’s not that simple. The final numbers you see in the newspapers — Candidate Smith has 37 per cent! Candidate Jones has 32 per cent! — are not the basic numbers the pollster has collected from his or her sample. Before the pollster releases final numbers, he or she does something called “weighting.”

What is weighting?

It’s the process of adjusting the numbers so that the poll reflects the makeup of the broader population.

Wait, pollsters are adjusting their numbers?

Yep. They have to.

In an ideal world, pollsters’ phone calls would produce perfect samples: exactly the same percentage of men and women as in Toronto as a whole, exactly the same percentage of old people and young people as in Toronto as a whole, and so on.

In practice, that never happens. They are calling people at random, after all, and on any given night, they can end up with too many men, too many old people, or too many of any other group of people. So they must weight.

Here’s a basic hypothetical example of how weighting works. Women make up about half of the population. If, in one poll, only a quarter of the people who answer the phone and participate in a poll are women, the pollster would count their responses for double in the final results.

Weights are usually much smaller than this. A pollster, for example, will want 49 per cent men and end up with 44 per cent men, so he or she will apply a small weight to make up the difference. If big weights are needed, the sample probably isn’t very good.

OK, I see how a pollster could do weighting to try to make sure the poll represents the city’s demographics. But old people are far more likely to go out and vote than young people. Do pollsters account for turnout factors like this?

Some do, some don’t. Forum is one of the pollsters that does.

When Forum does one of its polls on political issues — say the land transfer tax — it just does weighting to match the city’s population, like we explained above, and it ignores everyone’s likelihood to vote. You don’t have to be a voter to have an opinion on a tax.

But when Forum is polling the mayoral race, only voters matter, so the company weights the results to try to match the voting population rather than the general population as usual.

Here’s a basic hypothetical example.

Say that people age 65 and older make up 10 per cent of the city’s population but 15 per cent of people who usually vote in municipal elections. And say that, one night, only 5 per cent of the people who participate in a poll are 65 and older. For a poll on the land transfer tax, Forum would count their responses for double, to match their percentage of the actual population. But for a poll on the mayoral election, Forum would count their responses for triple, to match their expected percentage of the voting population.

But every election is so different from every other election. How do they know what voter turnout is going to be like?

Well, it’s very hard. Nanos doesn’t even try: his polls do not weight for expected turnout, since he thinks these “subjective assumptions” get pollsters in accuracy trouble. Even Bozinoff says, “I think it’s a mug’s game to try and predict in July who’s going to vote in October with any degree of certainty.”

Whether the pollsters are weighting for the regular population or weighting for the voting population, there’s a lot of adjusting going on here, and it’d make me feel better to know exactly how pollsters like Forum are doing their weighting in the Toronto election.

Loading... Loading... Loading... Loading... Loading... Loading...

That’d be superb, but they won’t tell you.

Many pollsters regard their weighting schemes as “proprietary.” Their business is about being most accurate, and they say they can’t disclose exactly how they achieve their accuracy.

“That’s kind of our secret recipe,” says Bozinoff. “All our competitors want to know what our weights are. It’s like Coke asking Pepsi, ‘Well, gee, what’s your formula?'”

Forum does its polling using low-cost robocalls — “interactive voice response” surveys in which people push buttons to answer questions asked by the recorded voice of an actor. Other companies, like Nanos, hire actual people to conduct real interviews. Isn’t that better?

Not necessarily.

The New York Times and some other U.S. news organizations refuse to publish automated polls, explaining that anyone of any age can pick up the phone and push buttons. Nanos argues: “The level of consistency in terms of a track record is not there, because these are new technologies.”

But the track record is growing. In several recent U.S. elections, automated polls have been roughly as accurate as polls that used human interviewers, Michael Traugott, a University of Michigan professor and former president of the American Association for Public Opinion Research, told the Star last year. In Canada, automated polls have been best in elections in Ontario, Alberta, British Columbia and elsewhere.

How did the polls do in the last Toronto mayoral election?

None of them were great, and some were bad, but the automated polls were best.

Ford ended up winning by 11.5 points over George Smitherman. EKOS’s final automated poll was most accurate — it had a 15-point lead for Ford. Forum’s final automated poll, completed a week and a half before the vote, gave Ford a six-point lead. The final polls from all of the pollsters that used other methods — Leger, Ipsos, Nanos, Angus Reid — showed a tight race.

This doesn’t necessarily mean the non-EKOS polls were bad: Ford might’ve picked up a lot of late-breaking support in the days after the polls were completed. But it reminds us, again, that we should approach poll results with caution.

Forum does its polls in one night. Other pollsters do their polls over multiple nights. Which is better?

Polling over multiple nights allows pollsters to call back people who didn’t answer the first time. Nanos, for example, does five call-backs. Since it is possible that the people who aren’t home on one particular night are significantly different than the broader population, the polling industry considers callbacks a “best practice.”

Bozinoff argues that one-night polls are best, because they allow us to pinpoint what was happening in politics on the day the poll was done. He insists callbacks don’t really matter no matter what the textbooks say.

“There is a science and there is an art to this,” he says. “The science is pretty clear about it. The art is, ‘Wow, how about the fact that you’ve done it both ways, and you’ve done it 10 times, and you’ve seen what works and doesn’t work, and you use your judgment.’ ”

The current polls show Ford in third place, but wasn’t he doing just as badly in the polls at this time in 2010?

Not at all. Ford was consistently ahead in the polls beginning in early August, and he was tied with Smitherman for the lead as early as June. He was only an underdog at the very beginning of the race.

David Soknacki and Karen Stintz are way back in the polls, 25 points or more off the lead. Is there any precedent that suggests they could come back and win?

Sort of. Naheed Nenshi was down 35 points in Calgary, in third place, a month before Calgary’s 2010 election. And David Miller was down 32 points to Barbara Hall, in fourth place, in a poll conducted just over two months before the 2003 election; he ended up beating Hall by 34 points.

But Toronto’s 2014 race is very different from both of those races. Most notably, there are three very well-known candidates in this election, and there is an incumbent with a loyal base. While Miller needed only a Hall collapse, Stintz and Soknacki need to overtake each of Ford, Tory, and Olivia Chow.