In the first two weeks of this month, as an autumn general election started to look more likely, 15 polls were carried out. They all aimed to do the same thing: estimate the support for each of Britain’s political parties, by asking some version of, “If there were a general election tomorrow, whom would you vote for?” Almost all put the Conservatives in the lead, with Labour in second place. But the polls disagreed on just how popular – or unpopular – those parties were. They put Tory support between 28% and 38%, while Labour ranged from 21% to 30% – the difference between a chaotic hung parliament, say, and a solid Tory majority.

Of course, if polls all said the same thing, there wouldn’t be so many of them. But when they vary so widely, what is the point of them? Over the last few years, as the polling industry has suffered a series of failures, that question has become more urgent. To recap: at the 2015 election, the polls didn’t see the Tory majority coming. Two years later, most of them made the opposite mistake: predicting a Tory majority when we got a hung parliament. In between, came the EU referendum – where the polls weren’t quite so far out, but most put remain ahead.

For their critics, these setbacks prove that polls are not an accurate way to read the popular mood. And those doubts have spread, something confirmed by the polls themselves. In 2017, Ipsos Mori found that 43% of people didn’t trust pollsters – an all-time high. The industry has come under fire, too, for selling the results of private polls to hedge funds on the day of the referendum; these were used to make bets against the pound.

But even if polls aren’t always trusted, their findings are still cheered, lamented, disputed and weaponised. They shape the way journalists talk about parties and politics; they put leaders under pressure. This is why they are used by campaigners – groups such as People’s Vote, which has commissioned a series of polls that highlight support for staying in the EU. Not that everyone is willing to accept the results. “My impression is that when people like poll findings, they seize on them and quote them,” says Peter Kellner, the former president of YouGov. “And when they don’t like them, they say: ‘Oh, they got the 2015 election wrong, they got the 2017 election wrong, it must be a load of bollocks.’” Pollsters themselves urge caution. “Polls aren’t magic; they can’t predict the future,” says Anthony Wells, YouGov’s director of political research. “But, clearly, they are the only good way we’ve got of measuring public opinion.”

Michael Gove and Boris Johnson the day after winning the 2016 EU referendum. Most polls had predicted a win for remain. Photograph: Getty Images

And the polls keep coming. This is the strange position pollsters now find themselves in: the turbulence of the times may have caught them out, but it has also made us more dependent on them, as we struggle to decipher the country’s thinking and what on earth might happen next.

After two general elections and two referendums in five years, pollsters are as sick of elections as the rest of us. “We’d like to get a rest sometime,” says Gideon Skinner, the head of political research at Ipsos Mori. ICM’s research director, Gregor Jackson, tells me that, if there were another high-profile failure,“We’d have to be honest and ask if there is a long-term future for political polling – in the narrow sense of accurately predicting the results of general elections.” When the next election comes, pollsters know they will face their biggest test yet.

***

On a miserable Saturday morning in July, I arrive at an Ipsos Mori call centre in Leith, Edinburgh. The main room – a sparse space overlooking a park – is filled with 50 people sitting at five banks of desks, each themed around a bright colour. Most of them are young: the call centre employs lots of students, as well as actors, musicians and comedians looking for casual work. They patiently make call after call.

Like most polling firms, Ipsos Mori devotes the majority of its work to market research. But I am here to sit in on telephone polling for its monthly Political Monitor: a voting-intention poll, with additional questions about respondents’ political views. This is the most high-profile work the firms carry out, often commissioned by newspapers, but it is also the least lucrative. “Most companies use it as a shop window to build their name,” says Deborah Mattinson, who runs the research consultancy Britain Thinks.

For its phone polls, Ipsos Mori calls a mixture of landlines and mobiles that have been selected at random. What surprises me, sitting in the call centre, is just how hard it is to convince people to participate. “That’s actually the biggest problem we have,” says Ian Douglas, a cheery man with white hair, who is the firm’s director of telephone project management. What has made phone polling increasingly challenging is “the number of people you have to convince it’s not a PPI call”, he says. Sometimes they slam down the phone or shout abuse. Interviewers are only expected to complete one 15-minute interview each hour; the rest of the time, they are being turned down or ignored.

The data shows increasing polarisation around Johnson. He is off-putting for many now in a way he wasn’t three years ago

Once they find someone willing to take part, the interviewers launch into their script: a set of questions written by Ipsos Mori’s political team in London. They can’t do anything that might influence the answers, like engage with an interviewee’s jokes or political asides, so some interviewers resort to a stiff, robotic delivery. Others make an effort to keep the interviews lively: one young man with jaunty diction, emphasises different words as he sets out the options on whether Britain will leave the EU without a deal: “Would you say it’s very unlikely, fairly unlikely…” It reminds me of Who Wants To Be A Millionaire?.

A voting-intention poll is usually based on interviews with between 1,000 and 2,000 people. But they can’t just be the first 1,000 who agree to take part – pollsters need a sample that is representative of the whole country. “The traditional analogy is eating a cake,” says Skinner. “You don’t need to eat the whole cake, you just need to get a good slice.” That means having the right proportion of men and women, 18- to 24-year-olds and over-65s, Scots and Londoners. So, pollsters set quotas for those different categories. Douglas tells me that it is easiest to persuade people over 45 who live in Scotland, Wales or the north of England to take part. It can take four or five days of polling to fill the final quota: 18- to 24-year-old men. They have learned that there are particular times when it is easiest to reach younger age groups: earlier in the week or Saturday mornings. When I arrive, at 11am on a Saturday, the main room is filled with the hubbub of interviewing. But as morning blurs into afternoon, it becomes quieter; by the time I leave, it is almost silent.

It is no surprise, then, that almost all voting-intention polls now take place online. It is cheaper, easier and quicker. Each agency has its own online panel: a group of people – more than 800,000, in the case of YouGov’s UK panel – who have agreed to take part in surveys, on politics or anything else, in exchange for a small payment.

Online polls have quotas to be filled, just like telephone polls. But it is hard to hit every one. So, once the polling has finished, pollsters adjust their results through a process known as weighting: if they needed to poll 120 people aged 25 to 34, but could find only 100, for example, they will make each response in that age group count for 1.2 people.

Polling firms weight their results for turnout, too: they take into account what someone says about how likely they are to vote or whether they voted in the previous election. It is this weighting that explains why pollsters can get such different results. “They’ve got a choice of levers to pull,” says Patrick Sturgis, a professor at the London School of Economics who led an independent inquiry into why the polling for the 2015 election went so wrong. “They pull a certain set of levers and it gives the Tories a five-point lead; they pull a different set and it gives the Tories a two-point lead.”

Theresa May calls for a general election on 8 June 2017, encouraged by polls predicting a healthy majority. Photograph: EPA

For decades, voters were split into two main groups – Labour and Conservative – divided by class. “Fifty, 60 years ago, most working-class voters voted Labour, most middle-class voters voted Conservative,” says YouGov’s Kellner. So, if a sample had the right proportion of working-class and middle-class people, pollsters had a decent chance of predicting an election accurately. That is no longer the case. “If you’re middle class, you’re only very slightly more likely to be Conservative than if you’re working class,” says Kellner. “If you’re working class, you’re only very slightly more likely to be Labour than if you’re middle class.”

The things that determine how people vote are always changing: to get a representative sample, pollsters must now pay particular attention to age, education, and, of course, views on Brexit. “The leave/remain camps have almost redefined how politics is seen,” says Tanya Abraham, who works on YouGov’s political team. “Because they can be any voter.” James Kanagasooriam, a partner in charge of polling at the lobbying firm Hanbury Strategy, tells me that the two traditional voting blocs have now fragmented into four groups of roughly equal size: as well as traditional Labour and Tory supporters, there is a coalition of liberal, metropolitan, remain-leaning voters, as well as core Brexit supporters. Leave voters can now back the Brexit party, remainers the Lib Dems.

The traditional voting-intention poll is actually rubbish: two questions to get at increasingly complicated information

“In my professional life, I’ve never known a time when voters have been so promiscuous with their votes,” says Andrew Hawkins, the founder of ComRes. ICM’s Jackson says: “Having a four-party system – and in particular the existence of the Brexit party – just makes it more difficult for polling agencies.” Most polling firms weight their data according to how people voted in the previous general election – but it is impossible to weight by a vote for the Brexit party, because it only launched in April.

It is even harder to predict what will happen in a general election, because it is dependent on whether Brexit happens before it takes place. “If we have left, and we haven’t run out of mangetout at Waitrose and people aren’t dying because they’re unable to get the right drugs, then I suspect there’d be a fillip in the polls for the Tories,” Hawkins says. If we don’t leave on 31 October, he suggests, the Conservatives will be in a much trickier position. Meanwhile, Jackson says: “The polling data is showing increasing polarisation in British views towards Boris Johnson. He’s off-putting for many people now, in a way he wasn’t three years ago.”

With so much fragmentation, no party looks likely to win a big majority – and when an election is tight there is little room for error. (A voting-intention poll has a margin of error typically understood to be three percentage points – if a poll puts the Conservatives on 33%, the party could be as low as 30% or as high as 36% – but this is often overlooked in the coverage.) “If you’re two points out when it’s close, you get it in the neck,” says Hawkins. “If you’re two points out when it’s not close, you don’t.”

***

Andrew Cooper is one of British polling’s grandees. The founder of Populus, he spent two years as David Cameron’s director of strategy at No 10. Like many firms, Populus did not come out of the 2015 election well. After that shock, Cooper resolved not to be caught out again. “We did a very, very thorough review,” he says. “It took lots of time, money and intellectual effort.” Eventually, Populus made a number of changes: it started weighting according to how interested people were in politics, for instance, rather than how they identified politically; to estimate turnout, it took into account demographic factors, and interviewees’ attitudes, in order to determine how likely different groups were to vote.

The 2016 referendum was its chance to put its new methodology to the test: the day before the vote, it published a poll putting remain on 55%. “That poll was the wrongest of them,” Cooper says flatly. Populus had made the same mistake as many firms: it underestimated just how many people who didn’t normally vote would turn out. The experience was doubly bruising for Cooper, as he was the official pollster on the remain campaign.

‘You have to convince people it’s not a PPI call.’ Photograph: Lol Keegan/The Guardian

This time there would be no attempt to put things right. “We came to the conclusion that the underlying methodological problems of voting-intention polls are so fundamental that we can’t solve them,” Cooper says. So that was that: since the referendum, Populus hasn’t carried out a single voting-intention poll. Instead, using data analytics, it looks at what Cooper describes as “the structure of political attitudes” – the patterns behind public opinion. (Similarly, while the Guardian continues to use ICM to carry out polls, the paper has shifted its focus to more in-depth research about public views and attitudes.)

“The traditional voting-intention poll is actually just a rubbish product,” adds Cooper. “It’s an attempt, by asking only two questions, to get at what is getting to be an increasingly complicated piece of information.”

This is the kind of statement one might expect to hear from a polling sceptic – someone such as Labour peer David Lipsey, who chaired a House of Lords committee that last year produced a critical report on political polling. It called for more oversight and suggested the media give less attention to voting-intention polls; at the time, Lipsey urged the industry to “get its house in order”. I meet him at his office, across the road from the House of Lords in Westminster. Protesters are gathered near his window; as a man repeatedly yells “Brexit means Brexit”, it feels as if public opinion is intruding very literally into our conversation.

If anything, Lipsey is surprised that the polling industry hasn’t attracted more hostile criticism. It wasn’t just the recent errors, he says, pointing to the election of 1992, when the polls mistakenly put Labour ahead, and to 1997 and 2001, when they hugely overestimated Labour’s lead. “In any case, if I predicted election results with a pin, I’d get it right half the time,” he says. Polling does have its uses, he concedes: it is effective at broadly gauging views on a wide range of subjects. But “the worst thing to use it for is predicting the results of elections.”

Polls reflect more emotional politics: leave’s ‘take back control’ had twice the resonance of remain’s economy warnings

Despite all the challenges that political polling faces, new players are still entering the industry. Last year, two veteran pollsters, Joe Twyman and Martin Boon, set up their own firm, Deltapoll. They thought politics was becoming more emotional, and that this was key to understanding how voters felt.

Delta uses something it calls “emotional resonance scoring” (ERS). A traditional poll might ask someone how far they agree with a statement; ERS tries to determine the intensity of their feeling, by measuring how quickly – and therefore how emphatically – they respond to it. Twyman gives the example of the EU referendum: polling showed that about as many people agreed with the remain campaign’s warnings about the economy as they did with leave’s “take back control” message; but ERS found the latter had twice the emotional resonance.

For now, Delta uses ERS to test the strength of voters’ feelings on politicians and parties; eventually, Twyman says, it will offer a new way of measuring voting intention.

Other firms are also experimenting with cutting-edge techniques to complement traditional methods. They show people speeches by politicians, then measure their heart rate or monitor their facial expressions, which are then recorded using a webcam and analysed by an algorithm to see how intensely they react to the different messages. But when I speak to people in the industry, the development they appear to be most excited about is a kind of modelling called multilevel regression and post-stratification (MRP), which YouGov used in the 2017 general election. Traditional voting-intention polls give you each party’s share of the vote for the country as a whole, rather than telling you how many seats they will get. But MRP enables polling firms to forecast election results on a constituency-by-constituency basis.

In 2017, YouGov polled about 50,000 voters a week, across the country. With those results, it created different voter types – a male professional in his early 40s who backed the Conservatives last time, say, or a female student voting for the first time – and calculated how likely each of those types were to back each party. By estimating how many people in each voter type lived in a constituency, it was able to forecast a result for every seat. And it worked, mostly: the model called 93% of seats correctly. When I ask Abraham how she felt that night watching the results come in, she sounds buoyant. “Great, actually,” she says. “Everyone just breathed a massive sigh of relief.”

The prime minister tells press he’d ‘rather be dead in a ditch’ than delay Brexit, against a backdrop of West Yorkshire police officers, earlier this month. Photograph: Reuters

At least political pollsters can take comfort from the fact that voting-intention polls are only a small part of their work. Kanagasooriam compares it to the tip of an iceberg. The real value of polling, he tells me, lies in all the other data. “Why do people vote a certain way? Who’s voting? Where they are on the issues, where they are on the parties, where they are on individuals, on messages.” He believes that, for all the emphasis on Brexit, voters continue to care about what they have always cared about: which party they trust to look after the economy; and which leader they like most.

Of everyone I speak to, Kanagasooriam seems the most enthusiastic about what polling can achieve. He energetically describes to me the sophisticated insights offered by data science. It doesn’t matter, he insists, that polling often fails to predict election results, because it keeps getting better at understanding what drives voters – and, at a time of upheaval, politicians need that information more than ever. He points to the Johnson government’s emphasis on tackling crime and investing in the NHS and the rest of the public sector – issues that have, over the past few years, ranked among voters’ top priorities. “This is the first time we have seen an administration actually pivot towards the things that matter for voters.” That’s the kind of impact polling can have, he suggests. “This is what affects stuff in Westminster.”

But voting-intention polls can affect Westminster, too. In 2014, 12 days before the Scottish referendum, the Sunday Times ran a YouGov poll showing a two-point lead for independence. It was only one poll – an outlier – but it rattled Britain’s political establishment and compelled David Cameron to launch a final intervention: he promised to give the Scottish parliament more powers if Scotland voted to stay in the UK. And it was encouraging polls in the early months of 2017 that helped convince Theresa May to call a general election, with disastrous consequences.

Listening in at the call centre feels, at times, like being given a direct line to interviewees’ thoughts. I wonder if they can really be so sure of where they stand on everything. At the time of my visit, Jo Swinson has been the Lib Dem leader for only five days, too soon for me to have an informed take on her as a potential prime minister, but on the phones, 22% of respondents say they strongly disagree that she could be good at running the country.

Even after a long career in polling, Ipsos Mori’s Douglas still seems fascinated by what each person has to say. “Your family and friends are a closed network – they usually have a lot of the same views,” he says, as we sit in Leith, dozens of his colleagues on the phone, hundreds of miles away from people at the other end of the line. “It’s easy to think it’s what the rest of the country is thinking, but often it’s not.” While we listen to interviews, he marvels at each surprising response. “The great British public tend to absolutely amaze you, time and time again.”

• If you would like a comment on this piece to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).