The polls keep coming, one after another. But the polls are all over the place. For example, they can’t agree on where the competing parties stand. One gives the Tories a 10-point lead, another gives Labour a 2% lead. Polling has never been an exact science, but political volatility, the growth of new polling firms and the extraordinary ubiquity of conflicting polls, has put it under new strains and new scrutiny.

Polling was once a specialised sector of market research. Now it is a niche area of the much bigger data industry, using the same Bayesian techniques of probabilistic analysis that stock markets employ in financial forecasting. Its customers include huge financial firms that can extract commercial advantage from the slightest margin of predictive accuracy.

When it comes to prediction, the neoliberal economist Frank Knight made an important distinction between “risk” and “uncertainty”. The data revolution has helped capitalism manage risk, but it manages uncertainty far less effectively. One byproduct of this is that, although we have more data than ever before, political outcomes are no easier to predict: hence, the current crisis of polling.

We can no more dispense with political polling than capitalism can give up on data

For pollsters, the risk of a “wrong” prediction is managed in their selection of who to interview, and how to weight the results. If young and poor voters didn’t turn out last time, they probably won’t this time. But in conditions of political uncertainty, these assumptions start to look like what they are: guesswork and ideology.

By relying on past outcomes to guide their assumptions, pollsters made no room for political upsets. Since poll numbers are often used as a kind of democratic currency – a measure of “electability” – the effect of these methodological assumptions was to ratify the status quo. They reinforced the message: “There is no alternative.” Now that there are alternatives, polling firms are scrabbling to update their models.

Yet, what do polls actually measure? In 1947, as modern polling was becoming a major industry, researchers canvassed American public opinion on something called the “Metallic Metals Act”. No such act existed, metallic metals being akin to ironic irony, or tautologous tautology. But 70% of respondents took a firm view for or against. They weren’t stupid: they were just acting as most of us do in an interview situation, under pressure to have definite views about something of which we may be uncertain, conflicted or even ignorant.

These researchers weren’t measuring opinion. They were producing it. This is what the polling industry does: that’s why it is an industry.

The artifice of the phone interview, or online survey, resembles no real-life circumstance in which opinions are formed. It is an assembly line designed to produce quantifiable opinions: that is, to express sentiments, preferences. This may make sense in the context of market research to measure consumer preferences. But formulating a preference, or even purchasing an item, is quite unlike casting a vote. The latter is closer to a major life decision – often rooted in collective experiences like class and race – than to a brand choice.

The methodology of polling implies that there exists a general will on any given issue – the sum of a quantity of individual opinions of roughly equal weight. These can be totalled up into a magic percentage. But one reason voting intention polls must weight their results is that not all opinions are equally informed, committed or even meaningful. Most of us have ambivalent or downright contradictory views on some subjects, which is why small adjustments in polling questions can produce such varying results.

So while polling, in conditions of political stability, can often accurately predict voting outcomes, its findings are less meaningful as a guide to “public opinion” on more complex issues. “Nothing is more inadequate,” wrote the sociologist Pierre Bourdieu, “for representing the state of opinion than a percentage.” “Public opinion” is a mirage.

There is, nonetheless, something authoritative about a round figure, which appears to brook no argument. Yet, the only test of polling is its predictive power. And predictions failed in 2015, 2016 and 2017. When this happens, there is a tendency in the media to explain by reference to misleading respondents: “shy Tories”, for example.

Perhaps some people deliberately obscure their real intentions. But consider what happened in the 2010 general election, with “Cleggmania”. For weeks, the Liberal Democrats surged, sometimes to first place, gaining strongly among young voters. In the end, they lost five seats.

Why did the polls not predict this outcome correctly? Most of us, if asked about an issue we aren’t sure of, or haven’t thought about, will cast our minds back to the news. Pollsters are measuring, as much as anything, the effects of recent news cycles. The media publishing them are reporting on the effects of their own coverage. They are short-term, however, and can recede by election day, by which time other assets such as street campaigning can make a significant difference. Cleggmania was precisely such a media phenomenon. The Lib Dems were not offering more to young voters in 2010 than in 2005: arguably, they offered less. But a surge of attention following Nick Clegg’s debate performances drove the party up the polls, only for most of the surge to burn out by election day.

Yet, most of us believe in public opinion. If we didn’t, the polling industry wouldn’t have a market. Polling is clearly useful for capturing trends in party support, or aspects of emerging moods, however partial. But political professionals working behind closed doors rarely make the mistake of treating polls, whether on voting intentions or issues, as democratic verdicts. Rather, the data is raw material to be worked on and shaped – used to guide publicity or election campaigns.

Because we believe in it, though, polling tends to become a self-fulfilling prophecy, particularly on social media – where each new poll is eagerly shared by supporters of whichever party appears to be surging this week. Like older forms of divination, the numbers give form to our desires and fears. They authorise our beliefs, legitimise our candidates and generate little waves of excitement. They allow us to blow attention bubbles around issues or parties, boosting the ratings further, and spawning yet another squee of excitement and feverish sharing of numbers. In moments of crisis, this can even allow small parties to game the system and generate attention and support as if from nowhere. If it weren’t for this, it is difficult to see how the Brexit party, a corporation with no members and a relatively small budget, could have won the European elections in May.

At its worst, polling can subvert democratic debate. Even a slight polling plurality is enough for politicians and newspapers to assert that “the people” want whatever they’re selling, producing the illusion of a non-existent consensus in favour of policies that have little real support.

We can no more dispense with political polling than capitalism can give up on data. For all its limits, the data usually “works” in the narrow sense that it quantifies an intention or sentiment, however fleeting or partial. The problem is the way the industry is supported by a belief in public opinion. If we can let go of that superstition, the numbers will have less power over us.

• Richard Seymour is a political activist and author; his latest book is The Twittering Machine