Whatever else people say about Joe Hockey’s unloved federal budget, it does have one irrefutable merit: it kills off the myth that Australian politics is driven by polling. For more than a decade now, there’s been a persistent idea that the state of Australian politics is somehow the people’s fault, that fear of focus groups keeps politicians hamstrung in what they can say, that party programs remain myopic for fear of being seen as “out of touch”. But beyond the cargo cult that surrounds Newspoll’s two-party preferred figures, public opinion is an irrelevance. Out in the wilds of policy, popular will is a nuisance, to be massaged, contained or bullied; or, if these don’t work, ignored. When a budget this unpopular is delivered, it’s at least refreshing to know that popular opinion has been comprehensively discarded. In a way, it’s a relief.

What Australians want, according to more qualitative polling, is a much more protectionist, statist but socially liberal nation than the one in which we live. The record levels of disaffection aren’t just caused by an unforgiving media environment or broken promises. They’re a sign that the views of the political class are diverging from those of mainstream Australia. People are retracting from politics – but it’s also retracting from them.

In the United States, what you might call the “bore in the bar” theory of democracy – that it’s all bullshit – is starting to look more persuasive. In academia, it’s called the “Economic Elite Domination” model: the unhappy idea that democracies are oligarchies in drag. This theory was once unpopular but is now resurging, partly on the back of disquieting research by two American political scientists, Martin Gilens and Benjamin Page. After an analysis of 1779 legislative outcomes over a 20-year period, the researchers determined that “economic elites and organised groups representing business interests have substantial independent impacts on US government policy, while mass-based interest groups and average citizens have little or no independent influence”.

Little or no independent influence. Stew on that for a moment.

Gilens and Page found that once you account for the preferences of “affluent” citizens, “the apparent connection between public policy and the preferences of the average citizen may indeed be largely or entirely spurious”. Seen this way, the shrivelling public involvement in politics isn’t a retreat from modernity and community, but a rational appraisal of how things are. After all, why give legitimacy to a system that just ignores you?

Australia isn’t the US, at least not yet. Our system lacks many of the pressure points of the American system where influence can be exerted: we don’t have unfettered political donations (not officially anyway) or congressional deadlocks; lobbyists don’t yet write legislation as a matter of routine; and our system of compulsory, preferential voting means the disenfranchised can’t get so fed up they quit politics altogether, even if record informal votes show many would like to. But we are becoming very fed up.

This is what happens now when the subject of “politicians” is brought up in a focus group. It’s described here by the public affairs researcher Scott Steel (aka blogger Possum Comitatus), but it could have come from almost any focus group held in the past few years:

“Let’s talk about politicians.” [Groans, chuckles and guffaws are the response every single time, regardless of age, gender or more complicated demography.] [More laughter.] “Give me a few words that you reckon most accurately describes politicians today.” “Just in it for themselves.” “Lying [expletives].”

“The two most popular expletives,” says Steel, “are ‘bastards’ and ‘dickheads’. Except for old ladies over 70 – they particularly like the word ‘mongrels’.”

This is our version of what the Canadian professor of politics Neil Nevitte calls “the decline of deference”. You can still catch some of that vestigial deference when elderly people call politicians “Mr”. But now those same senior citizens are also calling them mongrels.

How did it get to this? In her Quarterly Essay Great Expectations, Laura Tingle outlines some of the difficulties facing contemporary politicians. They’re weak in the face of a modern, globalised economy, but suffer intense media scrutiny at just this moment of impotence. Both the public and their representatives now have no clear, shared idea of what government is supposed to do in a deregulated market, and instead expect it to do everything. Everyone ends up disappointed. Shrinking revenues and an ageing population will “require us to forge a much more explicit new settlement, a much clearer social contract than the one we have had to date”. That this contract will end with Australia as a low-tax, small-government nation open to the world is taken as a technocratic inevitability.

The trouble for the political class is that this version of Australia is the opposite of how most of us want to live. Australians are suspicious of immigration. The public is extremely hostile to privatisation and foreign investment. We want the government to take measures, up to and including nationalisation, that will protect local jobs and manufacturing. We want more spending on health care and are willing to pay higher taxes to fund it. We support regulation, and we think big business has far too much power.

These positions are shared to a surprising extent across the political spectrum. Our leaders are frustrated by this dogged counter-vision, and the way the public clings to it. “The head members of both major parties,” as Guy Rundle puts it in an article for Crikey, “share a mutual sympathy at the stupidity of their own supporters in rejecting neoliberalism.” On the other hand, on social issues of gay marriage, voluntary euthanasia and abortion, these same leaders are decades behind “ordinary” Australians.

This dissonance helps explain why the last two election campaigns were so shambolic. They were failed sales jobs, repetitive, mendacious and joyless attempts to win over slivers of the population. The most recent, in 2013, managed to limbo even under the abysmal standard of the 2010 campaign. Several political veterans described it as the worst they had ever seen. The Labor stalwart and commentator Barry Jones, not a man for hyperbole, called it a “policy vacuum” and “the worst in our modern history for the debased quality of political discourse”. For an event supposedly tailored to what the people wanted, it left them completely dissatisfied.

When Kevin Rudd, back in the prime ministerial chair, announced additional government subsidies for apprentices’ toolboxes, no one thought it was because of a nationwide spanner shortage. It was a Hail Mary play three weeks deep, a sorry attempt to regain some traction with “blokes”. Special tax zones in the Northern Territory? Sure. Move a naval base to Queensland? Why not? But there was no detail. Like the Coalition’s Green Army of young workers and Direct Action plan to tackle climate change, these ideas came with a kind of in-built obsolescence, marked “for campaigning purposes only”. In effect, they were decoys.

The real policy platforms of the parties, particularly the Coalition, were the opposite – designed to be implemented, but kept hidden during the campaign. For the Coalition, that’s partly a feature of being out of office: since the failure of John Hewson’s ‘Fightback’ campaign in 1993, oppositions that take comprehensive, fully costed and detailed policy platforms to an election are considered deluded. Also, if voters had known what was in the budget, there would have been mayhem.

As the former Liberal senator Michael Baume advised in 2012: “Instead of having to defend the wide-ranging reforms of a complicated package … Abbott needs simply to be acknowledged as a credible alternative to a discredited Gillard government.” The kind of election Baume had in mind is a sorry one, an unpopularity contest won by the least unelectable.

If voters had known what was in the budget, there would have been mayhem

As our major political parties become more unstable – professionalised and no longer anchored to their traditional social values or bases – polling has become a sort of de facto electoral process. Advisers obsess over not giving the people what they don’t want, in a competition between leaders who contend: “you’re not sure who I am, but at least I’m not them”. Sometimes, like the Real Julia, their identity becomes unclear even to themselves. As politicians assure the electorate that they’ll never do anything unpopular, dishonesty has become routine.

Citizens have always grudgingly accepted that politicians lie, and they are willing to elect candidates who offer them scant policy detail. But when politicians lie about even that scant policy detail, it’s no longer clear exactly what elections in Australia are for. They’re not endorsements of judgement, because the judgement relies on trust. They’re not endorsements of policy, because we don’t know what it is. They are opening up a democratic deficit we can’t levy our way out of.

Instead of fronting up to the electorate, governments now invent a whole category of external bodies: commissions of audit, reviews, people’s assemblies, future summits. They create a kind of pseudo-consent, the illusion of consultation, objectivity and changed circumstances. They mimic the representative format of parliament, but do it in a way that’s both predictable and disposable. Unpopular policies already well planned seem to come from some external body, which is then quickly disbanded, and the government looks benign in comparison. Tony Abbott’s personal maxim that “it’s better to seek forgiveness than ask permission” has become our system of government.

The people referred to during the Howard years as “ordinary Australians” are dismayed by this process. But how much real power over policy they’ve lost is unclear – it’s possible they never had it in the first place. We don’t know. There’s no local equivalent of that Gilens and Page paper. Some domestic research shows politicians are at least concerned about the same issues as we are, even if their solutions might be different. These kinds of enquiries are scant, though. One participant in a Productivity Commission roundtable on population growth bitchily notes, “Australia lacks a culture in which social science research is valued as an important public resource.” We don’t know, in other words, and we don’t care.

For now, the only way to measure the gap between policy and the public is to look at the broader polling itself. The popular will is not an easy thing to synthesise this way. Gleaned from questionnaires, it’s often confusing and contradictory, hoodwinked by question wording, stymied by ignorance. Anyone who has been phone-polled (“How do I feel about Gladys Berejiklian?”) will recognise its limitations. It is not a platform for the creation of policy by itself. But however crude and confused it might be, an analysis of polling en masse does map the co-ordinates of disenfranchisement. It gives a voice to the largely silent half of a “national conversation”.

A national conversation. © Commonwealth of Australia

You also can’t undertake this trawl without dragging over a natural tension in democracy: just because something is popular doesn’t make it wise. Some of modern democracy’s earliest architects were very cautious about the idiocy of crowds, and warned against mainlining populism. “Pure democracy cannot subsist long nor be carried far into the departments of state – it is very subject to caprice and the madness of popular rage,” wrote the American founding father John Witherspoon. Instead, most countries settled on representative democracy, where candidates reflected the wishes of their constituents, but not always. It’s the philosophy contained in Edmund Burke’s famous line: “Your representative owes you, not his industry only, but his judgement; and he betrays, instead of serving you, if he sacrifices it to your opinion.” But he didn’t have to worry about Newspoll.

The sentiment is still recognisable in contemporary Australia. In the late 1990s David Marr interviewed the independent senator Brian Harradine, who held the balance of power in the upper house. Marr confronted him with a survey that showed majority support for euthanasia. So why was he stopping it? Harradine pointed out that a majority wanted the death penalty as well, 30 years after it was abolished. “Later I checked his figures and found he was right,” wrote Marr. It left a lasting impression on him, and he spent the Howard years thinking about “the false model of democracy as perpetual popularity contest”.

There are some places, but not many, where that model doesn’t look so false. Switzerland is one. Under the Swiss system of direct democracy, citizens can bring referenda by petition, have representatives removed mid-term and strike down laws they disagree with if they have the numbers. It’s a very stable and politically engaged country, but one that has developed inequities that are almost unique. At the local level, one administrative division refused to give women the vote until 1990, when an embarrassed federal court finally forced their hand.

It’s telling that the issue that attracts the most Swiss referenda is immigration. More than ten votes have been contested on the question since the 1970s. In recent years their results have ratcheted up towards intolerance. In 2009, a measure was passed that banned minarets on mosques, despite the opposition of the federal government. Earlier this year, the country imperilled its place in the passport-free Schengen travel zone by putting a quota on immigration from the European Union.

Australia would likely have remained a much smaller country had it been left up to the people. As George Megalogenis has put it, “immigration is the defining issue in the battle of wills between politicians and the polls, because voters, if given the chance, will always prefer fewer arrivals”. Right now, polling consistently shows that 40–50% of Australians think immigration rates are too high, while only 5–10% think they’re too low. Two decades ago, opposition to immigration was much more fierce. On some polling, desire for it to be stopped altogether reached as high as 70%. When Julia Gillard and Tony Abbott each established “small Australia” policies in 2010, they were altering a bipartisan position that had led to a sustained period of high immigration. They also weren’t alone – economic uncertainty has made immigration more electorally problematic all over the world.

Did Australians accept relatively high levels of immigration for so long because they’re tolerant? Australians themselves don’t think so – half think we’re a racist country. Instead, Paul Keating invented a strategy that was then expanded and perfected by John Howard: placate the business community with high levels of immigration, and crack down on asylum seekers to create the appearance of control. Kevin Rudd’s alteration of this gambit caused extreme anxiety among his colleagues.

“To state the obvious,” Julia Gillard wrote to him in a private email immediately before her leadership challenge in 2010, “our primary is in the mid-30s; we can’t win an election with a primary like that and the issue of asylum-seekers is an enormous reason why our primary is at that low level. It is an issue working on every level – loss of control of the borders feeding into a narrative of a government that is incompetent and out of control.”

The public has long been opposed to asylum seekers arriving by boat. Malcolm Fraser’s decision in the mid 1970s to take in Vietnamese boat people might be one of the most politically unpopular ever undertaken in Australia – it had less than 10% support when enacted. Even Fraser started talking about deportation once significant numbers of onshore arrivals started. The golden age of asylum policy in Australia wasn’t golden, and it was never popular.

But despite the heat on talkback radio, there’s some doubt about whether or not the issue of boat people is as decisive as Gillard thought it was. Around a third of people rate it as one of the most important issues, but it seems to have little effect on voter intention.

There’s another reason governments take to refugees with such relish: this is one of the few areas where they can demonstrate resistance to globalisation without confronting vested interests or the economically powerful.

Australians of all political affiliations loathe privatisation. Both major parties propose it continually.

When vested interests are involved, the decision-making process is very different, and the strength of public opinion can be safely discarded. Immediately before the Iraq War in 2003, only 6% of Australians supported military involvement without UN backing. But it’s impossible to imagine Peter Costello writing John Howard a vexed email about that. The Australian’s Paul Kelly, the same person who had described Howard as “‘the most knee-jerk, poll-reactive, populist prime minister in the past 50 years”, wrote that the hundreds of thousands of demonstrators “merely want Howard to behave according to the populist parody of him that they have created”. Somehow Howard was able to prevent his knee from jerking this time.

It’s the same in the economic sphere. Take privatisation. Australians of all political affiliations loathe it. The public remains unconvinced of the benefits of selling public assets, with support falling over time. Twenty-nine per cent support the imminent sale of Medibank Private and 54% oppose it, according to Essential Media polling conducted in January this year. Ideas such as selling the Snowy Hydro scheme and the Australian Rail Track Corporation, as suggested by the National Commission of Audit, are opposed by between 50% and 70% of the population. But privatisation is evergreen. Both major parties propose it continually, especially at state level.

When it comes to industry assistance, all the parties manage to not listen to their voters. The Greens, for example, are in favour of industry assistance, even though their supporters are among the most heavily opposed when it comes to bailouts for the car industry and airlines. But nationally, there was majority support for auto industry subsidies, even costly ones, right up to the point the industry collapsed.

Government subsidies for agriculture, renewable energy and manufacturing are very popular, while only a third of people think mining is in need of a leg up. If Qantas remains in trouble, half think the government should bail it out, either by partial or outright ownership. The word “nationalisation”, not uttered in political circles in a generation, is still a live concept in the community.

We support tariffs, subsidies, heavy regulation and barriers to entry. Joe Hockey’s recent decision to block the foreign ownership of GrainCorp reflected how politically poisonous such deals are for the Nationals, but opposition in the community was powerful and widespread, and not just in regional areas. There is almost unanimous support for rejecting foreign ownership of farmland for crops and livestock.

There’s a huge community constituency for gambling reform, but very little political support outside of the independents. Over 60% support mandatory pre-commitment technology, according to Essential Media polling. A majority oppose the building of a second Sydney casino, and majorities think online gambling, poker machines and sports betting need more regulation. Politicians have huge incentives to ignore this groundswell. Gambling interests donate heavily to political parties, and state governments collect vast amounts from gambling. In New South Wales and Victoria, it comprises almost 10% of taxation revenue.

In the past decade we’ve moved from being a society that wants less tax to being one that is willing to pay a little more, as long as the money is spent on social services. That really means healthcare spending, which has become almost an obsession as the population ages.

Support for other kinds of social spending is mixed. Welfare recipients are treated with suspicion (unless they’re middle class), and the arts and foreign aid are lower priorities. Spending on public transport is always listed as important in the community, and just as routinely ignored at state and federal levels. For a long time now Australians have believed that big business has too much power, and those numbers are climbing. Less than half think the same of unions, compared to 80% in 1979.

On social issues, politicians often join conservative commentators in claiming to represent the “silent majority”. But on the key social questions of our time they couldn’t be more out of step. Religious resistance to euthanasia or abortion is never described as “out of the mainstream”, but it is. (The same people never describe the royal family as “elite”.) Gay marriage is often described as if it’s a radical social experiment, but among the public the concept has quickly became completely normalised. In 2004 only 38% were in favour of same-sex unions. According to Galaxy polling, that’s now 64% and rising, and 75% believe they are inevitable. A majority of Christians, Coalition voters, rural and regional voters, and young people (81% of 18- to 24-year-olds) support it. The ALP’s party platform changed to reflect this in 2011, but its MPs still haven’t caught up.

Voluntary euthanasia attracts even more widespread support. On numbers alone, opposition to voluntary euthanasia is an “extreme” position. Up to 85% of us support it, with only 10% against in some polling, and these numbers vary little by location or gender. But politicians haven’t clamoured to reflect this vast majority; they’ve done the opposite. Every legislative attempt to introduce “dying with dignity” laws has failed or been overturned. There have been at least 13 efforts since 1995, most introduced by the same small set of legislators.

On the issue of abortion, Australia has been one of the most pro-choice societies in the world for decades. Today, just 4% of us say abortion should be banned in all circumstances. Fifty-seven per cent say women should be able to obtain an abortion readily – “when they want one” – and another third supports abortion in “special circumstances”. But only Victoria and the ACT have fully decriminalised abortion, and the state of debate in NSW, Queensland and Victoria looks more likely to erode these rights than strengthen them.

It’s a strange contradiction. In economic terms, most of us value a vision of an Australia from the past, protected from the uncertainties of deregulation, more closed to immigration, and with the government more heavily involved in the economy. On the social side, our vision seems decades into the future. Taken together, they describe a place that’s nowhere to be seen.

In political circles, the explanation for this gap between opinion and reality is passion and saliency. Take euthanasia, for example. The received wisdom is that this mass of support is flabby and inconstant, while the small opposition is noisy, persistent and well organised. Politicians who address the issue can only lose votes, not win them. But in reality, the repeated legislative failures spring from politicians who are much more religious than the public at large, and a small religious lobby that still exerts a significant amount of influence over them. One of the less ostentatiously powerful lobby groups in Australia, it is still strong enough to overwhelm the influence of 85% of the population.

Imagine now the power of real vested interests, the moneyed agents in Australian politics who dwarf the heft of a few god-botherers. Not only do they exert real influence in manifold ways but they often no longer need to do it directly: politicians themselves are drawn from the same managerial class as business leaders. As the social base of our political institutions has hollowed out, the train drivers, farmers and small-business owners on the backbenches are dwindling, leaving behind lawyers, business people and union officials. Parliament has always been richer, whiter and more male than most of Australia; now it belongs almost exclusively to a different class as well. It’s no surprise record numbers consider parliamentarians out of touch – they are.

None of this is to advocate for a direct democracy in the Swiss style. But if the political class is determined to change Australia’s social contract, it has to do so with some semblance of consent. It will need to put the popular will on par with powerful interests. It will also need to prosecute its arguments in election campaigns, not spring policies afterwards as a fait accompli. Right now, that doesn’t seem very likely. “People of privilege,” said the economist JK Galbraith, “will always risk their complete destruction rather than surrender any material part of their advantage.”

Hope, where it shows itself, is not within the two-party system but outside it. All over the Western world the two-party stranglehold on government is starting to loosen, and small and micro-parties are tapping into a powerful anti-political sentiment. In Australia we see it in resurgent support for the Greens and the Palmer United Party, as well as the micro-parties. The political class is already taking measures to ensure the success of these micro-parties is never repeated, on the grounds that the Senate election in 2013 “distorted the will of the voter”. With that rationale, it seems an unusual part of the system to target.

The smaller parties may be confused, ropey and occasionally crazy, but that doesn’t sound much different from the rest of us. They’re also robustly democratic in a way the major parties aren’t. Clive Palmer’s influence is less predictable, but in many ways his views are closer to those of the general public than his business interests might suggest. As the political commentator Tad Tietze put it, “Palmer’s electoral support comes from the fact he treats the other politicians with the same derision most people have for them.” And the same derision most politicians have for us.