What if democracy doesn’t work? What if it never has and never will? What if government of the people, by the people, for the people is a fairytale? What if it functions as a justifying myth for liars and charlatans?

There are plenty of reasons to raise these questions. The lies, exaggerations and fearmongering on both sides of the Brexit non-debate; the xenophobic fables that informed the Hungarian referendum; Donald Trump’s ability to shake off almost any scandal and exposure; the election of Rodrigo Duterte in the Philippines, who gleefully compares himself to Hitler: are these isolated instances or do they reveal a systemic problem?

Democracy for Realists, published earlier this year by the social science professors Christopher Achen and Larry Bartels, argues that the “folk theory of democracy” – the idea that citizens make coherent and intelligible policy decisions, on which governments then act – bears no relationship to how it really works. Or could ever work.

Voters, they contend, can’t possibly live up to these expectations. Most are too busy with jobs and families and troubles of their own. When we do have time off, not many of us choose to spend it sifting competing claims about the fiscal implications of quantitative easing. Even when we do, we don’t behave as the theory suggests.

2.8 million voters punished Al Gore for the floods and droughts of 2000 – ironic, given his position on climate change

Our folk theory of democracy is grounded in an Enlightenment notion of rational choice. This proposes that we make political decisions by seeking information, weighing the evidence and using it to choose good policies, then attempting to elect a government that will champion those policies. In doing so, we compete with other rational voters, and seek to reach the unpersuaded through reasoned debate.

In reality, the research summarised by Achen and Bartels suggests, most people possess almost no useful information about policies and their implications, have little desire to improve their state of knowledge, and have a deep aversion to political disagreement. We base our political decisions on who we are rather than what we think. In other words, we act politically – not as individual, rational beings but as members of social groups, expressing a social identity. We seek out the political parties that seem to correspond best to our culture, with little regard to whether their policies support our interests. We remain loyal to political parties long after they have ceased to serve us.

Of course, shifts do happen, sometimes as a result of extreme circumstances, sometimes because another party positions itself as a better guardian of a particular cultural identity. But they seldom involve a rational assessment of policy.

The idea that parties are guided by policy decisions made by voters also seems to be a myth; in reality, the parties make the policies and we fall into line. To minimise cognitive dissonance – the gulf between what we perceive and what we believe – we either adjust our views to those of our favoured party or avoid discovering what the party really stands for. This is how people end up voting against their interests.

We are suckers for language. When surveys asked Americans whether the federal government was spending too little on “assistance to the poor”, 65% agreed. But only 25% agreed that it was spending too little on “welfare”. In the approach to the 1991 Gulf war, nearly two-thirds of Americans said they were willing to “use military force”; less than 30% were willing to “go to war”.

Even the less ambitious notion of democracy – that it’s a means by which people punish or reward governments – turns out to be divorced from reality. We remember only the past few months of a government’s performance (a bias known as “duration neglect”) and are hopeless at correctly attributing blame. A great white shark that killed five people in July 1916 caused a 10% swing against Woodrow Wilson in the beach communities of New Jersey.In 2000, according to analysis by the authors 2.8 million voters punished the Democrats for the floods and droughts that struck that year. Al Gore, they say, lost Arizona, Louisiana, Nevada, Florida, New Hampshire, Tennessee and Missouri as a result – which is ironic given his position on climate change.

The obvious answer is better information and civic education. But this doesn’t work either. Moderately informed Republicans were more inclined than Republicans with the least information to believe that Bill Clinton oversaw an increase in the budget deficit (it declined massively). Why? Because, unlike the worst informed, they knew he was a Democrat. The tiny number of people with a very high level of political information tend to use it not to challenge their own opinions but to rationalise them. Political knowledge, Achen and Bartels argue, “enhances bias”.

Direct democracy – referendums and citizens’ initiatives – seems to produce even worse results. In the US initiatives are repeatedly used by multimillion-dollar lobby groups to achieve results that state legislatures won’t grant them. They tend to replace taxes with user fees, stymie the redistribution of wealth and degrade public services. Whether representative or direct, democracy comes to be owned by the elites.

This is not to suggest that it has no virtues; just that those it does have are not those we principally ascribe to it. It allows governments to be changed without bloodshed, limits terms in office, and ensures that the results of elections are widely accepted. Sometimes public attribution of blame will coincide with reality, which is why you don’t get famines in democracies.

In these respects it beats dictatorship. But is this all it has to offer? A weakness of Democracy for Realists is that most of its examples are drawn from the US, and most of those are old. Had the authors examined popular education groups in Latin America, participatory budgets in Brazil and New York, the fragmentation of traditional parties in Europe and the movement that culminated in Bernie Sanders’ near miss, they might have discerned more room for hope. This is not to suggest that the folk theory of democracy comes close to reality anywhere, but that the situation is not as hopeless as they propose.

Persistent, determined, well-organised groups can bring neglected issues to the fore and change political outcomes. But in doing so they cannot rely on what democracy ought to be. We must see it for what it is. And that means understanding what we are.