Democracy is tired, vindictive, self-deceiving, paranoid, clumsy and frequently ineffectual. Much of the time it is living on past glories. This sorry state of affairs reflects what we have become. But current democracy is not who we are. It is just a system of government, which we built, and which we could replace. So why don’t we replace it with something better?

This line of argument has grown louder in recent years, as democratic politics has become more unpredictable and, to many, deeply alarming in its outcomes. First Brexit, then Donald Trump, plus the rise of populism and the spread of division, has started a tentative search for plausible alternatives. But the rival systems we see around us have a very limited appeal. The unlovely forms of 21st-century authoritarianism can at best provide only a partial, pragmatic alternative to democracy. The world’s strongmen still pander to public opinion, and in the case of competitive authoritarian regimes such as the ones in Hungary and Turkey, they persist with the rigmarole of elections. From Trump to Recep Tayyip Erdoğan is not much of a leap into a brighter future.

There is a far more dogmatic alternative, which has its roots in the 19th century. Why not ditch the charade of voting altogether? Stop pretending to respect the views of ordinary people – it’s not worth it, since the people keep getting it wrong. Respect the experts instead! This is the truly radical option. So should we try it?

The name for this view of politics is epistocracy: the rule of the knowers. It is directly opposed to democracy, because it argues that the right to participate in political decision-making depends on whether or not you know what you are doing. The basic premise of democracy has always been that it doesn’t matter how much you know: you get a say because you have to live with the consequences of what you do. In ancient Athens, this principle was reflected in the practice of choosing office-holders by lottery. Anyone could do it because everyone – well, everyone who wasn’t a woman, a foreigner, a pauper, a slave or a child – counted as a member of the state. With the exception of jury service in some countries, we don’t choose people at random for important roles any more. But we do uphold the underlying idea by letting citizens vote without checking their suitability for the task.

Critics of democracy – starting with Plato – have always argued that it means rule by the ignorant, or worse, rule by the charlatans that the ignorant people fall for. Living in Cambridge, a passionately pro-European town and home to an elite university, I heard echoes of that argument in the aftermath of the Brexit vote. It was usually uttered sotto voce – you have to be a brave person to come out as an epistocrat in a democratic society – but it was unquestionably there. Behind their hands, very intelligent people muttered to each other that this is what you get if you ask a question that ordinary people don’t understand. Dominic Cummings, the author of the “Take Back Control” slogan that helped win the referendum, found that his critics were not so shy about spelling it out to his face. Brexit happened, they told him, because the wicked people lied to the stupid people. So much for democracy.

To say that democrats want to be ruled by the stupid and the ignorant is unfair. No defender of democracy has ever claimed that stupidity or ignorance are virtues in themselves. But it is true that democracy doesn’t discriminate on the grounds of a lack of knowledge. It considers the ability to think intelligently about difficult questions a secondary consideration. The primary consideration is whether an individual is implicated in the outcome. Democracy asks only that the voters should be around long enough to suffer for their own mistakes.

The question that epistocracy poses is: why don’t we discriminate on the basis of knowledge? What’s so special about letting everyone take part? Behind it lies the intuitively appealing thought that, instead of living with our mistakes, we should do everything in our power to prevent them in the first place – then it wouldn’t matter who has to take responsibility.

This argument has been around for more than 2,000 years. For most of that time, it has been taken very seriously. The consensus until the end of the 19th century was that democracy is usually a bad idea: it is just too risky to put power in the hands of people who don’t know what they are doing. Of course, that was only the consensus among intellectuals. We have little way of knowing what ordinary people thought about the question. Nobody was asking them.

Over the course of the 20th century, the intellectual consensus was turned around. Democracy established itself as the default condition of politics, its virtues far outweighing its weaknesses. Now the events of the 21st century have revived some of the original doubts. Democracies do seem to be doing some fairly stupid things at present. Perhaps no one will be able to live with their mistakes. In the age of Trump, climate change and nuclear weapons, epistocracy has teeth again.

So why don’t we give more weight to the views of the people who are best qualified to evaluate what to do? Before answering that question, it is important to distinguish between epistocracy and something with which it is often confused: technocracy. They are different. Epistocracy means rule by the people who know best. Technocracy is rule by mechanics and engineers. A technocrat is someone who understands how the machinery works.

In November 2011, Greek democracy was suspended and an elected government was replaced by a cabinet of experts, tasked with stabilising the collapsing Greek economy before new elections could be held. This was an experiment in technocracy, however, not epistocracy. The engineers in this case were economists. Even highly qualified economists often haven’t a clue what’s best to do. What they know is how to operate a complex system that they have been instrumental in building – so long as it behaves the way it is meant to. Technocrats are the people who understand what’s best for the machine. But keeping the machine running might be the worst thing we could do. Technocrats won’t help with that question.

Both representative democracy and pragmatic authoritarianism have plenty of space for technocracy. Increasingly, each system has put decision-making capacity in the hands of specially trained experts, particularly when it comes to economic questions. Central bankers wield significant power in a wide variety of political systems around the world. For that reason, technocracy is not really an alternative to democracy. Like populism, it is more of an add-on. What makes epistocracy different is that it prioritises the “right” decision over the technically correct decision. It tries to work out where we should be going. A technocrat can only tell us how we should get there.

How would epistocracy function in practice? The obvious difficulty is knowing who should count as the knowers. There is no formal qualification for being a general expert. It is much easier to identify a suitable technocrat. Technocracy is more like plumbing than philosophy. When Greece went looking for economic experts to sort out its financial mess, it headed to Goldman Sachs and the other big banks, since that is where the technicians were congregated. When a machine goes wrong, the people responsible for fixing it often have their fingerprints all over it already.

Historically, some epistocrats have tackled the problem of identifying who knows best by advocating non-technical qualifications for politics. If there were such a thing as the university of life, that’s where these epistocrats would want political decision-makers to get their higher degrees. But since there is no such university, they often have to make do with cruder tests of competence. The 19th-century philosopher John Stuart Mill argued for a voting system that granted varying numbers of votes to different classes of people depending on what jobs they did. Professionals and other highly educated individuals would get six or more votes each; farmers and traders would get three or four; skilled labourers would get two; unskilled labourers would get one. Mill also pushed hard for women to get the vote, at a time when that was a deeply unfashionable view. He did not do this because he thought women were the equals of men. It was because he thought some women, especially the better educated, were superior to most men. Mill was a big fan of discrimination, so long as it was on the right grounds.

To 21st-century eyes, Mill’s system looks grossly undemocratic. Why should a lawyer get more votes than a labourer? Mill’s answer would be to turn the question on its head: why should a labourer get the same number of votes as a lawyer? Mill was no simple democrat, but he was no technocrat either. Lawyers didn’t qualify for their extra votes because politics placed a special premium on legal expertise. No, lawyers got their extra votes because what’s needed are people who have shown an aptitude for thinking about questions with no easy answers. Mill was trying to stack the system to ensure as many different points of view as possible were represented. A government made up exclusively of economists or legal experts would have horrified him. The labourer still gets a vote. Skilled labourers get two. But even though a task like bricklaying is a skill, it is a narrow one. What was needed was breadth. Mill believed that some points of view carried more weight simply because they had been exposed to more complexity along the way.

Jason Brennan, a very 21st-century philosopher, has tried to revive the epistocratic conception of politics, drawing on thinkers like Mill. In his 2016 book Against Democracy, Brennan insists that many political questions are simply too complex for most voters to comprehend. Worse, the voters are ignorant about how little they know: they lack the ability to judge complexity because they are so attached to simplistic solutions that feel right to them.

Brennan writes: “Suppose the United States had a referendum on whether to allow significantly more immigrants into the country. Knowing whether this is a good idea requires tremendous social scientific knowledge. One needs to know how immigration tends to affect crime rates, domestic wages, immigrants’ welfare, economic growth, tax revenues, welfare expenditures and the like. Most Americans lack this knowledge; in fact, our evidence is that they are systematically mistaken.”

In other words, it’s not just that they don’t know; it’s not even that they don’t know that they don’t know; it’s that they are wrong in ways that reflect their unwavering belief that they are right.

Facebook Twitter Pinterest Some philosophers advocate exams for voters, to ‘screen out citizens who are badly misinformed’. Photograph: David Jones/PA

Brennan doesn’t have Mill’s faith that we can tell how well-equipped someone is to tackle a complex question by how difficult that person’s job is. There is too much chance and social conditioning involved. He would prefer an actual exam, to “screen out citizens who are badly misinformed or ignorant about the election, or who lack basic social scientific knowledge”. Of course, this just pushes the fundamental problem back a stage without resolving it: who gets to set the exam? Brennan teaches at a university, so he has little faith in the disinterested qualities of most social scientists, who have their own ideologies and incentives. He has also seen students cramming for exams, which can produce its own biases and blind spots. Still, he thinks Mill was right to suggest that the further one advances up the educational ladder, the more votes one should get: five extra votes for finishing high school, another five for a bachelor’s degree, and five more for a graduate degree.

Brennan is under no illusions about how provocative this case is today, 150 years after Mill made it. In the middle of the 19th century, the idea that political status should track social and educational standing was barely contentious; today, it is barely credible. Brennan also has to face the fact that contemporary social science provides plenty of evidence that the educated are just as subject to groupthink as other people, sometimes even more so. The political scientists Larry Bartels and Christopher Achen point this out in their 2016 book Democracy for Realists: “The historical record leaves little doubt that the educated, including the highly educated, have gone wrong in their moral and political thinking as often as everyone else.” Cognitive biases are no respecters of academic qualifications. How many social science graduates would judge the question about immigration according to the demanding tests that Brennan lays out, rather than according to what they would prefer to believe? The irony is that if Brennan’s voter exam were to ask whether the better-educated deserve more votes, the technically correct answer might be no. It would depend on who was marking it.

However, in one respect Brennan insists that the case for epistocracy has grown far stronger since Mill made it. That is because Mill was writing at the dawn of democracy. Mill published his arguments in the run-up to what became the Second Reform Act of 1867, which doubled the size of the franchise in Britain to nearly 2.5 million voters (out of a general population of 30 million). Mill’s case for epistocracy was based on his conviction that over time it would merge into democracy. The labourer who gets one vote today would get more tomorrow, once he had learned how to use his vote wisely. Mill was a great believer in the educative power of democratic participation.

Brennan thinks we now have 100-plus years of evidence that Mill was wrong. Voting is bad for us. It doesn’t make people better informed. If anything, it makes them stupider, because it dignifies their prejudices and ignorance in the name of democracy. “Political participation is not valuable for most people,” Brennan writes. “On the contrary, it does most of us little good and instead tends to stultify and corrupt us. It turns us into civic enemies who have grounds to hate one another.” The trouble with democracy is that it gives us no reason to become better informed. It tells us we are fine as we are. And we’re not.

In the end, Brennan’s argument is more historical than philosophical. If we were unaware of how democracy would turn out, it might make sense to cross our fingers and assume the best of it. But he insists that we do know, and so we have no excuse to keep kidding ourselves. Brennan thinks that we should regard epistocrats like himself as being in the same position as democrats were in the mid-19th century. What he is championing is anathema to many people, as democracy was back then. Still, we took a chance on democracy, waiting to see how it would turn out. Why shouldn’t we take a chance on epistocracy, now we know how the other experiment went? Why do we assume that democracy is the only experiment we are ever allowed to run, even after it has run out of steam?

It’s a serious question, and it gets to how the longevity of democracy has stifled our ability to think about the possibility of something different. What was once a seemingly reckless form of politics has become a byword for caution. And yet there are still good reasons to be cautious about ditching it. Epistocracy remains the reckless idea. There are two dangers in particular.

The first is that we set the bar too high in politics by insisting on looking for the best thing to do. Sometimes it is more important to avoid the worst. Even if democracy is often bad at coming up with the right answers, it is good at unpicking the wrong ones. Moreover, it is good at exposing people who think they always know best. Democratic politics assumes there is no settled answer to any question and it ensures that is the case by allowing everyone a vote, including the ignorant. The randomness of democracy – which remains its essential quality – protects us against getting stuck with truly bad ideas. It means that nothing will last for long, because something else will come along to disrupt it.

Epistocracy is flawed because of the second part of the word rather than the first – this is about power (kratos) as much as it is about knowledge (episteme). Fixing power to knowledge risks creating a monster that can’t be deflected from its course, even when it goes wrong – which it will, since no one and nothing is infallible. Not knowing the right answer is a great defence against people who believe that their knowledge makes them superior.

Brennan’s response to this argument (a version of which is made by David Estlund in his 2007 book Democratic Authority) is to turn it on its head. Since democracy is a form of kratos, too, he says, why aren’t we concerned about protecting individuals from the incompetence of the demos just as much as from the arrogance of the epistocrats? But these are not the same kinds of power. Ignorance and foolishness don’t oppress in the same way that knowledge and wisdom do, precisely because they are incompetent: the demos keeps changing its mind.

The democratic case against epistocracy is a version of the democratic case against pragmatic authoritarianism. You have to ask yourself where you’d rather be when things go wrong. Maybe things will go wrong quicker and more often in a democracy, but that is a different issue. Rather than thinking of democracy as the least worst form of politics, we could think of it as the best when at its worst. It is the difference between Winston Churchill’s famous dictum and a similar one from Alexis de Tocqueville a hundred years earlier that is less well-known but more apposite. More fires get started in a democracy, de Tocqueville said, but more fires get put out, too.

The recklessness of epistocracy is also a function of the historical record that Brennan uses to defend it. A century or more of democracy may have uncovered its failings, but they have also taught us that we can live with them. We are used to the mess and attached to the benefits. Being an epistocrat like Mill before democracy had got going is very different from being one now that democracy is well established. We now know what we know, not just about democracy’s failings, but about our tolerance for its incompetences.

The great German sociologist Max Weber, writing at the turn of the 20th century, took it for granted that universal suffrage was a dangerous idea, because of the way that it empowered the mindless masses. But he argued that once it had been granted, no sane politician should ever think about taking it away: the backlash would be too terrible. The only thing worse than letting everyone vote is telling some people that they no longer qualify. Never mind who sets the exam, who is going to tell us that we’ve failed? Mill was right: democracy comes after epistocracy, not before. You can’t run the experiment in reverse.

The cognitive biases that epistocracy is meant to rescue us from are what will ultimately scupper it. Loss aversion makes it more painful to be deprived of something we have that doesn’t always work than something we don’t have that might. It’s like the old joke. Q: “Do you know the way to Dublin?” A: “Well, I wouldn’t start from here.” How do we get to a better politics? Well, maybe we shouldn’t start from here. But here is where we are.

Still, there must be other ways of trying to inject more wisdom into democratic politics than an exam. This is the 21st century: we have new tools to work with. If many of the problems with democracy derive from the business of politicians hawking for votes at election time, which feeds noise and bile into the decision-making process, perhaps we should try to simulate what people would choose under more sedate and reflective conditions. For instance, it may be possible to extrapolate from what is known about voters’ interests and preferences what they ought to want if they were better able to access the knowledge they needed. We could run mock elections that replicate the input from different points of view, as happens in real elections, but which strip out all the distractions and distortions of democracy in action.

Brennan suggests the following: “We can administer surveys that track citizens’ political preferences and demographic characteristics, while testing their basic objective political knowledge. Once we have this information, we can simulate what would happen if the electorate’s demographics remained unchanged, but all citizens were able to get perfect scores on tests of objective political knowledge. We can determine, with a strong degree of confidence, what ‘We the People’ would want, if only ‘We the People’ understood what we were talking about.”

Democratic dignity – the idea that all citizens should be allowed to express their views and have them taken seriously by politicians – goes out the window under such a system. We are each reduced to data points in a machine-learning exercise. But, according to Brennan, the outcomes should improve.

In 2017, a US-based digital technology company called Kimera Systems announced that it was close to developing an AI named Nigel, whose job was to help voters know how they should vote in an election, based on what it already knew of their personal preferences. Its creator, Mounir Shita, declared: “Nigel tries to figure out your goals and what reality looks like to you and is constantly assimilating paths to the future to reach your goals. It’s constantly trying to push you in the right direction.”

Facebook Twitter Pinterest ‘Politicians don’t care what we actually want. They care what they can persuade us we want’ … Donald Trump in Michigan last week. Photograph: Chirag Wakaskar/SOPA/Rex/Shutterstock

This is the more personalised version of what Brennan is proposing, with some of the democratic dignity plugged back in. Nigel is not trying to work out what’s best for everyone, only what’s best for you. It accepts your version of reality. Yet Nigel understands that you are incapable of drawing the correct political inferences from your preferences. You need help, from a machine that has seen enough of your personal behaviour to understand what it is you are after. Siri recommends books you might like. Nigel recommends political parties and policy positions.

Would this be so bad? To many people it instinctively sounds like a parody of democracy because it treats us like confused children. But to Shita it is an enhancement of democracy because it takes our desires seriously. Democratic politicians don’t much care what it is that we actually want. They care what it is they can persuade us we want, so they can better appeal to it. Nigel puts the voter first. At the same time, by protecting us from our own confusion and inattention, Nigel strives to improve our self-understanding. Brennan’s version effectively gives up on Mill’s original idea that voting might be an educative experience. Shita hasn’t given up. Nigel is trying to nudge us along the path to self-knowledge. We might end up learning who we really are.

The fatal flaw with this approach, however, is that we risk learning only who it is we think we are, or who it is we would like to be. Worse, it is who we would like to be now, not who or what we might become in the future. Like focus groups, Nigel provides a snapshot of a set of attitudes at a moment in time. The danger of any system of machine learning is that it produces feedback loops. By restricting the dataset to our past behaviour, Nigel teaches us nothing about what other people think, or even about other ways of seeing the world. Nigel simply mines the archive of our attitudes for the most consistent expression of our identities. If we lean left, we will end up leaning further left. If we lean right, we will end up leaning further right. Social and political division would widen. Nigel is designed to close the circle in our minds.

There are technical fixes for feedback loops. Systems can be adjusted to inject alternative points of view, to notice when data is becoming self-reinforcing or simply to randomise the evidence. We can shake things up to lessen the risk that we get set in our ways. For instance, Nigel could make sure that we visit websites that challenge rather than reinforce our preferences. Alternatively, on Brennan’s model, the aggregation of our preferences could seek to take account of the likelihood that Nigel had exaggerated rather than tempered who we really are. A Nigel of Nigels – a machine that helps other machines to better align their own goals – could try to strip out the distortions from the artificial democracy we have built. After all, Nigel is our servant, not our master. We can always tell him what to do.

How the education gap is tearing politics apart | David Runciman Read more

But that is the other fundamental problem with 21st-century epistocracy: we won’t be the ones telling Nigel what to do. It will be the technicians who have built the system. They are the experts we rely on to rescue us from feedback loops. For this reason, it is hard to see how 21st-century epistocracy can avoid collapsing back into technocracy. When things go wrong, the knowers will be powerless to correct for them. Only the engineers who built the machines have that capacity, which means that it will be the engineers who have the power.

In recent weeks, we have been given a glimpse of what rule by engineers might look like. It is not an authoritarian nightmare of oppression and violence. It is a picture of confusion and obfuscation. The power of engineers never fully comes out into the open, because most people don’t understand what it is they do. The sight of Mark Zuckerberg, perched on his cushion, batting off the ignorant questions of the people’s representatives in Congress is a glimpse of a technocratic future in which democracy meets its match. But this is not a radical alternative to democratic politics. It is simply a distortion of it.

Adapted from How Democracy Ends by David Runciman, which will be published by Profile on 10 May. Buy it at guardianbookshop.com

David Runciman will join Decca Aitkenhead and Yascha Mounk at a Guardian Live event, Is this how democracy ends?, in London on Wednesday 30 May.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.