In a recent, laudatory newspaper profile of Boris Johnson, the Leave campaign’s leading spokesman was described as fighting “the forces of the establishment”. The newspaper which saw fit to describe the old Etonian MP and former mayor of London this way was that nest of chippy radicals, the Daily Telegraph.

These days, even the establishment is anti-establishment. Two weeks ago, Iain Duncan Smith – Conservative MP, former lieutenant in the British army, until recently a cabinet minister – also complained about the establishment, in the course of an address to the members of a private club in Belgravia.

Just as Steve Coogan’s oafish character Paul Calf used to claim he was a radical feminist because it was the only way to get his end away, even the most insidery of insiders now feel it necessary to portray themselves as alienated outsiders.

Jeremy Corbyn rose to the Labour leadership because his supporters believed him to be the only politician with the integrity to take on the country’s ruling class. The journalist Owen Jones has written a popular book called simply The Establishment, in which he pins the blame for the country’s troubles on a largely unaccountable, self-serving clique whose members rule Whitehall, the City, the law and the media.

It is traditional for the left to decry the establishment, but these days the right also rages against the machine. Nigel Farage – stockbroker, alumnus of Dulwich College – has built a successful political career on his anti-establishment credentials. Across the Channel, Marine Le Pen is doing something similar.

Hatred of the establishment is one of the few unifying forces in an age of polarisation. Anti-establishment leaders from either end of the ideological spectrum pay coy tribute to one another. In America, both Donald Trump and Bernie Sanders have achieved outrageous success by running against their respective party establishments.

A perennial aspect of human society is the tendency of the majority to rally around antipathy to an out-group, or enemy within – whether it’s witches, communists or immigrants. It is an oddity of the current moment that the out-group is the in-group. It is easy to conclude that this is because the establishment has become over-mighty. But a better explanation is that it has so little might left.

If it means anything, the establishment represents the concentration of power in an elite, the members of which are bound together by social ties, shared values and unspoken rules. The popular idea of it as an implacable, impenetrable club, able to impose its will on society at large was formed at a time when this was largely true.

Across the West, the capacity of ruling elites to govern nations has been shrinking for decades

In Britain, the term was popularised by Anthony Sampson’s 1962 bestseller, The Anatomy of Britain. A year after it was published, however, a metaphorical bomb exploded under the elites that Sampson had anatomised. Kim Philby fled to Moscow, revealing that he was the third man in the Cambridge spy ring (his fellow spies Burgess and Maclean had defected in 1951). As the historian Ben Macintyre has observed, the spy scandal wrecked the assumptions on which Britain’s ruling class had hitherto operated.

The Cambridge spies had attended the country’s most prestigious schools, and were comfortable in the leather armchairs of the Reform and Athenaeum. This shielded them from suspicion. Philby later described “a genuine mental block that stubbornly resisted the belief that respected members of the establishment could do such things”.

After the extent of their treachery was revealed, the intelligence community realised it could no longer rely on the old boy network, and the rest of the country was less inclined than it had been to defer to the privileged and high-born. Britain became a little more egalitarian, and its establishment went into a decline from which it never recovered.

In the same year that Philby’s crimes were exposed, an American president was assassinated. Subsequent investigations into JFK’s murder, along with the release of confidential documents, revealed the operation of a nexus of power, centred on the White House but operating out of view of the public. It wasn’t necessary to believe the conspiracy theories to marvel at the power wielded by a handful of swaggering politicians, bureaucrats, businessmen and gangsters.

In the 1960s, then, we learned to distrust this entity known as “the establishment”, and partly as a result, it opened up, becoming more porous, diverse and less arrogant. But its power continued to seep away, due to fundamental, long-term changes in the way we organise and communicate.

When Barack Obama was asked what surprised him most about becoming president, he replied to the effect that he hadn’t realised how difficult it would be to get anything done. He wasn’t just referring to political gridlock. Across the western world, the capacity of ruling elites to govern nations has been shrinking for decades.

In his 2013 book, The End of Power, Moisés Naím argued that our ideas about power are stuck in the mid-20th century. Most of us still think that if you hold high office, or sit at the head of a corporate boardroom, then you are clothed in immense power. But as the people who hold those positions will probably tell you, the ability of governments and corporations to make things happen is much more tightly constrained than it was in 1960.

In the last 40 years, the world has become vastly more complex, vastly more populous and vastly more intedependent. Bureaucracies, whether public or private, find it ever harder to cope with the variety of human behaviour and the speed with which those behaviours change. New media have enabled much more aggressive scrutiny from below, and increased the ability of small institutions and ad hoc groups like Occupy to organise against perceived threats and injustices.

The result is that, as Naím puts it, “being in charge isn’t what it used to be.” Bold top-down actions – a tax rise, a price cut – ramify unpredictably and often disastrously. Ministers, civil servants and company executives pull strings, only to discover that their string is attached to a booby trap – or to nothing at all.

Meanwhile, big changes to society are happening, without anyone actually deciding on them. Your children use technology you don’t understand. Your town is full of unfamiliar faces, speaking a different language. You cannot find a job as good as the one you used to have. Your daughter can’t afford to buy a house half the size of your own. These changes can be baffling. For many, they are enraging. Politicians and campaigners may direct this anger towards an aloof and omnipotent establishment. But the truth is that they are screaming at a ghost.

Ian Leslie is the author of Curious: The Desire to Know and Why Your Future Depends on it