In his 2014 book The Establishment, Owen Jones explained how and why Britain’s unequal, class-ridden system would always prevail. It was written at a time when the elite seemed to be thriving, despite having recently helped to trash the global economy. After a few lean years for “Davos man”, bank debt had effectively been nationalised. No one in power had gone to jail, while most of them seemed to be getting richer and richer. As Jones explained, the establishment was as dominant as ever.

Developments since then have sorely tested that view. After the vote for Brexit, David Cameron and George Osborne were suddenly cast adrift, while the Bank of England and captains of industry found themselves wondering who to support. The Conservative party – their political party, the only one they had ever supported – was following a course of action they thought would wreck the economy. Sterling plummeted and the FTSE 100 index stuttered. Industry investors began revolting and bankers relocating.

A year later, the establishment seemed to be recovering once again. And then came the snap June 2017 election. The Conservatives, with all their resources and an initial 20-point poll lead, lost their majority. Theresa May was outperformed by a badly dressed, pacifist republican with no money, no media support and a shadow cabinet that could fit in a phone box. The Tory party was left negotiating a Brexit deal with a dead duck leader, a hung parliament, and no idea of what outcomes the establishment wanted.

All of which suggests that it might be time to question whether the British establishment still functions as it once did. Yes, some members of the elite have become very rich. They are still united in their fear and loathing of leftwing ideas and ordinary people. They are still highly skilled when it comes to pursuing their self-interest. Their decisions still have powerful consequences that are widely felt. But they seem to be less able to exert control or predict what those consequences will be.

As an academic studying how power operates, I have spent the past 20 years researching elite figures in five areas associated with the modern establishment: the media, the City, large corporations, the Whitehall civil service and the major political parties at Westminster. After interviewing and observing more than 350 people working in or close to the top during that time, my sense of this evolving long-term crisis has become clearer. I have come to believe that the establishment is no longer coherent or collective or competent. Its failings are not only causing larger schisms, inequalities and precariousness in Britain; they also threaten the very foundations of establishment rule itself.

The idea of “the establishment” was popularised in the 1950s and early 60s. According to its key chroniclers, such as the historian Hugh Thomas and the journalist Anthony Sampson, most members of this elite network went to one of seven “Clarendon” boarding schools such as Eton, Rugby or Harrow. From there, they moved on to Sandhurst military academy or Oxbridge. They then glided effortlessly into a variety of powerful positions in private or public organisations. Political control operated through the great state institutions of the Church of England, Westminster, Whitehall and the armed forces. The economy was a stitch-up, coordinated through a public-private partnership of the Treasury, the Bank of England, the City and business leaders. The BBC and national press ensured that “the common people” accepted this state of affairs.

These elites reproduced and maintained their collective identity via exclusive social circuits. These included membership of expensive London clubs such as the Garrick and the Athenaeum. If they hadn’t already inherited a title, they would get one soon enough, ending their days in that gilded care home (or finishing school) known as the House of Lords.

Over the decades, the shape of the establishment changed. In accounts written in the 1990s by Jeremy Paxman, Andrew Adonis and Stephen Pollard, the power and influence of the Church, monarchy, aristocracy and army had clearly waned. Anthony Sampson wrote several surveys of the British establishment over five decades; in 2004, his final one recorded the stark decline of many Victorian-era institutions, and the power grabs made by new, abrasive business types and unassimilated foreigners. But despite these changes, other things remained the same – the prevalence of the Clarendon-Oxbridge conveyor belt, that same sense of shared elite interests, and so on.

What these authors did not notice was that beneath the surface, the social foundations and institutions of power were slowly weakening. Many of the traditional elements that once held the establishment together are degenerating, with little to replace them. This isn’t just a matter of a new establishment replacing an old one. It may be the end of the establishment as we have known it. (This, of course, will not mean the end of elites.)

This started to become clear to me when researching the civil service and business world in the later years of David Cameron’s coalition government. Interviews with former career mandarins revealed just how much Whitehall had changed. The service they had joined in the 1960s and 70s was the preserve of the establishment amateur. The vast majority had come from Oxbridge, having previously studied history, classics or something else that failed to equip them for managing an archaic state bureaucracy. Generalists ruled and specialists occupied the lower rungs.

City workers in London, 1954. Photograph: Alex Dellow/Getty Images

As Sir Alan Budd, a former economic advisor to the Treasury from 1970 onwards, explained, the department was not run by professional economists. Instead, those in charge deferred to the “Brown Book”, a government tome explaining how to use certain technical levers to respond to shifts in employment, inflation, etc. It was sort of a Dummies’ Guide to managing the British economy. “If you wanted to, say, reduce unemployment by 100,000, there were various ways of doing this,” Budd told me. “It was very like any textbook at university. The chapter on macroeconomics – this explained how you did things.”

Andrew Turnbull, a former head of the civil service, recalled how the economic shocks of the 1970s and the political ones of the 80s set in motion real change: even the British establishment began to realise it could no longer afford to be managed by privileged amateurs. “Meritocracy” and expertise – represented by grammar school education, the professions and PhDs – began dictating the new recruitment policy. “Gradually the classics and humanities people got replaced,” said Turnbull. “When I arrived we used to have people who were experts on Byron and musicians – rather refined people. Then, rather hard-nosed economists gradually took over, and the dominant culture became football and golf, rather than music.”

Sir John Gieve, a former deputy governor of the Bank of England, reflected on the culls that took place between the 1976 financial crisis and the early years of Margaret Thatcher’s first term as prime minister: “When I arrived at the Treasury, everyone was called Douglas, I seem to remember – and most of them left.” Around this time, the two top layers of the Treasury were removed, and a relatively junior member, who “was willing to challenge the orthodoxy and wasn’t a typical smoothy”, was elevated to the top of the department.

But it wasn’t just the civil service – a similar shift seems to have occurred across establishment strongholds from the 1980s onwards. Nothing better exemplified the declining power of the old establishment than Granada Media’s 1995 takeover of the exclusive Trusthouse Forte hotel chain. The leader of the hotel group, Sir Rocco Forte, was the son of Lord Forte, donated regularly to the Conservative party, and entertained the elite at his country estate. He had been out shooting grouse on the moors when the takeover bid was announced.

Gerry Robinson, who drove home the hostile bid, came from a poor Irish family and had become a dynamic new force in the corporate world. When he had earlier managed to capture Granada, John Cleese had called him “an upstart caterer”. That had only improved his standing in the City. Soon, a narrative formed about the takeover: flabby, inefficient old money was being run out of town by a new energetic breed succeeding on merit.

Chris Hopson, who headed up the Granada communications team, explained the symbolism of it all: “The shooting thing was a perfect analogy for our claims about how Forte was running the business – it summed up a company that was being run as a series of trophy hotels, that was an archaic family business, that wasted time on pomp and ceremony. It was too good to resist.”

The extent to which the traditional establishment had been replaced by people from different backgrounds was all the more pronounced when talking to 30 of the UK’s top business leaders nearly two decades later. Only a third of those I interviewed – 20 CEOS of FTSE 100 companies, and 10 CEOs from the top 100 private companies – came from a wealthy, upper-class background or had attended a public school. None had gone to one of the elite Clarendon schools. Only three had both a private school education and an Oxbridge degree. Several came from poor immigrant families.

Hardly any of them liked to draw attention to elements of privilege in their education or upbringing, and some proudly brandished their outsider status. One such was Andrew Owens, the co-founder and CEO of Greenergy, a fuel distribution business. Most people have never heard of it, but it has been listed as one of the UK’s top two largest private companies by turnover for several years. Owens’s father was a school caretaker and his mother a school cook. He told me his own school was “one of the worst” in Britain: “They set the bar low and they fell over it drunk on a Friday night. All they really cared about was beating Cardiff High in rugby.”

Asked about the reasons for his success, Owens says he is “fiercely tribal” when it comes to his own people and employees, but to no one and nothing else. He puts his achievements down to being quicker, more ruthless and more hard-headed than the big FTSE companies and the soft management that runs them. Who knows what John Cleese would have called him?

One word that came up several times was “disruptive”. “I don’t give a damn about losing,” Owens told me. “I do not associate anything I do with my personal self-image.” For that reason, he said, he finds it “very simple to take decisions that other people can’t take … so, we’ve been a disruptive technology in an area where people think you can’t be disruptive.”

Owens is ready to condemn slow, privileged types everywhere – in big business, in the financial sector, in politics and in government. But his most scathing comments are reserved for the civil service: “Whitehall is full of absolute idiots. It’s become a self-fulfilling black hole of hope,” he said. “This idea that you can get civil servants who are paid a fraction of the money that you could earn in industry, somehow making better judgments than industry. It’s nonsense.”

These interviews, together with longer-term survey data, reveal a couple of important trends. First, there has been a shift in elite power from the public sector to the private sector since the 1950s. This is a key theme in writings about the establishment since the Thatcher era. For Jeremy Paxman, the new “radical Toryism” was nearly as brutal towards the old establishment as it was to industrialised labour. In so doing, it had created “another elite, a new monied caste”.

Andrew Adonis and Stephen Pollard noted the emergence of a new “super class” of highly paid professionals in law, accountancy and other sectors. This produced a new “closely interlocked cadre”, separating itself from the rest. Adonis and Pollard – along with Robert Peston, writing in the wake of the 2007-8 financial crisis – have no doubt where real wealth and power have gravitated: the City of London and its networks, stretching out across the global financial system. These days, between a third and a half of FTSE 100 companies are led by non-British CEOs. The majority of shares are no longer owned by UK individuals or their pension funds, but are traded by large, unattached international investors.

Second, the automatic links between exclusive education, tradition, status, power and money, which once typified the establishment, have been broken. A far smaller percentage of those in power have taken the Clarendon-Oxbridge conveyor belt to the top. Exclusive London clubs lie empty or – worse still for elites – now allow women, foreigners and lower-class members to join. The members of the aristocracy, once liberally sprinkled across the boards of public institutions and corporations alike, have vanished from sight.

Many of those in the corporate and financial worlds do not have an elite education or a privileged past, but have amassed a great deal of money and influence. Those working at the top of state institutions are more likely to be of good establishment stock, and to receive honours – but also have less income and questionable influence.

This has left the various parts of the current establishment more disparate and more antagonistic towards each other. Dame Margaret Beckett witnessed this growing fragmentation while serving in Labour’s shadow cabinet the 1990s, and again in Tony Blair’s cabinet after 1997: “I remember John Smith [the Labour leader from 1992-94] gave a talk in the City at one point, and it was almost like the bride’s side and groom’s side – the people from the financial world, and the people from the industrial world, and they almost weren’t talking to each other. Civil servants, on the whole, hated you taking advice from people outside. One of my colleagues almost refused to see anybody from industry in case he was thought to have compromised himself in any way.”

With the educational and cultural links eroded, so too the shared values associated with the old establishment are disappearing. Earlier accounts observed that “public service” was a noble aspiration of the elite. As George Orwell noted in 1941, however useless the upper classes were, they believed in service to the nation: “One thing that has always shown that the English ruling class are morally fairly sound, is that in time of war they are ready enough to get themselves killed.”

Such a sense of duty and self-sacrifice is decidedly absent in the new elite. Instead, the values of those at the top are all about personal enrichment, individualism, enlightened self-interest and a reverence for the “wealth creators”. But such norms are antithetical to any sense of shared, collective interests. Selfish individualism and survival of the fittest are not a good basis for holding any group together – including the elite.

If the current manifestation of the establishment is no longer tied together by either shared class or collective interests, how does it maintain coherence? One answer is to be located in the ideas of neoliberalism: promoting the small state, the free-market system, low taxes and low regulation, globalisation and so on. Disparate modern elites now share an ideology that both justifies and maintains their wealth and positions of power.

But despite almost four decades of dominance over the political and economic system, neoliberalism no longer seems to provide such a stable basis for maintaining elite power and profit. All the clues are to be found in the City of London, now at the heart of British establishment power. It is in the capital’s financial sector that neoliberalism and self-interest have been dominant driving principles for far longer than anywhere else in UK. It is here that the establishment almost wiped itself out, and where it is most likely to finish the job in future.

Anyone who spends time researching the City can see the potential for things to collapse at short notice. It is a strange mix of old oak-lined establishment and shiny new steel-and-glass anarchy. No one there seems to trust anything or anyone else. Authors such as Philip Augar have documented how the death of “gentlemanly capitalism” followed the “Big Bang” of City deregulation in 1986. More recently, Joris Luyendijk, in his 2015 book about the world of banking, has shown how destructive an extended period of venal self-interest has proved to be for those who work there.

The City of London skyline. Photograph: Peter Macdiarmid/Getty Images

I have witnessed something similar while investigating of the City over the course of two decades. The most unnerving example was in 2004, while speaking to more than 30 top fund managers and financiers about their operations and the dot-com crash of 2000. What was so disturbing was realising that not much had changed since 2000, and that many could still see the potential for greater crashes ahead.

One of these was David Bailey, then a company chairman who had been in the City for 35 years. In that time, he had been a successful stockbroker, helped to set up London’s options market, advised on the technologies introduced during the Big Bang, and gone on to be a non-executive director in several big City firms. He was welcoming, friendly and blokey. He knew everyone and was happy to namedrop and gossip. He didn’t stop talking for two hours.

Prior to the Big Bang, Bailey said, the City was run by networks of old boys’ clubs, where trust and “my word is my bond” made things work. Competition was limited, slow and parochial. Collective interests were clear. The “big boys” set aside their differences and would chip in when a crisis threatened one of their local oligopolies. During the economic crises of the 1970s, the big institutions had come together, facilitated by the Bank of England, to save the larger banks and stabilise the London stock market. But this had all fallen apart since the 1980s. The old boys’ networks had been broken up. The banks were too big, too powerful and too foreign to play by the old rules.

“In the old days,” said Bailey, “every bank used to go and have tea with the governor of the Bank of England, probably on a monthly or quarterly basis. Well, do you think the boss of UBS or Dresdner Bank actually comes across and has tea with the governor of the Bank of England now? Bollocks. The Bank no longer has control of the banking structures, because it used to be done by a nudge and a wink, in the way that Britain has historically run the Commonwealth. But they don’t have the power anymore to say, ‘We collectively are going to save that business over there’, and it’s the same way in the London Stock Exchange.”

Another theme was the ignorance of those now in charge. Everything had become more complex and fast-moving. The maths deployed was becoming more advanced, but the base assumptions behind the financial models were becoming more simplistic and uncertain. Governments and regulators could not keep up. Investment banks did not know what their employees were up to, or how to manage the risks they were building up. Non-executive directors and fund managers knew nothing about the businesses they controlled: “The investment bankers don’t know how to manage these people,” Bailey told me. “And it has leached out into the wider society of all major companies. Putting the non-executives in charge of the business is a bit like taking me to the Olympics and saying: ‘Well, it’s a choice between David Bailey, who occasionally runs for the bus, and Darren Campbell, who trains seven days a week – and we’ll stick David Bailey in the 200 meters.’ Madness. The fund managers are in charge, but none of them know how to run a business.”

But, most of all, Bailey was concerned with the fact that no one seemed to be in control, or to be taking responsibility for what was happening: “Nobody is an owner. Everybody is a bloody employee. They’ve all got an incentive to pay each other more.” This was made worse by the fact that deregulation had left the City riven with conflicts of interest, enabling insiders to play all sides. The individual rewards could be astronomical, but the penalties for failure were relatively modest. And no one ever got caught.

Ultimately, the City was driven by greed and ruthless self-interest, but without personal responsibility. That meant the whole system was under threat. Bailey maintained a jolly, friendly manner, but also spoke in apocalyptic terms. “Terrifying”, “the end of capitalism” and “hugely vulnerable” were the phrases he used. The City was peopled by “predators”, “bandits” and “rabid dogs”, who rewarded themselves with huge salaries, bonuses and options, regardless of success or failure.

Remarkably, Bailey even predicted the likely causes of the financial crash and how it might play out, three years before it actually did: spiralling property prices and lending, derivatives, banks too big to fail and a Federal Reserve bailout. He was not the only one: a handful of economists and financiers – such as Nouriel Roubini, Ann Pettifor and Raghuram Rajan – went public with dire warnings of what they thought was to come. There were a lot more insiders who had their suspicions, but who chose to keep quiet. A lot of people – insiders and outsiders – still see more of the same ahead. But venal self-interest means most of them don’t care.

Yet the logic of neoliberalism and unbounded self-interest is as potentially destructive to the establishment as it is to the rest of society. After decades, its flaws and contradictions are becoming too large to deal with.

It is also the cause of another contradictory flaw that threatens the establishment. As both Robert Peston and Owen Jones have argued, the new regime – for all its individualist and anti-state rhetoric – still depends on the state. Elites require a rule of law, security, a transport infrastructure, an able workforce and social stability to function. But neoliberalism promotes an ever-smaller state and a poorer, less able employee pool, and nods through corporate and super-rich tax evasion on an industrial scale.

The international transiency of the new elite means they care little about the spaces, communities or workforces that are essential for servicing big corporations, as well as their personal needs. All of which suggests that the current manifestation of the establishment, if we can still call it that, has an extremely limited future.

Adapted from Reckless Opportunists: Elites at the End of the Establishment by Aeron Davis, which will be published by Manchester University Press on 12 March. Buy it for £9.99 at guardianbookshop.com

• This article was amended on 1 March 2018 to clarify how the financial sector reacted immediately after the 2016 EU referendum result.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

