In the summer of 2015, seven years after the financial crisis and with no end in sight to the ensuing economic stagnation for millions of citizens, I visited a new club. Nestled among the hedge-fund managers on Grosvenor Street in Mayfair, Number Twenty had recently been opened by accountancy firm KPMG. It was, said the firm’s then UK chairman Simon Collins in the fluent corporate-speak favoured by today’s top accountants, “a West End space” for clients “to meet, mingle and touch down”. The cost of the 15-year lease on the five-storey building was undisclosed, but would have been many tens of millions of pounds. It was evidently a price worth paying to look after the right people.

Inside, Number Twenty is patrolled by a small army of attractive, sharply uniformed serving staff. On one floor are dining rooms and cabinets stocked with fine wines. On another, a cocktail bar leads out on to a roof terrace. Gazing down on the refreshed executives are neo-pop art portraits of the men whose initials form today’s KPMG: Piet Klynveld (an early 20th-century Amsterdam accountant), William Barclay Peat and James Marwick (Victorian Scottish accountants) and Reinhard Goerdeler (a German concentration-camp survivor who built his country’s leading accountancy firm).

KPMG’s founders had made their names forging a worldwide profession charged with accounting for business. They had been the watchdogs of capitalism who had exposed its excesses. Their 21st-century successors, by contrast, had been found badly wanting. They had allowed a series of US subprime mortgage companies to fuel the financial crisis from which the world was still reeling.

“What do they say about hubris and nemesis?” pondered the unconvinced insider who had taken me into the club. There was certainly hubris at Number Twenty. But by shaping the world in which they operate, the accountants have ensured that they are unlikely to face their own downfall. As the world stumbles from one crisis to the next, its economy precarious and its core financial markets inadequately reformed, it won’t be the accountants who pay the price of their failure to hold capitalism to account. It will once again be the millions who lose their jobs and their livelihoods. Such is the triumph of the bean counters.

The demise of sound accounting became a critical cause of the early 21st-century financial crisis. Auditing limited companies, made mandatory in Britain around a hundred years earlier, was intended as a check on the so-called “principal/agent problem” inherent in the corporate form of business. As Adam Smith once pointed out, “managers of other people’s money” could not be trusted to be as prudent with it as they were with their own. When late-20th-century bankers began gambling with eye-watering amounts of other people’s money, good accounting became more important than ever. But the bean counters now had more commercial priorities and – with limited liability of their own – less fear for the consequences of failure. “Negligence and profusion,” as Smith foretold, duly ensued.

After the fall of Lehman Brothers brought economies to their knees in 2008, it was apparent that Ernst & Young’s audits of that bank had been all but worthless. Similar failures on the other side of the Atlantic proved that balance sheets everywhere were full of dross signed off as gold. The chairman of HBOS, arguably Britain’s most dubious lender of the boom years, explained to a subsequent parliamentary enquiry: “I met alone with the auditors – the two main partners – at least once a year, and, in our meeting, they could air anything that they found difficult. Although we had interesting discussions – they were very helpful about the business – there were never any issues raised.”

Facebook Twitter Pinterest A new ticker about the Lehman Brothers collapse in New York in 2008. Photograph: Alamy

This insouciance typified the state auditing had reached. Subsequent investigations showed that rank-and-file auditors at KPMG had indeed questioned how much the bank was setting aside for losses. But such unhelpful matters were not something for the senior partners to bother about when their firm was pocketing handsome consulting income – £45m on top of its £56m audit fees over about seven years – and the junior bean counters’ concerns were not followed up by their superiors.

Half a century earlier, economist JK Galbraith had ended his landmark history of the 1929 Great Crash by warning of the reluctance of “men of business” to speak up “if it means disturbance of orderly business and convenience in the present”. (In this, he thought, “at least equally with communism, lies the threat to capitalism”.) Galbraith could have been prophesying accountancy a few decades later, now led by men of business rather than watchdogs of business.

Another American writer of the same period caught the likely cause of the bean counters’ blindness to looming danger even more starkly. “It is difficult to get a man to understand something”, wrote Upton Sinclair, “when his salary depends upon his not understanding it.”

For centuries, accounting itself was a fairly rudimentary process of enabling the powerful and the landed to keep tabs on those managing their estates. But over time, that narrow task was transformed by commerce. In the process it has spawned a multi-billion-dollar industry and lifestyles for its leading practitioners that could hardly be more at odds with the image of a humble number-cruncher.

Just four major global firms – Deloitte, PricewaterhouseCoopers (PwC), Ernst & Young (EY) and KPMG – audit 97% of US public companies and all the UK’s top 100 corporations, verifying that their accounts present a trustworthy and fair view of their business to investors, customers and workers. They are the only players large enough to check the numbers for these multinational organisations, and thus enjoy effective cartel status. Not that anything as improper as price-fixing would go on – with so few major players, there’s no need. “Everyone knows what everyone else’s rates are,” one of their recent former accountants told me with a smile. There are no serious rivals to undercut them. What’s more, since audits are a legal requirement almost everywhere, this is a state-guaranteed cartel.

Despite the economic risks posed by misleading accounting, the bean counters perform their duties with relative impunity. The big firms have persuaded governments that litigation against them is an existential threat to the economy. The unparalleled advantages of a guaranteed market with huge upside and strictly limited downside are the pillars on which the big four’s multi-billion-dollar businesses are built. They are free to make profit without fearing serious consequences of their abuses, whether it is the exploitation of tax laws, slanted consultancy advice or overlooking financial crime.

KPMG abandons controversial lending of researchers to MPs Read more

Conscious of their extreme good fortune and desperate to protect it, the accountants sometimes like to protest the harshness of their business conditions. “The environment that we are dealing with today is challenging – whether it’s the global economy, the geopolitical issues, or the stiff competition,” claimed PwC’s global chairman Dennis Nally in 2015, as he revealed what was then the highest-ever income for an accounting firm: $35bn. The following year the number edged up – as it did for the other three big four firms despite the stiff competition – to $36bn. Although they are too shy to say how much profit their worldwide income translates into, figures from countries where they are required to disclose it suggest PwC’s would have been approaching $10bn.

Among the challenges PwC faced, said Nally, was the “compulsory rotation” of auditors in Europe, a new game of accountancy musical chairs in which the big four exchange clients every 10 years or so. This is what passes for competition at the top of world accountancy. Some companies have been audited by the same firms for more than a century: KPMG counts General Electric as a 109-year-old client; PwC stepped down from the Barclays audit in 2016 after a 120-year stint.

As professionals, accountants are generally trusted to self-regulate – with predictably self-indulgent outcomes. Where a degree of independent oversight does exist, such as from the regulator established in the US following the Enron scandal and the other major scandal of the time, WorldCom – in which the now-defunct firm Arthur Andersen was accused of conspiring with the companies to game accountancy rules and presenting inflated profits to the market – powers are circumscribed. When it comes to setting the critical rules of accounting itself – how industry and finance are audited – the big four are equally dominant. Their alumni control the international and national standard-setters, ensuring that the rules of the game suit the major accountancy firms and their clients.

The long reach of the bean counters extends into the heart of governments. In Britain, the big four’s consultants counsel ministers and officials on everything from healthcare to nuclear power. Although their advice is always labelled “independent”, it invariably suits a raft of corporate clients with direct interests in it. And, unsurprisingly, most of the consultants’ prescriptions – such as marketisation of public services – entail yet more demand for their services in the years ahead. Mix in the routine recruitment of senior public officials through a revolving door out of government, and the big four have become a solvent dissolving the boundary between public and private interests.

There are other reasons for governments to cosset the big four. The disappearance of one of the four major firms – for example through the loss of licences following a criminal conviction, as happened to Arthur Andersen & Co in 2002 – presents an unacceptable threat to auditing. So, in what one former big-four partner described to the FT as a “Faustian relationship” between government and the profession, the firms escape official scrutiny even at low points such as the aftermath of the financial crisis. They are too few to fail.

The major accountancy firms also avoid the level of public scrutiny that their importance warrants. Major scandals in which they are implicated invariably come with more colourful villains for the media to spotlight. When, for example, the Paradise Papers hit the headlines in November 2017, the big news was that racing driver Lewis Hamilton had avoided VAT on buying a private jet. The more important fact that one of the world’s largest accountancy firms and a supposed watchdog of capitalism, EY, had designed the scheme for him and others, including several oligarchs, went largely unnoticed. Moreover, covering every area of business and public service, the big four firms have become the reporter’s friends. They can be relied on to explain complex regulatory and economic developments as “independent” experts and provide easy copy on difficult subjects.

Left to prosper with minimal competition or accountability, the bean counters have become extremely comfortable. Partners in the big four charge their time at several hundred pounds per hour, but make their real money from selling the services of their staff. The result is sports-star-level incomes for men and women employing no special talent and taking no personal or entrepreneurial risk. In the UK, partners’ profit shares progress from around £300,000 to incomes that at the top have reached £5m a year. Figures in the US are undeclared, because the firms are registered in Delaware and don’t have to publish accounts, but are thought to be similar. (In 2016, when I asked a senior partner at Deloitte what justified these riches, he sheepishly admitted that it was “a difficult question”.)

Targeting growth like any multinational corporation, despite their professional status, the big four continue to expand much faster than the world they serve. In their oldest markets, the UK and US, the firms are growing at more than twice the rate of those countries’ economies. By 2016, across 150 countries, the big four employed 890,000 people, which was more than the five most valuable companies in the world combined.

The big four are supremely talented at turning any change into an opportunity to earn more fees. For the past decade, all the firms’ real-terms global growth has come from selling more consulting services. Advising on post-crisis financial regulation has more than made up for the minor setback of 2008. KPMG starred in the ultimate “nothing succeeds like failure” story. Although – more than any other firm – it had missed the devaluation of subprime mortgages that led to a world banking collapse, before long it was brought in by the European Central Bank for a “major role in the asset quality review process” of most of the banks that now needed to be “stress-tested”.

The big four now style themselves as all-encompassing purveyors of “professional services”, offering the answers on everything from complying with regulations to IT systems, mergers and acquisitions and corporate strategy. The result is that, worldwide, they now make less than half of their income from auditing and related “assurance” services. They are consultancy firms with auditing sidelines, rather than the other way round.

The big firms’ senior partners, aware of the foundations on which their fortunes are built, nevertheless insist that auditing and getting the numbers right remains their core business. “I would trade any advisory relationship to save us from doing a bad audit,” KPMG’s UK head Simon Collins told the FT in 2015. “Our life hangs by the thread of whether we do a good-quality audit or not.” The evidence suggests otherwise. With so many inadequate audits sitting on the record alongside near-unremitting growth, it is clear that in a market with very few firms to choose from, poor performance is not a matter of life or death.

Facebook Twitter Pinterest The ‘big four’ accountancy firms. Composite: Getty / Alamy / Reuters

These days, EY’s motto is “Building a better working world” (having ditched “Quality in everything we do” as part of a rebrand following its implication in the 2008 collapse of Lehman Brothers). Yet there is vanishingly little evidence that the world is any better for the consultancy advice that now provides most of the big four’s income. Still, all spew out reams of “thought leadership” to create more work. A snapshot of KPMG’s offerings in 2017 throws up: “Price is not as important as you think”; “Four ways incumbents can partner with disruptors”; and “Customer centricity”. EY adds insights such as “Positioning communities of practice for success”, while PwC can help big finance with “Banking’s biggest hurdle: its own strategy”.

The appeal of all this hot air to executives is often based on no more than fear of missing out and the comfort of believing they’re keeping up with business trends. Unsurprisingly, while their companies effectively outsource strategic thinking to the big four and other consultancy firms, productivity flatlines in the economies they command.

The commercial imperatives behind the consultancy big sell are explicit in the firms’ own targets. KPMG UK’s first two “key performance indicators”, for example, are “revenue growth” and “improving profit margin”, followed by measures of staff and customer satisfaction (which won’t be won by giving them a hard time). Exposing false accounting, fraud, tax evasion and risks to economies – everything that society might actually want from its accountants – do not feature.

Few graduate employees at the big four arrive with a passion for rooting out financial irregularity and making capitalism safe. They are motivated by good income prospects even for moderate performers, plus maybe a vague interest in the world of business. Many want to keep their options open, noticing the prevalence of qualified accountants at the top of the corporate world; nearly a quarter of chief executives of the FTSE100 largest UK companies are chartered accountants.

When it comes to integrity and honesty, there is nothing unusual about this breed. They have a similar range of susceptibility to social, psychological and financial pressures as any other group. It would be tempting to infer from tales such as that of the senior KPMG audit partner caught in a Californian car park in 2013 trading inside information in return for a Rolex watch and thousands of dollars in cash that accountancy is a dishonest profession. But such blatant corruption is exceptional. The real problem is that the profession’s unique privileges and conflicts distil ordinary human foibles into less criminal but equally corrosive practice.

A newly qualified accountant in a major firm will generally slip into a career of what the academic Matthew Gill has called “technocratism”, applying standards lawfully but to the advantage of clients, not breaking the rules but not making a stand for truth and objectivity either. Progression to the partner ranks requires “fitting in” above all else. With serious financial incentives to get to the top, the major firms end up run by the more materially rather than ethically motivated bean counters. In the UK in 2017, none of the senior partners of the big firms had built their careers in what should be the firms’ core business of auditing. Worldwide, two of the big four were led by men who were not even qualified accountants.

The core accountancy task of auditing can seem dull next to sexier alternatives, and many a bean counter yearns for excitement that the traditional role doesn’t offer. As long ago as 1969, Monty Python captured this frustration in a sketch featuring Michael Palin as an accountant and John Cleese as his careers adviser. “Our experts describe you as an appallingly dull fellow, unimaginative, timid, lacking in initiative, spineless, easily dominated, no sense of humour, tedious company and irredeemably drab and awful,” Cleese tells Palin. “And whereas in most professions these would be considerable drawbacks, in chartered accountancy they’re a positive boon.” Palin’s character, alas, wants to become a lion tamer.

The bean counter’s quest for something more exciting can be seen running through modern scandals like Enron and some of the racy early-21st-century bank accounting. One ex-big four accountant told me that if there was a single thing that would improve his profession, it would be to “make it boring again”.

Where once they were outsiders scrutinising the commercial world, the big four are now insiders burrowing ever deeper into it. All mimic the famous alumni system of the past century’s pre-eminent management consultancy, McKinsey, ensuring that when their own consultants and bean counters move on, they stay close to the old firm and bring it more work. The threat of an already too-close relationship with business becoming even more intimate is ignored. In 2016, EY’s “global brand and external communications leader” waxed biblical on the point: “You think about the right hand of greatness; actually the alumni could be the right hand of our greatness.”

From inboxing to thought showers: how business bullshit took over Read more

The top bean counter’s self-image is no longer a modest one. “Whether serving as a steward of the proper functioning of global financial markets in the role of auditor, or solving client or societal challenges, we ask our professionals to think big about the impact they make through their work at Deloitte,” say the firm’s leaders in their “Global Impact Report”. The appreciation of the profound importance of their core auditing role does not, alas, translate into a sharp focus on the task. EY’s worldwide boss, Mark Weinberger, personifies how the top bean counters see their place in the world. He co-chairs a Russian investment committee with prime minister and Putin placeman Dmitry Medvedev; does something similar in Shanghai; sat on Donald Trump’s strategy forum until it disbanded in 2017 when the US president went fully toxic by appeasing neo-Nazis; and revels in the status of “Global Agenda Trustee” for the World Economic Forum in Davos.

The price of seats at all the top tables is a calamitous failure to account. In decades to come, without drastic reform, it will only become more expensive. If the supposed watchdogs overlook new threats, the fallout could be as cataclysmic as the last financial crisis threatened to be. Bean counting is too important to be left to today’s bean counters.

Adapted from Bean Counters: The Triumph of the Accountants and How They Broke Capitalism by Richard Brooks, which will be published by Atlantic on 7 June, priced £18.99. Buy it for £16.14 at guardianbookshop.com

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.