Seeing the British establishment struggle with the financial sector is like watching an alcoholic who still resists the idea that something drastic needs to happen for him to turn his life around. Until 2008 there was denial over what finance had become. When a series of bank failures made this impossible, there was widespread anger, leading to the public humiliation of symbolic figures. But the scandals kept coming, and so we entered stage three – what therapists call "bargaining". A broad section of the political class now recognises the need for change but remains unable to see the necessity of a fundamental overhaul. Instead it offers fixes and patches, from tiny increases in leverage ratios to bonus clawbacks and "electrified ring fences".

Today's report by the parliamentary commission on banking standards (to which I gave evidence) is a perfect example of this tendency to fight the symptoms while keeping the dysfunctional system itself intact. The commission, set up after last year's Libor scandal, identifies all the structural problems and nails the fundamental flaw in finance today: "Too many bankers, especially at the most senior levels, have operated in an environment with insufficient personal responsibility." Indeed, as they like to say in the City, running a mega-bank these days is like "Catholicism without a hell", or "playing russian roulette with someone else's head".

In response, the commission proposes jailing reckless bankers. Restoring the link between risk, reward and responsibility is a crucial step towards a robust and stable financial sector. But the report's focus on individual responsibility is also dangerously incomplete because it implies that the sector is merely out of control. This plays into the narrative that things can be fixed by tweaking rules and realigning incentives; in other words, by bargaining.

In reality the financial sector is not out of control. It's beyond control. During the past two years I have interviewed almost 200 people working in finance in London: "front office" bankers with telephone-number bonuses as well as those in "risk and compliance" who are meant to stop them being reckless. I have also spoken to many internal and external accountants, lawyers and consultants.

The picture emerging from those interviews is of big banks not as coherent units run by top bankers who know what they are doing. Instead these banks seem, in the words of Manchester University anthropologist Karel Williams, "loose federations of money-making franchises". One risk analyst talked about her bank as "a nation engaged in perpetual civil war", while a trader said, "You have to understand, it's us against the bank."

I could give 50 similar quotes. Taken together, they leave but one conclusion: employees at the big banks themselves do not believe their top people know what's going on; the big banks have simply become too complex and too big to manage. If this is true, the solution is not so much to jail the top bankers when something goes wrong, it is to break up the banks into manageable parts. But the British establishment still seems incapable of accepting the notion that a bank that is too big to fail or manage is a also bank that is too big to exist.

The same seems to apply to the need to restore market forces in the financial sector: the second source of structural dysfunctionality. Imagine a restaurant had served up product as toxic as that which big banks, credit rating agencies and accountancy firms were churning out until 2008. You would expect that restaurant to have closed. You would also expect new restaurants to have opened up in the area. This is how a free market should work: competition drives out bad practices.

But where are the new credit-rating agencies, accountancy firms or big banks? Even worse, not only are there just four major accountancy firms, they are also financially dependent on the very banks they are supposed to audit critically. It's the same with the three credit-rating agencies dominating the market.

And it gets worse. Imagine that a restaurant in your neighbourhood made the kind of money paid to top employees in banking, credit-rating and accountancy firms. You'd expect people rushing to open more restaurants, and with that increased competition you'd expect wages to come down. Again, this is how competition works. There are thousands and thousands of young graduates aching to get into investment banking, so no shortage of prospective chefs. So where are the new players in high finance?

The reality is that global high finance is de facto a set of interlocking cartels that divide the market among themselves and use their advantages to keep out competitors. Cartels can extract huge premiums over what would be normal profits in a functioning market, and part of those profits go to keeping the cartel intact: huge PR efforts, a permanent recruiting circus drawing in top academic talent; clever sponsoring of, say, an ambitious politician's cycling scheme; vast lobbying efforts behind the scenes; and highly lucrative second careers for ex-politicians. There is also plenty of money to offer talented regulators three or four times their salary.

Capitalists have an expression for this, and it's "market failure". Here is the source of so many of the perversities in modern finance, and the solution is not only to denounce those who can't resist its temptations, it's to take away those temptations. That probably means smaller banks, smaller and independent accountancy firms and credit-rating agencies, simpler financial products, and much higher capital requirements.

Before studying bankers I spent many years researching Islam and Muslims. I set out with images in my mind of angry bearded men burning American flags, but as the years went by I became more and more optimistic: beyond the frightening rhetoric and sensationalist television footage, ordinary Muslim people go about their day like all other human beings. The problem of radical Islam is smaller and more containable than Islamophobes believe.

With bankers I have experienced an opposite trajectory. I started with the reassuring images in my mind of well-dressed bankers and their lobbyists; surely at some basic level these people knew what they were doing? But after two years I feel myself becoming deeply pessimistic and genuinely terrified. This system is highly dysfunctional, deeply entrenched, and enormously abusive, both to its own workers and the society it operates in. The problem really is exactly as bad as the "banker bashers" believe.

More from Joris Luyendijk's banking blog

'It's a proto-fascist ideology. Top bankers as predators and us as prey' – the latest interview in the blog