It's easy to dismiss Greg Smith, the former Goldman Sachs banker, as a man with a grudge. This could be a classic case of sour grapes, with Smith's claim that his former employers were "morally bankrupt" and described customers as "muppets" the result of him being overlooked in the dog-eat-dog world of investment banking.

In the circumstances, though, it doesn't really matter whether Smith was a principled whistleblower or someone not really up to the job getting his own back. Why? Because his accusations had the ring of truth about them: what he said chimes not just with what Goldman's customers think of the Wall Street behemoth, but with the deep public suspicion that big finance is lining its own pockets at the expense of Main Street.

That is an entirely justified suspicion and is a function of two developments – one long-term and one more recent. The big structural change on Wall Street and in the City of London has been for the investment banks to become bigger and more integrated. That means that one team goes round trying to persuade companies to float shares, a second team of analysts then pens supposedly independent analysis suggesting the shares are a great buy, while a third team of traders makes money every time the shares move up and down on the stock market. There is a massive conflict of interest here.

There were obvious dangers in this process that stemmed from the gradual dismantling in the final quarter of the 20th century of the curbs placed on Wall Street after the Great Depression in the 1930s. They were, however, masked until the second big development: the final crisis of 2007-08. This not only exposed – brutally and repeatedly – the follies of investment banks that had allowed their greed to get the better of their judgement, but also turned the big beasts of Wall Street into supplicants for taxpayer handouts.

Those financiers who had spent the previous 15 years demanding that Washington get off the back of business suddenly found that there was case for government, after all – even if that only amounted to a willingness to write big cheques that would guarantee their annual bonuses. Rightly, ordinary voters were disgusted: privatising gain and socialising losses is not the way American capitalism (or any other form of it) is supposed to work.

To rub salt in the wound, once the immediate crisis was over, Wall Street insisted that the burden of clearing up the financial mess that Wall Street created be shouldered by ordinary voters – through cuts in public spending.

Where does this leave us? There are three conclusions from all this. The first is that Goldman Sachs has suffered reputational damage from the Smith attack. The public was tolerant of the excesses of the investment banking fraternity when times were good: when house prices are falling and real incomes are being squeezed, it tends to be less forgiving. There is no great love lost between Goldmans and its corporate customers, either.

The second conclusion is that it is time there was more competition in the investment banking industry. Returns on capital are high, and the reason for that is that the barriers are high for potential new entrants. It takes a lot of capital to set up an investment bank, which is why the industry operates as a self-serving oligopoly, rather than as a channel for productive investment in the real economy.

The final conclusion, therefore, is that if the competition does not happen naturally through market forces (and there is no hint that it will), then a much tougher regulatory regime is needed – so that the corporate finance and trading arms of investment banks are split into different firms.

That would limit the opportunity for conflict of interest and would cut the cost of finance to business. Above all, it would tackle what has become a colossal conspiracy by the super-rich against the general public.