The boss of the influential Institute of Directors has launched a stinging attack on the pay packets handed out at Royal Bank of Scotland and Barclays, which together paid 523 of their bankers more than £1m in the scandal-hit year of 2012.

Simon Walker, the IoD director general, told an audience of public relations executives that the payouts were "unacceptable".

"Thousands of people in those two companies alone earn more even than the prime minister. This is in scandal-hit companies who have had a far from successful year," Walker said.

Barclays was fined £290m for rigging Libor last year while RBS paid £390m for attempting to rig the key benchmark rate while both banks were also hit by mis-selling scandals.

Walker spoke as one of the biggest fund managers in the City, F&C Asset Management, prepared to write to the chairman of nearly 60 financial firms around the world to warn that "the public perception of unjustified excessive pay can create long-term reputational damage".

The F&C fund managers are calling for "joined up" thinking in how banks are managed in terms of "ethical performance, risk management and the incentive system for bank executives - as expressed in terms of their remuneration".

"There is growing need for bank executive management, as well as bank boards, to promote appropriate corporate values," F&C will say. The fund managers give credit to Barclays and HSBC - fined £1.2bn for money laundering offences in the US - for adopting reviews of cultures and values.

Walker was referring to Barclays and RBS in his speech in which he also talked about the "serious cause for concern" caused by the public's lack of confidence in capitalism, in part caused by "rewards for failure" in pay levels.

"Now I recognise that Antony Jenkins [chief executive] and Sir David Walker [chairman] at Barclays are working hard to introduce a new culture in Barclays in particular. And as you can imagine given my job, I'm not a rabid anticapitalist. But even I believe that this state of affairs is unacceptable."

"Shareholder value has been destroyed, capitalism has been given a bad name, key measures of the market have been manipulated for cynical gains, taxpayers have shelled out billions to bail banks out, and yet vast rewards packages are still being handed out," Walker said.

"That is why I think it is understandable that many people are concerned about the state of rewards for failure, and the disastrous effects of short-termism in some businesses. Many IoD members are concerned about exactly the same things," he said.

Addressing the public relations audience, Walker said: "People outside the communications industry sometimes think that it is possible to spin anything – that the proverbial turd can indeed be polished, if you'll pardon the metaphor. But sometimes it can't – sometimes the underlying facts are the problem".

The fund managers at F&C call on banks "as a whole" to be accountable for "past transgressions" even when management has left after wrong-doings: "This means that regulatory or legal penalties should impact the incentive pool that is distributed to its executives." F&C warned about the impact of the EU plan to cap bankers' bonuses at one times their salary, or twice in some circumstances. Walker also picked up on this: "Just look at the EU's bonus cap – wrong-headed and counter-productive as it is, the behaviour of some of our banks makes it difficult for anyone to stand at their side to credibly fight against it. They are harming the whole of British business by focusing only on their own short-term self-interest".