The Conservative party needs to break its dependence on millionaires, the former Tory chancellor Ken Clarke has told the Observer, amid a growing furore over the tax affairs of the party’s donors.

After a week of some of the most intense fighting between the parties in recent years, Clarke said the Conservatives would be strengthened by loosening the hold of rich men on their financial survival.

He called on David Cameron to cap political donations and increase state funding of political parties to put an end to damaging scandals and rows. The Conservatives have been rocked in the past week by a potentially toxic combination of allegations of tax evasion by clients of the HSBC bank, whose chairman, Lord Green, became a Tory minister; tax avoidance by party donors; and leaked details of the secretive black and white fundraising ball.

On Saturday, Green stepped down from a financial services lobby group, TheCityUK’s advisory council, in order to avoid “damaging the effectiveness” of its efforts “in promoting good governance”.

Former HSBC boss Lord Green quits banking lobby group Read more

Clarke said that while he believed the current row over donors and tax avoidance was “artificial and bogus”, such episodic rows over the funding of political parties were feeding into the growing cynicism and distrust of the British political system.

He defended Cameron’s decision to attend the fundraising black and white ball in Mayfair, where guests included a series of controversial donors, but said the time had come for the prime minister “to put on his tin hat” and secure further state funding of parties, whatever the short-term public outcry.

Clarke, who was a cabinet minister until last July, said: “I think the Conservative party will be strengthened if it is less dependent on having to raise money from wealthy individuals. But there is no way any leader can avoid raising funds from large gatherings of that kind.

“What happens is that the Conservatives attack the Labour party for being ever more dependent on rather unrepresentative leftwing trade union leaders, and the Labour party spends all its time attacking the Conservative party for being dependent on rather unrepresentative wealthy businessmen. In a way both criticisms are true. And the media sends both up.

“The solution is for the party leaders to get together to agree, put on their tin hats and move to a more sensible and ultimately more defensible system.”

Clarke said that the move towards a donation cap should be made soon after the general election. He added: “It puts everything above allegations of conflict of interest if the parties are not beholden to individual people, individual interests, for large sums of money.

“You would have two or three days of ludicrous headlines of the kind we have at the moment and a year later it would all quiet down and it would seem like common sense. Those who deplore Ed Miliband for taking money from Unite, or deplore David Cameron for taking money from millionaires, should support the alternative.”

On Saturday Labour’s leader Ed Miliband accused the government of turning a blind eye to the financial affairs of the rich, and claimed the revelations over the industrial scale of tax avoidance at HSBC in Switzerland crystallised a “deeply divisive injustice”.

Figures published last week showed HMRC had taken forward just one prosecution out of more than 1,000 people with accounts at the Swiss branch of HSBC – the bank run by Green between 2006 and 2010 – who were known to have not paid due tax. One individual on that list was interviewed by HMRC in 2011 but not prosecuted, despite having paid no tax for 24 years.

A Tory donor named on the HSBC files, Lord Fink, initially threatened to sue Miliband if he repeated claims made in the Commons about his financial affairs. The former Conservative party treasurer subsequently admitted he was involved in “vanilla, bland” avoidance, adding: “Everyone does tax avoidance at some level.”

Announcing that a Labour government would launch an independent investigation into the culture and practices of HMRC with regard to tax avoidance, Miliband told a Welsh Labour conference in Swansea: “The government’s failure to tackle tax avoidance is no accident. It has turned a blind eye to tax avoidance because it thinks that so long as a few at the top do well, the country succeeds. It thinks that wealth and power fence people off from responsibility. It thinks the rules only apply to everybody else.”

In a reference to this weekend’s announcement by the Conservatives of further plans to squeeze those on benefits, and some disputed claims run by the Daily Mail that the Labour leader has been involved in tax avoidance, Miliband added: “David Cameron is a prime minister who may be strong when it comes to the weak, but he is always weak when it comes to the strong. We will act because we have a different vision of how our country succeeds.

“I believe we build the success of a country through the success of working people. It doesn’t matter how much I get attacked for this, I’m not backing down.”

A Conservative spokesman attacked the speech, claiming: “The culture and practices of HMRC went wrong under Labour – when top bankers paid lower tax rates than their cleaners, foreigners didn’t pay capital gains tax and the richest people routinely avoided paying stamp duty.”