The revolt over government plans to cap tax relief for charitable giving has spread to the cabinet and to some of the most influential voices on the Tory benches as the business secretary, Vince Cable, and Tory MP David Davis voiced their criticism on Thursday.

An increasingly rattled Treasury was also told by voluntary sector leaders that plans to build hospitals were having to be put on hold. In addition, Unicef reported that it was unable to raise some cash from a regular donor for emergency funds due to the proposed tax changes, suggesting that uncertainty over the proposals was having a chilling effect on some rich donors.

The chancellor, George Osborne, beset by criticism since he delivered the budget, announced plans to cap tax relief at £50,000, or 25% of a person's income, to stop wealthy people using the route to reduce their tax liability. The coalition argued the move was designed to curb tax avoidance by the wealthy.

But there has been a backlash from charities and arts organisations.

Steve Bubb, chair of the Association of Chief Executives of Voluntary Organisations, said Macmillan Cancer Support had warned that limiting tax relief would undermine attempts to fund its new London cancer centre.

The Arts Council England suggested that at least £80m in regular donations to its organisations was at risk, while a new £55m "matched funding" scheme with the Department for Culture, Media and Sport could also now be in jeopardy.

Other philanthropists warned that the premise of charitable donations was being tarnished by the government's repeated claims that much charitable giving was a form of tax avoidance.

A grassroots 'Give it back George' campaign was also spreading to local charities.The Treasury and No 10 have complained that EU law is inhibiting their ability to regulate bogus charities, based in member states, that are being used as vehicles to avoid paying tax. Two years ago, following a European court ruling, it was declared that UK taxpayers could claim tax relief on any gifts to charities based in the EU, not just in Britain.

That has raised Revenue & Customs suspicions about the validity of some charities operating outside Britain to which taxpayers can donate and claim relief.

Downing Street continued to send out frantic signals that it was listening to the criticism, but, unable to agree how to back down, partly owing to the complexity of the legal issues, it continued to send out mixed signals. The law is not due to be changed until next year, and ministers are stressing they will listen to concerns.

David Davis, a former chairman of the public accounts committee, on Thursday told the Guardian that ministers needed to rethink their plans.

"If the government's aim is to prevent tax avoidance, it would be better to ensure donations are approved by the Charity Commission. Or if the charity is based abroad, it should be subject to review by [Revenue & Customs]. If the government's aim is simply to raise money from philanthropists, the government should say so."

Under his chairmanship, the Commons public accounts committee issued a report a decade ago calling for the commission to be more robust in its approval of charities.

The Treasury had been hoping the argument would die down, but instead Cable, one of the toughest advocates of a crackdown on tax avoidance, kept the issue alive with the suggestion that the cap could damage university funding.

A spokeswoman said that Cable fully supported "the need to clamp down on abusive tax avoidance".

But, the spokeswoman said: "This should be separated from genuine charitable giving. Concerns have been raised with ministers, including Vince, by universities, and he is sympathetic to those concerns. We will make sure that what we are hearing from universities is fed back to the Treasury."

Nicola Dandridge, chief executive of Universities UK, said the cap could undo much of the progress universities had made in raising funding from private donors. "Universities raise considerable sums from philanthropic gifts, £560m cash in the last year," she said.

The Tory chairman of the culture select committee, John Whittingdale, also called for a rethink, saying: "The government has set great store by trying to increase philanthropy in support of the arts and good causes, and this clearly won't help.

"To say that we are going to put an arbitrary cap as the budget has done on the amount of tax relief that can be claimed just seems to me to send a very contrary signal to all the messages that the government has been putting out up to now."

The chief secretary to the Treasury, Danny Alexander, said the government would work with charities and philanthropists "to ensure the removal of the tax relief does not have a significant impact on charities which depend on large donations".

The Lib Dem minister added: "The policy is one we have set out for extremely good reasons."