Plans to give HM Revenue and Customs the power to dip into bank accounts to recover unpaid tax will leave people open to fraud and error, a Commons watchdog has warned.

The Treasury select committee, led by Tory MP Andrew Tyrie, said the current proposals are "very concerning" because people will be at risk of having money wrongly taken out by HMRC.

Around 17,000 a year people could be affected by the new tax collection powers, which are expected to raise around £100m a year.

The Treasury insists there are sufficient safeguards as HMRC will only be able to remove the money after four ignored requests for the tax, the money due is more than £1,000 and only if there is £5,000 in the account afterwards.

But the cross-party group of MPs raised concerns about HMRC's chequered record when it comes to errors.

"This policy is highly dependent on HMRC's ability accurately to determine which taxpayers owe money and what amounts they owe, an ability not always demonstrated in the past," the committee said. "Incorrectly collecting money will result in serious detriment to taxpayers. The government must consider safeguards, in addition to those set out in the consultation document, to ensure that HMRC cannot act erroneously with impunity. These might include the award of damages in addition to compensation, and disciplinary action in cases of abuse of the power."

It added: "The ability directly to have access to millions of taxpayers' bank accounts raises concerns about the risk of fraud and error, and this should also be covered by the consultation."

George Osborne, the chancellor, believes the measure can be justified because the Department for Work and Pensions already has the right to take money directly from people's bank accounts to pay child maintenance.

However, the committee pointed out that the parallel is not exact as the DWP is acting as an intermediary between two individuals, while HMRC would be acting in pursuit of its own objective of bringing in revenue for the exchequer. Tyrie said: "The proposal to grant the power to HMRC to take money directly from people's bank accounts is very concerning.

"People should pay the right amount of tax. But HMRC does not always ask for the right amount.

"Some taxpayers may find money taken from their accounts that later should be paid back. That would be unacceptable.

"Exceptional powers such as this require prior independent oversight. The government must demonstrate that it has dealt with the committee's concerns before proceeding."

After the budget, the Association of Chartered Certified Accountants branded the plans "seriously draconian" but have now concluded they are "less fearsome than first thought".

"On paper, the safeguards look relatively robust, and the reality is it is unlikely that anyone will be left penniless. The fact that there will be plenty of opportunity for those owing taxes to respond to HMRC and appeal before any funds are taken," said Chas Roy-Chowdhury, head of taxation at the ACCA, said earlier this month.

"There remain some concerns over how efficient HMRC can be in maintaining those safeguards, as well as whether this initiative is in fact a toothless one that will make no difference at all to collecting taxes owed. However, it is important now to have constructive engagement with HMRC to ensure the proposals are reasonable and proportionate."

The Treasury committee also repeated its concerns about the chancellor's Help to Buy scheme, warning the it is hard to spot bubbles in advance.

"The committee has expressed concerns about Help to Buy from the time of its announcement at last year's budget. The committee highlighted the risk that the primary effect of the scheme, at least in the short-to-medium term, could be to raise house prices.

"The committee has also drawn attention to the risk that withdrawal of Help to Buy may have a distorting effect on the housing market," the report said.

Three former chancellors – Lord Lawson, Lord Lamont and Alistair Darling – have now warned that the scheme needs revision and might be fuelling a house price boom.

The Organisation for Economic Cooperation and Development also this week called for a rethink, with Labour claiming Osborne is stoking a housing boom to gather votes before the election.

A spokesman for the Treasury said: "The government's long-term economic plan is to reduce the deficit so that we deal with our debts. It is therefore important that people pay the tax they owe on time. Although the vast majority do this, there is still a minority that chooses not to pay, despite being able.

"The proposed powers will give HMRC another tool to collect tax debt owed. The current consultation includes a range of safeguards to ensure the power is tightly targeted."