The comedian Jimmy Carr has issued an apology for his "terrible error of judgment" in using a tax avoidance scheme.

Carr confirmed that he had changed his tax affairs after the scheme was revealed this week. On Wednesday the prime minister had labelled the situation "morally wrong".

In a series of Tweets the comedian said: "I appreciate as a comedian, people will expect me to 'make light' of this situation, but I'm not going to in this statement as this is obviously a serious matter.

"I met with a financial advisor and he said to me: "Do you want to pay less tax? It's totally legal." I said: "Yes."

"I now realise I've made a terrible error of judgment.

"Although I've been advised the K2 tax scheme is entirely legal, and has been fully disclosed to HMRC [Her Majesty's Revenue and Customs], I'm no longer involved in it and will in future conduct my financial affairs much more responsibly. Apologies to everyone. Jimmy Carr."

Jimmy Carr tweets Photograph: Twitter

The apology follows a Times investigation which revealed that Carr, who has lampooned bankers for failing to pay their tax, protects £3.3m a year by channelling cash through the Jersey-based company K2, which then returns the money in the form of a loan. This is not subject to income tax.

The Treasury minister David Gauke welcomed Carr's decision to pull out of the scheme and urged other people involved in it to do the same. "I welcome what Jimmy Carr has said. I think it's the right step for him to take," he said. "All the focus has been on him. But of course there are many people who have invested in these schemes that are very artificial and contrived. They are clearly not in accordance with the intention of parliament and they are not fair on the general taxpayer who has to pay more as a consequence."

He added: "I hope others who have invested in such schemes, who maybe thought: 'Well, it's a great opportunity to reduce my tax bill,' also think carefully about it, think whether it is fair, whether it is right that they do so."

On Wednesday David Cameron criticised the situation, describing it as "morally wrong". The Times reports of Carr's financial arrangements suggested "straightforward tax avoidance", said the prime minister, and it was unfair on the people who paid to watch him perform that he was not paying his taxes in the same way that they did.

Cameron said he had not had time to look at other allegations relating to the singer Gary Barlow, who was given an OBE in the Queen's birthday honours list.

Speaking to ITV in a round of TV interviews during his trip to Mexico, the prime minister said: "I think some of these schemes – and I think particularly of the Jimmy Carr scheme – I have had time to read about and I just think this is completely wrong.

"People work hard, they pay their taxes, they save up to go to one of his shows. They buy the tickets. He is taking the money from those tickets and he, as far as I can see, is putting all of that into some very dodgy tax avoiding schemes.

"That is wrong. There is nothing wrong with people planning their tax affairs to invest in their pension and plan for their retirement – that sort of tax management is fine. But some of these schemes we have seen are quite frankly morally wrong.

"The government is acting by looking at a general anti-avoidance law but we do need to make progress on this. It is not fair on hardworking people who do the right thing and pay their taxes to see these sorts of scams taking place."

The Channel 4 standup is reported to have used a legal tax-avoidance scheme that enables members to pay income tax rates as low as 1%.

He spoke out amid claims that members of Take That – Gary Barlow, Howard Donald, Mark Owen and the band's manager, Jonathan Wild – invested at least £26m in another scheme, run by Icebreaker Management Services.

Earlier, a Downing Street spokeswoman had said Cameron backed George Osborne's description of aggressive tax avoidance as "morally repugnant".

Asked if Cameron was "disappointed" about the reports of the tax affairs of Barlow, who campaigned alongside the Tory leader at the general election, she said: "I just go back to the general point about tax, that everybody should pay the right level of tax.

"Separately, the jubilee celebrations were a great success and Gary Barlow played a big part in that."

Asked if the government might consider whether the tax affairs of individuals should be a factor when considering people for honours, she said: "I am not going to comment on individuals in terms of what's happening in a newspaper report."

Later, another Downing street source appeared to harden the government's position on celebrities' tax affairs, and said: "All tax avoidance schemes need to be addressed by HMRC whichever big-name stars are involved."

Separately, No 10 sources said a general tax avoidance rule, due to come in next year, was the best way of handling tax loopholes, rather than constantly chasing down every individual tax scheme.

Carr was confronted over his financial arrangements during a show in Tunbridge Wells, in Kent. Challenged by a member of the audience, who told him: "You don't pay tax," he replied: "I pay what I have to and not a penny more," the Times reported.

It came after he appeared to make light of the furore surrounding his tax payments. He asked: "Got through the papers? I haven't really been through the papers today. The Murdochs are after me …"

Carr is said to be one of more than 1,000 beneficiaries who shelter £168m from the taxman each year using K2.

Barlow, Donald, Owen and Wild are among almost 1,000 people who contributed £480m to 62 partnerships in music industry investment schemes, the Times reported.

An HMRC spokesman said it had successfully challenged an avoidance scheme run by Icebreaker LLP, winning on the main arguments in the tribunals. The spokesman said: "We are now preparing to litigate Icebreaker 2 but for legal reasons cannot say more at this time. We examine the implementation of avoidance schemes in detail and will not let any aspect of these cases go unchallenged."

HMRC said the K2 scheme was already under investigation.

The spokesman added: "We have taken firm action to protect the exchequer from unacceptable tax loss.

"We do not accept that the Icebreaker tax avoidance schemes have the tax effects their promoters claim – we have already successfully challenged one Icebreaker scheme in the tribunal."

HMRC could not confirm whether or not members of Take That and their manager had been involved in an Icebreaker 2 scheme.