The business secretary, Vince Cable, has admitted that his failure to pay a VAT bill on time due to an oversight is "a bit embarrassing".

But the Liberal Democrat minister, who has hit out at tax avoidance in the past, said the blunder would not prevent him from continuing to speak out about tax avoidance.

Cable's comments came after it emerged that he was hit with a £500 penalty from HM Revenue and Customs (HMRC) after the oversight on the tax on media work he had done before becoming a minister.

"I have not avoided paying tax," he said. "I paid in full all that was owed. The tax accountant has made a statement making this clear. I approached HMRC unprompted as soon as I realised I was liable for VAT on my earnings for 2009-10.

"The tax was paid in full and the matter closed within four weeks. I made no attempt to avoid tax – in fact I made every effort to pay what was outstanding as soon as it became clear I was liable for VAT.

"It's a bit embarrassing that this wasn't spotted earlier. None of this will stop me talking out against tax avoidance."

Downing Street said Cable had the prime minister's "full confidence".

"I understand Vince Cable has made a statement and the case is closed," a spokeswoman said. "The business secretary has rectified an oversight and the case is closed."

Cable, a former Lib Dem deputy leader, is said to have made a considerable sum from media work and book deals in the two years before he became a minister.

He earned an estimated £192,000 on top of his MP's annual salary – currently £65,738 – but failed to register it in time.

HMRC rules stipulate that if a person's turnover of VAT-liable goods and services over a 12-month period exceeds the current £73,000 threshold, they must register for the duty within 30 days.

Cable's aides said the error was spotted by his personal accountants in January while his tax return for the previous financial year was being finalised. The accountants notified HMRC officials and apologised before offering to pay any due tax.

He was then billed for a figure of less than £15,000 after a rebate on his charity donations was taken into account, according to his spokeswoman.

He is said to have sent a cheque to the Revenue by mid-February to make up the shortfall, and was let off with half the usual £1,000 penalty.

"HMRC waived 50% of the fixed penalty for late notification in recognition of the fact that I did approach them unprompted and my oversight was in good faith," Cable said.

His accountants, Myrus Smith, said: "There are no tax payments outstanding and none are being contested.

"When it became clear that Mr Cable's earnings had breached the level at which VAT was payable, an offer to settle immediately and in full was made to HMRC and this was duly accepted. The matter was dealt with within a month of the Revenue being informed in the first instance, and is now closed."

Cable has cut back his freelance work since becoming a minister in May 2010.