The boss of Lloyds Banking Group pledged to pull out of tax havens where the bank is not conducting genuine business at its annual meeting on Thursday where investors hit out against its "cosy' boardroom and "appallingly high" bonuses.

Chief executive António Horta-Osório said the 39%-taxpayer owned bank had embarked on a systematic review of "so-called tax havens" after a shareholder demanded to know why the bank was the seventh biggest user of such facilities.

Shareholder Anne Edmonds said: "I want to know when this will be stopped. Tax avoidance is legal and what Lloyds is doing is legal. But to me there is little difference between tax avoidance, which is legal, and tax evasion, which is illegal."

This was "very wrong", she said. "That money should be kept in the UK for the benefit of the UK."Horta-Osório agreed with her comments. "In 2012 alone we have closed 60 of those companies and that is more than 20% of the total. We are going to close all of them unless there are strong business reasons for our customers to keep them there," he said at the meeting in Edinburgh. He later clarified that "business reasons" did not mean "tax reasons".

Campaigners say Lloyds has 259 entities in tax havens, although the bank said this was out of date.

Chairman Sir Win Bischoff said Lloyds was the third biggest taxpayer in the UK. He added: "We do have for operational reasons some subsidiaries in what might be called tax havens."

Bischoff, who announced his retirement this week, faced a barrage of hostile questions over Lloyds' failure to pay a dividend since it rescued HBOS in 2008.

After Horta-Osório promised the bank would return to profit this year and "allow taxpayers' investment to be repaid", the shares closed near to 61p – the level at which the government agreed he will get his bonus if a third of the stake is sold.

This was cold comfort for many shareholders, who complained that shares still languished at around one-fifth of their pre-crisis high. "We suffered huge losses to save the taxpayer," said private investor Alexander Hopkinson-Woolley, who lambasted "the cosy coterie" on the board that failed to stop the HBOS takeover.

Another shareholder, David Harrison, slammed the bank for paying bonuses – totalling £1.2bn between 2010 and 2012 – while withholding dividends. "Many [executives] earn more in one year than some of us have earned in our lifetime."

The power of the institutional shareholders' vote – and that of the taxpayer – meant that Lloyds' pay deals were approved by 96%, although 6% withheld with their vote. Jim O'Neil of UK Financial Investments Limited was in the audience representing the taxpayer's 39% stake in Lloyds.

Bischoff said the chances of selling off the taxpayer stake had improved and confirmed that from September the TSB brand will return to the UK high street, as part of the bank's requirement to reduce its number of branches.

The bank had no timeframe on when it would resume paying dividends again, he told the Guardian after the meeting.