Shareholders have demonstrated their mounting anger over runaway boardroom pay, delivering a huge protest against Barclays pay policies – including the £17m package for chief executive Bob Diamond.

Nearly a third of shareholders failed to back the remuneration report at a sometimes hostile annual meeting in the Royal Festival Hall, London, where one shareholder warned of the damage to the bank's reputation because of its pay deals.

Shareholders also handed a severe rebuke to Alison Carnwath, the non-executive director who sanctioned the pay deals. More than one in five investors failed to support the re-election of Carnwath, a veteran of many boardroom battles, to the board – a huge protest given that directors usually expect near-unanimous support for their positions.

The bank's chairman, Marcus Agius, had to ask the assembled shareholders to behave in a "responsible" and "adult" manner after a number of heckles from the floor while Carnwath was attempting to defend the pay policies.

The bank's pay structure has run into controversy not just for the pay of Diamond, but also the fact that more was handed out to staff in bonuses last year – £2.1bn – than in dividends to shareholders – £700m. Shareholders have also been angered that the bank paid a £5.7m tax liability incurred by Diamond when he relocated from New York to London to become chief executive on 1 January last year.

The bank also published its first ever report on "citizenship" – one of Diamond's priorities – in which it revealed it had paid £300m in UK corporation tax. The bank reported £5.9bn of profits in 2011 but stressed it made £1bn of profits in the UK. Last year the bank was forced to admit it paid just £113m in UK corporation tax in 2009 – a year when it rang up a record £11.6bn of profits.

It also defended its use of Cayman Islands companies, saying that the total profit not taxed in these entities was less than £1m in 2010 and 2011.

The result of the vote, announced after a two-hour annual meeting attended by more than 800 private investors, stunned hardened City observers. Few had expected a protest vote of more than 20%, after a 10% protest last year, because the existence of a number of large, supportive investors would reduce the protest.

Business secretary Vince Cable welcomed the news, saying shareholders were "doing what they are supposed to do, which is holding executives to account". It coincided with the end of his consultation on handing shareholders more powers over executive pay. If his consultation had already become law, Barclays would have been one of the first companies to have to comply with his proposal that those failing to get 75% support on pay would be forced to issue an explanatory statement to the market.

Even after changing the terms of the bonuses for Diamond and the finance director Chris Lucas last week, only 73% of investors supported the remuneration report – 68.5% if deliberate abstentions were included. For Carnwath, fewer than 80% voted in favour - 79.15% – and deliberate abstentions drove this down to 77%.

While a veteran of many boardrooms, Carnwath has only been chair of the Barclays remuneration committee since July and suffered a rough ride as she read from a prepared script, saying the bank "reduced awards significantly in 2011". To this one shareholder shouted "not enough" to both laughter and applause.

She pressed on: "We will continue to seek to push down remuneration levels in the context of the environment in which we operate." To this there was a cry of: "Why have you only just woken up to this?" Agius insisted that bonuses had to be paid. He apologised, but only for his handling of the issue rather than the size of the deals. He said: "It's not an option to pay zero bonus. We would be so far out of line with our competitors that the commercial consequences would be dire."

One shareholder, Patrick Evershed – a former employee of fund manager New Star – at first spoke in support of the bank, then he said: "All the good work has been undone by the pay row". He added the bank was "dividing society … doing a lot of damage to the reputation of the bank."

Agius was repeatedly questioned by one shareholder about whether anyone had lost their jobs or bonuses over mis-selling payment protection insurance, for which the bank has taken a £1.3bn provision. Diamond intervened, to say: "There is no-one employed at Barclays today that was part of the management decisions that led to PPI."

The action in the hall was overshadowed by the size of the rebellion. Big investors such as F&C, Fidelity and M&G were among institutions to vote against.

Robert Talbut, chairman of the investment committee at the Association of British Insurers, whose members control a fifth of the stock market, said the result showed investors take executive pay seriously.

"Getting it right is an important part of a successful company. All banks face a challenge to improve their investment case by getting a better balance of returns to shareholders, payments to employees and capital retention," Talbut said.

David Paterson, head of corporate governance at the National Association of Pension Funds (NAPF), said: "The vote may have been passed, but the level of dissent about executive pay at Barclays needs to be taken seriously by the company and by the rest of the banking industry"

After the meeting, Diamond told the Guardian he conceded the row over pay was a "distraction" from his goals. "We have do more on our journey to our 13% return (on equity). That's my mission: 13% returns over time," Diamond said. The bank's return on equity - a measure of shareholder performance – was 6.6% at the end of 2011, lower than the cost of equity - a measure of cost of the operation - of 11.5%.

Agius said that he would "make it my job" to ensure another row over pay did not dominate the run up to next year's annual meeting.