An increase in the number of women in Britain's boardrooms will help drive down pay, David Cameron said on Friday as he expressed concerns about a 49% increase in earnings for top directors.

Amid warnings that senior non-executive directors are pushing up pay through a "closed shop", the prime minister said he was concerned about the dramatic increase.

"This is a concerning report, particularly at a time when household budgets are very tight and people have difficult circumstances – the increase in food prices and fuel prices, six million public sector workers on a pay freeze," Cameron said at the Commonwealth heads of government meeting in Perth.

The prime minister spoke after a report showed that total earnings for directors of FTSE 100 companies increased by 49% last year. The research by Incomes Data Services found that a FTSE 100 executive typically received an average of £2.7m in 2010.

Deborah Hargreaves, the chair of the High Pay Commission, told BBC Radio 4's Today programme that a closed shop of executives were driving up pay.

"I think it is very hard to justify these sorts of pay increases," Hargreaves said.

"When you think the average pay is going up 1% or 2%, it's not even meeting price rises. These pay packages have become so complex that executives don't even understand it themselves. We have got a closed shop here and someone needs to break it open."

The prime minister indicated some sympathy with this view, and went further by saying that increasing the number of women in boardrooms would help drive down pay levels.

"At the moment I think there is too much of a closed circle of people made available to be non-executives. By opening this up, by increasing the number of women in our boards – I think that would have a beneficial effect."

The prime minister said that executive pay would be brought down in three ways:

• Transparency: all figures should be published so that shareholders can compare and contrast salaries

• Accountability: the hand of shareholders should be strengthened so they know they are taking responsibility for remuneration

• Responsibility: "Boards have got to think whether when they are making pay awards: is this the right and responsible thing to do. Of course you have got to attract the best talent to run the business that you are accountable for as a non-executive director," Cameron said.

"[But] everyone, whether they are in public life or in private enterprise – they have got to be able to justify the decisions they make about pay."

The prime minister responded to a report in the FT, which said that the bonuses to civil servants had increased by £4m. He said the decision of ministers to take a 5% pay cut and then a freeze for the rest of this parliament and restrictions on public sectors earning more than the prime minister was acting as a "downward thrust" on pay.