David Cameron has defended his decision to appoint a fellow Etonian, Jo Johnson, as the new head of the policy unit in Downing Street, taking to six the number of ex-pupils from Britain's most exclusive public school in his inner circle.

Amid criticisms from within the Tory party over the appointment of so many Etonians to posts at the heart of his government, the prime minister said that the younger brother of the London mayor had an "immense brain" that would strengthen the policy unit.

In his most detailed comments on the appointment, Cameron told the World at One on Radio 4 on Wednesday: "I appoint people because they are good enough to do the job and they are the right person for that job. I have people around me who have all sorts of different backgrounds and all sorts of different schooling. The question is: are you going to be good enough to do the job?

"If you take Jo Johnson – someone who's got an immense brain, an immense talent, a very successful journalist at the Financial Times, wrote some excellent books before going into politics – I think he'll really bring a level of brainpower and analysis to the policy unit, which I think will be hugely welcome."

The prime minister spoke out after the Conservative MP for Totnes, Sarah Wollaston, expressed astonishment when Jesse Norman, another Etonian member of Cameron's new policy board, told the Times: "Other schools don't have the same commitment to public service."

Wollaston tweeted a link to a report of Norman's remarks, saying: "Words fail me." She later tweeted: "I'm not asked for policy advice, but just in case ... there are other schools & some of them even admit women."

The prime minister said it was right to have a range of people and he said he wanted to achieve a better balance of men and women. "The Conservative party is now much more open and much more diverse ... This is good progress. We still have a long way to go."

Cameron faced criticism when he followed the Johnson appointment by appointing the former brother-in-law of Earl Spencer to Downing Street. Christopher Lockwood, a senior journalist on the Economist whose sister used to be married to the brother of the late Diana, Princess of Wales, was named by the prime minister last year as one of the journalists he counted as a friend. Lockwood attended St Paul's School in London.

The prime minister answered the criticisms by saying that the second most important person in the government was William Hague, who attended a comprehensive school in Rotherham. Patrick McLoughlin, the transport secretary, was formerly a miner, he added.

"I don't really accept the caricature of this," the prime minister said of the criticisms. The other four Etonians in Cameron's inner circle after Johnson and Norman are Ed Llewellyn, his chief of staff; Oliver Letwin, the Cabinet Office minister; Rupert Harrison, the chancellor's chief economics adviser; and Sir George Young, the chief whip.

In his interview the prime minister served notice that he is bracing for bad results in Thursday's local elections. "It is mid-term. A government has had to make difficult decisions. We are responsible for making a series of difficult cuts and difficult choices. People understand that. But often it is not welcome and often it can lead to people feeling frustrated."