Universities should recruit more white, working-class boys in the same way they target students from other disadvantaged groups, the universities minister has suggested.

David Willetts says white, working-class teenage boys should be targeted alongside ethnic minorities and those from disadvantaged communities when universities look to attract students.

His suggestion comes against the backdrop of a drastic fall in the number of university applications from men.

Speaking to the Independent, Willetts said the Office for Fair Access (Offa), the university access watchdog, "can look at a range of disadvantaged groups – social class and ethnicity, for instance – when it comes to access agreements, so I don't see why they couldn't look at white, working-class boys".

Willetts said he would suggest the inclusion of white, working-class boys as a target group for recruitment in university access agreements, which universities have to sign to gain permission to charge higher fees, in a meeting with Professor Les Ebdon, the director of Offa.

Figures from last autumn's intake show a 54,000 fall in men applying to university, which is 13% down on 2011 and four times higher than the reduction in women applicants, the Independent said.

Just 30% of male school-leavers applied to university in 2012, compared with 40% of their female counterparts, according to Ucas.

Willetts told the Independent this was "the culmination of a decades-old trend in our education system, which seems to make it harder for boys and men to face down the obstacles in the way of learning … That is a challenge for all policymakers and parties."

He added: "I do worry about what looks like increasing underperformance by young men."

Dr Wendy Platt, director general of the Russell Group, which represents 24 top universities, said: "Universities cannot solve this problem alone.

"The root causes of the under-representation of students from disadvantaged backgrounds are underachievement at school and poor advice on the best choices of A-level subjects and university degree course."

Willetts said he was keen to protect universities' rights to select students but said he wanted to make sure white working-class boys had every opportunity for a fair shot at a place.

He told the BBC Radio 4 Today programme: "When the fees went up, one of the things we did with the extra money – which students don't pay themselves upfront – was say to universities, you have to put about a third of that money into reaching out and improving access. That's hundreds of millions of pounds now available. We want to see that used as effectively as possible.

"There have been summer schools for a long time but I want to see real summer schools which aren't just a kind of visit for an afternoon, which involve targeting schools that aren't sending pupils to university and saying, send your kids to this university for two or three summers, they can work in the labs, they can have some mentoring from current students, then they can raise their A-level performance, then they can get to university.

"I believe universities should look at merit and potential. They should look at who can achieve and benefit from coming to university. "There are groups that are underperforming; there is a shocking waste of talent of some young people who could really benefit from going to university that aren't going there," he said.