British universities are “not leftwing madrassas”, the higher education minister Sam Gyimah said in riposte to Toby Young, during a speech to vice-chancellors that was markedly gentler than the government’s recent rhetoric.

Gyimah, addressing the annual meeting of the Universities UK group in Sheffield, rebutted Young’s comments last month that students were being put off studying because universities were now “leftwing madrassas”.

The minister assured the vice-chancellors that they had his support, declaring: “I love our universities.”

But Gyimah warned that universities had a duty to retain public trust, and needed to avoid being seen as “ideological echo chambers”, producing irrelevant research or being distant from their communities.

“I know that many of you work hard to prevent this kind of turning inwards. Our best universities are not ivory towers. Still less are they ‘leftwing madrassas’, as one controversialist chose to describe them,” Gyimah said. “But ideological diversity, strong research cultures, engagement with the wider world and fair access are ongoing battles – and the price of failure will be very high.”

Young tweeted in August: “Wondering why applications to universities are down? Because universities have stopped being about the transmission and expansion of knowledge and become left-wing madrassas [Islamic schools].”

Shortly afterwards he turned his tweet into an article for the Mail on Sunday and claimed: “Our colleges have become seminaries of politically correct nonsense – leftwing madrassas whose purpose is not to disseminate knowledge and promote understanding but to suppress politically incorrect facts and stifle debate,” citing as evidence the small number of students who voted for Ukip.

His hostility may result from the fiasco over Young’s appointment to the executive board of the Office for Students, the new higher education regulator, which immediately attracted controversy for his lack of qualifications, exaggerated CV and long track record of offensive remarks.

Young eventually stepped down from the role, following criticism by the prime minister, Theresa May, and amid further controversy over his espousal of what he called “progressive eugenics”.

During his speech, Gyimah also politely rebuffed his fellow cabinet minister Michael Gove – who famously remarked that Britain had heard enough from experts.

Noting the upheaval involved in “unstitching a 40-year relationship with the EU”, Gyimah said: “It may not be fashionable to say it, but at times like this, we need experts more than ever. This is not the time for our universities to shrink back and sulk.”

The minister also signalled that he supported a more generous regime in offering visas for international students to study and work in the UK, which remains a source of controversy with the government and Conservative party.

“We also need to think seriously about international students and researchers. Out there, somewhere in the world, there are people, young and not so young, with the ideas and the potential to send shockwaves through the status quo,” he said.

“I want the UK to be where current and future generations come to see their groundbreaking ideas come to life and truly make a difference to the world.

“The forthcoming report of the Migration Advisory Committee on student migration offers us an opportunity to ensure our policy on student migration recognises the contribution that overseas students make to our universities, our balance of trade and our communities.”

Despite previously accusing some universities of being more interested in “bums on seats” than a valuable education, Gyimah said: “Given the state of debate on higher education in parts of the UK media, what I am about to say next may be controversial. But I will say it anyway: going to university is worth it.”

Nick Hillman, the director of the Higher Education Policy Institute who attended the speech in Sheffield, said it was warmly received. “People really liked the positive approach, which was a welcome contrast to some other politicians’ rhetoric about universities,” he said.

“The big outstanding question remains, however, whether current reviews will deflect him from his priorities. Not only is there Brexit, which he did mention; there is also the Office for National Statistics’ review of student loans, which could knock the sector for six almost by accident.”