A former higher education minister has accused the University of Oxford of failing to adequately tackle racism caused by an “unconscious bias” against black and disadvantaged applicants.

The Labour MP for Tottenham, David Lammy provoked an indignant response from the senior representatives of Oxford colleges, with one interrupting him to call his comments absolute nonsense during a debate at the university.

The event, a symposium on admissions held on Friday evening, had earlier heard the university’s vice-chancellor Louise Richardson suggest it might need to overhaul its admissions procedures to improve its record, with individual colleges giving way to more centralised admissions.

Her comments, however, were overshadowed by Lammy’s questioning of whether Oxford’s admissions interviewers were trained to recognise unconscious bias.

“You’ve got academics writing on it, yet I suspect the training is not actually happening across the system. We all tend to recruit in our own image,” Lammy told the audience.

“It’s not about teachers saying, ‘don’t go to Oxford’. It’s also about the academics saying ‘that young Somali girl, whose accent … ’” he said, until Peter Claus, the access fellow at Pembroke College, interrupted: “Absolute nonsense.”

“Are you saying there’s no unconscious bias? Are you suggesting that?” Lammy said.

Claus said admissions officer spend many hours on the interviewing and admissions process. “The interviews are but one part, one tiny part,” he said.

Lammy said: “All I can say is that, as a black politician serving the most diverse constituency in the country, I find it worrying that there’s a roar of, ‘oh we can’t possibly be racist’.

“When I speak to my constituents and they come into an institution like this and they are coming in for an interview with an academic, and they are coming from a tower block called Broadwater Farm, on the 15th floor, a Somali girl - I’m sorry but the burden is on this institution to demonstrate that there is no unconscious bias, and I’m concerned that you don’t believe that.”

The acrimonious session ended with Claus telling Lammy he was “misrepresenting the admissions process, that’s all”.

Lammy, who was higher education minister in the last Labour government, has criticised Oxford and Cambridge’s admissions in the past. In 2010 he published data highlighting Oxbridge’s poor record in admitting black British undergraduates, and revealed that Merton College, Oxford, had not admitted a single black student for five years.

He told the symposium that British higher education’s efforts on diversity still fell behind those of US universities such as Harvard. “That balance, between being an elite institution and being an elitist institution, is still work in progress for Oxbridge,” he said.

The event was held at Lady Margaret Hall, and showcased the college’s innovative admissions programme, in which talented students from disadvantaged backgrounds are admitted for a foundation year of study. The college’s principal is Alan Rusbridger, the former Guardian editor.

Richardson had earlier said that progress on diversifying Oxford’s student intake had been painfully slow, with the university’s strong college structure “fostering unnecessary duplication and uncoordinated small-scale activity”.

“I think it highly unlikely we will be able effect meaningful change unless and until we are willing to operate in an altogether more coordinated way,” she said, noting that her comments risked making her unpopular with college leaders.

The symposium was also given a preview of a major report into Oxford’s admissions process headed by Moira Wallace, the provost of Oriel college.

Wallace said: “We don’t start with a blank sheet, and we don’t start in a position where there are no disadvantaged people at Oxford. There are loads.”

Currently, one in 10 UK students at Oxford come from households earning £16,000 a year or less, and one in four come from households earning less than £42,000 a year.

“Actually there are lots of people from so-called disadvantaged backgrounds and less well-off backgrounds here already. But we’d like to see there be more,” Wallace said.