Ethnic minority Britons were subjected to nearly one-and-a-half million more stop and searches in the 10 years after the Macpherson inquiry than if the police had treated them the same as white people, the Guardian can reveal.

New research, carried out for the Guardian by criminologist Dr Ben Bradford of Oxford University to mark the 20th anniversary of the murder of Stephen Lawrence, found that the rate of stop and searches of black people doubled in a decade after 1999. This was despite the publication that year of Sir William Macpherson's landmark report on the Lawrence case, which found the biggest force, the Metropolitan police in London, to be "institutionally racist".

The research comes as Lawrence's family mark two decades since his death at the hands of racists. David Cameron has said that his death sparked a monumental change in society.

Cameron said: "The senseless killing of Stephen Lawrence in 1993 was a tragedy. It was also a moment that sparked monumental change in our society – change that has been brought about by the tireless efforts of Stephen's family in challenging the police, government and society to examine themselves and ask difficult questions.

"I believe that many of those questions have been answered: from improved community relations to more accountability in policing. Much has been achieved, but we know that more still needs to be done. We owe this to the memory of Stephen."

The Oxford research found that in the first year after Macpherson's report in 1999-2000, the stop rate for black people was 4.9 per 100 population. By 2009-10, the rate was 10.8 per 100 black people in Britain.

The rate also nearly doubled for Asian people, while for white people it only marginally increased from 1.5 to 1.6 stops per 100 citizens.

The research, based on analysis of official figures, reveals that the differing rates resulted in black and Asian people experiencing 1.478m "excess" searches in the decade after Macpherson. If white people had been stopped at the same rate as black people, police would have carried out 40mmore stops in the 10-year period.

A former Scotland Yard chief said police were "racially profiling" while a leading MP reacted to the research claiming it showed "policing by fear".

In an interview in the Observer, Lawrence's mother, Doreen, said police continued to harass young black men through stop and search despite them doing nothing wrong: "I think that is how a lot of young black men feel still today. But that doesn't stop them being harassed in stop and search, or if they drive a nice car it is always assumed they are drug dealers. That is as true now as it was then."

One of those who has been stopped repeatedly is Stuart Lawrence, the younger brother of Stephen.

Stuart, a teacher, has been stopped more than 20 times. He has no criminal record, is a teacher, and comes from a family who have been praised by figures including the prime minister.

In an interview with the Guardian last year, Stuart said: "There is no reason I can give, other than I am a young black man, who usually wears a baseball cap in my car, which is my god-given right."

The higher rate of stop and search for ethnic minorities is called disproportionality, and Bradford said: "When the pressure was on the police after the Stephen Lawrence inquiry, disproportionality was low; but as the media and political focus moved on, disproportionality grew again. "

Alleged racial discrimination in stop and search has been a long standing complaint. In the years since Macpherson, the police have declared themselves no longer to be "institutionally racist".

Ben Bowling, professor of criminology at King's College London said the research showed a "racial penalty" being exacted by the police on ethnic minorities in Britain: "It remains an example of what the Lawrence inquiry referred to as institutional racism. In my view, the police use of stop and search is simply not good enough."

The Met carries out 43% of the total stops in England and Wales. This skews the national figures, as more ethnic minorities live in the capital compared to the rest of the country.

The former Met police chief Brian Paddick said his bosses buried his warnings that officers were discriminating in stop and searches and community confidence was being damaged.

Paddick said officers, consciously or subconsciously, were racially stereotyping black people as criminal suspects: "This is racial profiling. The political will is not there and police culture needs to change.

"Officers are stopping innocent black people while criminals walk by smiling."

David Lammy, the Labour MP for Tottenham, the flashpoint for the 2011 riots, said the bedrock principle of British policing, that officers police by consent, was under strain because of stop and search: "It introduces policing by fear through the backdoor. The disproportionality breaks trust between the police and vast majority of law abiding ethnic minority people."

Chief Constable Alf Hitchcock, lead on equality, diversity and human rights for the Association of Chief Police Officers, denied police had been racially profiling and said: "The debate around the use of stop and search is complex. Without taking into account the nature of the search, where it takes place and the legislation under which it has been made, looking at disproportionality alone is too simplistic to tell us anything of value.

"Chief officers have been working with the Equality and Human Rights Commission to improve use of stop and search. A key aim is to identify where it may have a disproportionate impact and where action is required to address this. Her Majesty's Inspectorate of Constabulary has undertaken a comprehensive review of how all police forces operate stop and searches and we will learn from their findings."

Monday will see Doreen Lawrence attend a central London memorial service for her lost son. Her former husband, Neville, now lives in Jamaica where Stephen is buried. He said he would take fresh flowers to the grave, which he recently repainted as the 20th anniversary loomed. He said: "In England I could not feel free and for my own peace of mind I had to leave. I did not feel safe."