A two-bedroom flat on Notting Hill’s All Saints Road can fetch up to £2m these days. But half a century ago, when property prices were a small fraction of what they are now, the road was on the frontline of battles between the police and the black community.

Notting Hill and Brixton were the two main areas where Windrush migrants settled. The majority who arrived in Notting Hill came from Trinidad, and at the very heart of this community was the Mangrove restaurant, at 8 All Saints Road.

The Mangrove was established in 1968 by Frank Crichlow, an entrepreneur from Trinidad who became a community activist and symbol of black urban resistance in the face of police persecution.

He died in 2010, aged 78, but his legacy endures: this Sunday community activists, generations of Windrush residents, lawyers and musicians will gather to mark the 50th anniversary of the restaurant’s opening.

Crichlow previously ran the Rio Cafe in nearby Westbourne Park Road, a venue frequented by John Profumo and Christine Keeler. His new venture attracted artists, musicians and activists from around the world. Bob Marley, Marvin Gaye, Jimi Hendrix, Nina Simone, Diana Ross and the Supremes, and Vanessa Redgrave all flocked to the Mangrove to enjoy traditional Caribbean food, share ideas and discuss politics.

Although Crichlow was known for his strong anti-drugs stance, police continually raided the Mangrove looking for illegal substances – in one year alone the Mangrove was raided six times, although police found no drugs on the premises.

Matters came to a head when Crichlow and seven others, including the late writer and broadcaster Darcus Howe, were arrested and charged with a variety of offences including affray while protesting against police harassment. The trial exposed racism in the Metropolitan police 30 years before the Macpherson inquiry. All nine were acquitted of the key charges against them.

At the time Crichlow said: “It was a turning point for black people. It put on trial the attitudes of the police, the Home Office, of everyone towards the black community. We took a stand and I am proud of what we achieved – we forced them to sit down and rethink harassment. It was decided there must be more law centres and more places to help people with their problems.”

Crichlow set up the Mangrove Community Association to provide practical help for the community, from accommodation for older people to support for women coming out of prison.

31 October 1970: A demonstration takes place in Notting Hill organised by the Black Defence Committee. Photograph: Ian Showell/Getty Images

Police harassment of Crichlow continued, however. In 1979 he was charged with drugs offences but was again cleared. In 1988 Mangrove was raided again by police and Crichlow’s bail conditions prevented him from going near the restaurant for a year.

It never recovered from his prolonged absence, and ultimately closed. In 1992 he successfully sued the Met for false imprisonment, battery and malicious prosecution and received record damages of £50,000.

As well as providing an opportunity for celebration, the anniversary has resurfaced some darker memories. Clive Mashup Phillip, who will be one of those attending Sunday’s events, was a regular at Mangrove in its heyday and endured many struggles with the police himself.

“When I came to this country from Trinidad in 1961 I knew nothing about racism and I didn’t expect to find this kind of situation,” said Phillip, now 77. “Back home we showed people so much respect, especially white people. We came here to the mother country and found ourselves not wanted, abused and laughed at.”

He said that until 20 years ago he was regularly harassed by police. “We stood up and fought racism as a group. I felt safe being around Mangrove, it was like family. I was often charged with obstructing police officers. When I saw someone being abused by police in the street I would help to defend him and they did the same for me when I was in that situation.”

Phillip said he met many celebrities at Mangrove, including Bob Marley and Marvin Gaye. “I used to talk a lot with Marvin Gaye,” he said.

Jeb Johnson, 66, who describes himself as a Windrush baby, remembers Marley frequenting the restaurant. “He was a vegetarian and he used to eat rice with butter,” he said.

Johnson said a community hub like Mangrove was needed more than ever today. “We need something like this for our young people to give them an alternative to the life some of them have at present.”

Yvette Williams, a member of Justice4Grenfell, first got involved with Mangrove in the late 1980s, and helped to sort out people’s housing and immigration problems. She says some things have not changed, pointing to the circumstances of the Grenfell Tower fire as evidence. “These issues do not go away,” she said.

The community activist Lee Jasper, another Mangrove veteran, said: “The history of Mangrove embodied the spirit of British black civil rights resistance against racism and injustice. Our 50th-anniversary reunion on All Saints Road is a celebration of one of Britain’s foremost radical black political organisations. One day we will buy the entire street and turn All Saints Road into a world heritage site celebrating its unique black British history.”