Image copyright Allied Works Image caption Shortlisted: Allied Works design

It's a rare piece of green space with the backdrop of Parliament and a commanding view of the Thames.

Victoria Tower Gardens, London's smallest royal park, is a popular haunt for dog walkers, joggers, families - and also picnicking office workers, who use it to soak up the sun and get a breather from the hustle and bustle of city life.

But that could be about to change because this narrow strip of parkland is set to become home to a national Holocaust memorial with an underground learning centre.

By Holocaust Memorial Day 2021, organisers anticipate the £50m scheme will transform the park, which dates back to the 1870s and is fringed by trees and benches, into a tourist destination and education resource attracting more than a million visitors a year.

Image copyright Caruso St John, Marcus Taylor and Rachel Whiteread Image caption Shortlisted: Caruso St John, Marcus Taylor and Rachel Whiteread's sculpture

Image copyright Lahdelma & Mahlamäki Architects and David Morley A Image caption Shortlisted: Lahdelma & Mahlamäki Architects and David Morley A

A shortlist of 10 architects are currently competing in an international design competition launched by the UK Holocaust Memorial Foundation to create the new structure, with the winning team to be announced by the end of May.

Former Prime Minister David Cameron said the new monument would "show the importance Britain places on preserving the memory of the Holocaust", claiming it would represent "a permanent statement of our values as a nation" and something that would be visited "for generations to come".

But not everyone is happy about the plan. Some local residents, MPs and peers say that while they are fully behind the creation of a Holocaust memorial - and in particular a learning centre - they believe the project will destroy the park.

Image copyright Studio Libeskind and Haptic Architects Image caption Shortlisted: Studio Libeskind and Haptic Architects

Image copyright heneghan peng Image caption Shortlisted: heneghan peng

Barbara Weiss, the architect who refurbished the Russell Square headquarters of the Wiener Library - the oldest institution for the study of the Holocaust and genocide - questioned why it could not be placed with it. "I'm not against the memorial, I just don't want any building in our park, not even a hospital or an art gallery."

Ms Weiss is a leading light in the Save Victoria Tower Gardens campaign. She says the park is "absolutely unique, historic and gorgeous - to put something else there will totally change its character completely".

"It doesn't make a lot of sense to build a learning centre underground in an area beside a river in a flood area.

"The organisers are talking about one million extra visitors there - that's a lot of extra security. We would have people with machine guns and bag checks, and I know people who work in Parliament don't want that. They go to the park to get away from that pressure of feeling constantly monitored."

Image copyright Anish Kapoor and Zaha Hadid Architects Image caption Shortlisted: Anish Kapoor and Zaha Hadid Architects

Image copyright Adjaye Associates and Ron Arad Architects Image caption Shortlisted: Adjaye Associates and Ron Arad Architects

Lucy Peck, a retired architectural historian who lives nearby, said: "I'm not against a memorial at all, but there are bigger places in London that could take a project of this size much more easily. There's a superb Holocaust gallery less than a mile away at the Imperial War Museum, so why build another fairly similar thing here?"

The Imperial War Museum, a 15 minute walk from Parliament, was one of three locations out of 50 in the running as a Holocaust memorial site - until January 2016 when Mr Cameron named Victoria Tower Gardens as the preferred option.

Lucy Donoughue, the IWM's assistant communications director, said the museum - which is spending £15m on renewing and expanding its renowned Holocaust Exhibition and already attracts a million visitors a year - was not deemed central enough.

"While we were disappointed by this decision, we still remain hugely supportive of the initiatives laid out by the UK Holocaust Memorial Foundation," she said.

Veteran Conservative MP Sir Peter Bottomley, who has lived near Victoria Tower Gardens for more than 25 years, says he is unhappy two other sites - Potters Field on the south bank of the Thames between Tower Bridge and City Hall, and Millbank, next to Tate Britain - were ruled out.

Image copyright Foster + Partners and Michal Rovner Image caption Shortlisted: Foster + Partners and Michal Rovner

Image copyright John McAslan + Partners and MASS Design Group Image caption Shortlisted: John McAslan + Partners and MASS Design Group

Image copyright Diamond Schmitt Architects Image caption Shortlisted: Diamond Schmitt Architects

"Somewhere, somehow some unnamed person in Number 10 decided to substitute these three with Victoria Tower Gardens," he said. "You can't have a prominent memorial here - you've got to keep the garden. I would urge the government to pause, reopen the debate and rethink."

The park, which is listed Grade II and is partly inside a Unesco world heritage site, is no stranger to significant structures including August Rodin's bronze The Burghers of Calais, a statue of the Suffragette Emmeline Pankhurst, and a fountain commemorating the abolition of slavery.

A spokeswoman for the UK Holocaust Memorial Foundation, which is chaired by Sir Peter Bazalgette, said its mission had been to find "the most iconic location" for a national memorial and learning centre - and Victoria Tower Gardens, next to Parliament "fulfils that aim better than any of the almost 50 sites we examined".

She declined to say specifically why other sites had not been chosen except that "a lot of those were for commercial reasons".

"With cross-party support, we have made a promise to survivors that in Victoria Tower Gardens we will create a fitting national memorial as a permanent site of remembrance and an education centre to act more broadly as a voice against hatred and prejudice in the modern world, while respecting and enhancing the existing green space," she said.

"There can be nowhere more meaningful for such a powerful statement of our national values than next to Parliament, at the heart of our democracy. We want Britain's Holocaust survivors to know that we will not break our promise."

But Jewish Conservative peer Lord Wasserman, one of David Cameron's closest political allies who lives quite near the gardens, says it is not the right location for such a symbolic and important project.

"In particular, I am concerned that this will lead to massive resentment on the part of those ordinary Londoners who will be seriously inconvenienced by the additional traffic (vehicular and pedestrian) which the museum will generate," he said. "I'm also concerned about the the additional security risk associated with such a site."

Maja Turcan, whose parents were Holocaust survivors, says while a learning centre is needed "particularly at a time where anti-Semitism and hate crimes are increasing" - she questions why it could not be placed somewhere like Manchester "where there's a big Jewish community, but is also multi-ethnic and multi-cultural".

Image copyright Alamy Image caption Victoria Park Gardens as it is now

Aviva Trup, who manages Jewish Care's Holocaust Survivors Services - a centre which offers a programme of social, cultural and therapeutic events for Holocaust survivors in the UK - said "legacy and education is of upmost importance to our members".

She would not be drawn on whether Victoria Tower Gardens was the right place for the project, saying that "the most important thing is that the memorial is built in a central London location and is easy to access".

The UK Holocaust Memorial Foundation is currently running a public consultation exercise, with exhibitions across the UK featuring the proposed schemes until the end of April.

The winning design will be announced before the end of May by Sir Peter's jury, whose members include: London Mayor Sadiq Khan, Communities and Local Government Secretary Sajid Javid, TV presenters Loyd Grossman - also chair of the Royal Parks - and newsreader Natasha Kaplinsky.

The project should be open to the public by Holocaust Memorial Day on 27 January 2021.