“Amazing, most amazing position,” said a US government official in 1943 of British reluctance to help with a plan to rescue 70,000 Jews from a part of the Soviet Union under axis occupation. The British feared “the difficulties of disposing of any considerable number of Jews”. Or as the American paraphrased: “We let them die because we don’t know what to do with them.” In 1945 the Liberal politician Viscount Samuel described the British and international response to the Holocaust another way. “Out of that vast reservoir of misery and murder,” he told the House of Lords, “only a tiny trickle of escape was provided.”

These facts, recorded in Whitehall and the Jews 1933-1948 by the lawyer and historian Louise London, should be remembered. For everything that Britain has to be proud of in the defeat of Nazism, including a slightly less mean attitude to refugees than some other countries, and the children’s rescue programme, Kindertransport, the response to the displacement and slaughter of millions was to admit only by the thousands those trying to escape. The government feared immigrants taking British jobs, and social unrest. In terms that sound familiar now, it tried to distinguish political refugees from “economic” migrants. Much of the press backed them up. “The law of self-preservation”, said the London Evening News in 1938, “demands that the word ‘enter’ be removed from the gate.”

Inside the design by Caruso St John Architects, Marcus Taylor and Rachel Whiteread. Photograph: © Caruso St John Architects, Marcus Taylor and Rachel Whiteread & Malcolm Reading Consultants

The point is that there is complexity in this, as in many other aspects of the vast horror of the Holocaust, one that has to be recognised if, as is now proposed, a large British memorial is to be built. But such recognition wasn’t entirely obvious when David Cameron gave the United Kingdom Holocaust memorial his blessing a year ago, saying that it should “stand beside parliament as a permanent statement of our values as a nation”. His words suggest an element of self-congratulation in the project.

On Holocaust Memorial Day, 10 shortlisted designs for the memorial were made public by some of the biggest names in art and architecture, such as Lord Foster, Sir David Adjaye with Ron Arad, Anish Kapoor with Zaha Hadid Architects, Daniel Libeskind, Rachel Whiteread and Marcus Taylor with Caruso St John. They employ many of the tropes of modern memorialising – digging into the ground, mute forms, fractured walls, light descending from above, the gathering of stones to represent lost lives – but there is too little sign in either their briefing or in their design of deep thought about the memorial’s purpose. They don’t examine what it is to make this piece of work here and now.

The case for a memorial is strong, as the existing Holocaust memorial in Hyde Park is paltry and obscure, and offensively inferior to a nearby absurdity commemorating animals killed in war. The current project has grown out of a report by the prime minister’s Holocaust Commission, which sought the views of survivors and of organisations such as the Holocaust Educational Trust and the Wiener Library, and looked at other memorials around the world. The commission concluded that, with the passing of the generation that experienced the Holocaust, it was more important than ever to ensure that it will be remembered fully and well. Their recommendations included “national educational activity”, an online hub, action to ensure the permanent preservation of witnesses’ accounts, as well as a “striking and prominent new national memorial”.

It was decided that a memorial should include a “world-class learning centre”, so that “visitors can deepen their knowledge and understanding”, similar to the one at 9/11 Memorial and Museum in New York. Film production and special effects companies, Atlantic and Framestore, offered their services to help visualise the past. Sites were considered. Two, in the forecourts of commercial property developments, were thankfully rejected. The one chosen was Victoria Tower Gardens, a well-used public park next to the Palace of Westminster that has other monuments concerned with freedom and oppression: the Emmeline and Christabel Pankhurst memorial; Rodin’s sculpture The Burghers of Calais; a Victorian fountain in gingerbread gothic that remembers the abolitionist Thomas Fowell Buxton and the emancipation of slaves.

Some feature videos in which architects try to combine their sales techniques with a sensitive catch in their voice

A competition was launched, inviting entries from multi-disciplinary teams with enough size and experience to handle what is, with the learning centre included, a £40m scheme. The brief called for “an outstanding, ambitious, sensitive design that creates an emotionally powerful place for reflection and learning”, something “sombre but not shocking” to “convey the magnitude of what happened in a meaningful and comprehensible way” and “give visitors a deeper understanding of the Holocaust and its victims”. Eminent architects and artists responded and the shortlist was chosen.

Each step was right and proper, yet something has been missed along the way. By insisting on experienced teams, the competition lost the chance to make something like the Vietnam memorial in Washington DC, where the then 21-year-old student Maya Lin made the most potent and poignant memorial of modern times, a reflective stone wall, set into the ground, in which visitors can see both themselves and the names of every one of the dead. At the same time the jury contains only one architect and no landscape architect, too little for an endeavour where design is crucial.

A learning centre sounds good, except that it would be less than a mile away from the sober, substantial and powerful Holocaust exhibition at the Imperial War Museum. It’s not clear how one would help the other. Such a centre also inflates the scale of the project to the extent that it would effectively turn the whole gardens into a place of remembrance, at the expense of its current uses; even though the centre would be underground, the excavations needed to access it would change the green space completely. The brief asks the design teams to “enhance the gardens” but none of them do, because it’s an impossible request.

The scale of the Holocaust, it might be said, requires something that makes an impact. It shouldn’t slide in unnoticed. True, but neither should it come at the cost of something loved, as the gardens are. A memorial and a learning centre together, on this site, simply don’t work. Each element would compromise the other. Better to find another place, as prominent or more so, such as the expanse of grass called College Green, where TV companies currently like to interview politicians, or else reconsider the idea of a learning centre.

Powerful but problematic: Anish Kapoor’s inhabitable meteorite. Photograph: © Anish Kapoor and Zaha Hadid & Malcolm Reading Consultants

These issues suggest a lack of thinking through, as if the nobility of the cause made it unnecessary. Other assumptions insufficiently challenged include the location of a single grand memorial in the centre of London rather than creating several around the country, or learning from non-architectural forms of memorialising, such as Jeremy Deller’s use of living people to remember the Somme.

The biggest unexamined questions are what it means to build such a memorial now, in this country, a place affected indirectly. The brief offers bromides and generalities about being outstanding and sensitive. It mentions the other monuments in the gardens but doesn’t do much to invite reflection as to what the Holocaust reveals about Britain. It is far from raising the difficult question of contemporary treatment of refugees – whom Cameron called a “bunch of migrants” on the same day that he launched the memorial – compared with that in the 30s and 40s.

David Adjaye and Ron Arad’s design. Photograph: © Adjaye Associates & Malcolm Reading Consultants

The designs that follow might be predicted. They are mostly trite, some of them accompanied by awkward videos in which architects try to combine their usual sales pitch techniques with a sensitive catch in their voice. One talks cringingly of “our visitor journey”, as if this were a theme park. Kapoor proposes a large inhabitable meteorite – certainly powerful – but his architectural collaborators make it pompous by putting it on axis with a descending avenue of cypresses. Foster suggests a long straight line like a railway track going into brick rooms like gas chambers. This doesn’t feel right to me.

Several address themselves to the adjoining river Thames and the imposing mass of parliament’s Victoria Tower, but beyond that few engage with the surrounding gardens. An exception is the Whiteread/Taylor/Caruso St John submission, in which a translucent cast of the Buxton memorial, plus a scattering of smaller ones, bring light into a “hall of voices”, where you can hear survivors and witnesses speak. This is the only one that speaks both to the Holocaust and to the British experience of it.

Given that the project has got this far, the best thing might be to employ the most thoughtful team of the 10 and reconsider both the learning centre and the site. Then take a pause in proceedings and invite insights from as many as possible of the people who might offer them – survivors and their descendants, historians, writers, religious leaders. Otherwise it will only offer an official version, which is not enough.