Friedrich Wilhelm IV described his vision for Berlin’s Museum Island as a “cultural acropolis”; a sacred sanctuary for the arts and sciences that would cement the Prussian capital as the Athens of the north. Almost two centuries later, his classical aspirations have been fulfilled by British architect David Chipperfield, in the form of a dazzling white temple. Opening this weekend, after 20 years of planning, the James Simon Gallery stands as a €134m (£120m) Parthenon-on-Spree, forming a handsome new entrance to one of the world’s most important repositories of cultural treasures.

“We were quite nervous,” says Chipperfield, standing in the lofty new ticketing lobby, where stripes of sunlight flood in between the row of slender white columns outside. “The challenge was how to create something that was of its context and also of our time, in this incredibly sensitive location.”

He had good reason to be anxious. His first design, unveiled in 2006, was slammed by German critics as grossly misjudged for the Unesco world heritage site. The jumbled cluster of glass boxes was likened to “a glorified public toilet” by one prominent journalist, in a tirade published under the headline “Not like that, Mr Chipperfield!”

Facebook Twitter Pinterest Monumental … the view towards the main entrance. Photograph: © Ute Zscharnt for David Chipperfield Architects

The 65-year-old architect is no stranger to Berlin’s culture of fiery public debate. He worked for more than a decade to realise the acclaimed transformation of the Neues Museum, and is now engaged in refurbishing Mies van Der Rohe’s Neue Nationalgalerie, a holy of holies among architects. “It’s refreshing that architecture is subject to such robust debate here,” he says, lamenting what he sees as a lack of serious public discussion around the built environment in the UK. “They really hold your feet to the fire, which is painful at the time, but the work is better for it.”

His team went back to the drawing board and took classical cues from the neighbours, taking a lead from the broad entrance staircase of Friedrich Schinkel’s Altes Museum, as well as Friedrich August Stüler’s unifying colonnade, and the weighty stone plinth of the Pergamon Museum next door. These elements have been deftly synthesised into a structure that is entirely its own, a kind of stripped-back, etiolated classicism that is at once imposing and delicate. The result is a strangely scaleless structure that appears monumental from some angles, dinky from others, with its real purpose hidden below ground.

Named after one of the most prolific 19th-century Jewish donors to the museums’ collections (a gesture intended to recognise the many other such donors who were erased from history in the Nazi era), the building provides space for the practical functions that didn’t fit in the surrounding museums. It was a lengthy shopping list: ticketing, loos, cloakrooms, shop, cafe, auditorium and gallery for temporary exhibitions, some of which feel a bit squeezed into the narrow site. Chipperfield describes it pragmatically as “a sort of subway station”, serving as a central entrance hub with underground connections to the surrounding buildings. A tunnel now leads to the Neues Museum, and a big portal goes straight to the Pergamon, although the other links are a task for future generations.

Facebook Twitter Pinterest Pale, milky light … the upper foyer and main internal staircase. Photograph: Ute Zscharnt for David Chipperfield Architects

Anxious that it might be seen as a “dustbin of uses that didn’t fit elsewhere,” the architects have endowed their building with grandiosity in bucketloads: entering this functional underworld is more akin to ascending to a realm of the gods. A broad staircase leads up in three flights to a kind of temple mount, an elevated plateau where ticket desks and a cafe spill out on to a terrace overlooking the Kupfergraben canal. Chipperfield describes it as a “purposefully purposeless space”, a civic deck where people can come and take in the view for free and stroll between the surreally skinny columns – a line of 70 white concrete matchsticks that stand almost nine metres high but less than 30cm thick.

They are a striking sight, glimpsed at the end of views down the canal, but marching rows of thin, square columns conjure uncomfortable allusions here in Germany. The long colonnade has inescapable echoes of the work of Nazi architect Albert Speer, particularly his rally grounds in Nuremberg. “We’ve been called fascist in the past,” Chipperfield says, referring to the controversy surrounding his Museum of Modern Literature in Marbach, built in 2006, which also featured decidedly Speerian pillars. “Germans weren’t allowed to use columns after the war because they were so tainted by association. Being an English architect gave [my client] some relief – ‘Well, if he says we can do it, then it’s OK.’ We’ve tried to use the language in a very neutral, minimal way.”

Inside, there is a sophistication in the use of materials that Speer could only dream of. A huge wall of translucent white marble, just 6mm thick, casts a pale milky light into the entrance hall, while a perforated copper ceiling shimmers overhead. Steps lead down to a level of cloakrooms and loos, designed as a warm wooden world lined with book-matched walnut veneer – great tree-trunks thinly sliced and opened like the pages of a book, so the grain forms mirrored Rorschach patterns as it wraps around the walls.

Facebook Twitter Pinterest A ceiling like draped fabric … the museum’s auditorium. Photograph: Ute Zscharnt for David Chipperfield Architects

“We wanted the building’s concrete structure to be exposed, then lined with richly decorated surfaces formed by nature,” says Alexander Schwarz, design director of Chipperfield’s office in Berlin, a violin-maker-turned-architect whose attention to material detail oozes from every corner of the project. The 300-seat auditorium is a mini masterpiece, with concrete walls cast in pleated acoustic folds, and a ceiling that plunges down in three dark wooden waves like draped fabric. Elsewhere, wafer-thin steel benches are upholstered in pale leather, while dark patinated bronze brings a weighty touch to the doors (quite literally – they’re incredibly heavy to open).

It all sings with the seductive Chipperfield brand of restrained opulence, but there is a slight sense throughout the building that architectural gesture sometimes overruled practical function. The circulation spaces are expansive, but the new 600 sq metre gallery for temporary shows feels a bit shoehorned into a corner, uncomfortably long and narrow. Meanwhile, a charming staircase leads down to the water along the canal-facing facade, suggesting the romantic Venetian possibility of arrival by boat, but sadly it is nothing of the sort: no boats will be permitted to stop here. “It’s an affectation,” Chipperfield admits. “A human gesture to suggest the building dipping its toe in the water.” With a vocal campaign underway to see this stretch of the waterway transformed into a public swimming area, it could be the perfect place for bathers to pad up the steps to sunbathe on the terrace – a suggestion that makes the museum directors tremble at the thought of people arriving at their sacred cultural citadel in dripping trunks.

Which is a shame, because it could do with a touch of Berlin’s more bohemian, freewheeling spirit. It is an elegant and supremely well crafted building, but it feels a bit chilly, parked on the water’s edge like a superyacht. With luck, the swimmers will get their way, and the gleaming acropolis will live up to its intention of being a truly welcoming civic space.

The James Simon Gallery, Berlin, opens on 13 July.

• This article was amended on 8 July 2019. The original called Friedrich Wilhelm IV the Kaiser. He was King of Prussia.