I remember looking at Zaha Hadid's drawings for Rome's new museum of 21st-century arts a decade ago and wondering how on earth this structural adventure would ever be built. On paper, it looked like a surreal motorway intersection imagined by JG Ballard, or a wiring diagram plotted for the palace of esoteric giants. Her floor plans were some of the most mesmerising and challenging since Frank Lloyd Wright unveiled his seemingly improbable designs for New York's Guggenheim museum more than 50 years ago.

What was so radical about them? The walls of Hadid's new museum, unveiled to the public this month, not only curve but change in depth as they do so. There are moments where walls become floors and even threaten to become ceilings, diving and curving like bobsleigh tracks. (When I went there last week, Hadid told me she wanted the building's concrete curves to "unwind like a ribbon in space".) All of this means that the gallery has been an enormous challenge to build.

It took Wright 15 years to realise the Guggenheim; it has taken Hadid 10 to complete Maxxi, as the museum is known (a play on the Roman numerals for 21st century). There have been at least six changes of national government in Italy since the project was first announced in 1998, from left to centre to right, and the future of many such public projects has often seemed doubtful. But now here it stands, in the residential and military Flaminio district, almost exactly as Hadid and her team first imagined it.

Open to the public over the past two weekends as an architectural shell, the museum will launch fully next spring. Only then will it be possible to judge whether Maxxi, Hadid's finest built work to date, is a real success. Just how will the museum's curators make use of these extraordinary public spaces and gigantic galleries? What will go on show?

The truth is that although the museum, devoted to both architecture and art, has been busy collecting work by Anish Kapoor, Gerhard Richter, Francesco Clemente and many others (along with the archives of architects Carlo Scarpa, Aldo Rossi and Pier Luigi Nervi), this light-filled labyrinth is dedicated to the future. There is no great hurry to fill it, after all: there is the rest of the 21st century to go before the museum can be called complete.

Perhaps this is why Hadid has chosen to make Maxxi an almost modest, if not quite self-effacing, building from the outside. She says she hopes it will be fashion-proof. As you approach, it is only the big flags emblazoned with the name Maxxi that guarantee you have come to the right place. Instead, Hadid has reserved her architectural firepower for the interior.

The huge entrance lobby sets the tone, punching up through the height of the building and offering views into what appear to be ineffable depths. This is a museum of just a few heroic galleries, but with a variety of ways of reaching them. Daylight is ever-present; this can be blacked out if need be for exhibition purposes, though the sun is always held at bay, with light filtered through a two-tier system of roof-mounted louvres and screens. Artificial lighting is concealed wherever possible. If curators wish to divide the galleries, floating walls can be hung from the dark concrete ribs snaking throughout the building; these can also support sculpture weighing up to a tonne. The gallery's project architect, Gianluca Racana, says: "We didn't want anything – air-conditioning grilles or light fittings – to take away from the raw power of the spaces we've created, or from the art that will be on show."

This is a building of few colours: black, white, grey and the varied cream of exposed concrete. The walls and balustrades of the gallery's extraordinary stairs and passageways have been finished in the thick black primer used as an undercoat for new cars. (Highly durable and slightly rough to look at, the paint is surprisingly smooth to the touch.) The stairways rise up through the lobby, with their bare metal treads, disappearing mysteriously into the far recesses of the museum; the effect is cinematic – Piranesian, even – and wholly compelling.

There is a point on the first floor where you can choose to walk in one of three directions, between galleries, stairwells, liftshafts and lobbies. Two of these paths take you into the heart of the exhibition spaces, while a third projects you out of the main body of the museum, along a glazed walkway, allowing you to look in at the gallery as if from the outside – a haunting effect. "For me, it's like standing in [Rome's] Piazza del Popolo," Hadid says. "When you look north, you see the tridente [three streets set between two 17th-century baroque churches] offering you this sudden and thrilling choice of direction. Yet, coming south, all three streets lead back to the same single point."

This is a brave project, and little short of incredible in a city that has proved so deeply conservative over the past decade. In recent years, there has been little imaginative new architecture in Rome, least of all in the public sector. But, remarkably, Maxxi is funded by what is now the Ministry of Cultural Heritage and Activities, or, as it describes itself, "a laboratory for artistic experimentation and production that gives voice to the different languages of contemporariness". Rome's history is inexhaustible, but it is good to see the city moving forward.

In one sense, however, Maxxi is happily old-fashioned. It has been built on-site by local contractors using materials close to hand; Rome led the way when it came to concrete construction 2,000 years ago, and these ambitious new curved walls are made of Roman concrete. "It does sound odd when I say it," says Racana, "but this has been a little like building a medieval cathedral." And, like a medieval cathedral, the museum is in fact several structures gathered together. Tough new legislation ensuring the ability of new buildings to withstand seismic shock was put in place after the earthquake of October 2002, which rocked Italy's Molise and Puglia regions, and was felt in Rome. As a result, the museum consists of five separate buildings leaning against one another, designed to withstand powerful natural shocks.

Last week, the roof of Hadid's aquatic centre for the 2012 Olympics was unveiled, a wavy promise of things to come. Hadid won't be pressed on this, and says she will be happy to talk about the building only when it is complete, once the pools are filled and the swimmers are training. "All people want to do is talk about the budget, as if the rise in cost has been something we've caused. We haven't. We've done what we've been asked to do." Her hope, and that of the Olympic committee, is that the building will inspire Britain's sporting stars.

Likewise, I have a feeling that the energy and imagination of this new museum, its sense of intrigue and possibilities, will bring out the best in its curators. Who knows what twists and turns architecture will take in the course of the 21st century; for now, Hadid's gallery offers an exhilarating set of Roman walls to build upon.