When he got to Bilbao a month before it opened, says Frank Gehry, “I went over the hill and saw it shining there. I thought: ‘What the fuck have I done to these people?’” The “it” is the Bilbao Guggenheim museum, which made both its architect Gehry and the Basque city world-famous. Its achievement, measured in much-repeated metrics of visitor numbers and economic uplift, in global recognition and media coverage, in being, in effect, an Instagram sensation long before anyone knew what that might be, is prodigious. It revived belief that architecture could be ambitious, beautiful and popular all at once, yet Gehry has always said that its success took him by surprise.

The museum was opened 20 years ago this month, by the king and queen of Spain, since when it has become the most influential building of modern times. It has given its name to the “Bilbao effect” – a phenomenon whereby cultural investment plus showy architecture is supposed to equal economic uplift for cities down on their luck. It is the father of “iconic” architecture, the prolific progenitor of countless odd-shaped buildings the world over. Yet rarely, if ever, have the myriad wannabe Bilbaos matched the original. This is probably because it came about through a coincidence of conditions that is unlikely to happen again.

Despite Gehry’s protestations of surprise, it is a project that has fulfilled its original intentions with precision. Juan Ignacio Vidarte, the museum’s director, whose involvement dates back to the time of its inception in the early 90s, says that it was meant to be “a transformational project”, a catalyst for a wider plan to turn around an industrial city in decline and afflicted by Basque separatist terrorism. It was to be “a driver of economic renewal”, an “agent of economic development” that would appeal to a “universal audience”, create a “positive image” and “reinforce self-esteem”. All of which it pretty much did. It has been rewarded with a steady million visitors a year, the 20 millionth having arrived shortly before the 20th birthday.

Gehry, who beat two other architects in the competition to design the building, recalls that he was asked to design what was then not called an icon. He was nervous. “They said: ‘Mr Gehry, we need the Sydney Opera House. Our town is dying.’ I looked at them and said: ‘Where’s the nearest exit? I’ll do my best but I can’t guarantee anything.’” So he came up with the convulsive, majestic, climactic assembly of titanium and stone, of heft and shimmer, a cross-breed of palazzo and ship that also flips its tail like a jumping fish, that now stands on the bank of the river Nervión.

It was not a wholly new set of ideas – Sydney, indeed, had demonstrated the value of the transformative landmark, as had Paris with the Pompidou Centre. Frankfurt, Glasgow and Pittsburgh had striven to raise themselves with culture and/or museum-building. What set Bilbao apart was the degree of contrast between the city’s lowly status and the artistic and architectural ambition of its proposed flagship.

They found an ally in the Solomon R Guggenheim Foundation of New York, which had previous in commissioning icons from architects named Frank, in the form of Frank Lloyd Wright’s white spiralling museum on Fifth Avenue. It was then run by Thomas Krens, the holder of an MBA from Yale and a man formed by the risk-taking, deal-making culture of the 1980s. The Bilbao people had heard that the Guggenheim wanted to expand its European presence, and plans to do this by adding to the Peggy Guggenheim collection in Venice weren’t “going forward sweetly”, as Vidarte puts it, so they offered their battered city in place of La Serenissima. The Guggenheim, he says, liked their ideas and their seriousness.

Frank Lloyd Wright’s Guggenheim museum in New York. Photograph: Stan Honda/AFP/Getty Images

An agreement was worked out, an early instance of the international trading of museum brands that also engenders, for example, the forthcoming Louvre Abu Dhabi, whereby the governments of the city, and of the province and region in which Bilbao stands, would pay for construction, and contribute to acquisitions and running costs. The Guggenheim Foundation would lend its name, works from its permanent collections and its management and curatorship. The arrangement wasn’t universally popular – it was called “McGuggenheim”, an act of cultural imperialism paid for by the people it subjected – but it gave Bilbao access to the high-quality art without which a museum would be pointless.

The Canadian-born Gehry, now aged 88, was then in his 60s, and he had a high reputation in the architectural world for his imaginative reinterpretations of the everyday structures of his adopted home of Los Angeles. He didn’t have the celebrity he gained later (an appearance on The Simpsons, for example), and although he was known for the freedom of his forms, the public hadn’t yet seen much of the complex, multiply curving shapes which, since Bilbao, are assumed to be his trademark.

They were, however, brewing in some of his projects, especially an $82m house for the Ohio insurance magnate Peter Lewis – a trustee of the Guggenheim foundation from 1993 to 2005, its biggest donor, and the man who introduced Gehry to Krens — that was designed and redesigned but never built. Gehry and his office pioneered the use of CATIA, software originally developed for designing aircraft, which allowed elaborate shapes to be made without prohibitive cost. It enabled him to realise the Bilbao Guggenheim, as he is keen to point out, within its $100m budget. “I could have straightened everything out,” he says, but “it wouldn’t have cost less.”

The ability of computers to design unfeasibly elaborate buildings has since multiplied. It is a ubiquitous and defining feature of contemporary architecture. It has been an effective accomplice of post-Bilbao “iconic” architecture which, although sensible architects have been pointing out its weaknesses for almost as long as it has existed, and although it suffers from an obvious hyperinflation of shape – if everything looks abnormal it becomes normal – shows no sign of going away.

The Pompidou centre in Metz. Photograph: Centre Georges Pompidou-Metz/Roland Halbe/HO/EPA

This long-running craze would have happened in any case, but the Guggenheim gave it fuel. Its influence takes two main types. In one, public authorities – in West Bromwich, in Denver, in Metz – seek to use some version of the “Bilbao effect” to “kickstart” (as it is often horribly put) regeneration. In Spain, in the bubble years, cities became particularly fond of monuments whose appearance outran their content, architectural dumb blondes by Santiago Calatrava or Oscar Niemeyer or Peter Eisenman that looked especially redundant when the crash came. In the other type, private developers use funny shapes as marketing tools for their towers – see the skylines of Dubai or many Chinese cities, or London’s car boot sale of domestic gadgets.

What both approaches, public and private, have in common is the use of spectacle to distract attention. Public authorities might not want you to notice that their regeneration plans are flimsy. Developers typically use eye-catching design to justify their stretching of planning restrictions, or to obscure the fundamental sameness and ordinariness of their products, or to sell buildings before they are realised – in some cases too to deodorise the dirty money that pays for the projects.

The use of spectacle was also the basis of the most sustained critique of the generally lauded Guggenheim, that its powerful look makes it a poor setting for art. For the critic Hal Foster, speaking in Sydney Pollack’s film Sketches of Frank Gehry, the building trumps the art it is supposed to serve: “he’s given his clients too much of what they want, a sublime space that overwhelms the viewer, a spectacular image that can circulate through the media and around the world as brand”.

Gehry is familiar with the criticism and pushes back. All his professional life he has known and worked with artists. “In the beginning,” he says, “I thought architects should make neutral spaces for art. But my artists were saying: ‘Fuck off, we want to be in an important building. I want to go home and tell my mother I’m in the Louvre.’” He reels off the artists who he says liked the Guggenheim – Anselm Kiefer, Sol LeWitt, “even” Robert Rauschenberg. He says that a clique of museum directors, meeting in London, “passed a resolution that they should never build a building like Frank Gehry’s… they pretty much kept to it.” He claims that the same directors – “you know who they are” – told Cy Twombly never to show in Bilbao. He did, eventually, two years or so before his death. “Cy called me and said it was the best show in his whole life.”

Gehry is also keen to distance himself from the dumber aspects of the building’s architectural legacy. “I apologise for having anything to do with it,” he says. “Maybe I should be hung by the yardarm. My intention was not that it should happen.” Talk of the “Bilbao effect” makes him cringe – “it’s bullshit... I blame your journalist brethren for that.” He wants to stress an aspect of the design often overlooked by imitators, which is that it works hard to connect to its surroundings: “I spent a lot of time making the building relate to the 19th century street module and then it was on the river, with the history of the river, the sea, the boats coming up the channel. It was a boat.”

Another view of the Bilbao museum, which Gehry designed with the help of computer software intended for the aviation industry. Photograph: luisrsphoto/Getty Images

Vidarte, too, is uneasy about its influence. He’s “flattered”, but at the same time “concerned and nervous… many people are just trying to replicate its most superficial aspects.” The project was not about the building, he says, but also a “sustainable” plan for its management and content, and the regeneration of Bilbao was not just about the Guggenheim but also about investment in infrastructure and other urban essentials.

Alongside Gehry’s building stands a giant puppy, covered in living flowers. It was created by Jeff Koons and “underwritten” by Hugo Boss. Shortly before the opening three Basque separatist terrorists tried to disguise themselves as gardeners so as to plant explosives in it. They were foiled, but a policeman died in the ensuing shoot-out.

The incident was emblematic. It was evidence of the troubles that Bilbao was trying to escape and which have indeed diminished. The Koons-Boss pooch, charming and calculating at once, was an early manifestation of the sort of global, big-money, market-led, spectacular art culture that has now become familiar, for which the Bilbao Guggenheim was certainly a Trojan horse. Krens himself went on to be a controversial and ambitious protagonist of this culture, if not an entirely successful one – his Guggenheims planned for Rio de Janeiro, Las Vegas, Guadalajara, Taichung, lower Manhattan and Abu Dhabi have mostly failed to materialise, or stuttered if they did.

But to lay on the Bilbao Guggenheim the effects of its legacy is an injustice to what it was and is. From the political, cultural and commercial currents of its time, not all of them noble or elevated, it drew the energy to make a phenomenon that few people would wish away, least of all Bilbao itself. Its true lesson is that it can’t be copied, because it came from circumstances that were unique.

Five would-be icons

Centre Pompidou-Metz, France, designed by Shigeru Ban, 2010

As in Bilbao, a famous art institution created an architecturally conspicuous outpost in an unglamorous city.

The Public, West Bromwich, designed by Will Alsop, 2008

An attempt to revive the West Midlands with a “box of delights”, where people could both experience and make art. It is now a sixth-form college.

Centre Niemeyer, Avilés, Spain, designed by Oscar Niemeyer, 2011

A cultural centre designed by the celebrated Brazilian architect in his 90s, it closed for financial reasons soon after its completion, but later reopened.

Ordos museum, China, by MAD Architects, 2011

A museum and landmark for a city in the Gobi desert, whose redevelopment is famously underinhabited.

Louvre Abu Dhabi, designed by Jean Nouvel, 2017

Due to open in November, the latest marriage of a museum brand with an aspiring city, celebrated with dazzling architecture.