Melbourne's Federation Square was a controversial piece of architecture when it was built in 2002, with many people dismissing it as ugly and strange.

But when the Victorian Government revealed plans to demolish the southern building and replace it with a flagship Apple store, the overall public response was one of shock.

Federation Square is just 15 years old so is a few years away from being able to claim historical significance. So what makes a building worth preserving?

It turns out some of Melbourne's most-loved buildings were not always considered the architectural gems they are today.

Flinders St Station

Flinders Street Station was once considered ugly, but is now heritage-listed. ( ABC News )

It may surprise some people to discover that one of Melbourne's most iconic buildings was once considered hideous.

"In the 1970s Flinders St Station was considered ugly, dirty, crowded, outmoded and not a good work of architecture," architectural historian Rohan Storey said.

"But as soon as the railway commissioners proposed replacing it with a couple of giant office blocks with a carpark, and a station underneath, public anger emerged and public affection for the building emerged."

The plans were eventually shelved and the building began to be slowly recognised as a place of historic and architectural importance.

"I guess that was the beginning of recognising cultural significance, that a place can be important to people without necessarily being a fine piece of architecture," Mr Storey said.

"[But] we all now consider Flinders St station to be a fine work of architecture.

"You need some passage of time before you can say a place has accrued heritage value.

"You might be able to say something is recognised as a significant piece of architecture … but to have historic, social and cultural significance a place needs to be old, but how old depends on the place."

The Regent Theatre may have been demolished if not for public support. ( ABC News: Brad Ryan )

The Regent Theatre

The Regent Theatre, built in 1929 on Collins St, was one of Melbourne's most-loved cinemas. But it closed its doors in 1970 because it wasn't viable operating with a single screen. The City of Melbourne then bought it as part of its plans to build City Square.

"At the time it was threatened with demolition, it wasn't considered good architecture and was not considered worthy of heritage listing," Mr Storey said.

But public affection for the building was strong, and a Save the Regent committee was formed.

"The plans, a bit like Federation Square, got bigger and bigger and turned into a giant hotel with a forecourt to replace the Regent, and helping to pay for the square itself," Mr Storey said.

"And given the choice, Melburnians would have preferred to keep the theatre.

"The unions agreed and so a union ban was placed on its demolition, and it remained sitting there but empty for 25 years until it was finally restored."

After a lavish overhaul, it now hosts world-class stage and musicals shows like The Lion King, Wicked, King Kong and We Will Rock You.

Federation Square

Under the plans, the Apple store will replace Federation Square's southern building. ( Supplied: Victorian Government )

Federation Square was the result of an international design competition and the winning entry was seen by people within the design world as cutting-edge. But it was different to what was eventually built.

"The decision-makers and the architectural community thought it was a fantastic thing, it was in the early days of deconstruction which was the style that followed post-modernism," Mr Storey said.

"By the time it was finished, some Melburnians thought it was a horror, thought it was appalling and the worst thing they'd ever seen."

But architects were unhappy that the main free-standing shard — a four-storey building on the corner of Swanston St and Flinders St — wasn't built because there were concerns it would block the view of St Paul's Cathedral. It was instead cut down to one level.

"And even now, that stump of a shard is about to be demolished for the Melbourne Metro," Mr Storey said.

The move to demolish the southern building at the square and replace it with an Apple store sparked an uproar on social media.

"It's going to be a very interesting debate about the value of the architecture of Federation Square and its nature as a public square versus the supposed need for commercial return," Mr Storey said.

"Opinions change depending on the choice given, and in this case the design as built is an integrated whole all into the same style, and now we are faced with a choice between that and something that has an entirely different character plonked in the middle of it.

"So the immediate reaction is 'what are they thinking? It's going to ruin the aesthetics of the place', even if you didn't like the aesthetics to start with."

Queen Victoria Market

An artist's impression of a new greenhouse-inspired pavilion at the Queen Victoria Market. ( Supplied: Breathe Architecture )

One popular Melbourne spot facing the dilemma of old versus new is the 130-year-old Queen Victoria Market. The City of Melbourne is planning a $250 million upgrade of the site, including a large tower and a greenhouse inspired trading hall.

But traders and celebrities including actors Michael Caton, actress Sigrid Thornton and musican Dan Sultan have joined a community campaign against the upgrade.

It's an example of when the idea of what is ugly and what is beautiful, and the commercialisation of public spaces, come together to create strong debate, Mr Storey said.

"There's something old and familiar there, that doesn't look immediately like it needs to be tidied up or made better, so there's a fear of change and that change being bad," he said.

"There's not guarantee of what it will actually look like in the end or how it will operate."

Melbourne Museum

The Melbourne Museum is in Carlton but that wasn't the original planned location. ( ABC News )

The current Melbourne Museum, in the Carlton Gardens, was initially supposed to be built somewhere else. Construction started in the early 1990s along the Yarra at South Wharf, but when Jeff Kennett was elected premier the plans were thrown out. Instead, the site became the Melbourne Convention Centre — dubbed Jeff's Shed.

"I think everyone recognised the museum was not well-housed literally in the basement of a 19th century complex of buildings, that it needed new home … but the actual design and location was a controversial one," Mr Storey said.

There were concerns about the idea of a modern building behind the historic Exhibition Building and the beautiful green surrounds in Carlton.

"That was controversial at the time, and opposed by quite a few people, but it seems to be generally accepted now. It hasn't destroyed the gardens as it was feared, it hasn't expanded into the gardens," he said.

Melbourne's 'crazy, wacky, look-at-me' designs

Mr Storey said Melbourne has had a bold architectural culture since the 1980s which often raises the eyebrows in other parts of the country.

"As much as you try to avoid it you can't avoid it, you have to say whether something is good or bad. If it's it not both loved and hated then it's boring," he said.

"Other architects from around Australia, particularly Sydney, look at what we build in Melbourne, some of the crazy stuff, and wonder why the hell we're doing it.

"There seems to be a competition to design the most craziest, wacky, colourful, madly shaped, look-at-me designs. Whereas in Sydney they're more likely to build refined, simple white modernist things.

"The more interesting it is, the more likely it is to get a [building] permit, even if it's too tall or too big, or otherwise doesn't fit the planning scheme. It's an evolution from a desire to actually be interesting to a desire to look interesting."