This week the US government confirmed that the Federal Bureau of Investigation will be moving to a new facility in Maryland or Virginia – officially abandoning its hulking J Edgar Hoover Building in Washington, DC. The massive concrete structure on Pennsylvania Avenue now seems set for destruction, as even its few defenders concede that most citizens of DC react to the Hoover Building with unbridled hatred. Yet even unsuccessful buildings are part of our cities’ history, and part of the pleasure of being a city-dweller is finding buildings that you can love to hate, like these six:

Boston Government Service Center

Delicate anti-modernists love to hate Brutalism, the 1960s architectural movement whose large-scale concrete projects are under constant threat of destruction. But few Brutalist icons have as bad a reputation as this arm of Boston’s City Hall – an unfinished and unloved work by the great Paul Rudolph, chair of the Yale School of Architecture. Its overscaled pedestrian plaza was supposed to have been counterbalanced with a tower that never got built, and the architect’s signature striated concrete feels a little sinister on a building that houses a mental health facility. Saddest of all are the chain-link fences that now ring the plaza, giving the Brutalism haters even more ammunition to tear the thing down.

2 Columbus Circle, New York

Edward Durrell Stone’s squat, stubborn update of Venetian Gothic architecture was reviled, when it first opened in 1964, as an abandonment of all the modernist principles the architect propounded at his Museum of Modern Art around the corner. As the home of the campy Huntington Hartford Gallery (the no-abstraction-allowed collection was nothing special, while the leather-clad lounge upstairs, called “The Gauguin Room,” served pseudo-Polynesian pork and pineapple skewers), it seemed to revel in bad taste. Over time the so-called Lollipop Building became a surprisingly lovable work of architecture, but its supporters were too late – in 2005, after the city refused to grant landmark protection, the architect Brad Cloepfil tore off the façade and gutted the interiors to make way for its current tenants, the Museum of Arts and Design. The resultant mishmash has pleased no one.

EMP Museum, Seattle

Facebook Twitter Pinterest Photograph: Library of Congress

The Emerald City is home to the greatest work of contemporary architecture in America: Rem Koolhaas’s crystalline Seattle Central Library. Unfortunately it is also home to this thing – a fifth-tier multi-colored hodgepodge by Frank Gehry, which opened in 2000 and utterly failed to do for Seattle what his Guggenheim did for Bilbao. Commissioned by Microsoft billionaire Paul Allen, it was initially a museum devoted to rock music; the bizarre glass strips resting on the roof are, strange to say, a metonym for guitar frets. Some of Gehry’s work, such as his own house in Los Angeles, looks better with age. This one is the opposite: a monstrous relic of the CD-ROM era.

Centre Point, London

Smack in the center of the capital’s West End, looming over Tottenham Court Road and Oxford Street, Richard Seifert’s notorious tower outraged Londoners not only for its scale and its dreary, water-free fountains at street level, but because it remained empty for more than a decade after. (In 1974, homeless demonstrators occupied the vacant skyscraper, but they were turfed out before long.) Now that central London has been transmuted into a hollowed-out non-dom tax shelter and money laundering facility, Centre Point is now fulfilling its destiny. It is being converted into apartments, though who knows if the international buyers will ever move in.

Tour Montparnasse, Paris

Photograph: Steven Strehl

When it opened in 1973, towering over the 14th and 15th arrondissements in the south of the capital, almost no one had a good word to say about the first – and only – skyscraper to rise in central Paris. Well, they hated the Eiffel Tower and the Pompidou too, at first, but the Tour Montparnasse has only got worse with age. From a distance it glowers obnoxiously over Paris’s sea of Haussmannian gray, while at street level, tethered to a neighborhood-blighting shopping center, it’s even worse. Its most deleterious effect was to kill off the possibility of any tall buildings in Paris for a generation: the capital essentially banned skyscrapers until last decade, though a new project by Jean Nouvel in the southeast is now underway.

Palau de les Arts, Valencia

Designed at the start of the century by Santiago Calatrava, Valencia’s hometown hero-turned-villain, the opera house and concert hall on the bank of a dried-up river once seemed a symbol of Spain’s roaring economy and buoyant cultural scene. Now the bubble has burst, years of austerity have taken their toll – and Calatrava’s space ship is literally falling to pieces, the façade’s ceramic tiles dropping on the heads of passersby. (Inside the house the story isn’t any happier; this week Helga Schmidt, the Palau de les Arts’s too-long-put-upon director, announced she’s stepping down.) The city and its native son are now in court.

This article was amended on 1 August to correct that Paul Rudolph served as the chair, not the dean, of theYale School of Architecture.