Cities used to covet the Olympic Games. Hosting them meant glamour and prestige.

Not anymore.

Rome became the latest city to abandon a bid for the 2024 Summer Games when it withdrew on Tuesday because of worries over the cost.

Budget concerns have led city after city to drop their Olympic dreams in recent years. Hamburg, Germany, previously bailed on 2024. Stockholm and Krakow, Poland, pulled the plug on bids for the 2022 Winter Olympics, which were later awarded to Beijing.

An Olympic host city has to plan, pay for and construct massive infrastructure projects. Security costs can run into the billions of dollars. Thousands of hotel rooms must be built to house athletes and tourists.

Most of it happens on the taxpayer dime -- with little discernible economic benefit.

Elected leaders have argued that ticket sales, construction jobs and increased tourism outweigh the costs. But economists say the real return on hosting the Olympics is not so rosy.

Montreal, host city of the 1976 Summer Games, provides perhaps the best example of the long-term cost.

Before the games, the mayor declared: "The Olympics can no more lose money than a man can have a baby." He was wrong. Mismanagement and gross cost overruns left the city with $1.5 billion of debt that wasn't paid off until 2006.

Related: U.S. Olympic athletes aren't paid like champions

The Olympic stadium, known as the Big O, was turned into a ballpark and now sits largely idle. The citizens of Montreal took to calling it the Big Owe.

While Montreal is an extreme case, researchers at Oxford's Saïd Business School have found that spiraling costs are the norm.

"The games overrun with 100% consistency. No other type of mega project is this consistent regarding cost overrun," they wrote in an academic paper in 2013. "Other project types are typically on budget from time to time, but not the Olympics."

The Montreal experience went a long way toward scaring off potential host cities for 1984.

Only one, Los Angeles, made a bid, and consequently was able to dictate terms to the International Olympic Committee.

It pursued a new model that relied heavily on private financing. Los Angeles also used existing stadiums as venues, erasing one of the largest costs associated with hosting the Olympics. The result? A profit.

But the days of minimal public investment appear to be over. Russia is said to have spent an incredible $50 billion on the 2014 Sochi Olympics. China spared no expense in hosting the 2008 Beijing Games. Several stadiums built in Rio are expected to be used only lightly now that this summer's games are over.

The IOC will meet next year to select a host city for 2024. Only Budapest, Paris and Los Angeles are still in the running.

-- Livia Borghese contributed reporting.