Gabriel Metcalf has spent a lifetime thinking about cities. He spent 21 years at SPUR, the Bay Area urban think tank in San Francisco, and led the organization for 13 years. Holding that job, he said, was “the honor of my life.”

He’ll be leaving town at the turn of the year, bound for Sydney, where he’ll be head of the Committee for Sydney, a planning organization similar to SPUR. In a way, he’s following a dream. Metcalf and his wife have always wanted to live and work abroad. So he was delighted when a job offer came from Australia, “right out of the blue,” he said.

He’d been in Sydney a few times.

“I fell in love with it,” he said, “It’s a great city.”

He said the same thing about San Francisco, when he moved here 22 years ago, “when it was still possible to come here without a job and figure it out,” he once said.

Metcalf thinks San Francisco is “the most progressive of American cities.” But problems are overwhelming the city and the Bay Area. “Something has gone terribly wrong,” he wrote in SPUR’s magazine.

The success of the region has also brought failure — and economics are the key, he thinks.

“The economy has changed fundamentally in the last 50 years,” Metcalf said. “We have made the transition to the knowledge economy,” with thousands of new people and new jobs. The Bay Area has become a player on the world stage, with an economy that is larger than Sweden’s. He calls it “the reluctant metropolis.”

The region has become so attractive and successful that no one can afford to live here. Economic studies show that the median price for a house in San Francisco is $1.6 million — twice what it was five years ago. And it’s not just San Francisco. Only 18 percent of Bay Area residents can afford a medium-price house.

Metcalf believes it’s our own fault. The current generation — the voters, the leaders, the ordinary citizens — are to blame.

“You know what they say, San Franciscans love innovation as long as it doesn’t mean changes,” he said. Every development decision in the city — and the region as well — involves endless meetings, public hearings and workshops. Not to mention a permit system of Byzantine complexity.

“We’ve really gotten good at preventing bad things from happening,” Metcalf said, “but we are not good at making good things happen.”

It’s no secret that the result is that the region’s supply of housing has been overwhelmed by demand. Restrictions on zoning, coupled with construction costs that are among the highest in the country, mean that only one housing unit has been built in the Bay Area for every six new jobs.

Metcalf thinks that more dense housing is an answer. But zoning regulations are in the way.

“In most of the Bay Area, it is illegal to build anything but single-family houses,” he said. “What we are doing now is not working.”

The housing crisis is behind the Bay Area’s terrible traffic, long commutes and inadequate transportation as people push farther out in search of places to live. Metcalf is worried about the lack of big transit projects. In recent years, the region’s only major projects to be approved have been the Sonoma-Marin SMART train, a couple of BART extensions, increased ferry service and plans to electrify the 155-year-old Peninsula commute trains.

BART is talking about a second Transbay Tube in the next decade, but so far it’s only talk.

And projects seemingly take forever these days.

“Look how many years it has taken to study faster buses on Geary Boulevard,” Metcalf said. In fact, the Geary bus project has been in the works for 10 years.

Metcalf thinks the region can learn from the past, “particularly the generation that built BART.”

It took leadership to build BART, he said, in the face of technical problems and political opposition, especially from so-called progressive groups that argued BART would benefit only big business and developers.

Like all urbanists, Metcalf looks to the future. The numbers are sobering: The most recent Metropolitan Transportation Commission study envisions 300,000 new jobs in San Francisco over the next 20 years, part of a growth pattern in a booming mega-region of 4 million new residents that stretches from Monterey to Sacramento.

“People are going to come here,” he said. “There will be new immigrants from other parts of the country and from foreign countries. Our job is to make the region better for the next generation.”

In the meantime, he’s off to Sydney, a city very similar to San Francisco, where a combination of natural beauty and growth has created its own problems. He sees it as a challenge and an adventure.

How will it work out?

“Ask me in a year,” he said.

Carl Nolte is a San Francisco Chronicle columnist. His column appears every Sunday. Email: cnolte@sfchronicle.com Twitter: @carlnoltesf