Over the past few years, Richard Florida has had some explaining to do.

Through books and magazine cover stories, pricey speeches and consultations, the TED-talking University of Toronto professor popularized the early-aughts idea that faded cities could revitalize themselves by attracting the talented, intellectual types who made up what he called the "creative class." Lure some hip coffeeshops, create an "arts district," play up your gay friendliness, and watch the laptopping masses pour in.

Sixteen years after Florida published his first book, "The Rise of the Creative Class," that theory has proved half true. For many small, post-industrial cities without assets like big tech companies and universities, no amount of creative-class marketing would turn things around. Elmira, N.Y., for example, saw little return on its investment in the Florida program, as a 2009 story in the American Prospect detailed.

READ MORE: Are Southerners less creative?

But some cities — San Francisco and New York, Austin and Seattle and Washington — have seen the theory work entirely too well, as creative and techy types revitalized downtown neighborhoods to the point where only bankers and software developers can afford to live in them comfortably. Florida's critics jumped on the skyrocketing housing market as evidence that his creative class gospel might have backfired.

So at an appearance in Houston earlier this month, put on by the Kinder Institute and the Greater Houston Community Foundation, Florida knew he would face some skeptical questions. The audience didn't disappoint. How did he get his seminal theory so wrong, someone asked.

Florida didn't miss a beat.

"I got wrong that the creative class could magically restore our cities, become a new middle class like my father's, and we were going to live happily forever after," he said. "I could not have anticipated among all this urban growth and revival that there was a dark side to the urban creative revolution, a very deep dark side."

I could not have anticipated ... that there was a dark side to the urban creative revolution, a very deep dark side. ” — Richard Florida

Florida has been thinking a lot about that dark side lately, for a book he's publishing in the spring called The New Urban Crisis. It will talk about fixing the affordable housing shortage in desirable areas, connecting cities and their outlying suburbs with mass transit, raising the wages of service workers so they can afford the prices that are rising around them — to knit together a "patchwork metropolis," as he calls it. In short, many of the things that other people have been talking about for many years now, but wrapping it all together with a rhetorical flair that has been his calling card.

"The urban pessimists have a point," Florida says. "We neglected their point, which is that cities are gentrifying, people are being priced out, displaced from their homes. I think we need a new vision for cities that combines an optimistic viewpoint with an understanding of the challenges that re-urbanization brings."

So how did this self-promoting thought leader come to question his own theory of how the world was supposed to get better? He sat down for an interview after his speech to flesh out the story. As he tells it, he had wanted to include a chapter on inequality in his first book, but it was cut for length. After the book came out, the problem kept getting worse.

"Even though I saw the numbers, I was still naive about the extent to which this was occurring," Florida says. "And hopeful, just overly optimistic and hopeful, that this would somehow work itself out...I really had to introspect long and hard."

What really brought the shift was pointed criticism by a group of writers and social scientists who pointed out that Florida's "creative class" had either failed to reverse the decline of cities that had lost their economic engines to globalization and automation, or fueled gentrification and inequality in the places where it seemed to work.

Other research revealed the conditions that create pockets of poverty, and found a downside to ethnically mixed cities: People in different groups tend to live apart. "Here's Mr. Diversity, extolling the virtues of diversity in large cities," Florida says. "And what comes back to smash you over the head is that large diverse cities also incubate a horrific level of sorting and segregation."

A couple years ago, Florida acknowledged that the benefits of these creative clusters did not, in fact, spill over to the waiters and janitors who serve them. Rather, the newcomers reaped the benefits of their own skills, while wages stayed low for people who had lived there all along. The perennial Houston-booster Joel Kotkin, a professor at Chapman University outside Los Angeles who extols the benefits of low taxes and single-family sprawl, seized on the revelation as an admission that Florida was wrong all along.

READ MORE: Bayou City booster Joel Kotkin turns his affections to Austin, San Antonio

The resulting exchange, Florida says, is what prompted him to write his new book. He emailed Kotkin, telling him how much his work had influenced his thinking. Kotkin wrote back, trying to convey how limited he thought Florida's theories really were, since the majority of people don't actually want to live in the hip urban environments he envisions when they grow up and have kids. "You can't build an urban theory about where at best 10 percent of the population lives," Kotkin says.

Accordingly, Florida's new book is about an "inclusive urbanism," about investing in residents' skills rather than yuppifying their neighborhoods, about retrofitting suburbs for people who might want to be able to walk to a grocery store and piping them into the city with commuter rail. He doesn't, however, believe he was wrong to have encouraged cities to target creative professionals and high tech companies in the first place. While humbled, he's by no means contrite.

"If we stop clustering, if we seize up this influx of diversity and tolerance, we'll choke off our economic growth," Florida says. He also has little patience for the tribe of urban libertarians who advocate loosening development restrictions to boost housing construction and bring down prices, as if affordability were simply a matter of supply and demand. (It's not.)

"They drive me f-----g nuts!" Florida bursts out. "What happened to the urban left is it got captured by critical studies, the people who run around in geography departments and who've just given up reality. These are the people who think you're going to rebuild cities by deregulating land use. Welcome to Houston!"

Contra Kotkin, who credits Houston's relative affordability to its laissez faire regulation, Florida doesn't think Houston is a paragon of urban inclusivity. On a new index he's put together that ranks cities according to their levels of economic segregation and inequality, he says, Houston ranks third. Instead of just allowing unfettered housing construction as a one-shot solution, he advocates for more public investment in affordable rental housing, and land use rules that promote transit, to be financed by redirecting federal highway funds into light rail and high-tech buses.

In other words, it sounds very similar to the "smart growth" philosophy that has become commonplace in urban planning departments across the country. Probably more complete than his creative class manifesto, but more a product of how the field of urban policy has been changing ahead of him, rather than a pathbreaking philosophy to lead it.