The opinions expressed are his own.

The late 1990s was a wild time in Silicon Valley. The NASDAQ was soaring, and seemingly anyone could start a company, stick a .com at the end of its name, put together an IPO and retire a millionaire. The great boom ultimately took on a speculative character that led to wasted investments and the collapse of many poorly-grounded operations. But it was rooted in a surge of not-unrealistic optimism about the potential of the internet to change the world of business.

Among the striking features of the era, one of the most startling is this: the rate of high-tech entrepreneurship in Silicon Valley seems to have been below the national average from 1996 to 2000, according to a recent analysis of business creation during the tech boom. And from the late 1990s to the early 2000s — after the bust — Silicon Valley’s rate of high-tech entrepreneurship actually increased. How can this be? How is it that during the first great boom of the internet era, Silicon Valley was less of a hotbed for new firm formation than the country as a whole?

Economists Robert Fairlie and Aaron Chatterji suggest that the answer lies in the extremely tight labor market conditions that prevailed at the time. The tech boom was remarkably good for Silicon Valley workers. Average earnings rose by nearly 40% from 1997 to 2000 — more than twice as fast as the increase for the country as a whole. Non-salary compensation also soared, thanks to the popularity of stock options and the skyrocketing value of equity in tech firms. These generous pay increases made it unattractive for workers to leave established companies to strike out on their own. Entrepreneurship fell because life on salary was too lucrative to risk self-employment.

Why was pay so high? Rising productivity was a big reason, as were expectations (some more reasonable than others) of high and rising profitability across the tech sector. But these factors could just as easily have driven an increase in new firm formation and employment as a rise in salaries. Silicon Valley experienced more of the latter than the former because workers were scarce. During the late 1990s, the unemployment rate across Silicon Valley dropped well under 3%, eventually sinking to nearly half the national level. There was essentially no surplus labor in the whole of the region. Firms therefore had to bargain hard to hire qualified workers, and this meant giving up a substantial share of firm surplus in the form of salary, benefits, and profit-sharing. That, in turn, made it more attractive to be a worker than an entrepreneur.

And this brings us to the crux of the matter: why was the Silicon Valley labour market so tight? If the unemployment rate was so much lower than it was elsewhere in the country, and if compensation was rising so much more rapidly than elsewhere in the country, why weren’t people pouring into Silicon Valley from elsewhere in the country? More remarkably, why were people moving in the opposite direction?

If you can believe it, Silicon Valley’s main metropolitan centers were losing residents to other parts of the country during the Dot Com boom. From 1998 to 1999, for instance, San Francisco and San Mateo counties each lost a net of about 10,000 residents to other parts of the country. Santa Clara County, the heart of the Valley, lost a net of 30,000 residents. For the decade as a whole, the losses were even more substantial; roughly 176,000 more residents of Santa Clara county moved out than moved in during the 1990s. The Silicon Valley labor market was tight because even as wages were soaring, residents were decamping for other locations.

These departures weren’t simply workers without the skills to benefit from the rise of Dot Coms. The tech industry was flourishing around the country. The problem wasn’t an absolute shortage of talent. The problem was that the talent couldn’t be attracted to the heart of the boom.

The reason, you’ve no doubt already guessed, was housing costs. From 1997 to 2000, average earnings in the Silicon Valley area increased by nearly 40%. But from 1997 through the end of 2000, home prices in San Francisco nearly doubled, according to the Case-Shiller index of home prices. As fast as compensation was rising, it wasn’t keeping up with housing costs. And so even as the demand for skilled workers in Silicon Valley soared, residents trickled away to other locations. That made for a too-tight labor market, and that, in turn, squeezed out entrepreneurs.

And what did that mean for the American economy? The workers that moved elsewhere didn’t give up working. They found employment in other metropolitan areas, many of which developed thriving tech sectors. Those sectors weren’t fallow fields for new firm creation; entrepreneurship rates, remember, were higher nationally than in the Bay Area. But what the economics of metropolitan geography tell us is that many small collections of firms will often be less productive and less innovative than fewer, larger firm clusters. The forces that repelled workers from Silicon Valley, which was the intellectual heart of the country’s tech industry, reduced the potential economic impact of the tech boom. And in a not unimportant side effect, it reduced national productivity and total compensation in the economy.

During one of the great innovative periods in America’s recent history, high housing costs poured a bucket of cold water on the nation’s entrepreneurial capacity. That’s a problem. And what’s especially troubling about this example is that it wasn’t an isolated incident. On the contrary, the same forces that drove workers away from Silicon Valley during the tech boom appear to operate in a systematic fashion, undermining the productive potential of America’s great cities and holding back the country’s job creation machine. For two decades now, the country’s internal migration has amounted to a move away from productivity, and toward stagnation.

By now, the American economy should have come farther, developed more online business models, and created more jobs. The internet has been underexploited, if you can believe it. We have been wasting years and opportunities — in part because of things like the price of a 3 bedroom house in Santa Clara.

This is an excerpt from The Gated City, available as Kindle Single from Amazon.com. Copyright 2011, Ryan Avent.