San Diego County’s population grew in 2016, but not by much, as the high cost of living in the county appears to be taking its toll.

Housing costs have repelled many prospective migrants, and at the same time encouraged residents to relocate to Riverside County. Disproportionately, those leaving San Diego for Riverside are low-income people, not well-off homeowners chasing a bigger house.

According to census estimates, from 2010 to 2015, San Diego County’s population grew 1.2 percent per year, faster than California’s average and almost twice as fast as the national average. From 2015 to 2016, though, the growth rate was much slower – just 0.8 percent, barely above average. Other coastal counties offer a potential warning: Los Angeles and Santa Clara grew even slower, perhaps showing where San Diego could be headed if current trends continue.

But San Diego’s most worrying growth trend is in its migration patterns. Overall international migration remains healthy, but net domestic migration took a sharp negative turn last year. In 2016 alone, San Diego lost 8,300 people to the rest of the country; in the entire five-year period beginning in 2010, the census estimates the county only lost a total of 7,177 people.

There’s a hint about what’s going on in a less-used dataset: IRS migration data. The IRS keeps county-level data on when people and households moved, tracking individuals using tax exemptions. Unlike other sources, it keeps track of average incomes, making it invaluable in answering questions like “Are people who leave California likely to be poor people fleeing high costs, or rich people fleeing high taxes?” (The answer is neither. The average income for people who leave California is very close to that of people who stay in the state.)

One pattern is that people who migrate between rich regions and poor regions – in both directions – tend to have low incomes. San Diego County’s taxable income per capita, $36,900, is about the same as California’s. Riverside County’s average income is just $24,100. But the average income for people moving between Riverside and San Diego counties – in either direction – is just over $25,000. This represents large migration flows in both directions, but more people move from San Diego to Riverside (9,000 people) than in the other direction (7,000 people) in 2015. (There is no IRS data for 2016 yet, but like the census data, it will likely show more intense migration away from San Diego.)

In contrast, people migrating between San Diego County and other coastal counties tend to have middle or high incomes. Los Angeles County’s average income in 2015 was $36,000; people moving from Los Angeles to San Diego averaged $35,000, people moving in the other direction averaged $37,000. In farther-away regions, migrants aren’t just average, but are doing well: Average incomes for people migrating between San Diego County and the San Francisco Bay Area vary by year, but are on average in the $50,000 to $60,000 range.

The same pattern is true on the state level. California is a rich state; in the housing bubble years, and even in the ensuing bust, migration between the state and poorer Arizona and Nevada remained poorer than the average for Californians who stayed put. This has changed in the last few years, but migration with Arizona, Nevada and Texas remains on average somewhat poorer than state average. In contrast, migration with states like New York, Massachusetts, Washington and Pennsylvania is substantially richer than the state’s average. The per-capita income for California-New York migrants is about $70,000.

The reason for this discrepancy has to do with why people move. People who move between rich states (such as California and New York), or between rich regions (such as San Diego and San Francisco), do so for professional reasons. People who get finance jobs move to New York City, people who get tech jobs move to San Francisco, people who get jobs in biotech move to San Diego, etc.

Migration between rich and poor regions is different. People from poor regions can move to rich ones to try to earn more, regardless of their background. Service jobs pay better in San Diego than in Riverside or Imperial counties. The drawback is that on the coast, housing is more expensive, which turns away anyone who cannot afford the spiraling rents.

Riverside County, though, is not an outlying region: It is an exurb of Los Angeles, and increasingly of San Diego, as people drive until they qualify for a mortgage. In the postwar decades, Americans suburbanized in search of higher quality of life, and suburbs were richer than inner cities. Today, many California suburbs have higher average incomes, but new exurbs like Riverside tend to be poor, as people move there because when they’ve been priced out of employment centers.

For comparison, the situation in lower-cost Houston is not so stark. Harris County, home to Houston, has an average income per person of about $38,100. Its two biggest suburban counties are Fort Bend and Montgomery. Migrants between Harris and Montgomery counties earn about that much, too. There’s a bit of a disparity going to and from Fort Bend County, but not much. People moving from Fort Bend make about $31,000, and people going the other direction make about $34,800. In Houston, where rents are lower than San Diego, people suburbanize and exurbanize for a combination of old reasons (higher quality of life in the suburbs) and new ones (lower costs of living).

Yet people displaced from San Diego to Riverside have no good way of getting to San Diego jobs. The freeways are congested, driving such long distances is expensive and there isn’t an affordable public transit option to skip the congestion. Unlike Los Angeles, San Diego has no railroad going toward Riverside. The second phase of California high-speed rail could change this, as it connects San Diego to Los Angeles through the Inland Empire, but it is decades in the future, and expensive fares could discourage commutes. For the foreseeable future, people leaving the county for the Inland Empire will be facing long, unreliable, expensive commutes.

The only solution is for San Diego to create more affordable housing in the city and its inner suburbs, where commutes are shorter than in the Inland Empire. The county’s population growth remains healthy, but as it fills up, its growth rate is likely to keep falling, and more people will leave for the cheaper Inland Empire, or move to a different part of the country altogether.

Alon Levy is a Paris-based mathematician and public transportation policy writer.