A lot of people are moving out of California because they can't afford to live there anymore.

For every home buyer coming into the state, there are three Californians selling and moving elsewhere, according to data analysis firm CoreLogic.

"There is clearly a relationship between the migration patterns and home prices," said Sam Khater, deputy chief economist at CoreLogic. "The middle and lower middle can no longer afford to live in California."

California's housing market is one of the most expensive in the nation, with a median home price of $428,000 across the state. Prices have shot up 71% since 2011.

And a number of its local markets are prohibitively expensive. Of the five priciest housing markets in the country, four are in California, according to the latest data from the National Association of Realtors. San Jose tops the list with a median home price of $1 million.

Related: I pay more than half my income in rent

But incomes just haven't kept up with the home price bonanza for many people -- particularly for the working and middle class.

Despite strong job and wage growth, the Golden State has been losing residents for years and could be facing a shortage of middle-wage workers, according to a report from Beacon Economics released in March.

For instance, California's tech scene has been prosperous, but its benefits have been somewhat concentrated.

"All the gains aren't coming in support of the rest of the economy, they're coming at the expense of the rest of the economy," said Christopher Thornberg, founding partner of Beacon Economics. "Folks in the tech industry make craploads of money and they can afford to live there and will force out other people."

Plus, California is still a magnet for foreign buyers, who tend to be more affluent.

"It makes it harder for the average person to make a living there," said Khater. "So that means less teachers, fire fighters, retailer workers. It's causing the entire state to become more expensive."

To help make life more affordable, residents are trading the beaches and nice weather for states with more affordable housing markets and a lower costs of living. Places like Arizona, Texas and Nevada, according to Khater.

And when it comes to buying new homes, they're getting more for less. Last year, California migrants sold their homes for an average of $495,500, and only spent $315,000 in their new -- and often bigger -- houses.

"They are saving money and moving up market," said Khater. "You can increase your standard of living."

Related: Who's to blame for Portland's soaring home prices?

Karen Francis and her husband left the San Francisco Bay area in January after he was offered a job in Vancouver, Washington. While their move was career motivated, it's been good for their budget.

They now own a 2,600-square-foot home that is almost double the size of the place they were renting in California.

And it's not just their housing costs that have shrunk. The couple, which have an infant daughter, are also paying less in gas, utilities and services like garbage collection. And commute times are significantly better.

"You have to drive 20 minutes to get anywhere in Silicon Valley," the 28-year-old said. "Here, the drive to the store, church, gym where I workout, everything is 10 minutes or less."