Everywhere in San Francisco today, people are talking about the housing crisis. You can’t turn on the radio or read the news or see the rental listings without being reminded of the lack of affordability. Many of these stories focus on the prices themselves or on the growing influx of eager transplants to the City. Often overlooked is the story of the exodus of families and households from San Francisco.

It is, of course, natural for residents to occasionally move to a different city or state, but this should be a choice, not retreat brought on by economic or social coercion. Alarmist statements about the number of people moving in to San Francisco each year, as well as the urgent calls to build, build, build in order to meet the demand of that population growth, mask the true effects of population “churn.” A whopping 12% of the city’s population turns over every year—so while the news has focused on all the people moving in to the city, the other side of the equation is that approximately 50,000 residents have been leaving San Francisco annually over the last five years. The City Economist crunched some data for the Planning Department earlier this year, and showed the astounding annual magnitude of this migration in and out of the city.

However, even more alarming is that the underlying data on those who are moving into and out of San Francisco shows what we have long known through anecdotal evidence: that largely lower-income and people of color are being pushed out of the city and replaced by a wealthier, more homogeneous class of residents.

So, who is coming to San Francisco?

The American Community Survey, which is part of the US Census Bureau, is collected every year, and is the definitive data source for information on cities across the country. Using ACS data makes is possible to see population trends faster than waiting for the ten-year census. We focused on data collected after the end of the recession, from 2010 forward, to better understand the demographics of population change during these economic “boom” years, but examined data back to 2000 to establish a baseline.

The key takeaway is that San Francisco is becoming less diverse and increasingly a destination for upper income people. This should come as no surprise to any of us. But here are the numbers: through 2014, the most recent year for which ACS data is available, the median income for new San Francisco residents who moved from another state grew by nearly 40% over five years, after accounting for inflation and local cost of living. For example, in 2013 the incomes of people moving in were nearly twice as high as the incomes of those on their way out. To put that in terms of actual dollars and cents income, the number of households above $100,000 income per year grew by 6% while households making less than $100,000 per year shrank by 5%. These may seem like negligible percentages, but they represent approximately 10,000 low- and middle-income households that were essentially replaced in a single year by upper-income households.

This trend has a long precedent over the last 15 years, amplifying with each market cycle, and, if anything, the trend has likely further accelerated through 2015 and into 2016. These trends are evident both long term and over a shorter period, and if median shifts continue on the trajectory they have post-recession, the 2016 median income for in-migrants to San Francisco from other parts of the country or the world would be a whopping 280% of the income of out-migrants leaving for more affordable far-flung regions. The research arm of Trulia, the real estate website, even had an article on this issue earlier this year called “Priced Out” — it’s a growing national problem for urban communities across the country experiencing rapid gentrification in this booming real estate climate.

And who is being replaced?

But this is also more than just a story of income disparities or the affluent of all stripes displacing lower-income residents–the haves replacing the have-nots. There is also a very clear racial dimension. Populations that have historically been marginalized in the United States are again being negatively affected at disproportionate rates as San Francisco “grows” and gentrifies. Every year since 2010, the first year that the most accurate version of this data became available through the ACS, there has been a net loss of African-American residents from San Francisco. Over those five years, more than twice as many black residents moved out of San Francisco as moved in. That has amounted to a loss of as much as a quarter of the city’s black population since 2010.

This story is similar for other populations of color–a greater out-migration of residents than people of color moving into the city. By contrast, over this same five year period from 2010 through 2014, San Francisco gained almost 30,000 new residents who identify as white, the only ethnicity category on the ACS that showed more in-migration than out-migration every year.

The forces of capital are strong, and national and international demand for the chance to live in a city like San Francisco has soared, but this is not just an abstract debate about macroeconomic pressures. The “churn” from these migration trends is rapidly reshaping the city’s demographic picture. Some folks may celebrate “disruption” in all forms, but the data makes clear that this version of disruption is replacing and erasing one San Francisco and creating a drastically different one.

So, what can we do?

Yes, San Francisco will continue to change, grow, and develop. But we need to be honest that the face of San Francisco is changing, the city is rapidly gentrifying and the out-migration of working class and people of color communities is shocking. We have today not just a growing population but more significantly a replacement population. That’s 50,000 new residents a year in, and 50,000 existing residents out. Those are big numbers, and those are everyday San Francisco people. And it is a trend that no one should be comfortable with.

City policy may not be able to entirely reverse these winds, but it can make a difference. We have a responsibility as citizens to push for policy that gives all San Franciscans, especially those whose voices are often silenced, the tools and ability to hold fast against the economic maelstrom of development and displacement driven by real estate goals rather than public policy goals and San Francisco values.

We should push to fortify tenant protections such as rent control, and fight for restrictions on real estate speculation that drive up housing prices and incentivize evictions. We should fight to ensure that new developments are required to contribute to public needs in the form of maximum levels of affordable housing and fair-share of infrastructure costs. We should insist on “affordable” housing that is actually affordable to low- and middle-income San Franciscans – meeting families where they are, rather than playing over their heads for a “new middle class”. And we should reject moves to reduce affordable housing production and to price people out of affordable housing programs, like the San Francisco Realtors’ divisive Propositions P and U on this November’s ballot. Propositions P/U (“peeyew”!) are attacks on affordable housing and would simply add fuel to the fire of the displacement of people of color from San Francisco.

As San Franciscans, we have a lot at stake. We must stick together and fight back against out-migration and population replacement masquerading as benign population growth. We can and should work to ensure San Francisco remains an affordable city truly for all.

Peter Cohen, Fernando Marti and Max Arnell work with the Council of Community Housing Organizations