Over the years, one of environmentalism’s biggest drawbacks has been its blinding whiteness and the implied exclusivity of outdoor spaces. That’s changing, slowly, thanks to mainstream and upstart groups working to bring diverse populations into parks, onto trails, and into environmental politics and leadership.

But even as the green movement works toward building an outdoor community that reflects America’s demographics, anti-immigration and alt-right groups are using the environment as a weapon, citing overused public land, population growth, and pollution to keep immigrants out of the country.

This idea, taken to its most extreme, cropped up in several horrible mass shootings this summer. The gunman who killed 22 people at an El Paso, Texas, Walmart on August 3 cited environmentalism in his manifesto, as did the man who killed three at California’s Gilroy Garlic Festival on July 28. The shooter who killed 51 people in Christchurch, New Zealand, in March explicitly called himself an eco-fascist—essentially eugenics through a lens of landscape protection.

This kind of rhetoric has been building over the past several years. White supremacist Richard Spencer, in his statement that spurred the 2017 Unite the Right rally in Charlottesville, Virginia, cited the natural world as one of the reasons why the alt-right needed to grab power. That same year, conservative pundit Ann Coulter wrote in a Daily Caller article entitled “Choose Between a Green America and a Brown America” that immigrants were trashing the U.S. “Even people who don’t live in yurts can’t help but notice the environmental damage being done by hundreds of thousands of Latin Americans clamoring across the border every year, setting fires, dumping litter, spray-painting gang signs in our parks and defacing ancient Indian petroglyphs,” she argued. But this twisting of environmental ethics is implicitly xenophobic. It doesn’t necessarily mean fewer people using the earth’s resources; it means fewer people here.

Much of this sentiment is rising up on a wave of anxiety, says Betsy Hartmann, author of The America Syndrome: Apocalypse, War, and Our Call to Greatness—anxiety around demographic changes and the white majority losing power. But there’s also a long, dark history of environmentalism being used as a tool for racial gatekeeping, with a halo of doing good. Even Yosemite’s founding as a park hinged on pushing Native Americans and Mexicans out of the landscape, an early conservationist sentiment that tied racial purity to the purity of nature. And while protecting land and habitat is crucial, the oversimplified logic ignores big-picture global-population dynamics and the ways humans actually use resources and space.

This twisting of environmental ethics is implicitly xenophobic. It doesn’t necessarily mean fewer people using the earth’s resources; it means fewer people here.

So where does this rhetoric come from? The earliest seeds were planted by 18th-century demographer Thomas Malthus, who said that food production wouldn’t be able to keep up with population growth and at some point we’d be screwed. The Nazis used his logic as an argument for racial purity, and the idea became popular again in the 1970s as the oil crisis curbed resources and the baby boom spiked population numbers. One of the most insidious recent examples was John Tanton, an anti-immigration ideologue who founded groups like the Federation for American Immigration Reform in the late 1970s, and, according to documents unearthed by the Southern Poverty Law Center, infiltrated liberal organizations in a scheme to soften his message. The plan worked: in 1971, Tanton joined the board of the Sierra Club and pushed his anti-immigration agenda. The Sierra Club included population control as one of its tenets until the mid-2000s. Michael Brune, executive director of the Sierra Club, acknowledges that history, and says it shows how easily conservation can get twisted as an exclusionary technique. “It’s an antiquated idea from a climate perspective,” he says. “We can’t just draw a line on a map and say, ‘This place is protected.’ For instance, Yosemite is under siege from climate change. We have to address global problems, we can’t just solve them locally.”

That’s the key: on a global scale, tightening borders doesn’t change the pressure. Population growth and the environment have long been linked in most people’s minds—Bernie Sanders fielded a question from an audience member on this perennial topic during last week’s CNN Climate Town Hall. And it is indeed true that climate change and increasing populations are tightening the bounds of sustainability; the most recent report from the Intergovernmental Panel on Climate Change warns that a food crisis could be looming. But there’s a difference between population growth and consumption levels. The threat comes in how we consume resources, not where we live. Conflating population and consumption, says Hartmann, can be simplistic and dangerous. “I’m tracking climate-conflict alarmist language,” she says, “and the evidence for some of the assumptions is pretty slim or based in old colonial language.”

If we really wanted to carry this anti-immigration rhetoric to its logical conclusion, we’d cede all of our land and mineral rights to indigenous groups. Using anti-immigration rhetoric as a tool for racially motivated exclusion is flatly wrong, and it targets the wrong root issue.

A tricky tenet of environmentalism is preservation, trying to essentially hold a place intact as the world shifts around it. But the problem with that wilderness ethic of untouched landscapes is exclusion—and the historic injustice of who gets to live where. According to a recent article in The Atlantic, “The data suggests that there is no empirical linkage between immigration and environmental degradation, and some studies even suggest a negative correlation. Large corporations and the wealthy consume the most environmental resources, not poor immigrants.”

As Americans, we live in one of the richest nations on earth, and much of the real damage to our natural resources comes from unwise overuse. It’s happening now. The current administration has rolled back nearly 50 pieces of protective legislation, on everything from water quality to endangered species to methane emissions to land-use planning. That’s what we should be fighting, instead of fighting to keep people out.

Heather Hansman (@hhansman) is Outside Online’s environmental columnist and a frequent contributor to our Culture Notebook. She is the author of Downriver: Into the Future of Water in the West.