“We didn’t run from where we grew up. We aren’t afraid to be associated with the people who came up with us.”

That’s Richard Sherman of the Seattle Seahawks writing in defense of his friend, DeSean Jackson, who was cut from the Philadelphia Eagles amid reports of gang ties. Sherman isn’t trying to litigate the allegations or exonerate Jackson—he doesn’t know the details. But he doesn’t think it’s wrong for Jackson to associate with the men from his childhood.

And why would it be? Yes, some of them have criminal records—and for some, that includes gang activity—but leaving home is hard, and the social distance of wealth makes it even harder. As Sherman writes, “In desperate times for people who come from desperate communities, your friends become your family. I wouldn’t expect DeSean to ‘distance himself’ from anybody, as so many people suggest pro athletes ought to do despite having no understanding of what that means.”

I don’t know if Sherman sees it or not—my hunch is that he does—but in a few sentences, he’s put his finger on the pulse of something overlooked in our discussions of poverty and economic mobility as they relate to black Americans: neighborhood. Sherman’s experience of being pulled back to a poor neighborhood, even as he accumulates wealth, is common among blacks.

The difference for ordinary black Americans, as opposed to NFL stars, is that this has been a powerful driver of downward mobility. Just a quick comparison of black and white neighborhoods is enough to illustrate the particular challenges that face black families as they reach for middle class, or try to keep their position.

The key fact is this: Even after you adjust for income and education, black Americans are more likely than any other group to live in neighborhoods with substantial pockets of poverty.

As sociologist Patrick Sharkey shows in his book Stuck in Place, 62 percent of black adults born between 1955 and 1970 lived in neighborhoods that were at least 20 percent poor, a fact that’s true of their children as well. An astounding 66 percent of blacks born between 1985 and 2000 live in neighborhoods as poor or poorer as those of their parents.

A recent report from the Annie E. Casey Foundation affirmed this fact. According to the foundation, only 50 percent of black children live in neighborhoods with a poverty rate below 20 percent, compared to a national average of 74 percent. Of course, the other way to say this is that 50 percent of black children live in neighborhoods with poverty rates of greater than 20 percent.*

How does this stack up to white families? Here, Sharkey is indispensable: Among white children born through 1955 and 1970, just 4 percent live in high poverty neighborhoods. Or, put another way, black Americans live with a level of poverty that is simply unknown to the vast majority of whites.

It’s tempting to attribute this to the income disparity between blacks and whites. Since blacks are more likely to be poor, it stands to reason that they’re more likely to live in poor neighborhoods. But the fact of large-scale neighborhood poverty holds true for higher-income black Americans as well. Middle-class blacks are far more likely than middle-class whites to live in areas with significant amounts of poverty. Among today’s cohort of middle- and upper-class blacks, about half were raised in neighborhoods of at least 20 percent poverty. Only 1 percent of today’s middle- and upper-class whites can say the same.

In short, if you took two children—one white, one black—and gave them parents with similar jobs, similar educations, and similar values, the black child would be much more likely to grow up in a neighborhood with higher poverty, worse schools, and more violence.

This is an outright disaster for income mobility. According to Sharkey, a child with roots in a high poverty neighborhood has a 64 percent chance of moving down the income ladder over the course of his or her lifetime. Given their circumstances, blacks face a reversal of their gains over the last generation.

Simply put, the persistence of poor neighborhoods is a fact of life for the large majority of blacks; it’s been transmitted from one generation to the next, and shows little sign of changing. All of which raises an obvious question: Why do blacks have a hard time leaving impoverished neighborhoods?

“When white families advance in economic status,” writes Sharkey, “they are able to translate this economic advantage into spatial advantage by buying into communities that provide quality schools and healthy environments for children.” The same isn’t true for black Americans, and some of the answer has to include present and ongoing housing discrimination. For example, in one study—conducted by the Department of Housing and the Urban Institute—black renters learned about fewer rental units and fewer homes than their white counterparts.

Once you grasp the staggering differences between black and white neighborhoods, it becomes much easier to explain a whole realm of phenomena. Take the achievement gap between middle-class black students and their white peers. It’s easy to look at this and jump to cultural explanations—that this is a function of black culture and not income or wealth. But, when we say middle-class black kids are more likely to live in poor neighborhoods, what we’re also saying is that they’re less likely to have social networks with professionals, and more likely to be exposed to violence and crime.

This can have serious consequences. Youthful experimentation for a white teenager in a suburb might be smoking a joint in a friend’s attic. Youthful experimentation for a black teenager might be hanging out with gang members. As Mary Pattillo-McCoy writes in her book Black Picket Fences: Privilege and Peril Among the Black Middle Class, “Youth walk a fine line between preparing for success and youthful delinquent experimentation, the consequences of which can be especially serious for black youth.”

DeSean Jackson is still an NFL player, and—as a player for Washington, D.C.’s professional football team—will make a tremendous amount of money. He can keep his connections to his friends, he can live in the same neighborhood, if he wants, and downward mobility won’t be a pressing concern.

For millions of more ordinary black Americans, however, the opposite is true. Even with more income and more education, they’re stuck in segregated neighborhoods. Yes, there isn’t much mobility for anyone, but that fact is especially true for blacks. Indeed, when someone says that America has a “racial hierarchy,” this is what they mean: Whether times are good or bad, blacks remain at the bottom.

Correction, April 4, 2014:This article origially misstated that 50 percent of black children live in neighborhoods with poverty rates of greater than 50 percent. They live in neighborhoods with poverty rates greater than 20 percent. (Return.)

