Get breaking news alerts and special reports. The news and stories that matter, delivered weekday mornings.

There is no actual barrier dividing the predominantly black and poor Baltimore neighborhood where Freddie Gray lived and ultimately died from the more affluent and white enclaves in and surrounding the city.

But those left to pick up the pieces following riots in the community where buildings were looted and torched after Gray’s death say there are obstacles there nonetheless.

Much of the anger expressed during the unrest in Baltimore was about the death of yet another black man in the custody of police officers. But there was also underlying frustration over racial and socioeconomic forces that have stymied opportunities for minorities in Baltimore's poorest neighborhoods — areas that saw some of the worst violence in years in April, after Gray's death.

“There’s two Baltimores — a white-centered Baltimore and a black-centered Baltimore,” said Tommy Bourne who was born and raised in the city and has lived in three different neighborhoods in Baltimore. “The more affluent the neighborhood, the more access.”

A new Obama administration rule aimed at ending racial segregation across the country — coupled with a recent Supreme Court decision preserving the Fair Housing Act’s scope in matters of discrimination — could have an impact in places like Baltimore, a city where racially segregated housing patterns have been entrenched for decades.

But residents of Baltimore and housing policy experts say that all depends on the determination of the nation's cities to tackle the issue honestly and aggressively and the federal government’s willingness to get tough with communities that violate policies.

“The assumption is that every jurisdiction wants to integrate and they don’t have the info to do so and that’s a very optimistic viewpoint,” said Richard Rothstein, a research associate at the Economic Policy Institute and author whose work examines the history of state-sponsored residential segregation and racial educational inequities.

“It’s a good first step," he said "but I wouldn’t personally get too excited."

According to the Department of Housing and Urban Development, the new rule "sets out a framework for local governments, states, and public housing agencies (PHAs) to take meaningful actions to overcome historic patterns of segregation, promote fair housing choice, and foster inclusive communities that are free from discrimination." The rule, a long-sought goal of civil rights advocates, will require cities to examine their housing patterns for racial discrimination trends and periodically publicly report their findings.

HUD will provide instructions and data to local communities to help figure out neighborhood patterns of segregation and integration and racial and ethnic concentrations of poverty. Cities will then set benchmarks on helping reduce the inequities and, in some of the most egregious cases, a portion of the billions in federal funding allocated each year could be withheld.

The new rule will take effect after 30 days, but will be phased in over time.

Related: Obama Administration Announces Fair Housing Rule

The public housing apartments and historically segregated streets of Chicago were the chosen backdrop for the announcement made by HUD Secretary Julián Castro last Wednesday, July 8.

Chicago Mayor Rahm Emanuel said it's no surprise his city was chosen to make the announcement.

"We have a long history as it relates to fair housing," Emanuel said while standing at the site that once housed Stateway Garden's eight high-rise public housing buildings, which along with Cabrini-Green and the Robert Taylor Homes were once some of the country's most striking symbols of urban blight. "I think this is the perfect place not just because of the history of Chicago but because of the future we are building today."

Proponents of the new rule commend the Obama administration for taking the step. But they also said they’ll be keeping a close eye on whether cities use that data to institute the type of change that results in a reversal of deeply-ingrained racial housing trends.

"We think the new fair housing rule is a very positive step that has the potential to impact even cities as deeply segregated as Baltimore. But whether that actually happens depends on whether local, state and federal officials use the data that HUD will provide to inform major changes in how programs operate," said Barbara Sard, vice president for housing policy at the Center on Budget and Policy Priorities. ​

Advocates are further encouraged at the timing of the rule which was announced after the U.S. Supreme Court last month rejected an attempt to narrow the scope of the Fair Housing Act, one of the nation's most important civil rights laws.

The Morning Rundown Get a head start on the morning's top stories. This site is protected by recaptcha

In the 5-4 vote, the "court acknowledges the Fair Housing Act's continuing role in moving the nation toward a more integrated society," the court wrote. The ruling said a housing discrimination case can be based on disparate impact on minorities based on the federal fair housing law which was enacted in 1968,

Some conservative critics are calling the administration’s new rule aimed at tracking and reversing housing discrimination a failed attempt at "social engineering."

"American citizens should be free to choose where they would like to live and not be subject to neighborhood engineering and gerrymandering at the behest of an overreaching federal government. Local zoning decisions should be made by local communities, not bureaucrats in Washington, D.C.," Rep. Paul Gosar, an Arizona Republican said Friday in a statement after lawmakers in the House approved his amendment aimed at blocking the Obama administration rule.

Related: Baltimore Riots: Violence Scarred a City Dealing With Decline for Decades

An Urban Institute map of the nation’s income inequality trends by census tract over the past two decades shows the stark picture of the types of patterns the Obama administration is hoping to reverse.

The areas deemed "most affluent" are in blue. The least affluent are gray.

Baltimore is nearly entirely one gray mass on the map.

As of 2013, the Census reported that blacks make up a little over 63 percent of Baltimore and whites make up 31.6 percent.

"I grew up calling a strip of Baltimore the white highway," Bourne said. "You can drive from Charles Street — it goes from downtown Baltimore all the way to the county — and you will not pass anything but predominantly white neighborhoods."

Devin Allen's photo on the cover of Time magazine. Devin Allen/TIME Magazine / Devin Allen/TIME Magazine

Currently, Bourne resides in Canton, an area close to downtown Baltimore which he describes as "predominantly white neighborhood," with "young white professionals" and immigrants.

"What we’re finding now is that there’s a push to repopulate the city," he said. "However when we look at the ways it’s happening and the areas that it’s happening in, it’s almost as if they’re welcoming white folks back into the city as opposed to welcoming anyone into the city that would want to come."

Bourne says after the "white flight" decades ago from Baltimore, blacks became the majority population, but now there’s a shift — one he noticed as he moved from neighborhood to neighborhood.

The type of pattern Bourne describes is also documented in the "Thompson v. HUD" case which charged the Housing Authority of Baltimore County and HUD of violating the Fair and Housing Act with a racially segregated system of housing.

In the expert testimony of the case, law professor john a. powell (who writes his name in lowercase) used a map of “communities of opportunity” based on a 14-point criteria which included low crime and transportation access. On the map, Baltimore was depicted as having the lowest opportunity whereas its surrounding areas were considered high opportunity.

The trend was also clear to the court which found HUD programs "failed to achieve significant desegregation” in the Baltimore region.

Rothstein of the Economic Policy Institute said undoing the impact of policies—many of them put in place by the federal government generations ago as a way to racially segregate—will be a herculean task.

Volunteers work to help clear out the debris from a CVS in Baltimore, Tuesday, April 28. Maryland's governor vowed there would be no repeat of the looting, arson and vandalism that erupted Monday in some of the city's poorest neighborhoods. Lloyd Fox / AP

Such policies included crafting a public housing program in lieu of pushing aggressively for integrated neighborhoods and giving loans to contractors to build suburbs around cities on the promise they would not sell to blacks, he said. It also included steering African Americans into central city neighborhoods where they were forced to pay higher rents for homes than whites because of the supply and demand imbalance.

The overcrowding, combined with a lack of city services, weaker performing schools, a dearth of grocery stories with affordable healthy food options and similar amenities, turned these neighborhoods into slums, said Phil Tegeler, executive director of the Poverty and Race Research Action Council.

“White families came to associate African American families with slum conditions,” Rothstein said.

So even if African American families were able to move to communities with better services, schools and housing stock, there was “white flight” out of these areas, Rothstein said.

Under President Richard Nixon, then-HUD Secretary George Romney, Mitt Romney’s father, tried to address this by denying funds to communities that refused to desegregate.

"He did withdraw federal funds from three suburban communities, Baltimore being one of them," Rothstein said. "There was a backlash and Nixon reined him in."

Some fair housing advocates worry that, faced with the same backlash over the new rule's implementation, future presidential administrations might follow suit. They also worry that cities might opt not to take HUD funding rather than face potential increased federal scrutiny.

Bourne said there are geographical displays of segregation in the Baltimore. Charles Street, Bourne explains, separates the city east from west.

Adding to the divide, Bourne said there are subtle sentiments of an unwillingness to intermingle.

He feels a decision to squash the $2.9 billion developmental rail project, “The Red Line,” is indicative of these attitudes. The Red Line was intended as a mass transit train traveling from Woodlawn in West Baltimore to Bayview in East Baltimore.

“A lot of people are happy the governor put the kibosh on the project, but personally I would have found it very useful,” he said.

There is some indication that the type of rule the Obama administration is implementing could have an impact in Baltimore.

“The recent regional plan Baltimore and surrounding communities adopted under a similar policy (as part of the Sustainable Communities initiative) is some evidence that a data-driven planning process can induce real change,” said Sard at the Center on Budget and Policy Priorities. ​

The rule could also force cities to question issues of neighborhoods being "separate but unequal," Tegeler of the Poverty and Race Research Action Council said.

"Cities and towns are going to have to put on the table disparities between white communities and communities of color on schools, city services, grocery stories, bike trails, parks and crime rates," he said.

With the new ruling, Bourne says he is hopeful, but there are more issues deeply entrenched that contribute to Baltimore's housing disparities.

“They’ve gotten around making it strictly about race,” he said. “[The rule] goes a good way to stop the more overt things but there’s a lot of deeper issues we have to look at.”

Bourne believes there are many more steps needed to lift and eradicate the invisible fences in Baltimore.

“There’s an extreme divide and it’s almost as if folks have those invisible fences,” he said. “Even if you take the fence off, it’s like psychologically built in to you. It’s like, ‘I know that I don’t belong past this point and so I’m not going to have any business past this point.’ It’s very sad.”