(CNN) President Barack Obama's election in 2008 broke a racial barrier that set soaring expectations for an era of improved race relations.

Now, as he departs office, those hopes have largely evaporated. Tensions between African-American communities and police departments have deteriorated following a slate of high-profile shootings of unarmed black men. The man who will replace Obama in January was a leading peddler of the racially-tinged "birther" myth. A majority of Americans now say relations between blacks and whites have worsened since Obama took office.

Obama has said he never believed his election could completely erase centuries of racial conflict in America. But the decline in racial ties is nonetheless an ironic legacy for the first African-American president, one Fareed Zakaria explores in the CNN Special Report "The Legacy of Barack Obama" airing Wednesday at 9 p.m. ET.

In interviews, Obama and those who worked closely with him identify a strain of racial bias that hardened against the President, even as his election crumbled a historic racial wall.

"I think there's a reason why attitudes about my presidency among whites in Northern states are very different from whites in Southern states," Obama told Zakaria. "Are there folks whose primary concern about me has been that I seem foreign, the other? Are those who champion the 'birther' movement feeding off of bias? Absolutely."

Obama said he didn't view racism as a major component of mainstream Republican opposition to his policies. Instead, he said it exists on the political fringe. Those who have worked for him, however, do identify race as a factor in consistent Republican efforts to stymie Obama's agenda in Washington.

JUST WATCHED Years of President Obama defending Obamacare Replay More Videos ... MUST WATCH Years of President Obama defending Obamacare 02:00

"It's indisputable that there was a ferocity to the opposition and a lack of respect to him that was a function of race," said David Axelrod, Obama's former senior adviser and now a CNN senior political commentator.

He recalled a moment when a powerful Republican said to him, "you know, we don't really think you should be here, but the American people thought otherwise so we're going to have to work with you."

Republicans fiercely reject the charge that race played a role in their opposition to Obama's agenda, insisting their differences are ideological. But a racial undercurrent has charged anti-Obama sentiments, even in debates over policy, from the beginning of his first term.

In a confluence of events, the first racial controversy of his presidency -- a dust-up involving the arrest of a black Harvard professor on his own porch -- coincided with the angry backlash to Obama's proposed reforms to health care, a debate that took on racist undertones during public protests.

The episode involving Henry Louis Gates in Cambridge, Massachusetts, came in July 2009, seven months into Obama's presidency. After Gates was arrested by a white police officer who accused him of disorderly conduct, Obama alleged during a press conference the police acted "stupidly" -- an assessment that drew anger from the right.

Glenn Beck, the conservative commentator, accused Obama of hating white people, and called the President a racist. A GOP congressman introduced a resolution calling on Obama to apologize.

The President did emerge two days later to concede he "could have calibrated those words differently." He invited Gates and the police officer for beers on the White House South Lawn to smooth things over.

Police Sergeant James Crowley, second right, of Cambridge, Massachusetts, speaks with Harvard Professor Henry Louis Gates Jr., second left, alongside Obama and Biden as they share beers on the South Lawn of the White House in July 2009.

But the incident instilled in Obama a cautionary approach toward race in the ensuing years of his presidency -- disheartening some of his supporters.

"Black moral witness falls silent because if the President can't talk about this without being sent to the woodshed, to be on an equal basis with some random cop, it's over," said Van Jones, who worked in the Obama White House during the Gates incident and is now a CNN political commentator.

Obama largely avoided discussing race for the next three years. But the topic was unavoidable in 2012 when the killing of unarmed Florida teenager Trayvon Martin spurred new discord between police departments and African-American communities.

JUST WATCHED Obama: Trayvon Martin could have been me Replay More Videos ... MUST WATCH Obama: Trayvon Martin could have been me 03:29

Obama emerged to say if he had a son, "he would look like Trayvon," a deeply personal sentiment that thrust him into the center of a fresh debate over race in America.

The ensuing years only saw the tensions deepen as repeated violent incidents drew scrutiny across the country. In places like Ferguson, Missouri, and Baltimore, protests erupted over the police treatment of black men.

Some wanted Obama to say more in support of the emerging Black Lives Matter movement. Others accused him of siding with protesters at the expense of law enforcement. It was a balancing act that became a near-constant presence during his second term in office.

An inflection point came in June 2015, when Obama traveled to Charleston, South Carolina, to mourn the nine victims gunned down during Bible study at the Emanuel African Methodist Episcopal Church. The crime was explicitly racist in nature -- the shooter declared he was there to "kill black people" -- and the President's eulogy evolved into a meditation on race at a moment of national introspection.

JUST WATCHED Obama sings 'Amazing Grace' during eulogy for pastor Replay More Videos ... MUST WATCH Obama sings 'Amazing Grace' during eulogy for pastor 01:44

Obama's speech moved beyond just grief for the victims -- the President stepped directly into a national conversation about race in which he plays a central role. He declared the Confederate flag a symbol of racial oppression, and praised the renewed urgency in removing it from the South Carolina State Capitol.

At the end of his remarks, the President paused in the silent arena hall before singing the opening strains of "Amazing Grace." It was a moment that resonated deeply as the country came to terms with a shocking crime.

It was also a continuation of Obama's long public reckoning with race in America, which began well before he entered office. He detailed his upbringing in his first book, "Dreams from my Father," writing about searching for an identity as the son of a black Kenyan father and a white American mother, raised in Hawaii and Indonesia.

Those identity questions have followed him to the White House, where his status as the first American-American president has brought with it oversized expectations for his ability to cure the nation's racial divides.

"He never ran to be the first black president. He ran to be the president of the United States and he happens to be black," said Axelrod. "He needed to become a force for healing, and finding the right way to do that was something that he wrestled with."

That explanation hasn't always sat well with some prominent black leaders, who say Obama should have spoken more forcefully about the plight of black Americans.

Obama responded by delving into law enforcement practices, creating commissions and issuing recommendations to police departments for how to treat suspects and handle volatile situations. He took steps to reverse the high rates of incarceration among African-American men, including granting clemency to hundreds of people convicted of non-violent drug crimes. And he instituted the My Brother's Keeper mentorship program for young black men, a mission he says he'll continue when he leaves office.

But those efforts haven't improved Americans' view of race relations in the final days of Obama's presidency. A CNN/ORC survey in October showed 54% of Americans believe relations between blacks and whites have gotten worse since Obama became president. Fifty-seven percent of whites, and 40% of blacks, held that view.

JUST WATCHED 11 years of President Obama discussing race Replay More Videos ... MUST WATCH 11 years of President Obama discussing race 03:10

Trump's election isn't likely to improve matters. A Pew Research Center survey taken after November's election found nearly half of US voters believe race relations will worsen under Trump. Only 25% said they would improve.

That's hardly surprising given the racially charged presidential election that Trump won. His opponents accused him of stoking racist fears during his "birther" phase, which he falsely claimed had originated with Hillary Clinton.

Speaking in July -- as the campaign was entering a bitter stretch that included accusations of racist language -- Obama described his own views of his racial legacy.

"More than anything, what I hope is that my voice has tried to get all of us as Americans to understand the difficult legacy of race; to encourage people to listen to each other," he said during a news conference in Poland.

"The legacy of slavery and Jim Crow and discrimination didn't suddenly vanish with the passage of the Civil Rights Act or the Voting Rights Act, or the election of Barack Obama," he said. "Things have gotten better -- substantially better -- but we've still got a lot more work to do."

"If my voice has been true and positive, then my hope would be that it may not fix everything right away," he continued. "That's OK. We plant seeds, and somebody else maybe sits under the shade of the tree that we planted. And I'd like to think that, as best as I could, I have been true in speaking about these issues."

CORRECTION: This story has been updated to accurately reflect the year in which Trayvon Martin was killed.