(CNN) Rand Paul's intensive effort to redeem himself -- and the Republican Party -- with minorities seemed to be on the verge of cratering.

He joked Tuesday to conservative radio host Laura Ingraham that he was happy his train didn't stop in the riot-scarred city of Baltimore . It was exactly the type of tone-deaf remark the Kentucky senator's group of black advisers urged him to avoid as he seeks to expand the GOP's outreach to minorities.

They immediately knew Paul's comment would pose a problem. They checked with other African-Americans to see how it was being perceived.

On Wednesday afternoon, Paul spoke for about an hour with black advisers who told him what they were hearing. The Republican presidential contender told them he understood the concern and expressed regret for his words.

"He said 'OK, I understand what everyone is saying, you're right. I shouldn't have said it that way,'" said Elroy Sailor, a senior adviser and director of strategic planning for Paul. "He recognizes that people listen and hear things differently. Certain words resonate with different constituencies.'"

The episode underscores Paul's complicated history with race. This is the man who openly questioned central tenets of the Civil Rights Act before he later voiced unequivocal support. He's partnered with New Jersey Sen. Cory Booker to push for criminal justice reform but also spoke this week of his sympathy for the "plight of the police."

Influence

Perhaps more importantly, Paul's comments -- and his response -- speak to the influence an informal group of black advisers will have as he tries to prove that his eclectic blend of conservative fiscal policies paired with a focus on criminal justice and civil liberties can expand the reach of the Republican Party.

Paul, who declined to be interviewed for this story, spent years forming this group of black advisers. The process began shortly after he bombed during a 2013 speech at Howard University. His tone was lecturing as he offered a history lesson on Republicans to some of the nation's brightest black students. Some accused him of "whitesplaining" politics.

JUST WATCHED 2016 candidates on Baltimore Replay More Videos ... MUST WATCH 2016 candidates on Baltimore 02:10

After the speech, he went to Louisville, Kentucky, to engage more and learn from his mistakes, a process that even some of his supporters say is incomplete. But interviews with several of his closest advisers and informal consultants reveal an imperfect politician trying to understand the gritty realities of African-American communities.

"He wasn't talking about Republicans freeing the slaves or anything like that," Christopher 2X, a community activist in Louisville's West End, recalled of a 2013 meeting at Plymouth Community Outreach Center, which is in one of the city's poorest areas. "He said to them, 'I don't understand your world. I'm a white male. I'm trying. Forgive me if I don't understand your pain.'"

Some of what Paul did to build relationships with minorities played out in public, like when he went to historically black Simmons College and fielded questions from students, asked for their votes, and copped to the GOP's failings as the cameras rolled.

But much of his work happened behind closed doors.

He studied the case of William Warley, a black resident who successfully challenged Louisville's segregated housing policy in 1917. He forged a bond with Kevin Cosby, the president of Simmons, who also pastors a 10,000-member church. And he connected with community activists like 2X, who served time in prison for a drug conviction and tested Paul's comfort zone by bringing four teenage boys to meetings and putting the senator on the phone with rapper Master P.

Days after his Howard speech, Paul reached out to prominent black Republicans including Sailor, former Republican National Committee Chairman Michael Steele and former Oklahoma Rep. J.C. Watts. They each helped Paul make more connections to black communities in cities like Detroit, Atlanta and Chicago, setting up meetings with barber and beauty shop owners, pastors and civic leaders where Paul listened more than he spoke.

'Skeptical'

"I was a little bit skeptical based on some things I've heard and I've seen from other Republicans," said Watts, who has since endorsed Paul. "I wanted someone to pick up on that Jack Kemp model and I wanted him to understand that it's the justice issues, or the injustice, that keep black people from voting Republican. He has listened and learned and has been able to take on things that most Republicans would be afraid of."

Paul's black advisers told him to avoid lecturing on the highlights of the Republican Party and black voters, and to listen instead and drill down on policy.

"In meetings, I've heard Republicans say to me that black people are Republicans, they just don't know it yet. I don't need you to tell me I'm conservative because I go to church. What I like about Rand Paul is that he doesn't make that presumption," Steele said. "He has taken affirmative steps to become more aware of how black people view certain issues. But he has been forthright about what he is willing and capable of doing."

JUST WATCHED Paul: Clinton did not do her job properly at State Dept. Replay More Videos ... MUST WATCH Paul: Clinton did not do her job properly at State Dept. 02:04

The education of Paul included conversations with presidents from historically black colleges and universities, policy papers recommended by NAACP leaders, the documentary "Prison State" and the book "The New Jim Crow," which has become the secular bible among the new generation of civil rights activists.

In the Senate dining room, and sometimes in his office, he huddled with civil rights leaders like Wade Henderson, who leads the Leadership Conference on Civil and Human Rights, and Hilary Shelton of the NAACP. The discussions, sometimes over breakfast, touched on education, foreign policy, health disparities, voting rights and criminal justice reform and helped inform Paul's approach to policy.

Paul introduced in 2014 the REDEEM act with Booker, a bill that would allow adults convicted of nonviolent offenses to have their records sealed, which would allow for an easier transition from jail back into the working world. He has also introduced a bill to restore the voting rights of felons in federal elections -- an estimated 1 in 13 African-Americans aren't able to vote as a result of laws in various states, including Kentucky.

That package of bills puts him on the same side as prominent civil rights groups that have been pushing for similar efforts for decades. And in her recent policy speech on criminal justice reform, Hillary Clinton name-checked Paul.

"He has a healthy understanding that voting is the language of democracy and if you don't vote, you don't count," said Henderson, who recently met with Paul. "His willingness to explore these issues and to engage them intellectually is encouraging."

Yet Henderson was dismayed by Paul's comments on Baltimore, where he pointed to the absence of fathers and the "lack of a moral code in our society," as the root cause of the unrest.

"His comments strongly reinforce the skepticism that many in the black community have about the sincerity of his outreach and his understanding of the complexity of inequality in American life," he said. "Outreach to the black community is important, but a doctrinaire conservative response to Baltimore's unrest is no substitute for a willingness to learn about systemic injustice or to confront the harsh realities of police abuse. I expected more from him."

Ferguson

After last year's unrest in Ferguson, Missouri, Paul seemed to exhibit a singular fluency in talking about race and racism, a boldness not typically seen among Republicans. Responding to those protests, which erupted after a policeman killed an unarmed black teenager, Paul wrote an op-ed that pointed to the disparate treatment meted out to African-Americans by the criminal justice system.

"Three out of four people in jail for drugs are people of color. In the African-American community, folks rightly ask why are our sons disproportionately incarcerated, killed, and maimed?" he wrote in November 2014.

He made similar comments in Iowa in June 2014.

"If you look at the war on drugs, 3 out of 4 people in prison are black or brown. White kids are doing it, too. In fact, if you look at all the surveys, white kids do it just as much as black and brown kids," he said to an overwhelmingly older, white crowd. "But the prisons are full of black and brown kids because they don't get a good attorney, they live in poverty; it's easier to arrest them than to go to the suburbs."

New civil rights movement

Paul's approach comes as young African-Americans across the country are increasingly engaged in a new civil rights movement, with some seeing a natural fit between libertarianism and criminal justice reform if Republicans changed their tone.

"Republicans and conservatives talk about the overreaching federal government and freedom and liberty, but it's often vague. It's intellectually dishonest for a conservative to dismiss Ferguson and Eric Garner but decry big federal government at the same time," said Eugene Craig, who heard Paul speak in March at Bowie State University, a historically black school in Maryland. "I'm a young black man and I have more concerns about being pulled over in the car than I do about a raid from the EPA."

There are big doubts that Paul's efforts will pay off politically. In Louisville, there's skepticism about Paul among traditional civil rights leaders, who see him as an opportunist.

Raoul Cunningham, president of the state and Louisville branch of the NAACP, keeps a file of newspaper clippings on Paul that portray him as a convenient convert to the cause of restoring voting rights to felons. The organization wrote a letter urging Paul to vote for Loretta Lynch for attorney general. Paul voted no last week, joining with all but 10 Republicans in opposing the first black woman nominated for the post.

Cunningham said he has "heard Paul talk about making efforts to broaden the party, but that's about it."

At Simmons, where Paul has appeared on campus beside the college's president, there is a similar sentiment among students and professors, who questioned Paul's interest in the school and the surrounding West End neighborhood.

"He's visited," said Elaine Walker, a Simmons student. "That's all."

Paul's campaign will be a test of whether the work he started two years ago can reshape the Republican primary electorate and power him to the GOP nomination. In many ways, he is borrowing from George W. Bush, who focused on a key issue that resonated with black voters -- same-sex marriage -- and white conservatives.

His aides are focused on Southern states that have sizable black populations like Georgia and Mississippi as well at states like Florida and Michigan.

Over the next weeks, the Paul campaign will hold a major fundraiser in Atlanta among African-American business leaders and focus on building support on the ground in early states. But the ambitions are relatively modest.

"If we can go from 3 to 4% in a field of 10 to 12 candidates, that could make a difference," Sailor said. "After this election, whether we see huge turnout or not, you will see a change in the way Republicans engage the African-American community. Victory for this campaign is changing the discourse."