Charlottesville, Virginia(CNN) Daryl Davis' home is filled with memories of the days he has spent with the Ku Klux Klan.

He is not a member of the white supremacist organization. He can't be. Davis is the descendant of slaves. He's a blues musician who has learned how to lift hate out of hearts, even from those who in other times might have been hell-bent on killing him or anyone who looked like him.

You could consider Davis a KKK whisperer, a man who's spent decades talking to Imperial Wizards, Grand Dragons and rank-and-file Klansmen.

Daryl Davis looks at his collection of KKK robes, given up by Klansmen he has befriended.

His questions started as a 10-year-old in the 1960s, when he was confronted by racism in the Belmont suburb of Boston and wondered "How can you hate me? You don't even know me."

As a man, Davis took that question directly to Klansmen, and some women, too.

Sometimes, it can get ugly. Davis, now 59, has been kicked and attacked. But mostly he listens. Even as some people spew hate. He listens. Thinks. And responds.

Occasionally, Davis is the first black person they've ever spoken to.

Thirty years of these meetings has left him hopeful, not hateful. His closet is filled with dozens of KKK robes and memorabilia given to him by those whom he has inspired to leave the Klan. Not because he demanded it, cajoled or threatened them. But, he says, because they learned from him.

"They're done, they're done," he says of the men who've given up the robes they used to wear so proudly. "As a result of meeting me and having these conversations, not overnight, but over time."

This week, Davis drove to Charlottesville, Virginia, where racism turned violent this summer when white supremacists clashed with counterprotesters, leading to one death and several injuries.

Heather Heyer, 32, was killed last August when a car slammed into a crowd of people protesting against hatred.

He was to meet the Imperial Wizard of the Rebel Brigade Knights, Billy Snuffer. He was in Charlottesville with other Klansmen to attend the hearing of a colleague charged with shooting a gun during the "Unite the Right" rally that was also where James Fields is accused of killing Heather Heyer and injuring many more.

"What's going on man, how you doing?" Davis asks as Snuffer extends a hand. The two men embrace. One the descendant of slaves, the other a proponent of the Confederate South who thinks the mixing of races is against God's will.

Imperial Wizard Billy Snuffer of the Rebel Brigade Knights, left, discusses the Klan, Nazis and hate with Daryl Davis.

Snuffer wears a cap with the Confederate flag on it, along with the phrase "Heritage not hate."

They chat in the courthouse square of this college town. Snuffer has his stance -- the Klan, he says, is not racist, though it has some bad actors and a bad history. Now, he's trying to take the organization back to its roots, and he has no time for neo-Nazis or skinheads.

Davis listens, despite knowing the vile and violent history of the Klan and its penchant for hanging, raping and terrorizing black people in America.

He pushes back gently as he tries to steer the conversation toward shared ground on this, his second meeting with Snuffer. He credits Snuffer for at least talking with him, a black man.

"As Americans, your Confederate history is as much a part of my history as my black history is a part of yours," Davis says.

"Sure it is," Snuffer agrees.

His point is that he wants to honor the history and heritage -- and to defend monuments of Confederate leaders such as Gen. Thomas "Stonewall" Jackson, whose statue is now shrouded in Charlottesville.

As they talk, something unusual happens. Both men realize they share the same feeling about the "Unite the Right rally." Both say it had nothing to do with saving Confederate statues.

"This was all about starting a race war," Davis says. Snuffer replies: "You're exactly right."

Snuffer and Davis are an odd and highly visible pair, standing at the site of one of the flashpoints of the confrontation in Charlottesville. And they attract attention.

Protesters shout down the conversation between Davis and Snuffer.

"F**king a**holes over here trying to make apologies for the legacy of racism," a woman yells at them. "You f**king a**hole. Do not come into our city and try to rewrite history. We know what this history is."

Davis tries to defuse the situation and engage the woman. He wants her to know where he's coming from. Why he's talking with a Klansman.

She can't stand it. It's too raw. The fear and pain felt by some residents is still very real. Emotions still run extremely high.

Davis wants to explain. He wants to tell her about his work, the robes that show how he has converted Klansmen. How he's trying to find middle ground. It fails.

"Get the f**k out of Charlottesville," the woman shouts.

"You're a person of color. Why the hell are you talking to them?" she asks.

"That's right, I am a person of color," Davis responds. "But I'm also an American and so is this man right here." Suddenly, he is defending an Imperial Wizard of the KKK.

The loss of Heather Heyer still seems fresh and raw in Charlottesville.

This isn't the first time Davis has gotten criticism from another person of color for talking to the KKK. Years ago, an NAACP branch member derided him for what he was doing.

But for change to happen, Davis believes you have to recognize the similarities, even with people who hate you.

A small crowd gathers now, surrounding Davis and Snuffer. Police approach as the shouting continues and separate the groups.

Davis and Snuffer head to the courthouse for a hearing related to the violence in Charlottesville last August.

The bluesman stays with the Klansman, continuing the conversation, perhaps finding some brotherhood in them both being targets of the vitriol as they wait to go into court for the hearing.

Davis has an idea for one of the cases being heard here today. He wants to suggest to the judge that he take the man accused of shooting into the air near a school to the National Museum of African American History and Culture in D.C., in the way that courts sometimes mandate education for a first drunk-driving offense.

"Why not send them to learn something about the very people that they profess to hate or they profess to be inferior?" Davis asks.

It is all part of the conversation and bridge Davis is hoping to build. He knows he will not change Snuffer overnight. And he realizes some residents of Charlottesville may not be ready to see a black man trying to make sense of what the Klan has to say for itself.

But Davis says he hopes and believes Snuffer will go home and reflect on the day. That he will remember Davis was screamed at as much as he himself was. How Davis tried to defend him, to find a way to have a conversation. That he sat in court for two hours alongside him.

As the sun sets in Charlottesville, Davis heads home, hoping he has planted a seed that will one day change a mind.

On his drive, Davis' phone rings. It's the Imperial Wizard calling to make sure the musician has gotten home safely. He had just witnessed a black man stand up for him against protesters. A caring call. Perhaps the seed was sprouting.

"So it begins," Davis says.