(CNN) Blame President Trump for his tepid moral response. Call the neo-Nazis and white nationalists thugs. Fill your Facebook and Twitter accounts with moral outrage.

But the tragedy that took place in Charlottesville this month could not have occurred without the tacit acceptance of millions of ordinary, law-abiding Americans who helped create such a racially explosive climate, some activists, historians and victims of extremism say.

It's easy to focus on the angry white men in paramilitary gear who looked like they were mobilizing for a race war in the Virginia college town. But it's the ordinary people -- the voters who elected a reality TV star with a record of making racially insensitive comments, the people who move out of the neighborhood when people of color move in, the family members who ignore a relative's anti-Semitism -- who give these type of men room to operate, they say.

Riot police form a line of defense in front of the statue of Confederate Gen. Robert E. Lee in Emancipation Park, recently renamed from Lee Park.

Riot police form a line of defense in front of the statue of Confederate Gen. Robert E. Lee in Emancipation Park, recently renamed from Lee Park.

Counterprotesters line the route taken by white nationalists and neo-Nazis during the "Unite the Right" rally. After clashes with anti-fascist protesters and police, the rally was declared an unlawful gathering and people were forced out of Emancipation Park, formerly called Lee Park and home to a controversial statue of Confederate Gen. Robert E. Lee.

Counterprotesters line the route taken by white nationalists and neo-Nazis during the "Unite the Right" rally. After clashes with anti-fascist protesters and police, the rally was declared an unlawful gathering and people were forced out of Emancipation Park, formerly called Lee Park and home to a controversial statue of Confederate Gen. Robert E. Lee.

A woman is treated for exposure to pepper spray during clashes between white nationalists and counterprotesters at Emancipation Park.

A woman is treated for exposure to pepper spray during clashes between white nationalists and counterprotesters at Emancipation Park.

White nationalists, neo-Nazis and members of the alt-right movement exchange volleys of pepper spray with counterprotesters as they enter Emancipation Park.

White nationalists, neo-Nazis and members of the alt-right movement exchange volleys of pepper spray with counterprotesters as they enter Emancipation Park.

A white nationalist is seen with a cut below his eye suffered during clashes with counterprotesters at Emancipation Park .

A white nationalist is seen with a cut below his eye suffered during clashes with counterprotesters at Emancipation Park .

A counterprotester uses a lighted spray can against a white nationalist at the entrance to Emancipation Park.

A counterprotester uses a lighted spray can against a white nationalist at the entrance to Emancipation Park.

A counterprotester strikes a white nationalist with a baton during clashes at Emancipation Park, where white nationalists are protesting the removal of the Confederate Gen. Robert E. Lee monument.

A counterprotester strikes a white nationalist with a baton during clashes at Emancipation Park, where white nationalists are protesting the removal of the Confederate Gen. Robert E. Lee monument.

A woman receives first aid after a speeding car slammed into this silver convertible as it navigated through a crowd of counterprotesters.

A woman receives first aid after a speeding car slammed into this silver convertible as it navigated through a crowd of counterprotesters.

People fly into the air as a vehicle drives into a group of people demonstrating against a white nationalist rally after police cleared Emancipation Park in Charlottesville, Virginia, on Saturday, August 12.

People fly into the air as a vehicle drives into a group of people demonstrating against a white nationalist rally after police cleared Emancipation Park in Charlottesville, Virginia, on Saturday, August 12.

That was the twisted formula that made the Holocaust and Rwanda possible and allowed Jim Crow segregation to survive: Nice people looked the other way while those with an appetite for violence did the dirty work, says Mark Naison, a political activist and history professor at Fordham University in New York City.

''You have to have millions of people who are willing to be bystanders, who push aside evidence of racism, Islamophobia or sexism. You can't have one without the other,'' Naison says.

"We are a country with a few million passionate white supremacists -- and tens of millions of white supremacists by default," he says.

Many people prefer to focus on the usual suspects after a Charlottesville happens -- the violent racial extremists who are so easy to condemn. Yet there are four types of ordinary people who also play a part in the country's racial divisions, Naison and others say:

No. 1: The 'down-low' segregationists

Many of the white racists who marched in Charlottesville were condemned because they openly said they don't believe in integration or racial equality.

But millions of ordinary white Americans have been sending that message to black and brown people for at least a half a century.

They send it with their actions: They don't want to live next to or send their children to school with black or brown people, historians say.

Busing, a nationwide campaign to end school segregation by shipping students of color to white schools, collapsed in large part because of fierce opposition by white parents. "White flight" -- white families fleeing city neighborhoods after people of color moved in -- helped create the modern suburbs.

Think only white supremacists oppose integration? These ordinary white Boston parents protested busing in 1974.

This isn't the Jim Crow segregation that one reads about in the history books. It's the covert or "down-low" segregationist movement that has shaped much of contemporary America since overt racism became taboo in the 1960s, says David Billings, who wrote about growing up white in the segregated South in his memoir, "Deep Denial: The Persistence of White Supremacy in United States History and Life."

"Across the country, white people withdrew from the 'public' sphere and migrated to 'whites only' suburbs to evade racial integration," Billings wrote. "The word 'public' preceding words like 'housing,' 'hospital,' 'health care,' 'transportation,' 'defender,' 'schools,' and even 'swimming pool' in some parts of the country became code words that meant poor and most often black and Latino. The word 'private' began to mean 'better.'''

This white separatism continues today. Whites move out so often when nonwhites move in that sociologists have a name for the phenomenon. It's called "racial tipping."

This separation also occurs in the private lives of many white Americans, according to one pollster. In 2013, Robert Jones, CEO of the Public Religion Research Institute, made a splash after conducting a survey where he found that 75% of whites in the United States didn't have a single person of color in their social circle -- they only had white friends.

We are a country with a few million passionate white supremacists -- and tens of millions of white supremacists by default. Mark Naison, a political activist and history professor at Fordham University

Jones polled a complex subject. Many people of color self-segregate as well, and some American neighborhoods are so segregated that residents never come in contact with people of other racial or ethnic groups.

Yet some white Americans are driven by the same impulses that drove some of the white racists in Charlottesville -- racial separation.

"White people in the past century and a half have made a conscious effort to resegregate themselves," says Edward Ball , author of "Slaves in the Family," a memoir about coming to terms with learning his family owned slaves.

"We have to work hard to make our social lives reflect our values, because white people do not choose the company of people of color generally," he says.

Ball once wrote that "unconsciously or inadvertently, all of us white folks participate in forms of supremacist thought and activity."

The angry white men in Charlottesville were just being open about their white supremacy. Ball says he wasn't surprised by their boldness.

"Their climate is now better for them," he says.

No. 2: Those who say 'yes, but...'

President Trump's critics blasted him for not coming out strong enough against the white racists who marched in Charlottesville. Trump initially denounced the "egregious display of hatred, bigotry, and violence on many sides." It was the "many sides" qualifier that infuriated some people. They wanted an unequivocal denunciation of racism from a leader.

Trump's "many sides" response, though, wasn't that abnormal in the context of US history. It used to be the norm for white political leaders to draw a moral equivalence between racists and those who suffered from their acts of brutality, historians say.

It's the "yes, but" rhetorical maneuver -- condemn racism but add a qualifier to diminish the sincerity of what you just said.

White families fleeing people of color helped create suburban America. It's called "white flight."

The Rev. Martin Luther King Jr. ran into this "yes, but" response so much that he wrote about it in his "Letter from Birmingham Jail."

He wrote:

"I have almost reached the regrettable conclusion that the Negro's great stumbling block in his stride toward freedom is not the White Citizens' Counciler or the Ku Klux Klanner, but the white moderate who is more devoted to 'order' than to justice; who prefers a negative peace, which is the absence of tension, to a positive peace, which is the presence of justice; who constantly says: 'I agree with you in the goal you seek, but I cannot agree with your methods of direct action...'"

President Dwight Eisenhower took the "yes, but" approach when he complained he couldn't move too fast to comply with the Supreme Court's decision to integrate schools because people had to respect the Southern way of life, says Carol Anderson, author of "White Rage" and a professor of African-American studies at Emory University in Atlanta.

A recent Washington Post article gave other examples: When Southern governors like Orval Faubus of Arkansas and Earl Long of Louisiana were pressured in the 1950s to end segregation, they called both the NAACP and the White Citizens' Councils, a rabid segregationist group, "extremists."

The Rev. Martin Luther King Jr. once said moderate whites who refused to take a stand on racial justice angered him more than the KKK.

"You get that equivocation," says Anderson, "that trying to make a system that absolutely strips people of their humanity on par with people demanding their humanity."

That "yes, but" approach is often used today to discredit the grievances of the Black Lives Matter movement, another professor says. Whenever an unarmed black or brown person is shot by police, some deflect the issue by saying, "Yes, but all lives matter."

"When a police officer shoots an unarmed black person, even then it's controversial to say racism is a factor," says Erik Love, a sociologist at Dickinson College in Pennsylvania. "We say, 'Why don't we talk about these other issues. What about the crime rate, what police officers need to protect themselves.' And suddenly we're not talking about race anymore."

No. 3: Those who choose chaos

There's a famous line from the classic film, "Casablanca." A police officer is closing down a casino, declaring, "I'm shocked -- shocked -- to find that gambling is going on in here!" -- all while pocketing his casino winnings as they're being handed to him on the sly.

That line could apply to Trump supporters who say they're frustrated by the President's statements on race since Charlottesville erupted.

How could you be shocked?

"This is who he is, this is what he does," says Anderson, the Emory University professor. "'Mexicans are rapists and criminals.' That's what he said in his first speech. Their complicity comes in the form of self-denial instead of owning it."

For those who say they voted for Trump despite his intolerance, Anderson offers this analogy: Minister Louis Farrakhan.

Farrakhan is a leader in some parts of the black community because of his message of self-help and black empowerment. He reached peak popularity in the 1990s, but he also preached anti-Semitic, anti-white, anti-Catholic and anti-homosexual rhetoric . And the organization he leads, the Nation of Islam, has taught that white people are inherently evil.

Fordham professor Mark Naison says America is at a scary place now where open, communal warfare could erupt.

"If he was running for office and black people voted in droves for him, the narrative would be, 'They're supporting a racist,''' she says.

Ta-Nehisi Coates, an acclaimed writer on race relations, made a similar argument after Trump's election in an interview with Vox's Ezra Klein , where he responded to commentators who said not all white voters who supported Trump endorsed all of his ideas.

"As my buddy said, is that what you said to the followers of Louis Farrakhan? No, nobody says that to the followers of Louis Farrakhan. No, they blasted him as an anti-Semite, which he is, and say, 'how can people follow this bigoted message?' That's the ultimate testament -- that you could be Donald Trump and be President. There is no black person who could have the kind of vices Donald Trump has and, hell, be governor. Maybe you could be mayor somewhere."

Many voters knew Trump would bring something else to the Oval Office -- chaos. That's why they chose him. He's their first reality TV president, one writer says.

Many voted for Trump because they liked the persona he cultivated as the star of "The Apprentice." Reality TV rewards characters who say rude and reprehensible things, characters are often cast as racial stereotypes, and those who provoke the most chaos get the most attention, says Joy Lanzendorfer, author of the Vice article, "How Reality TV Made Donald Trump President."

"He would say horrible things about people, act out and break the rules, but people weirdly respected it," she says. "They said he was a winner, and that's how a winner wins."

It's not, however, how many would want a nation's leader to handle a racial crisis.

No. 4: Those who look the other way

Ari Kohen knows something about the cost of hate. When he looked at images of neo-Nazis chanting "Jews will not replace us!" in Charlottesville, he thought of his grandfather, Zalman Kohen. He was living in rural Romania in 1944 when the Nazis rounded him up with the help of his neighbors and sent him to a death camp.

This is who he is, this is what he does. 'Mexicans are rapists and criminals.' That's what he said in his first speech. Their complicity comes in the form of self-denial instead of owning it. Carol Anderson, author of "White Rage," on Trump voters who expressed dismay at his conduct after Charlottesville

His grandfather survived, moved to the United States and lived until he was 90. But he never returned to Romania, says Kohen, an associate professor of political science at the University of Nebraska-Lincoln.

"He could never forgive his neighbors," he says. "These were people who, maybe they didn't love Jews, but these were people who lived next to each other. They knew his family and he knew their family. The idea that they could all stand by while life was completely and forever changed for large portions of their community -- he could never understand it."

Many scholars have been vexed by the same question. When they examine genocidal events like the Holocaust, many come to the same conclusion:

Never underestimate the ability of ordinary people to look away.

Some do it with family members. Kohen says the hundreds of white racists who descended on Charlottesville must have family or friends who noticed their behavior beforehand. He suspects that some refused to confront them.

"There's this wink and nod, everyone knows that this person is going down a dangerous path and people passively go along with it," he says. "They don't want to rock the boat. This is family or a friend. It's hard to distance yourself from people you care about."

When ordinary people refuse to speak out, a country can descend into violence -- as it did in Northern Ireland.

This passivity extends to how people react when their country's leaders become intolerant, others say. Once you see it coming, you have a duty to act, says Naison, the activist and Fordham professor.

"If you don't speak up when this sort of ideology is being promoted at the highest level, you end up being complicit in the actions taken by its more extreme adherents," Naison says. "Once the demons are unleashed, you've become a co-conspirator."

Naison says he doesn't think most Americans realize how dangerous it is in their country right now. He's warned people who voted for Trump.

"I told these guys, you can't control this; you're playing with fire," Naison says. "Open, violent communal warfare is scary. You can't control it. Look at what happened in the Balkans, Northern Ireland, Israel."

There's also evidence, though, that millions of ordinary Americans from all walks of life don't want that kind of America. Heather Heyer, the demonstrator who lost her life in Charlottesville, was a young white woman who marched in solidarity with black protesters. Millions of Americans have since taken to the streets or social media to stand against what happened there.

Former President Barack Obama even weighed in with a photo and quote that's become the most liked tweet ever on Twitter.

Obama quoted Nelson Mandela, the South African leader who knew something about hate and reconciliation. In his 1994 autobiography "Long Walk to Freedom," Mandela wrote:

"No one is born hating another person because of the color of his skin or his background or his religion. People must learn to hate, and if they can learn to hate, they can be taught to love, for love comes more naturally to the human heart than its opposite."

Yet to get to that place Mandela talked about, it may be necessary to not just look at the usual suspects people condemn when racial violence spills into public view.

If you want to know why those white racists now feel so emboldened, it may help to look at all the ordinary people around you, your neighbors, your family members, your leaders.

But first, start by looking at yourself.