(CNN) An unapologetic Joe Biden defended his remarks about working with segregationist senators on Wednesday, telling reporters that "there's not a racist bone in my body."

"Apologize for what?" Biden told CNN when asked about Booker's call outside a fundraiser Wednesday evening in Rockville, Maryland.

"Cory should apologize," he said. "He knows better. There's not a racist bone in my body. I've been involved in civil rights my whole career."

Biden's brief remarks were his first comments following a controversy that erupted after he pointed to two late segregationist senators as examples of colleagues he could work with during an era where "at least there was some civility" in the Senate.

At the fundraiser Tuesday, Biden recalled being a member of the Senate in the 1970s with Southern Democrats who opposed civil rights and desegregation. He specifically named Sens. James Eastland of Mississippi and Herman Talmadge of Georgia, who Biden called "one of the meanest guys I ever knew."

"I was in a caucus with James O. Eastland. He never called me 'boy,' he always called me 'son,' " Biden told donors.

"Well, guess what? At least there was some civility. We got things done. We didn't agree on much of anything. We got things done. We got it finished," Biden said. "But today, you look at the other side and you're the enemy. Not the opposition, the enemy. We don't talk to each other anymore."

Speaking to reporters on Wednesday evening, Biden said he "could not have disagreed with Jim Eastland more" and that he "ran for the United States Senate because I disagreed with the views of the segregationists -- there were many of them in the Senate at the time."

"The point I'm making is you don't have to agree. You don't have to like the people in terms of their views, but you just simply make the case and you beat them. You beat them without changing the system," he said.

Booker shot back in an interview with Don Lemon on "CNN Tonight" later Wednesday evening.

"I was raised to speak truth to power," Booker said. "And that I will never apologize for doing that and Vice President Biden shouldn't need this lesson."

"Somebody running for president of the United States, somebody running to be the leader of our party, should know that using the word 'boy' in the way he did can cause hurt and pain. We need a presidential nominee and leader of our party to be sensitive to that," Booker added.

At a fundraiser Wednesday night, Biden brought up Eastland and Talmadge again -- but this time did not tout the civility of the era in which the two Southern segregationists were his Senate colleagues.

Instead, Biden pointed to the late Massachusetts Sen. Ted Kennedy, a progressive icon, helping him land a spot on the Senate Judiciary Committee.

"And we had to put up with the likes of like Jim Eastland and Hermy Talmadge and all those segregationists and all of that. And the fact of the matter is that we were able to do it because we were able to win -- we were able to beat them on everything they stood for," Biden said. "We in fact detested what they stood for in terms of segregation and all the rest."

Biden's Democratic presidential rivals sharply criticized his remarks Wednesday.

"You don't joke about calling black men 'boys.' Men like James O. Eastland used words like that, and the racist policies that accompanied them, to perpetuate white supremacy and strip black Americans of our very humanity," Booker said in a statement.

"Vice President Biden's relationships with proud segregationists are not the model for how we make America a safer and more inclusive place for black people, and for everyone. I have to tell Vice President Biden, as someone I respect, that he is wrong for using his relationships with Eastland and Talmadge as examples of how to bring our country together," he added. "And frankly, I'm disappointed that he hasn't issued an immediate apology for the pain his words are dredging up for many Americans. He should."

"To coddle the reputations of segregationists, of people who if they had their way I would literally not be standing here as a member of the United States Senate, is -- I think it's just it's misinformed," Sen. Kamala Harris told reporters.

"To suggest that individuals who literally made it their life's work to take America back on the issue of race is a real problem for me," the California Democrat said.

Other leading contenders were similarly critical Wednesday evening.

"I'm not here to criticize other Democrats, but it's never OK to celebrate segregationists. Never," said Sen. Elizabeth Warren of Massachusetts.

Former Rep. Beto O'Rourke of Texas said in an interview with "Pod Save America" that Biden's remarks were not a good example of the civility the former vice president was calling for.

"For the vice president to somehow say that what we're seeing in this country today is a function of partisanship, or a lack of bipartisanship, completely ignores ... the legacy of slavery and the act of suppression of African Americans and communities of color right now," O'Rourke said.