Washington (CNN) The chair of the House Democratic Caucus is standing by his remarks referring to President Donald Trump as "the grand wizard of 1600 Pennsylvania Avenue" but declined to call him a racist Wednesday.

"Do you regret going that far in your comments on Monday?" CNN's Alisyn Camerota asked New York Democratic Rep. Hakeem Jeffries on "New Day."

"Not at all," he replied, later adding, "We've got to have an opportunity for at least one day a year to have a candid, if sometimes uncomfortable, conversation about race."

During a Martin Luther King Jr. Day event on Monday, Jeffries told the crowd, "These are challenging times in the United States of America -- we have a hater in the White House, a birther in chief, the grand wizard of 1600 Pennsylvania Avenue."

Jeffries told CNN Wednesday that "of course, I don't believe that the President is a card-carrying member of the KKK."