WASHINGTON -- "Not all" Republicans are racist, said Rep. Steve Israel (D-N.Y.) on Sunday, but "to a significant extent, the Republican base has elements that are animated by racism, and that's unfortunate."

Israel's comment was in response to a question from CNN's Candy Crowley, who asked the chairman of the Democratic Congressional Campaign Committee about remarks by Attorney General Eric Holder this week. In a speech to a civil rights group, Holder questioned his treatment by Republican lawmakers at a House Judiciary Committee hearing, and implied that race may have played a role.

House Minority Leader Nancy Pelosi (D-Calif.) also suggested this past week that racism was a factor in the Republican party's opposition to immigration reform. "I think race has something to do with the fact that they're not bringing up an immigration bill," Pelosi told reporters, adding, "I've heard them say to the Irish, if it were just you, this would be easy."

Israel didn't directly comment on Pelosi's remarks, pivoting instead to the difficulty of passing a comprehensive immigration reform package in the face of unified Republican opposition. "All we need is 20 Republicans, just 20 to vote for that bill, and it will be law, and we don't have to have this debate anymore," Israel said.

The debate over immigration reform threatens to divide the Republican party and weaken their electoral chances in November and in the 2016 presidential race. Appearing on CNN with Israel, Rep. Greg Walden, the chairman of the National Republican Congressional Committee, said Israel's comments about race and his party and immigration were "wrong and unfortunate," but then quickly transitioned to less thorny topics.