WASHINGTON (Reuters) - Senior U.S. Republicans distanced themselves on Sunday from Donald Trump’s comments about a Mexican-American judge, saying they were worried that the tone of his presidential campaign could enrage Latinos, who are a growing U.S. voting bloc.

Trump has accused U.S. District Judge Gonzalo Curiel of harboring a bias against him in lawsuits involving fraud allegations against Trump University, the New York business man’s now-defunct real estate school.

The presumptive Republican nominee has suggested Curiel’s Mexican-American heritage had influenced the judge’s opinion because of Trump’s campaign pledge to build a wall on the U.S.-Mexico border.

Senate Majority Leader Mitch McConnell said he “couldn’t disagree more” with Trump’s comments about the judge.

“I am concerned about the Hispanic vote, America is changing,” McConnell told “Meet The Press”. “I think it’s a big mistake for our party to write off Latino Americans. I am concerned about that and I hope he will change his direction on that.”

Democrats have accused Trump of racially tinged rhetoric about Latinos, including his description of Mexican immigrants as “criminals and rapists” in the speech he gave a year ago launching his campaign.

Such rhetoric has exacerbated friction between Trump and Republican party leaders such as McConnell and House of Representatives Speaker Paul Ryan. For the past few years, the party has been trying to broaden its appeal with Latino voters and senior Republicans are concerned that Trump’s comments could cost the party votes, not only in the presidential race but in congressional races as well.

Former House of Representatives Speaker Newt Gingrich, who has been mentioned as a possible running mate to Trump, called Trump’s comments about the judge “inexcusable.”

“This is one of the worst mistakes Trump has made,” Gingrich told Fox News.

Republican U.S. presidential candidate Donald Trump arrives to a campaign rally in Redding, California, U.S. June 3, 2016. REUTERS/Stephen Lam

Curiel was born in Indiana to Mexican-immigrant parents.

“He is a member of a club or society very strongly pro Mexican, which is all fine. But I say he’s got bias. I want to build a wall,” Trump said in an interview on Sunday on “Face the Nation”.

Asked if he believed a Muslim judge would be biased against him based on Trump’s call for a temporary ban on Muslims entering the country, Trump replied, “It’s possible. Yes.”

McConnell said America is a nation full of immigrants -- pointing out that his wife, former Labor Secretary Elaine Chao, came to the United States when she was eight and didn’t speak English.

“All of us came here from somewhere else,” McConnell said.

Senator Bob Corker, who chairs the Senate Foreign Relations Committee, said on “ABC’s This Week” that he did not “condone the comments” that Trump made about Curiel.

Legal scholars on the right and the left have criticized Trump for attacking the judge, saying it could harm judicial independence should he become president.

Former Attorney General Alberto Gonzales, who is of Mexican descent, was one of the few Republicans to defend Trump. Gonzales argued in a column in The Washington Post that Trump should be allowed to question a judge’s fairness, saying that questioning a judge is crucial to ensuring public trust in the courts.

But when asked about the racial element in an interview on Sunday on Fox News, Gonzales criticized Trump.

“I certainly would have taken a different approach,” he said. “Whenever you say something about a judge’s nationality, I think it demeans the office and hurts the judiciary as a whole.”