Washington (CNN) Top Republican officials and donors are increasingly worried about the threat Donald Trump's attack on a judge's Mexican heritage could pose to their party's chances in November -- and about the GOP's ability to win Latino votes for many elections to come.

Trump is under fire for repeatedly accusing U.S. District Judge Gonzalo Curiel, who is overseeing a lawsuit involving Trump University, of bias because of his Mexican heritage. Those concerns intensified Sunday after Trump said he would have the same concerns about the impartiality of a Muslim judge.

House and Senate GOP leaders have condemned Trump's remarks about Curiel, while donors have openly worried that losing Latino voters could doom them in key down-ballot races. Other important party figures, including former Speaker Newt Gingrich, are urging Trump to change his combative, confrontational style before it's too late.

Veteran Republican strategist Rick Wilson warned this weekend that GOP leaders who have endorsed Trump "own his politics."

"You own his politics," Wilson wrote in a column for Heatstreet , adding later, "You own the racial animus that started out as a bug, became a feature and is now the defining characteristic of his campaign. You own every crazy, vile chunk of word vomit that spews from his mouth."

The GOP's deepest fear: A Barry Goldwater effect that could last far longer than Trump's political aspirations.

Goldwater, the Arizona senator who was the 1964 GOP nominee and a leader of the conservative movement, alienated a generation of African-American voters by opposing the Civil Rights Act -- opening the door for Democrats to lock in their support for decades. Republicans fret that Trump could similarly leave a stain with Latino voters.

'Concerned'

"I am concerned about that," Senate Majority Leader Mitch McConnell, R-Kentucky, said Sunday.

"America is changing. When Ronald Reagan was elected, 84% of the electorate was white," McConnell said on NBC's "Meet the Press." "This November, 70% will be. It's a big mistake for our party to write off Latino Americans. And they're an important part of the country and soon to be the largest minority group in the country."

"I hope he'll change his direction on that," said McConnell, who first made the Goldwater comparison last week in an interview with CNN's Jake Tapper

That hasn't happened yet. In interviews Sunday, Trump wouldn't back away from his assertion that Curiel's parents' birth in Mexico has left the judge angry over Trump's proposal to build a wall along the U.S.-Mexico border and biased in the legal case over Trump University. Trump even went further, saying on CBS' "Face the Nation" that he'd have similar concerns over a Muslim judge, since he has proposed banning all Muslims from entering the United States.

Trump's remarks led to condemnations from the same leading Republicans that in recent weeks have embraced him -- and accepted that the party's fate in November is inextricably linked to his.

"I don't agree with what he had to say," McConnell said.

"This is a man who was born in Indiana," McConnell said of Curiel. "All of us came here from somewhere else. Almost all Americans are either near-term immigrants like my wife, who came here at age 8 not speaking a word of English, or the rest of us whose ancestors were risk-takers who came here and made this country great. That's an important part of what makes America work."

House Speaker Paul Ryan, just a day after announcing his endorsement of Trump, bashed him on a Wisconsin radio station

"Look, the comment about the judge, just was out of left field for my mind," Ryan said Friday on WISN in Milwaukee. "It's reasoning I don't relate to, I completely disagree with the thinking behind that."

The criticism from McConnell and Ryan was predictable: Both preside over GOP majorities that are threatened thanks to competitive races in Latino-heavy states like Arizona, Nevada and Florida.

More surprising was the condemnation from Gingrich, who has transparently jockeyed for a spot on Trump's ticket.

"I don't know what Trump's reasoning was, and I don't care," Gingrich told The Washington Post . "His description of the judge in terms of his parentage is completely unacceptable."

Gingrich was even sharper on "Fox News Sunday," calling Trump's remarks "inexcusable."

Trump responded to Gingrich's critique on Monday, telling "Fox and Friends" that the former House Speaker's comments were "inappropriate."

'One of the worst mistakes'

"This is one of the worst mistakes Trump has made," Gingrich said.

Tennessee Sen. Bob Corker, the Foreign Relations Committee chairman who has provided key Republican support for Trump's foreign policy stances and is also often named as a prospective vice presidential candidate, rebuked Trump's comments about the judge on ABC's "This Week."

"I think that he's going to have to change," Corker said of Trump's overall behavior and campaign tactics.

Sen. Ben Sasse, a Nebraska Republican who has been outspoken about his opposition to Trump, tweeted Monday: "Public Service Announcement: Saying someone can't do a specific job because of his or her race is the literal definition of "racism."

Public Service Announcement:

Saying someone can't do a specific job because of his or her race is the literal definition of "racism." — Ben Sasse (@BenSasse) June 6, 2016

Trump's campaign downplayed the impact of his assertion that the judge's Mexican heritage could preclude him from delivering fair rulings in the Trump University case.

A Trump official said the remarks are "no reason to celebrate, (but) no reason to panic" -- an indication there is concern inside the campaign but Trump's aides don't believe it's damaging long-term.

Another campaign adviser laughed when asked if Trump officials can talk to the candidate about watching what he says.

Alberto Gonzales , who led the Justice Department under President George W. Bush from 2005 to 2007, wrote in a Washington Post op-ed Saturday that Curiel's Mexican heritage shouldn't be enough to disqualify him from overseeing the case. But, Gonzales said, Trump is entitled to a fair trial, and the appearance of impropriety could be enough for him to reasonably request that Curiel recuse himself.

Trump thanked Gonzales for his support.

Inside the Republican Party, campaigns and donor circles, fear over the damage Trump's remarks could do to the party's relationship with Latino voters was palpable.

"Awful," a top Republican official said of Trump's attack on the judge. "We are all beside ourselves."

The official went on to say that "you have to feel for Paul Ryan," who had just announced his support for Trump.

Depth of concerns

In a series of interviews with donors, fundraisers and congressional officials, the depth of the concerns about what Trump's latest attacks underscore become clear.

"Honestly? My worst fear. Call me stupid -- I was one of the guys who figured he'd do the whole pivot thing," said one donor, referring to an often-used strategy of moving more to the middle after securing the nomination.

The donor, who had been active for several candidates during the primary, said he was "ready to get in line" once Trump signed the joint-fundraising agreement last month with the RNC. The bold names associated with the joint agreement -- people like businessman Woody Johnson -- were enough of a sign, the donor said.

Now? "Not so much."

But it may be bigger than that, according to several GOP officials. Republicans are defending 24 seats in the Senate while holding a slim four-seat majority. While the House majority is significantly more robust -- 58 seats -- there are members in that chamber who saw their seats move into riskier positions the day Trump locked up the nomination.

The solution -- one that top GOP officials on Capitol Hill have been repeating in the weeks since -- has been to make sure top donors dump cash into the down ballot races.

Up to this point, they've done just that. One fundraiser with ties to one of the two primary GOP congressional super PACs said donors have been "burning up the phone lines" trying to figure out how to help protect GOP majorities in Congress.

The primary Senate GOP super PAC, the Senate Leadership Fund, had more than $16.3 million on hand at the end of April, the last time numbers were reported with the FEC. The group raised more than $4 million in March and April alone -- a number that, according to the fundraiser, will increase "significantly" in the months ahead.

The top House super PAC, the Congressional Leadership Fund, nearly doubled its 2015 fundraising in the first quarter of 2016 alone.

"The concern is -- do we get to the point that all the money in the world doesn't matter?" asked another donor, who said his whole goal this cycle was to protect House and Senate candidates. "We're obviously not there right now, but stupid s--- like this really makes you wonder."

Democrats are certainly trying to make each Trump comment sting. The party's House and Senate campaign committees are firing out a steady clip of press releases attempting to tie each vulnerable candidate to Trump. Democrats make clear those comments will be featured heavily in the fall in attack ads.

Perhaps more noticeably, over the weekend, talks between top GOP figures about the future of the party have become more urgent. Several Republican officials pointed to McConnell's comments to Jake Tapper on CNN last week, where he first voiced concern about Trump's effect on Latino voters mirroring that of Goldwater's effect on black voters.

Yet those same officials watched McConnell go to great lengths not to say that Trump's attacks on the judge in the Trump University case were racism.

"That was just painful," said one Republican official who served in George W. Bush's administration. The official added that the reality is McConnell -- and Ryan and every Republican in a leadership position or facing an election challenge -- "will be stuck dealing with the latest Trumpism every interview of every day, of every month until November."

CORRECTION: An earlier version of this story misstated which committee Sen. Bob Corker is the chairman of. It is the Senate Foreign Relations Committee.