The former Republican presidential nominee Mitt Romney has said he will consider voting for the Libertarian party in the presidential election instead of Donald Trump, in a stark example of how far the GOP establishment is from accepting the billionaire leading its bid for the White House.

Romney, who lost to Barack Obama in the 2012 presidential election, even said the decision to vote against his party would be “very easy” if Bill Weld, another former Massachusetts governor who has fundraised for Romney, was at the top of the Libertarian ticket.

“If Bill Weld were at the top of the ticket, it would be very easy for me to vote for Bill Weld for president,” Romney told CNN’s Wolf Blitzer on Friday. The Libertarian nominee is currently former Republican governor of New Mexico Gary Johnson, with Weld as his vice-presidential pick.

Romney, who is hosting his annual ideas retreat in Utah, where speaker of the house Paul Ryan and Republican national committee chair Reince Priebus are among the attendees, added he was going to try to “get to know Gary Johnson better and see if he’s someone who I could end up voting for. That’s something which I’ll evaluate over the coming weeks and months”.

On Saturday Trump responded to Romney’s criticism through his usual medium, Twitter. “Mitt Romney had his chance to beat a failed president but he choked like a dog,” Trump wrote. “Now he calls me racist – but I am least racist person there is.”

The news came at the culmination of a week of bad news for the Trump, who spent much of the week battling his own party over his racially charged comments. Under pressure to appear more presidential, Trump responded to primary results on Tuesday with a victory speech delivered via a teleprompter – a device he has spent months maligning his rival politicians for using.

“I will make you proud of the party,” Trump promised. “For those who voted for someone else, in either party, I will work hard to earn your support.”

Senior party figures, especially those facing close races in November, appear to be formally distancing themselves from the Trump campaign. Romney has already made it clear that he will be skipping the GOP convention in Cleveland in July, joining other former Republican presidents and presidential candidates, including senator John McCain, and former presidents George W and George HW Bush, staying away from Trump’s inauguration as the GOP’s standard-bearer.

An Iowa state senator this week became the first Republican elected official to leave the party over Trump, likening the presumptive nominee to Hitler. On Tuesday, Mark Kirk, a Republican senator from Illinois who is considered to be one of the most vulnerable to losing his seat, announced he would not support the real estate businessman.

The Republican speaker of the house Paul Ryan, who had previously endorsed Trump in a tepid op-ed for the Janesville Gazette, also struck out at him and his comments against the federal judge presiding over the Trump University case.

Trump has said that judge Gonzalo Curiel was biased against him because he is “we, think, Mexican.” Curiel is actually from Indiana. Ryan said on Tuesday that these comments were “the textbook definition of a racist comment” which “should be absolutely disavowed”.

Donald Trump has faced criticism from senior Republicans over his remarks against judge Gonzalo Curiel. Photograph: John Moore/Getty Images

Wisconsin governor Scott Walker also appeared unsure about his support for the Republican nominee for president, telling local radio station WKOW on Tuesday that “it’s just sad in America that we have such poor choices right now”, and condemning the presumptive nominee’s comments on the Trump University judge.

And on Bloomberg’s politics podcast on Friday, senate majority leader Mitch McConnell refused to rule out rescinding his endorsement of Trump, and said that he needs to pick an experienced vice-presidential nominee “because it’s pretty obvious he doesn’t know a lot about the issues”.

Unlike his former colleagues, Romney has no formal position in the GOP, and is under no obligation to support the party’s nominee. In fact, Romney has been consistently outspoken against Trump and his rhetoric; he told the Wall Street Journal in May that “I wanted my grandkids to see that I simply couldn’t ignore what Mr Trump was saying and doing, which revealed a character and temperament unfit for the leader of the free world”.

But nonetheless, the fact that the party’s most recent nominee for president was considering voting for a candidate in a rival party shows just how much Trump’s candidacy is already tearing the party asunder, even as his campaign makes rhetorical calls for unity.

Romney is one of the highest-profile Republicans to repudiate the GOP’s presumptive nominee for president.

“I wish everybody in the Republican party had rejected Mr Trump and chosen someone else,” Romney told Blitzer when asked about Ryan’s decision to endorse Trump.

“But my choice is different than that of the other people, and I’m certainly not going to argue with them about their choice.”