(CNN) Republican lawmakers and administration aides found themselves again Wednesday weighing the costs and benefits of remaining loyal to President Donald Trump, whose equivocal statements about neo-Nazis and white supremacists marked a dramatic shift in presidential rhetoric.

By Wednesday afternoon, most appeared to have made their calculation: deserting Trump now could only harm — and not help — their agendas or political fortunes.

Republican leaders in Congress, including House Speaker Paul Ryan and Senate Majority Leader Mitch McConnell, released statements affirming their disavowal of white supremacist groups and neo-Nazis — but not explicitly condemning Trump, who said Tuesday there were "very fine people" protesting in Charlottesville amid the torch-bearing marchers.

Within the White House, Trump's aides privately expressed indignation at the derailed news conference, which unraveled on cable television Tuesday afternoon and has been replayed endlessly since.

But they, too, stopped short of declaring their consternation publicly, determined instead to remain focused on their agenda and keep the President occupied.

Trump himself has remained largely silent on the matter. But inside the glassed-in confines of Trump Tower — where he remained inside for nearly two days straight — the President was defiant in the wake of the ensuing backlash, according to two people who visited the building on Wednesday.

He's plunging forward "without regret," one of those people said, firmly believing the media and East Coast elites are unfairly hyperventilating about the Charlottesville remarks. The two people said it is similar to the posture he took during challenging moments of his winning presidential campaign.

Some Trump aides described themselves as shocked at the President's remarks. But the more common sentiment Wednesday was dismay — at Trump's words, but also at his repeated inability to remain on the intended message.

For many of the operatives and policy experts who signed on to work for the Republican President, it was a familiar feeling. There have been stumbles and setbacks throughout Trump's presidency that prompted head-shaking and grumbles from even the most unwavering aides.

The shadows this week are darker, the advisers concede, with a greater chance of obscuring whatever message the President hopes to advance as he returns to Washington from his working vacation next week.

No signs of departures from Trump world

But despite those anxieties, there have been no signs of defections from Trump's inner circle. As of Wednesday afternoon, no members of Trump's administration — including high-profile Jewish advisers Gary Cohn and Jared Kushner — announced they were resigning. Instead, officials described a heads-down attitude among Trump's staff, despite the growing controversy.

White House chief of staff John Kelly — who watched grimly as Trump held forth in the lobby of Trump Tower — was frustrated at how Trump's appearance played out, including the President's unplanned decision to take questions, a person familiar with the matter said Tuesday.

But a day later, Kelly has urged staffers to remain focused on their work rather than stew in recriminations about the press conference gone wrong, one official said.

"We have work to do and we're going to do it," said one administration official when asked about the mood inside the White House.

The two people who visited Trump Tower on Wednesday, speaking about an internal debate on condition of anonymity, said the reaction to Trump's remarks is so far breaking along similar lines that previous arguments inside Trump's orbit have fallen.

The President's sons and those directly around him believe he is in the right and saying out loud what many of his supporters believe. Yet his newer advisers are troubled by the reaction and frustrated that Trump doesn't seem to recognize the damage his remarks pose to his political agenda.

Kushner and his wife, Ivanka Trump, are vacationing in Vermont, but have communicated with the President. They have not responded to CNN requests for comment. First Lady Melania Trump was at Trump's golf resort in Bedminster, New Jersey, during the press conference and remains there.

Cohn was "enraged" at having to stand at the President's side during his statement, an associate of his told CNN. By Wednesday, however, his anger gave way to being "disappointed" and "embarrassed" in the wake of the brutal news coverage. This friend said he wouldn't be surprised if Cohn quit, but doesn't believe he has made a decision and could also very well stay.

Marc Short, the White House legislative affairs director, was on Capitol Hill Wednesday afternoon on a previously scheduled visit, talking about Trump's agenda and the plan for September.

And Stephen Miller , Trump's policy adviser and speech-writer, hosted a mid-morning conference call with conservative operatives and some Capitol Hill aides to discuss Trump's upcoming legislative agenda, a person familiar with the matter said. Miller warned those who joined the call not to stray from his intended topics, which didn't include the President's remarks about Charlottesville.

Trying to move on

Attempting forward momentum, the White House announced Trump would travel Friday to Camp David, the presidential retreat in Maryland's Catoctin Mountains, to meet with members of his national security team to discuss strategy in Afghanistan.

Vice President Mike Pence, traveling in South America, said he was cutting his trip short to join Trump at Camp David. But he, too, offered only support for the President.

"The President has been clear on this tragedy and so have I," Pence said during an appearance alongside Chile's president. "I spoke at length about this heartbreaking situation on Sunday night in Colombia and I stand with the President and I stand by those words."

On Wednesday morning, however, there were virtually no voices backing Trump on morning television, despite a document circulated by the White House on Tuesday evening providing talking points to regular surrogates for the President.

"Everything will be harder now," a senior White House official said, but dismissed the flood of apocalyptic suggestions from observers of both parties.

In Congress, most Republican lawmakers appeared cautious in their approach to Trump's remarks. While virtually none came out in support of his statements, few called him out by name, either.

Those who did were regular critics of Trump's tactics.

"Mr. President, I encourage you to try to bring us together as a nation after this horrific event in Charlottesville. Your words are dividing Americans, not healing them," said Sen. Lindsey Graham, the South Carolina Republican who ran against Trump for the GOP presidential nomination in 2016.

After a high-profile failure to repeal the Affordable Care Act , congressional Republicans and the White House are under pressure to deliver on promises to reform the US tax system and advance an infrastructure package — items that will require some element of cooperation between the White House and Capitol Hill.

More pressing is next month's deadline to raise the nation's debt ceiling, which will require Republican congressional leaders to both wrangle their own members — some of whom want to pair the increase with spending reforms — and to reach an agreement with Trump.

Failure to strike a deal would cause the nation to default on its debts, roiling financial markets and causing catastrophic effects on the economy — a potential outcome that has some lawmakers, including McConnell, wary of overly harsh treatment of Trump.

McConnell was privately upset with the President's handling of the episode , according to a source close to the Republican leader. The source said McConnell was deeply concerned that Trump is reopening long-festering racial tensions, something that could fan the flames ahead of demonstrations expected this month in Lexington, Kentucky.

McConnell -- who released a statement Wednesday that declared "there are no good neo-Nazis" but stopped short of criticizing Trump by name -- is taking a cautious approach after Trump lashed the GOP leader repeatedly last week, according to the source.

McConnell, the source said, did not want to immediately attack Trump for fear that it would look like retribution for their fight last week.