For the first time in his presidency, Donald Trump is facing significant criticism from Republican officials and conservative groups who are rattled by his ban on immigrants and refugees from Muslim-majority nations, questioning his domestic policy agenda and worrying about what steps the New York billionaire might take next in the name of nationalism.

By Sunday evening, more than a dozen GOP members of Congress had spoken out against Trump’s executive order on immigration. Among them were an array of the party’s most influential figures. The top Republican in the Senate, Mitch McConnell, said the United States should not implement a religious test. Sen. Rob Portman of Ohio said the plan to strengthen vetting of refugees was itself not vetted. And the political and policy groups led by Charles and David Koch offered their first public criticism of Trump, whose candidacy the billionaire brothers found so unpalatable they sat out the 2016 election.

The wave of criticism marks the end of a startlingly brief honeymoon period for a new President who has been in office for scarcely a week, and even set the White House on defense as it backtracked on the ban applying to green-card holders. And while much of the blowback was driven by Trump’s immigration orders, the controversial plans he has on the horizon suggest the rest of his term could be just as rocky.

The emerging rifts come amid mass protests in cities around the U.S. against an executive order that would block millions of people from entering the United States. Legal permanent U.S. residents were detained at airports, refugees were trapped en route to the United States and judges from coast to coast stepped in to stop the unprecedented White House action. The chaos knocked the White House back on its heels and prompted Trump on Sunday night to release a defense of the policy.

“This is not a Muslim ban, as the media is falsely reporting,” Trump said in a statement released by the White House. “This is not about religion—this is about terror and keeping our country safe.”

The sentiment did little to calm skittish conservatives, who have already grown tired of the theatrics and hysterics. From removing the Director of National Intelligence and the Chairman of the Joint Chiefs of Force from the National Security Council, to trolling news organizations on Twitter, the precedent-breaking new President was testing the patience of Republicans who had hoped he might change tactics and tone once he was in the Oval Office.

But there is no changing a 70-year-old billionaire. And by Sunday some of the party’s most influential figures had begun to break publicly with the Republican President. “We cannot be partisan. We can’t say, ‘OK, this is our party, right or wrong,’” Charles Koch said Sunday as he gave a pep talk to his deep-pocketed pals who plan to spend as much as $400 million heading toward the 2018 midterm elections. The network’s official position, taken Sunday with no caveats, was that Trump’s immigration ban was anathema.

During a later session at the Koch retreat near Palm Springs, Calif., the co-chairman of the policy and politics network told donors not to expect Trump to get a pass, especially if he goes after specific groups of people or adds red ink to government budgets.

“We have the courage to oppose bad policies that will only harm people’s lives, regardless of who proposes it,” Brian Hooks said. “Remember: A trillion-dollar government stimulus was a bad idea under Democrats. It’s a bad idea when a Republican proposes it.” Hooks, one of Charles Koch’s top aides, vowed that the Koch network would “hold all politicians accountable, regardless of political party.” Put another way: Stand with Trump at your own risk, lawmakers.

To be sure, the number of Republicans to publicly excoriate the new President is still relatively low. Silence reigned for most of the weekend as protests raged. And there were few signs that Trump was ready to bend in any meaningful way in the face of criticism. If anything, the criticism may only convince Trump to step up attacks on his opponents and the media. His first public comment on Sunday morning, after a day of striking protests, was a broadside at The New York Times.

Yet it is clear Trump will not have an unconditional coalition behind him. Rep. Charlie Dent of Pennsylvania called the move “ridiculous.” Sen. Ben Sasse of Nebraska said the order gives terrorists a win because they can claim the United States just equated all Muslims with jihadists. Rep. Dan Newhouse of Washington said many immigrants are “having their lives needlessly disrupted.” Sen. Bob Corker, a finalist to be Trump’s Secretary of State, joined fellow Tennessean Sen. Lamar Alexander in calling for changes to this policy. All are Republicans and come from across the ideological spectrum.

At the same time, conservatives are building blockades on Trump-style fiscal policy. “I really don’t like it,” Sen. Mike Lee of Utah said of Trump’s border tax plan, which could add a 20% tax on good and materials imported from Mexico. (Companies are most likely to pass the cost along to American consumers.) Asked later about Trump’s moves to shut out immigrants from seven countries with Muslim majorities, Lee tried his best to dodge. “I wasn’t aware that I’d lose my First Amendment rights after walking out this door,” he said gamely as as he left reporters behind.

Until now, the prospect of sweeping policy changes under unified Republican government had largely swept aside the tensions between Trump and members of his party. Trump rode a populist wave to victory, and many lawmakers are skittish about being the next target of a tweeted tirade. Some lawmakers are hoping Trump proves pliable on policy, or that he defers to Vice President Mike Pence or White House Chief of Staff Reince Priebus. Still, there’s an unknowable risk of sitting on the sidelines and hoping things turn out just fine. It’s simply not what’s in the DNA of the outside groups who pushed the party to the right during the Obama administration.

Jason Pye, a spokesman for the conservative advocacy group FreedomWorks, said that there is plenty to like about Trump’s policy agenda, from his early push to gut Obamacare to his pledge to usher in sweeping regulatory reforms. But Pye said he was troubled by plans to shell out billions to build a border wall without corresponding spending cuts, invest up to $1 trillion in infrastructure and impose tariffs while doing little to tackle entitlement programs like Social Security and Medicare. It’s a far cry from the conservative mantra during the Obama years, when the GOP insisted on offsetting all new discretionary spending—including for emergency disaster relief and unemployment insurance for the needy—with reductions elsewhere. There was no mention of doing the same for Trump’s proposed border wall.

Republicans “spent the last eight years complaining about budget deficits,” Pye says. “It makes us look like hypocrites.” During the Bush administration, he added, “Congressional Republicans abandoned any sort of fiscal restraint they claimed to have. I’m worried that just because the man in the White House has an ‘R’ next to his name that we’re going to do it all over again.”

Other conservative groups echoed the sentiment. The Koch-backed Americans for Prosperity wrote a letter to House Ways and Means chair Kevin Brady complaining that the GOP’s border-adjustment plan amounted to a “whopping tax hike.” Club for Growth spokesman Doug Sachtleben says the proposal is “really a bad idea.” The free-market group opposes some of Trump’s other trade ideas as well. “We don’t think getting the country involved in the trade war is a good idea,” Sachtleben says. “The notion that you punch first with a tariff threat is just not good for the economy.”

But for now, disagreements on fiscal policy have taken a backseat to the backlash over the immigration ban. Even Trump’s allies struggled to excuse the hastily composed order. Rep. Jason Chaffetz, a Utah Republican, told reporters at the Koch summit that he appreciated Trump’s intentions to secure borders. But, he added, he had no idea what Trump was thinking when it came to residents who have green cards. “I don’t understand what they’re trying to do,” Chaffetz said.

The rupture in GOP unity, coming so soon after Trump took office on Jan. 20, portends bigger fights to come. Many of the big-ticket items on Trump’s domestic agenda are sure to ruffle feathers among budget hawks. Airports aren’t cheap to rebuild, and bridges, roads and tunnels aren’t free, either. The widespread protests against the immigration moves suggest Trump’s critics are energized, if not organized—and that not all Republicans will blindly have Trump’s back.

“This executive order sends a signal, intended or not, that America does not want Muslims coming into our country,” Senators John McCain and Lindsey Graham said in a joint statement. “That is why we fear this executive order may do more to help terrorist recruitment than improve our security.”

In typical fashion, Trump brushed them off as weak on immigration and “always looking to start World War III” in a tweet. He ordered the White House, too, to release a statement defending the President’s moves. Two of his top advisers convened a conference call late Sunday to further brief reporters and dispute coverage of the order as a ban on Muslims.

Yet there are signs that patience with the President is wearing thin. Breaking with Trump carries political risks. But some Republicans are beginning to believe that not doing so would be even riskier.

Get our Politics Newsletter. The headlines out of Washington never seem to slow. Subscribe to The D.C. Brief to make sense of what matters most. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Philip Elliott at philip.elliott@time.com and Alex Altman at alex_altman@timemagazine.com.