It doesn’t take much to mint an icon in this political climate. Cliven Bundy became one nearly overnight. The story of Bundy’s battle against federal bureaucrats fit neatly into a resonant narrative: the defiant land-owner taking a stand against government overreach.

As word of Bundy’s refusal to pay the federal government to graze his herd on public land spread, more than 1,000 armed sympathizers descended on his Nevada ranch in the desert outside of Las Vegas. When the U.S. Bureau of Land Management abandoned its effort to seize Bundy’s cattle, the rancher, 68, was celebrated as a hero in certain right-wing circles. Supporters compared the Battle of Bunkerville, Nev., to the American Revolution; there was even a hashtag, #AmericanSpring. With his ten-gallon hat and gruff rhetoric, Bundy was an irresistible symbol of a certain frontier ideal.

The reality was much different. Bundy’s herd of cattle has been illegally grazing on federal land for more than 20 years. He owes the government more than $1 million, which he refuses to pay because, he says, he does not recognize federal authority to collect it. While some conservative media outlets rushed to canonize Bundy, the vast majority of elected Republicans steered clear of the standoff, perhaps because the facts suggested Bundy was less a patriot than a deadbeat.

Or worse. Speaking to supporters on Saturday, Bundy digressed into a discussion of race. “I want to tell you one more thing I know about the Negro,” Bundy said, according to Adam Nagourney of the New York Times:

Mr. Bundy recalled driving past a public-housing project in North Las Vegas, “and in front of that government house the door was usually open and the older people and the kids — and there is always at least a half a dozen people sitting on the porch — they didn’t have nothing to do. They didn’t have nothing for their kids to do. They didn’t have nothing for their young girls to do. “And because they were basically on government subsidy, so now what do they do?” he asked. “They abort their young children, they put their young men in jail, because they never learned how to pick cotton. And I’ve often wondered, are they better off as slaves, picking cotton and having a family life and doing things, or are they better off under government subsidy? They didn’t get no more freedom. They got less freedom.”

These remarks will surely dim Bundy’s spotlight. The few national politicians who flocked to his cause have already denounced the remarks. Nevada Senator Dean Heller, who had praised Bundy’s supporters as “patriots,” released a statement Thursday morning calling his views on race “appalling.” Kentucky Senator Rand Paul, who said Bundy’s case raised a “legitimate constitutional question” about federal authority, called his remarks offensive. “I wholeheartedly disagree with him,” Paul said.

Conservative media and political outfits which had promoted Bundy’s cause fell silent. Fox News ignored the remarks, though journalist Greta Van Susteren, who has featured the story, released a statement condemning Bundy’s remarks. Americans for Prosperity’s Nevada branch, which also latched onto the ranch rebellion, condemned Bundy’s comments in a statement to TIME. “I think most people would agree that spending over a million dollars to chase ‘trespass cattle’ in the Nevada desert is a poor use of tax dollars,” says spokesman Zachary Moyle. “It’s important to note that our opposition to wasteful government spending in no way lends support to offensive remarks made by Mr. Bundy or anyone else.”

Calls to Bundy’s ranch and to a mobile phone belonging to his family went unanswered Thursday. Craig Leff, a spokesman for the BLM, told TIME the agency will “continue to pursue this matter administratively and judicially.” The Battle of Bunkerville is over. Now the backlash has begun.

This story was updated at 5:35 p.m. on April 24 to include comments from Americans for Prosperity

Write to Alex Altman at alex_altman@timemagazine.com.