Outside right-leaning groups are firing back at Speaker John Boehner after he attacked their intentions and questioned their commitment to conservative principles on Wednesday.

“Speaker Boehner’s real problem here isn’t with conservative groups like FreedomWorks, it’s with millions of individual Americans who vote Republican because they were told the GOP was the party of small government and fiscal responsibility,” Matt Kibbe, the president of FreedomWorks, said in a statement.

Kibbe went on: “Once again Republicans, led by John Boehner, are working with Democrats to increase spending yet again on the taxpayers’ tab while promising ‘savings’ down the road. We know how this movie ends. How can leadership credibly promise spending cuts later, after agreeing to a plan that rolls back the sequester savings promised two debt increases ago? There’s a predictable pattern here.”

At a press conference Wednesday, a visibly angry Boehner said conservative groups who oppose the two-year budget deal stuck by Rep. Paul Ryan (R-WI) and Sen. Patty Murray (D-WA) are “using our members and they’re using the American people for their own goals. This is ridiculous.” Moments earlier, during a closed-door meeting, Boehner told House Republicans that the well-funded and influential organizations “aren’t acting out of principle, and they’re not trying to enact conservative policies. They’re using you to raise money and expand their own organization,” he said, according to a source in the room.

FreedomWorks — along with the like-minded Club For Growth, Heritage Action and Americans For Prosperity — began speaking out against the budget deal, based on reports about what was in it, before it was officially announced Tuesday. They all oppose the final agreement and are pushing lawmakers to vote against it. In response to Boehner’s remarks, the groups made clear they would carry on with their fight.

“Americans are deeply concerned about the direction of the country,” Dan Holler, a spokesman for Heritage Action, told TPM when asked to respond to Boehner’s remarks. “Over the next few days, lawmakers will have to explain to their constituents, many of whom are our members, what they’ve achieved by increasing spending, increasing taxes and offering up another round of promises waiting to be broken. That will be a really tough sell back home. Meanwhile, we’ll continue fighting to achieve our goal, which is create an America where freedom, opportunity, prosperity and civil society flourish.”

Club For Growth issued a statement in response to Boehner’s remarks, but didn’t mention him by name, and instead lit into the budget agreement he and Ryan are vouching for.

“We stand with Marco Rubio, Ted Cruz, Tom Coburn, Rand Paul, members of the Republican Study Committee and every other fiscal conservative who opposes the Ryan-Murray deal,” Chris Chocola, the group’s president. “After carefully reviewing the budget deal, on which we never commented until it was complete, we determined that it would increase the size of government. We support pro-growth proposals when they are considered by Congress. In our evaluation, this isn’t one of those.”

These conservative groups have been a major thorn in Boehner’s side, using their wealth and activist base to turn House Republican members against his efforts to pass legislation, frequently griping that such proposals aren’t conservative enough. GOP leaders intend to bring up the bill for a vote on Thursday, and are confident they have the votes to pass it.