Speaker of the House John Boehner dismissed a move this week by a Republican member to oust him from his post as “no big deal.” But rumbles on the Right have clearly moved the Speaker off his game—his golf game.

“If I go down to see President Obama, the Right begins to wonder what I’m up to,” Boehner told the Golf Channel. “The Left begins to wonder what the president is up to. The president has suggested, ‘Hey, do you think it is too much trouble if we play golf again?’ And I have to look at him and say, ‘Yes, because everybody gets bent out of shape, worried about what we are up to, when all we are really gonna do is play golf.’”

The resolution by Rep. Mark Meadows contends that “the Speaker has, through inaction, caused the power of Congress to atrophy, thereby making Congress subservient to the Executive and Judicial branches, diminishing the voice of the American People.” Instead of using the power of the gavel to push conservative policies, Meadows maintains that the speaker uses it to squash nonconformist conservative members.

Boehner railed against blogs and talk radio in his discussion with the Golf Channel’s David Feherty. He claimed that when Republicans took over the Congress in 1994 “one radio talk show that nobody had ever heard of” broadcasted. He points to the existence today of “hundreds of radio talk show hosts all trying to outdo themselves right—going further right, and further right, and further right.”

He worries what such people would make of an outing on the links with the Golfer in Chief.

Golf Digest ranked Boehner, who played a round with Tiger Woods a few years back, as the ninth best golfer in Congress.