David Cameron has a sharp sense of humour and often peppers his conversation in private with words that would make his mother, a highly respectable retired JP, blush.

But is his locker room banter making our Dave a tad un-prime ministerial? Some Tory MPs were slightly surprised this week when the prime minister referred to the parliamentary expenses body as a "four letter word".

Eyebrows were raised when the prime minister joked about the Independent Parliamentary Standards Authority (IPSA) at a meeting of the 1922 committee on Wednesday evening in an attempt to show he felt the pain of Tory MPs. I am told the prime minister said words to the effect of:

We all know what we think of IPSA. It is a four letter word.

Many MPs laughed at a mildly funny joke which Cameron had delivered without uttering a word that would have horrified his mum. But some Tories were not amused. This is what one told me:

The language David used at the 1922 committee to describe IPSA was quite un-prime ministerial.

Perhaps Cameron make a mildly risqué joke because he was nervous ahead of the meeting. Tories said the prime minister turned on IPSA at the 1922 meeting – describing the body as "anti-family" – after a lengthy debate at senior levels of the party.

The two key figures in the debate were Steve Hilton, the prime minister's director of strategy, and Patrick McLoughlin, the chief whip. This is how the debate went, according to my moles:

• Hilton, who was instrumental in shaping Cameron's decisive response to the expenses scandal in the early summer of 2009, argued for the prime minister to tread carefully in confronting IPSA. One Tory says:

Steve was decisive in persuading David to take up the expenses issue so strongly in 2009, destroying the careers of around 20 MPs in the process. He did that because he was terrified that Labour would get ahead of David. But the thinking was very short term. It was just: let's get through the election and think about the consequences afterwards. But he is responsible for the consequences.

• McLoughlin told the prime minister he had to confront IPSA after MPs recently passed a motion calling on the body to change its way by 1 April or face changes. My Tory says:

The McLoughlin argument runs deeper than Tory MPs moaning about their expenses. It is about MPs from all sides not being able to fulfill their duties properly. It is not just about waiting a long time for expenses to be reimbursed. It is uncertainty over staffing budgets. IPSA means that parliament isn't working properly. That was the argument Patrick McLoughlin put forward. That was decisive and it won the day.

And how does McLoughlin feel after winning an argument against Dave's inner circle? He's will proceed with care after mildly critical stories started popping up.

Tories are saying Cameron may eventually regret his attack on IPSA. One says:

It is quite extraordinary that the PM has decided to associate himself so closely with the future of IPSA. He should have left it to ministers. It means that if a new system is set up and there are scandals then he will be blamed.

But the chief whip advised that the prime minister had to intervene after MPs duffed up Sir George Young, the leader of the commons, at a meeting of the 1922 committee last week. In a riveting account of the meeting in the Sunday Telegraph, Melissa Kite quoted a Tory saying that some MPs went "berserk".