The co-chairman of the Conservatives has denied describing party activists as "swivel-eyed loons" after rumours circulated on the internet that he was the source for remarks widely published over the weekend.

Lord Feldman said he was taking legal advice after posts on Twitter implied he was the senior Tory quoted anonymously in several national newspapers. The mystery Tory made the remarks at a party dinner event – allegedly in earshot of journalists – after being asked about the decision of 116 party MPs to defy the prime minister and vote in favour of an amendment regretting the absence of an EU referendum in the Queen's speech.

The unnamed figure is reported as saying: "It's fine. There's really no problem. The MPs just have to do it because the associations tell them to, and the associations are all mad, swivel-eyed loons."

In a statement Feldman, who was a friend of David Cameron at Oxford University, said: "There is speculation on the internet and on Twitter that the senior Conservative party figure claimed to have made derogatory comments by the Times and the Telegraph is me.

Lord Andrew Feldman. Photograph: conservatives.com

"This is completely untrue. I would like to make it quite clear that I did not nor have ever described our associations in this way or in any similar manner. Nor do these alleged comments represent my view of our activists. On the contrary in the last eight years of working for the party, I have found them to be hard-working, committed and reasonable people. They are without question the backbone of the party. I am very disappointed by the behaviour of the journalists involved, who have allowed rumour and innuendo to take hold by not putting these allegations to me before publication. I am taking legal advice."

The remarks threaten to inflame the incendiary row between Conservative grassroots and Cameron's inner circle, including its many former Eton schoolboys, who are criticised as being "out of touch". Feldman was at a dinner of the Conservative Friends of Pakistan on Wednesday at the Intercontinetal hotel in Westminster where the remarks were said to have been made. However, those sitting near him are said by sources close to Feldman to be willing to publicly deny hearing anything similar to the comments reported.

The Conservative party went on the attack on Saturday, suggesting similarities between the Twitter rumours around Feldman and the defamatory claims wrongly connecting former party treasurer Lord McAlpine to child abuse allegations.The case of former chief whip Andrew Mitchell, who is contesting claims that he called police officers at the gates of Downing Street "plebs", was also cited.

Grant Shapps, the co-chairman of the Conservative party, said: "He [Feldman] works very closely with the party volunteers. I believe him when he says that he did not say that about our fantastic volunteers," he told the BBC. "We have seen these rumours flying around the internet, we have seen it with Lord McAlpine and Andrew Mitchell, both of whom were later in the clear."

James Kirkup, the Telegraph journalist who reported the remarks, tweeted: "I have seen Lord Feldman's statement. I stand by my story."

Mitchell, now on the backbenches, appeared on Sky News to offer his support. He said: "It looks to me tonight as if there's a full-on media storm staring on all of this and we should bear in mind that the man at the centre of it, Lord Feldman, says it is untrue and if Lord Feldman says it is untrue then I believe him. We should avoid a rush to judgment.

"None of us think what has been suggested in the media today. Having worked with Lord Feldman I can tell you that this is not his view about activists and I would be very surprised if he did say such a thing.

"It is all very well making these points by innuendo, pointing the finger at the man who has made it clear he didn't say those things. I don't think anyone around David Cameron thinks these things.

"I don't think there is anyone senior in the party or junior in the party who believes anything of the sort about our activists.

"If anyone said such a thing I thing it would be a disgraceful thing to say, completely untrue, and Lord Feldman has made it clear that he didn't say it."

The Ukip leader, Nigel Farage, seized on the remarks allegedly made by a senior Tory. Farage, who claimed to know the identity of the Tory, tweeted on Friday: "If you are a Conservative supporter who believes in Ukip ideas then your party hates you. Come and join us."

"

Those posting comments on the Conservative Home blog on Saturday were unforgiving. Sandy Jamieson wrote: "We activists are all 'mad, swivel-eyed loons'. Of course we are – we elected David Cameron as leader."

Another poster, with the username Doppel1800, wrote: "The cliquey Cameroons are on a completely different planet which even their choice of insults betrays."

Downing Street is under pressure because the Tory is said to be well known to the prime minister for many years. He or she is due to play a significant role in the party's preparations for the general election. The Times, Daily Telegraph and the Daily Mirror, which all reported the remarks and which say they know the identity of the Tory, declined to name the senior member of the prime minister's circle."

The publication of the remarks, which were made during the week that the prime minister was in the US, is particularly embarrassing for Cameron. They come after No 10 aides expressed fury with Philip Hammond, the defence secretary, who criticised the government for devoting so much time to gay marriage legislation.

A Downing Street spokesman said: "It is categorically untrue that anyone in Downing Street made the comments about the Conservative party associations and activists reported in the Times and the Telegraph."