The former Conservative cabinet minister Lord Tebbit has suggested that people who visit food banks are at the same time spending their money on junk food.

Speaking in the House of Lords, Tebbit said there was a "near infinite demand" for valuable goods given away free and asked ministers to "initiate research into junk food sales in areas where people are [relying] for basic food on the food banks".

He made the comments moments after a Tory environment minister, Lord de Mauley, drew gasps from the chamber by saying food banks are not a scandal but a sign of Britain's charity and ministers should not seek to "interfere" in their use.

During the debate on food banks, several Labour, Liberal Democrat and crossbench independent peers condemned the rise in their usage. Statistics from the Trussell Trust, which oversees more than 400 UK food banks , show 614,000 adults and children received help from its food banks in the first nine months of 2013-14, compared with 350,000 for all of 2012-13.

Asked whether he agreed that the need for food banks was a scandal, De Mauley, a hereditary peer and former banker who went to Eton, said: "Britain has a great tradition of charitable giving, and it would be a bad day on which we started to interfere with that."

Pressed on whether he really meant that, the minister said: "I think I have just answered that, my Lords. Britain has a great tradition of charitable giving, and it would be a great mistake to interfere with that." He did, however, decline to commission the research into junk food requested by Tebbit, saying that there has been a decrease in the number of households reporting that they felt unable to afford food compared with 2007.

Earlier, Tebbit called for David Cameron to be replaced as Tory leader by the next election. In a speech to the Bow Group, he said: "It's not about whether we go left or right but whether we will be a top-down, narrowly focused party run by an elite, or a bottom-up party run by like-minded, anti-statist minded people.

"I hope that change can be brought about and that before the general election we can feel like Conservatives again and feel like the prime minister is a son of Thatcher and not a son of Tony Blair."

Tebbit, who is famous for suggesting in 1981 that the unemployed should get on their bikes to find work, also warned the bedroom tax could lose the party the next election.

"I think we introduced that rather without thinking it through very well, and I think that's costing us," he said.

• This article was amended to correct the date of Lord Tebbit's "on your bike" speech