Proposals to legalise gay marriage should be delayed to let the government focus on policies "that matter", the defence secretary, Philip Hammond, has said.

The government is consulting on the plans and people have until next month to submit their views. The prime minister has publicly offered his strong backing to the measure but it was not included in last week's Queen's speech.

Hammond told the BBC's Andrew Marr Show: "Clearly it's not the number one priority. If you stop people in the street and ask them what their concerns are, they'll talk to you about jobs and economic growth, they'll talk to you about the level of the wages they're earning, wanting to see real growth in wages again."

He added: "There is no legislation in the Queen's speech [on gay marriage]; there's a consultation going on, and we should look at, listen to what people are saying in response to that consultation.

"But I think the government has got to show over the next couple of years that it is focused on the things that matter to the people in this country – not just the short-term things but the long-term things as well: the reform of our education system, changing the welfare system – which is like turning around a super-tanker – changing the welfare system so that work always pays and people have an incentive to work."

Hammond's comments came after Tim Loughton, the children's minister, wrote to a constituent to say he was opposed to gay marriage. "Marriage as a religious institution cannot be anything other than between a man and a woman. I do not see why we need to change the law, especially at this time when there are so many other important matters for the government to be addressing," Loughton wrote.

Senior Conservative sources have claimed there are at least four cabinet ministers and several junior ministers who will vote against any proposal. But those within No 10 say Cameron is reluctant to back down because he sees gay marriage as a key symbol of his plans to modernise the Tories' image.

In the wake of poor local election results this month, Gerald Howarth, a defence minister, said the prime minister had no mandate for the policy. However, Lynne Featherstone, the Liberal Democrat Home Office minister who is overseeing the gay marriage consultation, said it was about "how, not whether" the measure was introduced. Eric Pickles, the communities secretary, said he supported gay marriage and it was a "question of good manners".

Trevor Phillips, head of the Equality and Human Rights Commission, called on Cameron to get on with legalising gay marriage. He told the BBC: "My message to David Cameron and friends is get on with it and my message to other people is get over it. Get on with something that really matters and the country absolutely cares about. This is not the ground on which to fight, the battle for the place of religion in public society which is an important battle, but this is not the place to fight it."