David Cameron has said the Conservatives would like to cut inheritance tax, suggesting the party may go into the next election reviving its abandoned pledge to raise the threshold to £1m.

The prime minister signalled that a Tory-led government would make it a priority to take less tax from people's estates, as he believes "in people being able to pass things down the generations and on to their children".

He made the comments when asked about his 2007 pledge to raise the threshold at which the tax is payable, from £325,000 up to £1m, which the coalition has not done. He said the Conservatives would have to address the issue in their manifesto for the election next year.

"Would I like to go further in future? Yes, I would. I believe in people being able to pass money down through the generations and pass things on to their children. I think you build a stronger society," he said in a speech near Brighton on Monday.

"Inheritance tax should only be paid for by the rich. It shouldn't be paid for by people who have worked hard and saved, and bought a family house in, say, Peacehaven [a town near Brighton]. So the ambition is still there and I would like to go further. It's better than it was, but it didn't make it into the coalition agreement. It's something we will have to address in our manifesto."

George Osborne's 2007 announcement that only millionaires would pay inheritance tax under a Conservative government was credited with heading off the threat of the then prime minister Gordon Brown calling a snap election.

Soon after the coalition came to power, Cameron made it clear he could not deliver on the policy, which was popular with the core Tory vote. The Conservatives and Liberal Democrats agreed to delay inheritance tax cuts to prioritise income tax reductions.

One problem with Cameron's hint is that Jeremy Hunt, the health secretary, last year announced a freeze on the threshold for paying inheritance tax until at least 2019 to pay the bill for capping elderly care costs at £75,000 per person. If a Conservative government were to raise the threshold any sooner, it would mean rewriting the entire social care funding deal, which took years to agree.

Cameron also hinted that he was in favour of keeping universal benefits for pensioners, including free bus passes, television licences and winter fuel allowances. This was in contrast with Labour and the Lib Dems, who have said they would look at whether it is fair for these benefits to go to the wealthy.

"We will set our policy for the next parliament at the next election. I don't want to prejudge that. But the only thing I would say is that people think you save lots of money by not giving these benefits to upper-rate, top-rate taxpayers," Cameron said.

"You save a tiny amount of money and you always introduce another complexity into the system. We made our promises for this parliament, we kept our promises, I'm proud of that."

The prime minister was tackled on the issue as he addressed pensioners at a Saga event, after Osborne announced a wave of pension reforms at the budget last week. His main policy, which appears to have given the Tories a boost in the opinion polls, was to allow people to take lump sums out of their pensions, rather than having to buy annuities.

Some have raised concerns that people could blow their pensions too early or invest them badly, but Cameron said the fear was "deeply condescending".

"They are not irresponsible people, they are responsible people, that's why they saved in the first place," he said.

Cameron also revealed some immediate changes to allow savers to spend their pensions more freely, rather than waiting until the law is changed in 2015.

From Thursday, the government will cut the minimum yearly income required to access pension savings flexibly to £12,000, raise the limit on how much can be accessed to 150% of an equivalent annuity, double the total pension savings that can be taken as a lump sum to £30,000 and increase the size of a small pot that can be taken as a lump sum.