Work and pensions secretary clarifies comments, saying he is neither encouraging nor discouraging people to return benefits

Iain Duncan Smith, the work and pensions secretary, insisted he was neither encouraging nor discouraging wealthy pensioners to hand back their universal benefits such as the free bus pass, free TV licence or the winter fuel allowance.

He had been reported in the Sunday Telegraph as supporting the charitable voluntary move as a way of reducing the deficit, but on Monday he said: "I am not encouraging people to hand it back or keep it."

He told BBC Radio 4's Today programme: "This is a bit of a silly story, which people have tried to elaborate, when I didn't say very much at all.

"All I said in answer to a question, [is that] there's always been the position that if somebody wants to hand the money back if they don't use it that's up to them.

"But I'm not making that a policy position; it's just there, it's always been available for them to do – that's it."

He added the government had provided a hotline for people to pay back such benefits, but added he was proud of what the government had done to help pensioners. Asked if he would give up his own benefits as he's asking others to, he responded: "I'm not asking them to give up their allowances."

The cost of these benefits is roughly £3.5bn, but David Cameron has ruled out means-testing them.

The idea of wealthy pensioners returning their benefits to the government was immediately rejected as unworkable by many of Duncan Smith's cabinet colleagues, and his aides stressed his suggestion in a Sunday newspaper interview was not a policy, but an aspiration.

Nick Clegg said the idea did not make sense.

The row reflects the tensions within the cabinet over how to control welfare spending. Osborne, in a major innovation, is planning to put a cap on some working-age benefit payments. So far he has capped at £18,200 a year the amount an individual can receive in benefits.

On Sunday, Cable praised Duncan Smith for raising the issue of universal benefits for pensioners, but said: "My party's view is that we've got to address this in a more systematic way rather than just relying on individual conscience.

"The government should approach the issue of these fringe benefits properly, either by suggesting that they should be taxable or restricted to, say, older pensioners who are less fit.

"I think Iain Duncan Smith is acknowledging that there is an issue here, that relatively affluent people of pensionable age have done relatively well in very difficult times, and it would be fair to acknowledge that. And some people give it back, some people will fund charities, others will do nothing. At least he's flagging an important question, and it's important that he's raised it."

Ken Clarke, the Tory minister without portfolio, said it was not feasible to hand the money back, before adding a different gloss to the government's spending and growth strategy.

He said on Sky's Murnaghan programme: "We're cutting spending more slowly than almost any of the other western Europe sovereign debt-infected countries; we're cutting our public spending slower than President Hollande's government in France and he was elected on an anti-austerity programme, but he is cutting much more than we are."

He said the economy was still bumping along the bottom, and it may take another three years to recover.

• This article was amended on 30 April 2013. The original stated that "So far he has capped at £26,000 a year the amount an individual can receive in benefits". This has been corrected.