Iain Duncan Smith, the embattled work and pensions secretary, is refusing to meet leaders of the rapidly expanding Christian charity that has set up more than 400 food banks across the UK, claiming it is "scaremongering" and has a clear political agenda.

The news will fuel a growing row over food poverty, as church leaders and the Labour party accuse ministers of failing to recognise the growing crisis hitting hundreds of thousands of families whose incomes are being squeezed, while food prices soar.

Responding to requests for a meeting from Chris Mould, chairman of the Trussell Trust, which has provided food supplies to more than 500,000 people since April, Duncan Smith has dismissed claims that the problems are linked to welfare reforms and attacked the charity for publicity-seeking. In his most recent response on 22 November, Duncan Smith made clear that he had received enough letters from the trust and referred Mould to his previous answers. His deputy, Lord Freud, the minister for welfare reform, also explicitly rejected an invitation for talks on 30 August, telling the trust's chairman that he was "unable to take up your offer of a meeting".

Mould, whose organisation is struggling to keep up with ever-increasing demand for its services, said that he and his army of volunteers could not understand why ministers were refusing to listen to their suggestions for easing the plight of the hundreds of thousands of people in desperate need.

"To them, it doesn't make sense. It doesn't make sense to me either. We are deeply disappointed, but we are as open as ever to meet ministers in the hope that perhaps the new year will bring a fresh approach to what could so easily have been a fruitful dialogue." Mould also told Duncan Smith he is not opposed, for political reasons, to welfare reform.

In 2010, the Trussell Trust provided food to around 41,000 people, but in the past eight months the number has increased to more than half a million, a third of whom are children.

Mould first wrote to Duncan Smith in June, saying that many of the problems people were facing could be tracked back to changes in their benefits, and to delays in the payment of them.

Duncan Smith began his reply by criticising the "political messaging of your organisation", which "despite claiming to be nonpartisan" had "repeatedly sought to link the growth in your network to welfare reform". He said his department's record in processing benefit claims had improved and should do so further with the introduction of universal credit.

He rejected any suggestion that the government was to blame. "I strongly refute this claim and would politely ask you to stop scaremongering in this way. I understand that a feature of your business model must require you to continuously achieve publicity, but I'm concerned that you are now seeking to do this by making your political opposition to welfare reform overtly clear."

The standoff will further anger church leaders who were incensed by reports last week that the government had turned down a potential pot of £22m of EU funding for food banks, on the grounds that the UK did not want to be told by Brussels how to spend money for European structural funds.

Last week, in a Commons debate on food poverty, Conservative MPs recognised the gravity of the situation. Steve Baker, the Tory MP for Wycombe, said it was a "scandalous indictment of the safety net that is the welfare state" that so many people could be left hungry. "Some 12,000 children in Buckinghamshire live in income poverty and one in five children in Wycombe go to bed hungry – that increases to one in three in some parts of my constituency," he said.

The shadow secretary of state, Rachel Reeves, said: "Iain Duncan Smith should meet with the Trussell Trust and others who are on the frontline of dealing with the growing problem of food poverty in Britain, rather than burying his head in the sand.

"It's David Cameron and Iain Duncan Smith's failure to tackle the cost-of-living crisis which has led to half a million people relying on food banks since April this year.

"Under the Tories, it's tax cuts for the rich and food banks for the poor. Only a Labour government would act to freeze energy prices, help more employers to pay a living wage, and expand free childcare to help more parents earn a living for their family."

The Department for Work and Pensions said: "The benefits system supports millions of people who are on low incomes or unemployed and there is no robust evidence that welfare reforms are linked to increased use of food banks. In fact, our welfare reforms will improve the lives of some of the poorest families … with the universal credit making 3m households better off – the majority of these from the bottom two-fifths of the income scale.

"The Trussell Trust itself says it is opening three new food banks every week, so it's not surprising more people are using them. They also agree that awareness has helped to explain their recent growth.

"The government has taken action to help families with the cost of living, including increasing the tax-free personal allowance to £10,000 which will save a typical taxpayer over £700, freezing council tax for five years and freezing fuel duty."

Meanwhile, the number of households living in bed and breakfast accommodation has increased by 124% since 2010, according to figures compiled by Labour from official statistics.

The statistics show that illegal use of B&B accommodation, where families are staying longer than six weeks, has risen by 800%.