Middle-class Britain has been shocked by the hidden reality of welfare ghettos revealed by TV programmes such as Benefits Street, Iain Duncan Smith is expected to say as he welcomes a Bank of England report claiming that his welfare-to-work reforms are bearing fruit.

In a speech on Thursday marking the 10th anniversary of the formation of the Centre for Social Justice thinktank, the work and pensions secretary is due to say: "With income inequality under Labour the worst for a generation, whilst the middle-class majority were aware of the problems in poor communities, they remained largely unaware of the true nature of life on some of our estates.

"We let these problems be ghettoised as though they were a different country. Even now, for the most part they remain out of sight – meaning people are shocked when they are confronted with a TV programme such as Benefits Street."

The Channel 4 show has caused huge controversy, with some claiming the programme makers misled residents of the street about the concept for the series, while others say it has revealed the reality of a life trapped in worklessness and benefit dependency.

Nick Clegg sidestepped an invitation to visit the Birmingham street featured in the programme after one of the residents featured in it called his LBC radio phone-in. The deputy prime minister said he would love to meet Dee Kelly but could not make a commitment to visit James Turner Street.

Clegg said he thought that critics of Benefits Street on both sides of the argument were mistaken. He said: "The impression I have had is you have one side of the argument saying this just shows the whole benefit system is rubbish, we should withdraw all benefits and all people are scroungers living off the state. There is another extreme of people who say it was outrageous that Benefits Street was ever made and it is demonising people on welfare. I strongly suspect both of those caricatures are wrong. We want a welfare system which is compassionate."

Duncan Smith argues that blame for those trapped on benefits should not be attached to benefit claimants but on a welfare system that traps them on the dole because work does not pay.

He will say: "The reality is that our welfare system has become distorted, no longer the safety net it was intended to be. Too often it is an entrapment – as it has been for a million people left on incapacity benefits for a decade or more, or the more than 4 million abandoned on out-of-work benefits even before the recession."

He also says the coalition should have tried to reform welfare even if there had been no need to reduce the deficit."The left would frame this debate as a rigid dichotomy: on the one hand, those opposing cuts, decrying all savings as an assault on the poor and vulnerable; on the other, those urging that the whip be cracked harder, clamping down on spending and making deeper cuts."

He insists his mission remains the same as when he first became convinced of the need to tackle poverty and worklessness: "In neighbourhoods blighted by worklessness ... where gangs were prevalent, debt and drugs the norm ... families broken down ... those living there had one thing in common: they were for the most part dependent on the state for their daily needs.

"And its very worst, the present system makes criminals out of those trapped in its clutches. Faced with losing up to 94 pence of every pound they earn because of how benefits are withdrawn, too many end up in the shadow economy or working cash in hand.

Such behaviour can never be condoned, but it is a tragic state of affairs – and a mark of how far the current system has failed – that people should feel pushed into crime by having their aspirations to make a living penalised. Surely the system should deliver for people who want to work hard and play by the rules."

He claims that the left, with the problem of social breakdown, believe the most sympathetic approach is to sustain the poor on slightly better incomes – the accepted wisdom of the last government being that poverty is about money, and more state money should solve it.

Duncan Smith claims: "Too often for those locked in the benefits system, that process of making responsible and positive choices has been skewed – money paid out to pacify them regardless, with no incentive to aspire for a better life.

"Our real success has been to reframe the argument – challenging a narrative beloved of the left … which focuses so exclusively on how much is being spent on welfare that it risks overlooking the real question: it is not about how much goes into the benefit system, but what difference it makes to people at the other end."

• The photograph on this article was changed on 23 January 2014 to better reflect the nature of the article.