Like many of the five million people now watching the controversial Channel 4 show, Benefits Street, I only took a look at it because of the furore in the media. Primed to expect ghastly "poverty porn", I saw instead a sad and touching documentary, which takes the time to offer a nuanced depiction of people usually talked of as troubling, unwelcome statistics, and only put under the microscope when some great tragedy occurs.

As with all fly-on-the-wall documentaries, it involves personal exploitation of those it features. In recent years there has been much debate over the use of non-professional people to provide popular television. There has been angst about the nastiness of watching talent-show hopefuls whose ideas about their own abilities are startlingly inflated and worries too about how contestants in reality game shows seem quickly to forget that they are being constantly filmed, and have no say in how that material is edited. All well rehearsed.

Broadly, by nature, reality formats are exploitative. But they are an aspect of the culture now, for better or worse, and suggesting that some people are too vulnerable to make up their own minds about whether to take part is patronising. As for those who declined to take part, but whose street has nevertheless become notorious, there is no denying that they have been dealt an unfortunate hand.

Some critics argue that Benefits Street is not an accurate portrayal of James Turner Street. It exaggerates the number of people out of work, who of course are the people who are around to be filmed anyway. That's more than likely. But I think the show does demonstrate that whatever the proportion of jobless people there are on this street, there are enough to form a critical mass that establishes living on benefits as normal.

My feeling is that the show probably disturbs many on the left because it confirms the cliches they often find unhelpful. Everyone's skint, but everyone smokes. Quite a few people have iPhones, laptops, computer games. They pick up bits of work, sometimes criminal work, and don't declare it. They never seem to cook "proper" food. They drink. They take drugs, and are often either active or recovering addicts. Their houses and gardens are a mess. They swear in front of the kids. They offer no boundaries to protect the kids from the adult chaos around them, with predictable results. Even the ones who say they enjoy their lives are dependent on anti-depressants. They complain all the time that their benefits aren't enough.

Who knew all this? Everybody, surely? The real problem is that in some observers that knowledge inspires compassion, while in others it inspires contempt. The voices on the left who condemn the programme do so out of a strong belief that contempt, not compassion, will be nurtured in the eyes of the viewer. I think that's something of a counsel of despair.

The exposure of those who took part in Benefits Street, and the unwanted attention thrust on those who did not, can and should be fashioned into something that can be considered a worthwhile legacy. Simply insisting that it should never have been broadcast will not do that.

For example, hostile critics of the poor often declare such existences to be a "lifestyle choice". The idea is, somehow, that sitting about in squalor, beer can and ashtray at your side as you listlessly watch TV, unable to cope with the washing up, let alone the demands of raising a family, as high-interest debt piles up, is in fact something enviable, a merry-go-round of careless fun that the poor sods with comfortable houses, decent jobs, hobbies and interests, and a clear idea about what they can achieve in the future, would jump at if only their iron self-discipline and unwavering self-respect would allow them such liberation.

Benefits Street, however, clearly depicts lives that no one would choose, if they were in a position to perceive viable alternatives. The people featured are stuck. Only benefits keep them alive, and they know no alternative to that. Even when, with help, some hope sparks up, it is soon extinguished.

In the third episode – the only one I've seen – Mark did find a job, for example. How smart he looked in his suit, how hopeful and pleased. But the job didn't last long because like all the work he's ever had, he wasn't earning a single penny from it. Not a good way to teach people that "work pays".

The other thing that comes over strongly in Benefits Street is the level of disengagement from human culture. No one on the street makes things, or fixes things, or even seems to have a passion for music or movies, or even the telly. They live in a basic, primitive way, and none of the services that support them seems to want to assist them, even in small ways, in escaping from that, even if only for an hour or two.

The other great fear of the left is that the right will leap on Benefits Street as evidence of their inverted "effect and cause" argument. The right tends to argue that people live the way they do because benefits exist. The left believes,that benefits exist because people live the way they do. A left that cries foul and calls for censorship of Benefits Street, only assists the right in its speciously overturned logic.

The single piece of media coverage that made me decide I'd better watch the show was reporting of the comments work and pensions secretary Iain Duncan Smith made about it. In keeping with the effect-and-cause argument, he insists that Benefits Street illustrates how benefits create ghettos, as if economic ghettos had never before existed in human history.

And anyway, had Conservative policies such as the flagship right-to-buy scheme been a success, then Benefits Street would now be gleaming under the loving attention of proud owner-occupiers. Right-to-buy was supposed, after all, to banish "sink estates" and by extension "sink communities".

When I lived in council housing, as a child, there were already "problem families" and people whose homes were smelly chaos. There always have been. But the demographic spread of people in that community was far wider than the social housing cohort is now. That critical mass, in which the wider exception – such as living on benefits – becomes the local norm, established itself during the last Conservative government, not under the last Labour government. The Tories were warned that an attack on social housing would further ghettoise the socially excluded, and for Duncan Smith now to describe James Turner Street as a Labour-created ghetto is disgustingly disingenuous. Labour's great failure was in continuing Conservative housing policies, and many others, not in turning away from them.

Longstanding progressive arguments – for intensive early intervention for families who aren't coping; for education that helps children to discover their talents and nurture their life-skills, not just to be employable; for work that is secure and well-paid; for homes that are socially subsidised without the need to have had a child to get one; for community arts outreach – these are arguments whose force and urgency are backed by Benefits Street, not threatened by it. The left should stop wringing its hands, and start making those arguments.

• This article was amended on 27 January 2014 to delete two sentences that suggested that it was Iain Duncan Smith, the work and pensions secretary, who rose to prominence as a teenager at a Conservative Party conference when it was, in fact, William Hague, the Foreign Secretary in 1977.