Is feminism "destroying the family"? Plenty of people appear to believe so. Earlier this week, "crop-haired" Katherine Rake, who recently ended a seven-year stint as head of feminist campaigning organisation the Fawcett Society, was accused of continuing to work towards this sinister agenda – her easy-care hairdo cited by the Daily Mail as proof of her malicious intent.

Rake upset the socially conservative by declaring in her first speech as head of the Family and Parenting Institute (a government-funded organisation that we don't need anyway) that "the days of the typical family are numbered". This statement of statistical-projection fact drew outrage from David Cameron, who lambasted Labour for its continuing hostility to marriage.

Ed Balls, the minister for children, schools and families, offered a retort. He's not personally hostile to marriage – he is, after all, married himself, to "crop-haired" work and pensions secretary Yvette Cooper – but he is against the Conservative idea that unmarried parents should be treated as "second-class" citizens.

It's a horrible idea, of course, and its horrible reality is still a raw, living memory. Only days ago, there was a new outpouring of abhorrence – and an apology from Kevin Rudd, the Australian prime minister – over one of the most ghastly manifestations of the treatment of "unmarried parents". Who could listen to the stories of children separated from their mothers, told they were orphans, and transported to exploitation in Australia (a practice that ended only four decades ago) without a horrified shudder?

Yet however scrupulously one may reject the idea of unmarried parents as "second-class citizens", class – as a cultural expression of economic status – continues to dog the "debate". Single parents do tend to be "second-class citizens", with 47% living below the poverty line. Statistically, their children do less well at school, and are overwhelmingly more likely to enter the criminal justice system, or to run into mental health or addiction problems.

Middle- and upper-class people are more likely to have their children later, when they are financially stable. Middle- or upper-class teenage pregnancies are more likely to end in abortion than teenage pregnancies among poor families. And so on. Delayed and structured parenthood, even marriage, is now, broadly, the choice of people – men and women – who have real choices. Early and unplanned parenthood is now, broadly, the choice of people – men and women – who look around them and perceive their options are narrow.

Feminism has liberated aspirational and affluent women, because it has ensured that women who are capable of financial independence are more able to achieve it, within marriage or outside it – although the continuing gender pay gap (much-monitored by Rake when she was at the Fawcett Society) is testament that even this success has been highly circumscribed.

I don't imagine that 1970s feminists envisaged the rejection of educational opportunities, the refusal of family planning options, and the graft of bringing up children alone and on the breadline, as brave-new-world female choices for a post-liberation era, any more than they envisaged the advent of boob-jobs, pole-dancing supermodels and store-card bankruptcy. Actually, it is a big fat irony that feminists are now obliged to defend the right of lone mothers to stay at home with their young children, when the initial idea was to liberate women from the obligation to, well, stay at home with their young children.

Notwithstanding his dislike of the idea of unmarried parents as second-class citizens, Ed Balls has long been a senior member of a political party that has directed policy – admittedly without great success – at pressurising unmarried parents, in particular, into work. The Conservatives believe that, with rings on their fingers (placed there with the help of a tax break), the mothers who have caused Labour such angst would have stayed at home very much more happily. But that is only because of their blind adherence to the middle-class portrait of the pre-feminist world.

I don't believe in that world, in which women got married, then gladly gave up work. Both of my grandmothers worked. My father's mother worked in a factory that made ladies' foundation garments, supporting her five surviving children alone after her husband, an asthmatic miner, had died young of lung disease.

My mother's mother left "service" after she had married. But as the wife of a landless rural worker and mother of seven, she kept chickens, grew vegetables, made jams and chutneys, baked, cooked, sewed and knitted interminably, all year round. My aunts worked, all 10 of them, especially the widows – in offices, as barmaids, in shops, as cooks, even though they were married, even though they had children. My mother, like her sisters and her sisters-in-law, stayed at home when we children were small. But she returned to work, first part-time, then full-time, as soon as she could.

So, when Fay Weldon ruffles feathers, as she did at a literary festival this week, by declaring that one of the down-sides of feminism is that it has "made wage-slaves" out of mothers, I can only shake my head. Mothers always had access to wage-slavery, and those mothers who had husbands who drank the pay-packet, or handed over risible "housekeeping", were particularly glad of their "pin money". Feminism gave women much greater access to, or at least hope of, "careers". Its rejection of marriage was bound up with the fact that a married woman was a woman who could forget about professional progress.

Now feminists defend their distrust of marriage, as if out-of-work mothers, living with a succession of boyfriends with whom they have a succession of children, was the intention and the dream all along. It wasn't, and other social factors, such as the sexual revolution, the mass exodus from religious belief, and changing patterns of work (including the annihilation of the earning power of manual-labouring males), have played just as large a part in "promoting" alternative family structures as feminism.

For Conservatives, the feminist rejection of the taboo against unmarried motherhood has been fuelled by a welfare state that "puts children first", creating a society in which the female poor have little incentive to work because they can get an income and a home from breeding. There is certainly some truth in this argument, but it ignores the counter-intuitive fact that throughout human history, and in our own era too, the poorest people are always the ones who have children most frequently, whether there is a welfare state to offer incentives or not.

So, if David Cameron is worried about his recent poor showings in the polls, he should reconsider his lazy homilies about the magic of marriage, and take a look at the possibility that "broken societies" are often teeming with very broke people.