Embarrassment about sex was once the greatest cause of British double standards. Now children, the result of sex, fuel the hypocrisies of perfidious Albion. According to the received wisdom of our times, we are living through a phenomenal feminist advance. Women have stormed the bastions of male privilege and cracked, if not smashed, the glass ceiling.

But the success of women is based on exploitation of women. The households of privileged families and a large proportion of the middle class could not function without the labour of servants fleeing the Brezhnevian tower blocks of eastern Europe, Asian slums and poverty and oppression the world over. Our social structure would collapse were it not sustained by shadowy legions of women, living outside the law as illegal immigrants or living outside the welfare state in the black economy.

Stern left-wing writers in the 20th century told their readers that their wealth depended on the exploitation of far-flung colonies. Lectures on how our clothes and trainers come from the sweatshops of the east remain a standard of liberal journalism today. But the line between the rich and the poor worlds does not run between north and south or east and west but through British homes, touching the most intimate aspects of family life – the care of children and the happiness of marriages. Small wonder there is a gap between public pronouncements and private behaviour.

The contradictions of Baroness Scotland are exemplary. On the one hand, she is a feminist from a working-class immigrant family who has overcome the formidable prejudices of the law to become Britain's first woman attorney-general. On the other, she employed Loloahi Tapui from Tonga as a housekeeper for six months, even though she was here illegally.

If she were not a law officer who had "inadvertently" hired an illegal immigrant, you could say that the critics' concentration on the baroness was sexist. Richard Mawhinney, her barrister husband, was as responsible and as likely to benefit from the arrangement. The second wave of feminism did not meet a significant backlash because its triumphs suited him and other upper-middle-class men very well. The modern power marriage depends on both husband and wife bringing in high incomes throughout their careers. Without servants, one partner, almost certainly the wife, would have to give up earning to look after the children. As important, without servants there would be no domestic peace. Imagine the petty, grating arguments if Patricia Scotland and Richard Mawhinney came home from demanding jobs to find that the lavatories needed to be scrubbed and the washing pulled out of the machine and ironed. Many middle-class marriages would have cracked under the pressure of chores, had not the magic of globalisation conjured housekeepers from Poland, Russia, Sri Lanka, the Philippines and, as we now know, the scattered islands of the Tongan archipelago.

In the US, Scotland would have gone as soon as the story broke. The discovery that an illegal immigrant is rearing the children of a public figure is the surest way to destroy a political career. However, in Britain, last week I sensed a reluctance among the better sort of journalist to go for the kill. A nagging feeling of "there but for the grace of God" restrained them and they neither realised nor cared that their self-interested silence allowed injustice to flourish.

British women turned their backs on live-in service because they could not abide the loss of independence. You do not need a vivid imagination to picture the isolation of the foreigners who have replaced them. Many illegals can barely speak English and, in any case, cannot appeal to help from the authorities for fear of being deported. Even the feminist charity Kalayaan, which deals with migrants staying legally in Britain on domestic worker visas, reports that an eye-wateringly high number of the servants it sees are not allowed out of the house and are beaten or raped.

Prosecutions are rarer than hen's teeth, but occasionally Kalayaan or Liberty will find a woman bold enough to bring an action against an exploitative employer. Recently, they gave us the story of a Nigerian brought to London to work for a solicitor. The lawyer took her passport, withheld her wages and subjected her to two years of threats and violence. After a final battering, she ran away. A neighbour phoned the police, who refused to investigate. Nevertheless, Liberty managed to take her before an employment tribunal, which awarded her £90,000 for unpaid wages and in compensation for unfair dismissal and race discrimination.

I am not saying that such cases are commonplace, merely that they are the inevitable consequence of the black domestic economy. I am adamant, however, that many comfortable British families do not believe that domestic workers have as much right as they do to demand maternity, holiday and sick pay from employment tribunals. In their hearts, they see nothing wrong with firing a nanny because she is pregnant.

Bridget Anderson, of Oxford University and one of the leading authorities on migrant labour, tells me how she became a traitor to the Oxford middle class when she helped a neighbour's au pair join a trade union. Not only did her neighbours refuse to speak to her from that day on, they told everyone else in the street never to let her into their homes for fear of the subversive ideas she would put into their nannies' heads.

Write like this and professional women accuse you of wanting them to stop working, which I do not want to do, or of painting them as callous bitches happy to abandon their children to the care of poor strangers, which I do not want to do either.

All I want is an honest admission that our apparently egalitarian society is kept going by the labour of the vulnerable and the voiceless and that it would be no more than decent if the incessant arguments about whether women in the City should receive seven-figure rather than six-figure bonuses were accompanied by a determination to end the suffering of the poor women on whose unreported work so much of our world depends.