Do any of us really identify ourselves as members of "hard-working families"? As a rhetorical label used by Labour politicians, it is not winning votes, as critics have pointed out. In a country where 70% of us still identify as working class, most people would agree with Len McCluskey that "ordinary, working class" is a better description of the majority of voters.

"Hard-working families" implies we're only entitled to citizenship (or, as the Tories would have it, the odd game of bingo) if we can prove we're working our fingers to the bone. But no one can work all the time: if you're a pensioner, a single parent, sick, or there is no work to be had, then you're in trouble. And most of us know this, because we're related to them. Sit my extended family around a table and you'd have white- and blue-collar workers, the sick, the old, people in council housing, and families with two cars and a nice house but large debts to pay for them. This is replicated all over Britain. There is no static "underclass" and neither is there a robust middle class: instead, there are a lot of people who have to work for a living and, because of that fact, choose to identify as working class.

There's another reason why the appeal to "hard-working families" is an empty abstraction. Most people don't see hard work as a virtue. They identify as working class because they have to work, not because they want to. Two recurring conversations within my family and among the people I spoke to for my book The People are what they'd do if they won the lottery, and how they can afford to spend less time at work and more with those they love. This is a sensible attitude. Hard work causes stress, poor health and early death. And hard work has never solved poverty. We work longer hours now than we've done for 50 years, yet the gap between the rich and poor has never been wider.

The real history of the working class is of avoiding working for "them" any more than is necessary. That desire shaped collective campaigns throughout the last century, such as the TUC's demand for holidays with pay – a fight finally won in the late 1930s. It was an aspiration that also galvanised workers in ways we don't often hear about. The desire to control how much time was spent working for other people's profits provoked thousands of men to try to get into reserved jobs in factories during the second world war, to avoid armed service: not because they were cowards, but because fighting in 1914 had brought no benefits for ordinary people.

During both world wars millions of women used the expansion of munitions work as a means of escaping domestic service, which paid poor wages and often demanded a seven-day week. In 1945 Labour's guarantee of full employment and a welfare state won the party millions of votes. But the real gains of the postwar years were delivered by ordinary people themselves. It was they who mobilised to improve working conditions and, importantly, to reduce the amount of time they spent at work. The workplace militancy of the decade after 1968 was provoked by workers' frustration that, despite technological innovations, low pay made them reliant on overtime in order to afford a holiday or a car.

People aren't afraid of hard work. It was the understanding that people wanted more control over their time that drove the Tories' electoral successes over the last century. Grammar schools were initially popular with parents who hoped an education would offer their postwar babies what apprenticeships offered an earlier generation: a set of skills that would prove an important bargaining tool, something they could use to negotiate with employers, or to set up on their own. Similarly, in 1979 Margaret Thatcher's promise to make it easier for people to strike out on their own, regardless of their background, by starting their own business and owning their own homes proved very appealing. But promises of social mobility and self-preservation have failed, because only a few can ever possess the wealth and opportunity in a capitalist society. Grammar schools didn't increase most people's opportunities, bankruptcies rose in the 1980s and owner-occupiers suffered record levels of repossession in the 1990s and are burdened with huge, unsustainable debts.

Solidarity, on the other hand, has delivered important victories. Over the past century these have included better working conditions, shorter working hours, an expanded public sector that gave us better jobs and care, democratically controlled housing and free education. Working-class people's ability to mobilise politically has been attacked by the state, but the desire to help each other out has not died – it's just that its only outlet is now in worrying about children's and grandchildren's uncertain futures. Parents are aware that their homes, cars and savings for their children's higher education rely on a highly insecure labour market, in which permanent contracts are elusive.

By showing that collective effort can bring huge gains for all of us, the left could justify the redistribution of income and property, which is the only way to create a truly classless society. Utopian maybe, but less ludicrous than suggesting that "hard-working families" can overcome the inequality perpetrated by a powerful elite determined to hang on to their privilege.

• Selina Todd is author of The People: The Rise and Fall of the Working Class 1910-2010