Michael Young was an inconvenient child. His father, an Australian, was a musician and music critic, and his mother, who grew up in Ireland, was a painter of a bohemian bent. They were hard-up, distractible and frequently on the outs with each other. Michael, born in 1915 in Manchester, soon found that neither had much time for him. Once when his parents had seemingly forgotten his birthday, he imagined that he was in for a big end-of-day surprise. But no, they really had forgotten his birthday, which was no surprise at all. He overheard his parents talk about putting him up for adoption and, by his own account, never fully shed his fear of abandonment.

Everything changed for him when, at the age of 14, he was sent to an experimental boarding school at Dartington Hall in Devon. It was the creation of the great progressive philanthropists Leonard and Dorothy Elmhirst, and it sought to change society by changing souls. There it was as if he had been put up for adoption, because the Elmhirsts treated him as a son, encouraging and supporting him for the rest of their lives. Suddenly he was a member of the transnational elite: dining with President Roosevelt, listening in on a conversation between Leonard and Henry Ford.

Young, who has been called the greatest practical sociologist of the past century, pioneered the modern scientific exploration of the social lives of the English working class. He did not just aim to study class, though; he aimed to ameliorate the damage he believed it could do. The Dartington ideal was about the cultivation of personality and aptitudes whatever form they took, and the British class structure plainly impeded this ideal. What would supplant the old, caste-like system of social hierarchy? For many today, the answer is “meritocracy” – a term that Young himself coined 60 years ago. Meritocracy represents a vision in which power and privilege would be allocated by individual merit, not by social origins.

Inspired by the meritocratic ideal, many people these days are committed to a view of how the hierarchies of money and status in our world should be organised. We think that jobs should go not to people who have connections or pedigree, but to those best qualified for them, regardless of their background. Occasionally, we will allow for exceptions – for positive discrimination, say, to help undo the effects of previous discrimination. But such exceptions are provisional: when the bigotries of sex, race, class and caste are gone, the exceptions will cease to be warranted. We have rejected the old class society. In moving toward the meritocratic ideal, we have imagined that we have retired the old encrustations of inherited hierarchies. As Young knew, that is not the real story.

Young hated the term “welfare state” – he said that it smelled of carbolic – but before he turned 30 he had helped create one. As the director of the British Labour party’s research office, he drafted large parts of the manifesto on which the party won the 1945 election. The manifesto, “Let Us Face the Future”, called for “the establishment of the Socialist Commonwealth of Great Britain – free, democratic, efficient, progressive, public-spirited, its material resources organised in the service of the British people”. Soon the party, as it promised, raised the school-leaving age to 15, increased adult education, improved public housing, made public secondary school education free, created a national health service and provided social security for all.

As a result, the lives of the English working class were beginning to change radically for the better. Unions and labour laws reduced the hours worked by manual labourers, increasing their possibilities of leisure. Rising incomes made it possible for them to buy televisions and refrigerators. And changes, partly driven by new estate taxes, were going on at the top of the income hierarchy, too. In 1949, the Labour chancellor of the exchequer, Stafford Cripps, introduced a tax that rose to 80% on estates of £1m and above, or about £32m in contemporary inflation-adjusted terms. (Disclosure: I’m a grandson of his.) For a couple of generations afterward, these efforts at social reform both protected members of the working classes and allowed more of their children to make the move up the hierarchy of occupations and of income, and so, to some degree, of status. Young was acutely conscious of these accomplishments; he was acutely conscious, too, of their limitations.

Just as happened in the US, college attendance shot up in Britain after the second world war, and one of the main indicators of class was increasingly whether you had been to university. The middle-class status of meagerly compensated librarians reflected a vocational requirement for an education beyond secondary school; that the better-paid assembly-line workers were working-class reflected the absence of such a requirement. Working-class consciousness – legible in the very name of the Labour party, founded in 1900 – spoke of class mobilisation, of workers securing their interests. The emerging era of education, by contrast, spoke of class mobility – blue collars giving way to white. Would mobility undermine class consciousness?

These questions preyed on Young. Operating out of a community studies institute he set up in Bethnal Green, he helped create and nurture dozens and dozens of programmes and organisations, all attending to social needs he had identified. The Consumers’ Association was his brainchild, along with its magazine, Which?. So was the Open University, which has taught more than 2 million students since Young founded it in 1969, making it the largest academic institution in the UK by enrolment. Yet education mattered to him not just as a means of mobility, but as a way to make people more forceful as citizens, whatever their station – less easily bulldozed by commercial developers or the government planners of Whitehall. Late in life, he even set up the School for Social Entrepreneurs. Over the decades, he wanted to strengthen the social networks – the “social capital”, as social scientists say these days – of communities that get pushed around by those who were increasingly claiming a lion’s share of society’s power and wealth.

What drove him was his sense that class hierarchies would resist the reforms he helped implement. He explained how it would happen in a 1958 satire, his second best-seller, entitled The Rise of the Meritocracy. Like so many phenomena, meritocracy was named by an enemy. Young’s book was ostensibly an analysis written in 2033 by a historian looking back at the development over the decades of a new British society. In that distant future, riches and rule were earned, not inherited. The new ruling class was determined, the author wrote, by the formula “IQ + effort = merit”. Democracy would give way to rule by the cleverest – “not an aristocracy of birth, not a plutocracy of wealth, but a true meritocracy of talent.” This is the first published appearance of the word “meritocracy”, and the book aimed to show what a society governed on this principle would look like.

Facebook Twitter Pinterest ‘Education mattered to Young not just as a means of mobility, but as a way to make people more forceful as citizens, whatever their station.’ Photograph: Getty

Young’s vision was decidedly dystopian. As wealth increasingly reflects the innate distribution of natural talent, and the wealthy increasingly marry one another, society sorts into two main classes, in which everyone accepts that they have more or less what they deserve. He imagined a country in which “the eminent know that success is a just reward for their own capacity, their own efforts”, and in which the lower orders know that they have failed every chance they were given. “They are tested again and again … If they have been labelled ‘dunce’ repeatedly they cannot any longer pretend; their image of themselves is more nearly a true, unflattering reflection.”

But one immediate difficulty was that, as Young’s narrator concedes, “nearly all parents are going to try to gain unfair advantages for their offspring”. And when you have inequalities of income, one thing people can do with extra money is to pursue that goal. If the financial status of your parents helped determine your economic rewards, you would no longer be living by the formula that “IQ + effort = merit”.

Those cautions have, of course, proved well founded. In the US, the top fifth of households enjoyed a $4tn increase in pretax income between 1979 and 2013 – $1tn more than came to all the rest. When increased access to higher education was introduced in the US and Britain, it was seen as a great equaliser. But a couple of generations later, researchers tell us that higher education is now a great stratifier. Economists have found that many elite US universities – including Brown, Dartmouth, Penn, Princeton, and Yale – take more students from the top 1% of the income distribution than from the bottom 60%. To achieve a position in the top tier of wealth, power and privilege, in short, it helps enormously to start there. “American meritocracy,” the Yale law professor Daniel Markovits argues, has “become precisely what it was invented to combat: a mechanism for the dynastic transmission of wealth and privilege across generations.”

Young, who died in 2002 at the age of 86, saw what was happening. “Education has put its seal of approval on a minority,” he wrote, “and its seal of disapproval on the many who fail to shine from the time they are relegated to the bottom streams at the age of seven or before.” What should have been mechanisms of mobility had become fortresses of privilege. He saw an emerging cohort of mercantile meritocrats who can be insufferably smug, much more so than the people who knew they had achieved advancement not on their own merit but because they were, as somebody’s son or daughter, the beneficiaries of nepotism. The newcomers can actually believe they have morality on their side. So assured have the elite become that there is almost no block on the rewards they arrogate to themselves.

The carapace of “merit”, Young argued, had only inoculated the winners from shame and reproach.

Americans, unlike the British, don’t talk much about working-class consciousness; it is sometimes said that all Americans are, by self-conception, middle class. But this, it turns out, is not currently what Americans themselves think. In a 2014 National Opinion Research Center survey, more Americans identified as working-class than as middle-class. One (but only one) strand of the populism that tipped Donald Trump into power expressed resentment toward a class defined by its education and its values: the cosmopolitan, degree-laden people who dominate the media, the public culture and the professions in the US. Clinton swept the 50 most educated counties, as Nate Silver noted shortly after the 2016 election; Trump swept the 50 least. Populists think that liberal elites look down on ordinary Americans, ignore their concerns and use their power to their own advantage. They may not call them an upper class, but the indices that populists use to define them – money, education, connections, power – would have picked out the old upper and upper-middle classes of the last century.

And many white working-class voters feel a sense of subordination, derived from a lack of formal education, and that can play a part in their politics. Back in the early 1970s, the sociologists Richard Sennett and Jonathan Cobb recorded these attitudes in a study memorably titled The Hidden Injuries of Class. This sense of vulnerability is perfectly consistent with feeling superior in other ways. Working-class men often think that middle-class and upper-class men are unmanly or undeserving. Still, a significant portion of what we call the American white working class has been persuaded that, in some sense, they do not deserve the opportunities that have been denied to them.

They may complain that minorities have unfair advantages in the competition for work and the distribution of government benefits. Nevertheless, they do not think it is wrong either that they do not get jobs for which they believe they are not qualified, or that the jobs for which they are qualified are typically less well paid. They think minorities are getting “handouts” – and men may feel that women are getting unfair advantages, too – but they don’t think the solution is to demand handouts for themselves. They are likely to regard the treatment of racial minorities as an exception to the right general rule: they think the US mostly is and certainly should be a society in which opportunities belong to those who have earned them.

Facebook Twitter Pinterest A still from an Open University maths lecture, first broadcast in January 1971. Photograph: Open University/PA

If a new dynastic system is nonetheless taking shape, you might conclude that meritocracy has faltered because, as many complain, it isn’t meritocratic enough. If talent is capitalised efficiently only in high tax brackets, you could conclude that we have simply failed to achieve the meritocratic ideal. Maybe it is not possible to give everyone equally good parenting, but you could push more rigorously for merit, making sure every child has the educational advantages and is taught the social tricks that successful families now hoard for their children. Why isn’t that the right response?

Because, Young believed, the problem was not just with how the prizes of social life were distributed; it was with the prizes themselves. A system of class filtered by meritocracy would, in his view, still be a system of class: it would involve a hierarchy of social respect, granting dignity to those at the top, but denying respect and self-respect to those who did not inherit the talents and the capacity for effort that, combined with proper education, would give them access to the most highly remunerated occupations. This is why the authors of his fictional Chelsea Manifesto – which, in The Rise of the Meritocracy, is supposed to serve as the last sign of resistance to the new order – ask for a society that “both possessed and acted upon plural values”, including kindliness, courage and sensitivity, so all had a chance to “develop his own special capacities for leading a rich life”. Even if you were somehow upholding “IQ + effort = merit”, then your equation was sponsoring a larger inequality.

This alternative vision, in which each of us takes our allotment of talents and pursues a distinctive set of achievements and the self-respect they bring, was one that Young had learned from his schooling at Dartington Hall. And his profound commitment to social equality can seem, in the mode of schoolhouse utopias, quixotic. Yet it draws on a deeper philosophical picture. The central task of ethics is to ask what it is for a human life to go well. A plausible answer is that living well means meeting the challenge set by three things: your capacities, the circumstances into which you were born, and the projects that you yourself decide are important. Because each of us comes equipped with different talents and is born into different circumstances, and because people choose their own projects, each of us faces his or her own challenge. There is no comparative measure that would enable an assessment of whether your life or my life is better; Young was right to protest the idea that “people could be put into rank order of worth”. What matters in the end is not how we rank against others. We do not need to find something that we do better than anyone else; what matters, to the Dartingtonians, is simply that we do our best.

The ideal of meritocracy, Young understood, confuses two different concerns. One is a matter of efficiency; the other is a question of human worth. If we want people to do difficult jobs that require talent, education, effort, training and practice, we need to be able to identify candidates with the right combination of aptitude and willingness and provide them incentives to train and practice.

Facebook Twitter Pinterest Michael Young in 1949. Photograph: Getty

Because there will be a limited supply of educational and occupational opportunities, we will have to have ways of allocating them – some principles of selection to match people to positions, along with appropriate incentives to ensure the necessary work gets done. If these principles of selection have been reasonably designed, we can say, if we like, that the people who meet the criteria for entering the schools or getting the jobs “merit” those positions. This is, to enlist some useful philosophers’ jargon, a matter of “institutional desert”. People deserve these positions in the sense in which people who buy winning lottery tickets deserve their winnings: they got them by a proper application of the rules.

Institutional desert, however, has nothing to do with the intrinsic worthiness of the people who get into college or who get the jobs, any more than lottery winners are people of special merit and losers are somehow less worthy. Even on the highest levels of achievement, there is enormous contingency at play. If Einstein had been born a century earlier, he might have made no momentous contributions to his field; a Mozart who came of age in the early 20th century and trained on 12-tone rows might not have done so either. Neither might have made much use of their aptitudes had they grown up among the Amazonian Nukak.

And, of course, the capacity for hard work is itself the result of natural endowments and upbringing. So neither talent nor effort, the two things that would determine rewards in the world of the meritocracy, is itself something earned. People who have, as The Rise of the Meritocracy bluntly put it, been repeatedly “labelled ‘dunce’” still have capacities and the challenge of making a meaningful life. The lives of the less successful are not less worthy than those of others, but not because they are as worthy or more worthy. There is simply no sensible way of comparing the worth of human lives.

Put aside the vexed notion of “merit”, and a simpler picture emerges. Money and status are rewards that can encourage people to do the things that need doing. A well-designed society will elicit and deploy developed talent efficiently. The social rewards of wealth and honour are inevitably going to be unequally shared, because that is the only way they can serve their function as incentives for human behaviour. But we go wrong when we deny not only the merit but the dignity of those whose luck in the genetic lottery and in the historical contingencies of their situation has left them less rewarded.

Yes, people will inevitably want to share both money and status with those they love, seeking to get their children financial and social rewards. But we should not secure our children’s advantages in a way that denies a decent life to the children of others. Each child should have access to a decent education, suitable to her talents and her choices; each should be able to regard him- or herself with self-respect. Further democratising the opportunities for advancement is something we know how to do, even if the state of current politics in Britain and the US has made it increasingly unlikely that it will be done anytime soon. But such measures were envisaged in Young’s meritocratic dystopia, where inheritance was to hold little sway. His deeper point was that we also need to apply ourselves to something we do not yet quite know how to do: to eradicate contempt for those who are disfavoured by the ethic of effortful competition.

“It is good sense to appoint individual people to jobs on their merit,” Young wrote. “It is the opposite when those who are judged to have merit of a particular kind harden into a new social class without room in it for others.” The goal is not to eradicate hierarchy and to turn every mountain into a salt flat; we live in a plenitude of incommensurable hierarchies, and the circulation of social esteem will always benefit the better novelist, the more important mathematician, the savvier businessman, the faster runner, the more effective social entrepreneur. We cannot fully control the distribution of economic, social and human capital, or eradicate the intricate patterns that emerge from these overlaid grids. But class identities do not have to internalise those injuries of class. It remains an urgent collective endeavour to revise the ways we think about human worth in the service of moral equality.

This can sound utopian, and, in its fullest conception, it undoubtedly is. Yet nobody was more practical-minded than Young, institution-builder par excellence. It is true that the stirrings of Young’s conscience responded to the personal as well as the systemic; dying of cancer in a hospital ward, he worried whether the contractor-supplied African immigrants who wheeled around the food trolleys were getting minimum wage. But his compassion was welded to a sturdy sense of the possible. He did not merely dream of reducing inherited privilege; he devised concrete measures to see that it happened, in the hope that all citizens could have the chance to develop their “own special capacities for leading a rich life”. He had certainly done exactly that himself. In the imaginary future of The Rise of the Meritocracy, there was still a House of Lords, but it was occupied solely by people who had earned their places there through distinguished public service. If anyone had merited a place in that imaginary legislature, it would have been him.

Is the British establishment finally finished? Read more

That was far from true of the House of Lords he grew up with, which was probably one reason why his patron Leonard Elmhirst declined a peerage when offered one in the 1940s; in the circles he moved in, he made clear, “acceptance would neither be easy for me to explain nor easy for my friends to comprehend”. So it is more than a little ironic that when Young, the great egalitarian, was offered a peerage in 1978, he took it. Naturally, he chose for himself the title Baron Young of Dartington, honouring the institution he had served as a trustee since the age of 27. As you would expect, he used the opportunity to speak about the issues that moved him in the upper house of the British parliament. But there is a further, final irony. A major reason he had accepted the title (“guardedly”, as he told his friends) was that he was having difficulties meeting the expense of travelling up to London from his home in the country. Members of the Lords not only got a daily allowance if they attended the house; they got a pass to travel free on the railways. Michael Young entered the aristocracy because he needed the money.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

• This article originally appeared in the New York Review of Books. See nybooks.com. It was amended on 25 October 2018: the 1945 Labour government raised the school leaving age to 15, not 16.