Fathers of boys have a perennial subject of conversation. Is it us, our parenting skills or the wider society that is creating so many disaffected, troubled and disengaged young men? One friend of mine has hit on the theory that boys have lost their fear at the same time as having misplaced any sense of ambition. With the two great animators of human motivation – fear and greed (at its best, ambition) – knocked out we have a growing army of underperforming, unnecessarily idle and too often unemployed young men. Unable to motivate themselves, they just stew.

Young men's underachievement when compared with women has multiple causes – but a growing inability to find motivation, whether growing up on a social housing estate or in a top boarding school is, I think, one of them. The roots of this simultaneous collapse in ambition and loss of fear are hard to fathom and much contested, but they are there for all to see. It has been apparent in boys' GCSE and A-level results for some time. Now, as the Observer reports, it is extending into work. Graduate male unemployment on the most recent figures is rising so much faster than women's that it is more than half as high again – 17.2% compared to 11.2%.

Women graduates seem to be more useful to employers, according to the Higher Education Policy Institute. Girls are more diligent. They try harder at university. And now it seems women graduates try harder to find work – and to stay in it. High unemployment is going to be the story of the next few years and it will be men who will take the greater hit. The number of men in employment is already nose-diving, touching new lows – and worse lies ahead. Thanks to a credit crunch, a breathtaking programme of spending cuts and a stagnant world economy, the prospect is for unemployment now to rise above 3 million, only to fall very slowly, if at all – whatever the government's Panglossian forecasts.

The differing impact on men and women began to manifest itself in the 1980s. "Gizza job," intoned Yosser Hughes in the landmark TV drama the Boys from the Blackstuff; it became the emblematic expression of the age and painted the Conservatives as socially pitiless for two decades. The problems with men in the decade ahead will cast those years into the shade. Yosser is but a forerunner of an army. We will lose a generation of young men, trapped into permanent adolescence, idleness and a collapsed sense of self-worth. They will not be much use to society, the women who are looking for partners and fathers for their children, or themselves.

The question is, why? The standard left-of-centre response is that the economy needs to deliver more opportunity – the fall-back position of so many of the candidates for the Labour leadership, who want the party to rediscover social democracy or even socialism. Of course the unnecessarily aggressive programme of deficit reduction the coalition has embarked on will prove to be a strategic mistake. The budget that was allegedly to restore financial market confidence has triggered a stock market rout as investors fear prolonged economic stagnation. Of course the country needs to invent much better economic institutions than we have to support enterprise and job creation. And of course the state has to accept a much greater role in acting as employer of last resort in today's economic circumstances – a contemporary version of Roosevelt's New Deal – than the coalition government has accepted.

But calls for public action will not win popular support unless they also address the question of individual motivation. Without it public action will be seen, however irrationally, as part of the problem rather than the solution. It will be depicted as saving people, and boys and young men in particular, from the consequences of their disengagement at other people's expense. Too many families and too many people are shaking their heads over the behaviour of their sons and young men. A government that wants to win and sustain broad support needs to have plausible answers.

Affluence and a sense of entitlement with little reciprocal obligation partly explains what has happened. No rights without obligations was one of Blair's principles, and he was correct. One of the reasons that boys do not try harder is that the penalties for disengagement are so low – indeed, there are even rewards, at least in the sense that if you don't try, you can't fail. Much better to smoke dope, hang out and obsessively play computer games all day. Society is affluent enough to allow this to happen without anybody starving, and in the last resort it will provide a safety net, however threadbare and tightly means-tested. There is little fear in this de-risked universe.

The great male demotivator is the risk of loss of face – much more important to the masculine self-image of being masterful and in control than to a woman. The dynamic part of the economy is increasingly knowledge-based, and its jobs require intelligence, application and adaptability, whether in advanced manufacturing or the creative industries. For decades boys could follow their fathers into jobs that were routine and demanded little of them – in factories, offices or mines – and which did not expose them to the risk of public failure. These routine jobs are ever rarer. Boys now survey an education system and labour market where more is demanded of them – and the threat of loss of face if they don't measure up is more acute. A growing number react by refusing to take part. But what about ambition, the sheer hunger for any kind of achievement?

Here the celebrity culture along with winner-takes-all salaries such as those in football or in banking have undermined motivation. To try to work hard is a mug's game. The middle-class boy who diligently works his way up in a company or starts a business is a dupe; far better to try to make tens of millions in the City with zero risk – or not do anything. Everyone knows that investors, shareholders and governments underwrite any losses, while executives capture dynastic fortunes without risking any money of their own. City and corporate excess has become a demotivator for everyone else. Working-class boys who yearn for the same excess hope to be discovered by celebrity television, with one poll reporting that 11% of 16- to 19-year-olds are "waiting to be discovered" by a reality TV programme. An astonishing 26% think a rich career in sport, entertainment or the media is open to them. Why go down to the Jobcentre?

And if women are prospering better in this brave new world, it is because they respond to the same signals by fearfully redoubling their efforts to avoid failure. The dark side of all that female diligence and hard work is the epidemic of eating disorders and mental health problems afflicting women. Neither gender is reporting an increase in happiness. A political debate cast between madcap deficit cutters and those labour leadership candidates who want to rediscover the virtues of a backward-looking "socialism" opens up none of these issues. We badly need better.