Ipsos Mori survey of adults in 20 countries finds that those in richer countries fear their nations' best days are behind them

Adults in parts of the developing world are far more optimistic than their counterparts in rich nations, where the majority feel that young people will live a worse life than current generations, according to a major new survey.

The research, conducted in 20 countries by Ipsos Mori, found most young adults in Europe, North America, Japan and Australia fear that their nations' best days are behind them.

However, in the so-called Bric nations (Brazil, Russia, India and China) as well as Turkey, where standards of living have risen in recent years, the poll found that many more people believe the next generation will be better off than their own.

In a stark warning, the head of the world's foremost economic research institute, the Organisation for Economic Co-operation and Development (OECD), described such pessimism among western youth as "politically explosive" while Harvard historian Niall Ferguson, warned of a clash of generational interests as ageing societies paid for themselves by "consuming at the expense of the young and the unborn".

The survey of just over 16,000 adults asked: "To what extent, if at all, do you feel that today's youth will have had a better or worse life than their parent's generation?" Some 42% responded that life would be worse, compared with 34% who thought it would be better.

But the average total hides wide variations. In China 81% of the populace (and 78% of those questioned who are under 30) believe the lives of the youngest generation will be better than their own.

In European countries recently afflicted by negative growth rates and continuing austerity measures, positive responses fall dramatically to 16% in Spain, 13% in Belgium and just 7% in France.

Even in countries such as Sweden and Germany, which have weathered the financial crash with relative success, people are twice as likely to be pessimistic than optimistic about the future lives of their young.

In Britain, the survey found just 20% of the populace think today's youth will have a better life than their parents. More than half (54%) think it will be worse. In recent years high private rents, increasing housing prices and university tuition fees combined with a dramatic decline in wages have sparked a debate in the UK political mainstream about the life chances of young adults. The government has insisted that its austerity policies are necessary so future Britons aren't unfairly burdened with massive national debt. But the Labour leader, Ed Miliband, has warned that this risks jeopardising the so-called "promise of Britain.

Young adults around the world were more pessimistic about their own prospects than the population as a whole by a margin of 5%.

The OECD secretary-general, Ángel Gurría, said there was nothing more dangerous and destabilising than an embedded sense of futility among young adults.

The Paris-based multinational organisation has repeatedly found that since the onset of the global slump, young people and children have replaced the elderly as the group "at greater risk of income poverty" in the 33, mainly developed countries it represents (pdf).

In their most recent 2013 publication, they found average relative income poverty rose by 1.6% to 13.8% among youth between 2007-10. At the same time, relative income poverty actually fell from 15.1% to 12.5% among the elderly.

Last month the Guardian reported that just under half of Europe's adults under 30 (some 37 million people) were living with their parents.

"Nothing is more politically explosive, more dangerous and more destabilising than having a whole generation of frustrated young people," Gurría told the Guardian.

"It is shameful enough that the current crisis has made it particularly difficult for young people to find jobs and kick start their careers. What would be tragic is if the very trait that we count on the young to infuse into our societies – optimism – were to somehow become permanently scarred. We can't afford that."

Jim O'Neill, the economist who identified in 2001 that the Bric nations were the most likely to prosper during the last decade, said he was not surprised that young people in these nations and other large emerging countries such as Turkey were hopeful about their futures.

"One of the strongest signs of this is that many young people from these nations who have been educated overseas are returning home as they see their opportunities there as bigger and brighter compared with the chances and competition they face in the so-called developed countries, where opportunities seem so much more limited," he said.

Ferguson, who most recently authored, The Great Degeneration: How Institutions Decay and Economies Die, warned that pay-as-you-go welfare systems would lower living standards of future generations because of the increased cost of an ageing society.

"While it is once again fashionable to think in terms of inequality in terms of class (or income deciles), the reality is that many of the most salient conflicts of interest in western democracies are now between generations rather than classes. Ageing societies with welfare states designed for an earlier era are increasingly consuming at the expense of the young and the unborn.

"Not only will my children's generation pay higher taxes and receive lower benefits over their lifetimes than my generation but they also find themselves competing in a globalized labour market quite unlike anything experienced by the baby-boomers," Ferguson said.

Bobby Duffy, managing director of Public affairs at Ipsos Mori, said the findings represent "a continuing significant shift in global optimism."

"Optimism in western developed economies is hard to come by – particularly when people are asked to think about the future for their young people. In some ways, this should be no surprise, coming on the back of five plus years of dire economic conditions in many western economies.

But, he added that the "data suggests something more deep-seated". "Even those economies that have fared pretty well, such as Sweden and Germany, are far from brimming with confidence for their younger generations," Duffy said.

"The assumption of an automatically better future for the next generation is gone in much of the west – and this could have far-reaching implications for how we engage with national and international politics and economics."

Case studies

Ding Zhouyang, 24, graduate student in journalism at Sichuan University in Chengdu

My generation grew up with information. When we were young, we watched TV, we used computers, we surfed the web. My parents' generation grew up in a completely closed society. So we have a much more international perspective than they did. We're far more tolerant of different views and changing social mores. My father is a construction engineer. His university assigned him a job on graduation – he originally worked in a state-owned enterprise, and switched to a private company in 2003 after the government reformed his sector. He believes in the value of hard work and perseverance. My mother is a chemist. She puts far more value on stability.

I'm much more like my father – I don't believe there's such a thing as "happily ever after", only one challenge after another. And to overcome these challenges you must learn to rely on yourself.

My generation's biggest problem is that we're impetuous – we have trouble concentrating, and don't have quite the same tolerance for struggle that our parents did. Because of the one-child policy, few of us have brothers and sisters, and most of our needs are already cared for – we lead very stable lives. I really do believe that things are getting better, both for my age group and for society as a whole. At least, we're getting more independent. Most of us can choose to do what we want to do. And even if we don't know exactly what that is, we can at least stay away from things that we don't want to do. We're getting freer and freer to decide.

Justin Wolfers. Photograph: Justin Wolfers

Justin Wolfers, 24, Sydney, Australia

My grandparents immigrated to Australia in the 50s, after living under occupation or as refugees during the second world war, and they worked hard to ensure my parents were imbued with the privileges that they themselves were not: proper schooling, university, and the peaceful, if sleepy, affluence that an upbringing in Sydney provides. I can't say, though, that my parents' world views, or their overall attitudes, are in line with the idea that those who grew up in the 70s were revolutionaries.

They are conservative both socially and fiscally, value personal security, and don't really talk about the state of the world.

Maybe this libertarian, insular influence has led me to have grander aspirations for my generation, but our leaders couldn't be blander or less inspiring. The notion of us mining and pipelining our own Great Barrier Reef seems to be one big (all too real) metaphor for the ways Australia's vibrancy and diversity is being drained. The politicking is becoming meaner, and the policy lines are thinner and more reactionary.

Such a despondent state hasn't inspired revolution in my generation, though, at least not in the masses-in-the-streets sense. We've grown up to expect to vote without confidence, wondering perhaps if trust is a notion borrowed from a time when there was less money (less oil) in elections.

There is infinite space on the internet for us to be disaffected, offended, and distracted, but we must cut through that – through the deluge of things to be disappointed about – and realise that working hard, having passion and compassion, and not conforming to left/right is a way forward in itself.

In some ways our generation values the symbols of our parents' generation more than they did. I go cycling in the mountains to make sure the Australian geography is a part of me, and understand that those who assure us of certainty, safety, are having us on, and that scepticism – saying no, no, we can do better – is itself a form of optimism.