So here we are, forming an orderly queue at the slaughterhouse gate. The punishment of the poor for the errors of the rich, the abandonment of universalism, the dismantling of the shelter the state provides: apart from a few small protests, none of this has yet brought us out fighting.

The acceptance of policies that counteract our interests is the pervasive mystery of the 21st century. In the US blue-collar workers angrily demand that they be left without healthcare, and insist that millionaires pay less tax. In the UK we appear ready to abandon the social progress for which our ancestors risked their lives with barely a mutter of protest. What has happened to us?

The answer, I think, is provided by the most interesting report I have read this year. Common Cause, written by Tom Crompton of the environment group WWF, examines a series of fascinating recent advances in the field of psychology. It offers, I believe, a remedy to the blight that now afflicts every good cause from welfare to climate change.

Progressives, he shows, have been suckers for a myth of human cognition he labels the enlightenment model. This holds that people make rational decisions by assessing facts. All that has to be done to persuade people is to lay out the data: they will then use it to decide which options best support their interests and desires.

A host of psychological experiments demonstrate that it doesn't work like this. Instead of performing a rational cost-benefit analysis, we accept information that confirms our identity and values, and reject information that conflicts with them. We mould our thinking around our social identity, protecting it from serious challenge. Confronting people with inconvenient facts is likely only to harden their resistance to change.

Our social identity is shaped by values that psychologists classify as extrinsic or intrinsic. Extrinsic values concern status and self-advancement. People with a strong set of extrinsic values fixate on how others see them. They cherish financial success, image and fame. Intrinsic values concern relationships with friends, family and community, and self-acceptance. Those who have a strong set of intrinsic values are not dependent on praise or rewards from other people. They have beliefs that transcend their self-interest.

Few people are all-extrinsic or all-intrinsic. Our social identity is formed by a mixture of values. But psychological tests in nearly 70 countries show that values cluster in remarkably consistent patterns. Those who strongly value financial success, for example, have less empathy, stronger manipulative tendencies, a stronger attraction to hierarchy and inequality, stronger prejudices towards strangers and less concern about human rights and the environment. Those with a strong sense of self-acceptance have more empathy and greater concern for human rights, social justice and the environment. These values suppress each other: the stronger someone's extrinsic aspirations, the weaker his or her intrinsic goals.

We are not born with our values. They are shaped by the social environment. By changing our perception of what is normal and acceptable, politics alters our minds as much as our circumstances. Free, universal healthcare, for example, tends to reinforce intrinsic values. Shutting the poor out of it normalises inequality, reinforcing extrinsic values. The rightward shift that began with Thatcher and persisted under Blair and Brown, whose governments emphasised the virtues of competition, the market and financial success, has changed our values. The British Social Attitudes survey shows a sharp fall over this period in public support for policies that redistribute wealth and opportunity.

This shift has been reinforced by advertising and the media. Their fascination with power politics, their rich lists, their catalogues of the 100 most powerful, influential, intelligent or beautiful people, their obsessive promotion of celebrity, fashion, fast cars, expensive holidays: all inculcate extrinsic values. By generating feelings of insecurity and inadequacy – which means reducing self-acceptance – they also suppress intrinsic goals.

Advertisers, who employ plenty of psychologists, are well aware of this. Crompton quotes Guy Murphy, global planning director for JWT: marketers "should see themselves as trying to manipulate culture; being social engineers, not brand managers; manipulating cultural forces, not brand impressions". The more they foster extrinsic values, the easier it is to sell products. Rightwing politicians have also, instinctively, understood the importance of values in changing the political map. Margaret Thatcher famously remarked that "economics are the method; the object is to change the heart and soul".

Conservatives in the US generally avoid debating facts and figures. Instead they frame issues in ways that appeal to and reinforce extrinsic values. Every year, through mechanisms that are rarely visible and seldom discussed, the space in which progressive ideas can flourish shrinks a little more. The progressive response has been disastrous.

Instead of confronting the shift in values, we have sought to adapt to it. Once progressive parties have tried to appease altered public attitudes: think of all those New Labour appeals to middle England, often just a code for self-interest. In doing so they endorse and legitimise extrinsic values. Many greens and social justice campaigners have also tried to reach people by appealing to self-interest: explaining how, for example, relieving poverty in the developing world will build a market for British products, or suggesting that, by buying a hybrid car, you can impress your friends and enhance your social status. This tactic also strengthens extrinsic values, making future campaigns even less likely to succeed. Green consumerism has been a catastrophic mistake.

Common Cause proposes a simple remedy: that we stop seeking to bury our values and instead explain and champion them. Progressive campaigners, it suggests, should help to foster an understanding of the psychology that informs political change and show how it has been manipulated. They should also come together to challenge forces – particularly the advertising industry – that make us insecure and selfish.

Ed Miliband appears to understand this need. He told the Labour conference that he "wants to change our society so that it values community and family, not just work" and "wants to change our foreign policy so that it's always based on values, not just alliances … We must shed old thinking and stand up for those who believe there is more to life than the bottom line". But there's a paradox here, which means that we cannot rely on politicians to drive these changes. Those who succeed in politics are, by definition, people who prioritise extrinsic values. Their ambition must supplant peace of mind, family life, friendship – even brotherly love.

So we must lead this shift ourselves. People with strong intrinsic values must cease to be embarrassed by them. We should argue for the policies we want not on the grounds of expediency but on the grounds that they are empathetic and kind; and against others on the grounds that they are selfish and cruel. In asserting our values we become the change we want to see.

• A fully referenced version of this article can be found on George Monbiot's website