Think America is all about guns, cultural conservatism and brittle religion? Think again. Think America is split 50/50 between this stereotyped old America and more socially liberal new America? Think again. American values are shifting and shifting fast. In the new America the split is much closer to a 60/40.Old America is being left behind.

In any recent election, a Democrat who proclaimed his social liberalism and had to defend an equivocal economic record would have been routed. Yet, today Obama is competitive in a close race. Clinton was elected as a different kind of Democrat who would be tough on welfare. When he flirted with social liberalism he soon realised he was walking towards an electoral precipice and tacked back to the then centre –"no more something for nothing".

Obama may have many personal qualities but the reason he is not toast in this presidential election is simple: there are not only many more socially liberal Americans than even a few years ago but many Americans have let go of the notion that the only thing that matters is economic success. New America is asking what old Europe has asked for generations: how can we make our society fairer? Americans remain sceptical of the state and fiercely independent minded. They are therefore unlikely to reach for the solutions beloved of old Europe, but increasingly Americans want a better society not a bigger porch.

How do we know this? In 1977, Ronald Inglehart wrote The Silent Revolution. In it he described the generational transformation in American values, as a new post materialist generation was supplanting its predecessor. The use of social psychology to understand core beliefs has since become common place. This year, Cultural Dynamics, who produce the British Values Survey, conducted an American Values Survey. A similar survey was also conducted in 2004.

For those of you unfamiliar with the Cultural Dynamics segmentation at its most reductive, any population can be divided in three based on dominant motivations. Settlers are psychologically conservative and focus more on security, tradition and culture. Prospectors are more aspirant and tend to care most about status and being successful, they can be psychologically conservative or liberal. Pioneers are more likely to be post-materialists, and to think in terms fairness and justice. They are typically more socially liberal.

In the 2004 survey the US remained an outlier among developed nations – its population was more aspirant and focused on wealth creation than any other major industrial nation. Over half were prospectors. Today the picture looks very different. Indeed, less than a third now class as prospectors. Instead, it is the more socially liberal pioneer that dominates. In the 2012 survey half of Americans class as pioneers.

How did this happen? Undoubtedly, demographic change provides part of the answer, as the ranks of college graduates have swelled and immigration patterns have changed. Younger Americans, Latinos and most black voters are more likely to be socially liberal than older white voters. But demographics alone cannot account for this. It seems as though after the collapse of Lehman Brothers, Americans suffered from a collective bout of existential angst. The conclusion for many was that the American dream could no longer just be about the riches at the end of the freeway. It had to be about the richness of lives shared while on the freeway.

What does this mean for American Politics? The Cultural Dynamics survey finds that 31.4% of Americans identify as Democrats, 27% as Republican and 30.8% as independents. Of course, Republican identifiers tend to be older and are more likely to be white than Democrats. They are also much more likely to say they are religious.

Overall, independents are very evenly spread across the values groups, the only value that defines them collectively is ‘self directed,’ and this is very much a core American cultural value. Republicans do well amongst the socially conservative settler and less well among pioneers and Democrats do least well amongst settlers but better among prospectors and pioneers.

Both those who identify as right-leaning and moderate Republicans have similar values, they embrace values like "security", "conformity", "propriety" and "tradition", right-leaning Republicans more strongly so. We can also see that the Republicans do pick up a certain kind of socially liberal supporter – libertarians who "don’t do government".

Democrats who identify as right-leaning look very different from those who identify as left leaning. Right-leaning Democrats are more likely to be Prospectors and pick up values like "power" and "visible success", plenty are still socially conservative. But left-leaning Democrats are much more likely to pick up one value above all else – "universalism" – an overarching belief in fairness. Indeed, 42% of left leaning Democrats are drawn from one particular subgroup that might best be described as ultra Pioneers (the most post materialistic and focused on fairness). This group of ultra Pioneers now accounts for a staggering one quarter of the US population and Republicans barely get a look in, gaining just half the support of Democrats from this group.

When Clinton was standing for election, he had to pay far more attention to right-leaning or potential right-leaning Democrats to keep his election chances alive but Obama can "play to his base" because his base is now vast. In fact, Obama’s biggest challenge is to motivate disillusioned left leaning Democrats to vote. If he fails to do this defeat beckons.

Today’s America is more receptive than ever to the social liberalism first proffered by Kennedy and the Good Society programmes of Lyndon B Johnson. "Freedom", always core to the American lexicon, now has to share the stage with "fairness". Old Europe can no longer erroneously content itself with the belief that whilst America owns "prosperity" it owns "fairness". And old Republicans can no longer console themselves with the belief that they represent the silent majority, the real America, because the new America looks very different from the old one.

Nick Pecorelli is Associate Director of Demos. For more on the American Values Survey, follow this link.