It must surely be one of the most beguiling and evocative posters of the 1970s. High above the Earth, floating serenely among the stars and loosely tethered to a speeding spaceship, Yuri Gagarin smiles out at us and salutes. The first man in space is dressed in brilliant communist red, and emblazoned on his helmet are the letters CCCP (the Russian initials for the USSR). Above the skies, he looks around and tells us what he can see, or rather what he can not see: Boga Nyet!: There is no God! Below him are the toppling towers and domes of churches and mosques, left behind and condemned to imminent collapse by the soaring achievements of Soviet science. The old religions are withering away. Reason and research have raised humanity to a new idea of heaven – we can now all join Gagarin in an achievable paradise, empty of divine beings, and full instead of starry promise.

A photograph taken 42 years later, in May 2017, is an equally compelling image of our own decade. It shows what popular songs long regarded as perhaps the nearest thing to heaven on earth – Paris in the springtime. But this is no romantic glimpse of the world’s favourite tourist destination. We are in the working-class suburb of Clichy. In the square in front of the mairie, beneath the French tricolour and the stars of the European Union, and under the watchful eyes of the police, God is being defiantly worshipped on the public highway. Hundreds of Muslim men are kneeling in prayer to protest at the closure of their unauthorised mosque. The secular French state, its republican values built on the basis of an absolutely non-negotiable laïcité (secularism), is being peacefully but fundamentally challenged by a group claiming the right to be loyal French citizens while asserting that they (and millions of others) also have rights as members of a religious community. As the notorious row about the burkini showed just a few years ago, these are not claims that France finds it easy to accommodate.

Both images are entirely characteristic of their time, with resonances far beyond their place. The Soviet poster was of course devised as propaganda, but the assumptions behind it were very widely shared. In the 1970s, most politicians in the US and western Europe, as in the USSR, broadly believed that scientific advance, material progress and growing prosperity would lead to the continuing retreat of faith from the public realm. Just as important, nearly all shared the view later articulated in Bill Clinton’s 1992 presidential campaign, that the most significant political determinant would ultimately be “the economy, stupid”.

Russia now defines itself loudly and proudly as Orthodox. Putin is ostentatiously devout. Even the KGB has its own church

All that has changed. In the Russia that has emerged from the wreckage of the USSR, state atheism is decidedly a thing of the past. The country now defines itself loudly and proudly as Orthodox. President Putin is ostentatiously devout. The Cathedral of Christ the Saviour in Moscow, dynamited on Stalin’s orders in 1931, has been meticulously rebuilt. Even the KGB has its own church. Beyond Russia, the competitive materialisms of the cold war have been in large measure dislodged, or at least recast, by a different discourse. To an extent rarely seen in Europe since the 17th century, faith now shapes a significant part of the global public debate.

The whole of the Middle East is caught up in murderous conflicts that are articulated and fought out in religious as much as economic terms. In Indonesia and Nigeria, Myanmar and Egypt, communities are attacked and individuals killed on the pretext that the practice of their faith makes them aliens in their own country. India, whose constitution enshrines the state’s equidistance from all religions, is convulsed by calls for the government to assert an explicitly Hindu identity, with grave consequences for the hundreds of millions of Indians who are Muslims, Christians or belong to other faiths. In many countries, among them the US, immigration policy – which usually means the case against immigrants – is often debated in the language of religion.

Facebook Twitter Pinterest French protest … Muslim men pray in the street after their mosque is closed in Clichy in March 2017. Photograph: Bertrand Guay/AFP/Getty Images

Even in a largely agnostic Europe, the French street protest is part of a similar trend. The Bavarian first minister has recently urged the presence of the cross in official buildings as the marker of a Catholic Bavarian identity (even though the Catholic church opposed the idea). In 2009, a nationwide referendum in Switzerland decided to ban the building of minarets; while for the last few years thousands have been marching regularly in Dresden to protest against the alleged “Islamisation” of Europe. The most populous country on Earth, China, claims that its national interests, the very integrity of the state, are threatened by the exiled spiritual leader of Tibetan Buddhists, the Dalai Lama, a man whose only power is the faith he embodies.

Belief is back. Around the world, religion is once again politically centre stage. It is a development that seems to surprise and bewilder, indeed often to anger, the agnostic, prosperous west. Yet if we do not understand why religion can mobilise communities in this way, we have little chance of successfully managing the consequences.

If one had to choose a tipping point, a specific moment at which this change crystallised, it would probably be the 1979 Islamic revolution in Iran. Deeply shocking to the secular world, it appeared at the time to be pushing against the tide of history: now it seems instead to have been the harbinger of its turning. After decades of humiliating intervention by the British and the Americans, dissenting Iranian politicians – many of them far from devout – saw in the forms of Iranian Shi’ism a way of defining and asserting the country’s identity against the outsiders. The mosque, even more than the bazaar, was the space in which new national narratives could be devised and in which all of society could engage. Ten years later, the Lutheran churches of East Germany played a comparable role for those resisting and ultimately overthrowing an oppressive state sustained in office by the armies of the Soviet Union. In both cases, faith structures (literal and metaphorical) were for a time seen not as oppression, but as frameworks for freedom.

If one had to choose a tipping point, a moment when this change crystallised, it would be the 1979 Islamic revolution in Iran

Many states since then have followed a similar path. In a way that could hardly have been imagined 60 years ago, the reassuring politics of prosperity have in many parts of the world been replaced by the often violent rhetoric and politics of identity, articulated through religion. This should not surprise us. It is a return to the pattern prevalent in almost all societies through history; for along with language and nationalism, religion is one of the most powerful forces for shaping and energising a community. Not because it is a structure of control (though it is often crushingly that), but because it gives the group a narrative of who they are and what, together, they can become. Religion in its social dimension is less about escapism or consolation, than about hope, new behaviour and the challenge of struggling towards the future.

“We tell ourselves stories in order to live.” Joan Didion’s famous sentence is not a reflection on religion, but it speaks to exactly that compelling need we all have for stories that give shape and significance to our individual lives. Religious stories are part of a similar strategy, but for communal survival. Addressing the conundrums of life and death, they offer not explanation but meaning. As much about the future as the past, they provide a society with a narrative that goes far beyond the self, embracing the living, the dead and those still to be born in one continuing story of belonging. Not surprisingly, evidence from the ice age onwards suggests that societies with such narratives appear better equipped to overcome threats to their existence, to survive and to flourish. At the beginning of the 20th century, the French sociologist Émile Durkheim argued that there can in fact be no society unless it has such shared overarching stories, what he called “an idea that it constructs of itself”: that shared communal idea – or ideal – and its narratives were the precondition of a functioning society. We are, as a group, the stories that we tell about ourselves. If, for whatever reason, we lose or forget those narratives, we collectively no longer exist. In the rituals and ceremonies in which they are enacted, the society may be worshipping a remote, perhaps even a non-existent, god: it is also, Durkheim suggested, venerating an ideal version of itself, which may one day be made real.

Such narratives of faith can create wonderfully potent symbols of solidarity. One example: Hindus in Bengal celebrate every autumn the return of the goddess Durga and her triumph over the forces of evil. To mark the festival they make each year a new statue of the deity. These painted representations of the goddess are made, not by a solitary artist, but in effect by the whole community. Supported by a wicker frame, the bulk of the body is composed of clay. But great efforts are made to incorporate bits of earth from all the different parts of the locality, from rich areas as well as poor. By convention, for the final mix, a priest should beg for some soil from a sex worker’s house, and there should ideally also be some clay from the banks of the Ganges, as Durga has a close connection to the river. The result is that in the temporary statue of the goddess every part of the community – rich and poor, weak and strong – is represented and honoured, and directly connected to the endlessly renewing, life-giving river. The image Durga comes to “inhabit” during her festival that the people venerate thus carries within it the whole place and the whole community, physically and symbolically present. It is an image endlessly made, destroyed and remade by the people themselves. Few political structures have found metaphors so emotionally compelling for a society in which everybody has their place.

Facebook Twitter Pinterest The goddess Durga represents the whole place, the whole community … the Durga Puja festival, India. Photograph: Arun Sankar K/AP

The power of such narratives, beliefs and rituals to sustain communities through danger and across centuries is a recurrent fact of history. It is a central part of the story of the Jews after the destruction of the Temple in Jerusalem in 70AD and the brutal campaigns of Hadrian; of the people of Ethiopia, alone in Africa in resisting the colonial invaders; and of the African slaves transported to the Americas. In such circumstances, religion offers an architecture of meaning in which people may find shelter and hope. For many today, in areas of economic disruption and dysfunctional states, it may be the only architecture available. And it is surely part of the reason why across much of the world, belief is back.

Religion: why faith is becoming more and more popular Read more

It is a view of religion with which many Europeans struggle. It is not just that the rational mind rejects as scientifically untrue the theological or mythological frameworks of faith, or that beliefs have over millennia been consciously manipulated by rulers and priests to support the powerful, and to exclude and persecute whole sections of society. The Holocaust and the many moral failures of both Catholic and Protestant churches in the face of Nazi evil have been followed by their long institutional refusals to embrace widely shared ideals of equality. Many find it hard to see here forces that will shape a better world, which perhaps explains the growing focus across the west on the individual quest for truth and for private spirituality.

But this risks blinding us to the power of narratives that articulate an ideal, that offer fulfilment in the context of a community, make demands on everyone, and – above all – hold out hope. It may be a matter for regret, it may represent a failure of secular politics, but it should certainly be no surprise that so many societies now see in such narratives of faith their best way forward.

• Living With the Gods by Neil MacGregor is published by Allen Lane (RRP £30). To buy a copy for £25.80 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.