While secularism is on the increase, immigration from Poland and the Indian subcontinent could reverse this trend

A blue plaque on a white stucco house just off the seafront in Brighton is a rare monument to atheism in a country where religion is a minority belief.

It marks the former home of George Holyoake, the last man to be jailed for refusing to believe in God and an overlooked hero of the secular cause.

Holyoake, a free-thinking radical, was jailed in Cheltenham in 1842 after suggesting, at the end of a lecture on socialism, that religion was a luxury the poor could not afford. The town's conservative establishment prosecuted him for his outspokenness – one priest called it devilism – and it is said he was threatened with being taken from Cheltenham to Gloucester jail in chains.

After his release from prison, he retreated for the last part of his life to Brighton. It was an appropriate refuge: the city is now, according to demographers, the least religious place in Britain.

Yet even today Holyoake would stand out as an exception in Brighton – a man prepared to speak confidently about his lack of belief rather than fudge the issue.

Most people in the Sussex city do not go to church: 27% said they had no religion in the last census. But most still described themselves as Christians. In Britain, cultural ties remain strong, even as belief fades.

"People here look at you a bit strangely if you say you are a regular churchgoer," says Bill McIlroy, a member of Brighton's humanist and secular association. "But while many people here don't believe, they have still got a misplaced respect for the church."

That sense of tradition frustrates campaigners against the influence of what is now politely described as the faith community.

"People cling to the idea of religion as a source of morality," says Terry Sanderson, president of the National Secular Society. "There is a general apathy: people don't want to make a fuss."

Many Britons have at best a shallow belief in God: the most recent British Social Attitudes survey found that just a third of the population held firm religious beliefs, with another third deeply sceptical and the final third uncertain.

But far fewer are prepared to go further and describe themselves as openly atheist. It was telling that Nick Clegg caused a stir when he did so soon after becoming Liberal Democrat leader.

"People prefer to talk about spirituality rather than religion, which can mean anything you want it to," says Sanderson.

Even Holyoake – who went on to invent the term secularism – professed himself more sceptical about religion than opposed. "You cannot be an atheist – you do not look like one," his memoirs record the magistrate telling him during his trial for blasphemy.

Holyoake held firm: "Though sorry to say what might outrage them or look like obstinacy, yet out of respect to my own conscience I must say that I was an atheist."

But by the end of his life he preferred to use the newly devised term agnostic. That still best fits the mainstream British view.

According to demographer David Voas at Manchester University: "The part of the population that is properly religious is a minority, but so is that part which is overtly secular. In the middle is an informal group of fuzzy people who don't really care."

Britain is not, as some think, the most secular nation in Europe; it is less so than Scandinavia and parts of the former Soviet bloc.

Migrant communities are the most committed – even long-standing ones, such as Irish Catholics who dominate what Voas calls "the British bible belt running from Merseyside to Wigan pier".

Most other people are less engaged, although measuring belief is difficult and the data conflicting.

Much depends on how questions are asked and whether they refer to broad cultural identity or practical belief.

The census, which records the highest level of religious belief, assumes people are adherents of one religion or another and actively asks them to choose from a list.

In 2001, more than two-thirds selected Christian and well under a fifth no religion. That, says Voas, overstates the reality of British religious belief. Asked to choose Christian rather than Muslim or Hindu, people see it as an ethnic identity. Voas argues that the British Social Attitudes survey, which found over half the population holding no strong religious views, is a better measure.

Other polls go further. In a 2006 Guardian/ICM poll only 33% described themselves as religious, against 63% who said they were not – including a majority who described themselves more broadly as culturally Christian.

Women and older people were more likely than average to believe in a god, but overall only 13% said they went to a place of worship at least once a week. More people saw religion as a force for harm than for good.

That suggests a country where religion is on the retreat. "There is a huge change – the trend is towards secularism," says Sanderson.

Others, pointing to the impact of migration, from Catholic Poland as well as the Indian subcontinent, are less sure.

The country's religious makeup is fragmenting as the Church of England declines. "It is not the case that Britain is getting more religious," says Voas.

The pope will find himself on an island full of doubters.