It's more than 40 years since a bunch of young protesters broke into a government bunker and published the documents about the state's secret preparations for the possibility of nuclear war they found there. The Spies for Peace, as they called themselves, were cheered on by many in the press and in the peace movement. As they included my late father, I grew up knowing their identities - although none of them broke cover while they were alive. When I wrote about them in the past, I mused on whether they had actually achieved anything. What I concluded was that, although they and other protesters of the 60s never achieved all they had set out to do, they did help to change the culture. The difference, politically, between the pre-60s generation and the post-60s generation was partly a shift from deference to intransigence.

Civil liberties - such as the rights to liberty and assembly and freedom of speech - are ancient rights enshrined in common law as well as modern rights enshrined in international human rights legislation. But if they are not to be eroded by the political class whose desire is always to acquire greater power then people need to remain vigilant. In the 60s, the cultural direction suggested that any attempt by the government to increase its power over the people would automatically be challenged. Many ordinary people decided they were no longer going to bend the knee automatically to the government.

Now we have changed direction. Despite a pervasive air of cynicism about our political masters, there is little challenge to the secretive and authoritarian bent of those who rule us. As Helena Kennedy neatly put it last year: "What we have forgotten is that the state is there courtesy of us and we are not here courtesy of the state." Although sporadic resistance flares up, the rights and liberties of ordinary people are increasingly under threat. There is, most famously, the curtailment of the right to peaceful protest. The government has used powers contained in the Serious Organised Crime and Police Act 2005 to prevent "unauthorised" demonstrations within a kilometre of parliament. The 1997 Protection From Harassment Act, updated in the 2005 act, has been used in attempts to shut down protests against militarism and climate change.

Powers such as these, or those conferred on the police by the Terrorism Act 2000 to stop and search in a "designated" area even without "reasonable grounds", are generally accepted as the price we must pay for our safety in an increasingly hostile world, even though their use goes way beyond the needs of security. Similarly, the growing powers of the government to hold information about us, including the power to keep a permanent database of the DNA of everyone who has ever been arrested - even if not guilty - are rarely challenged.

It is partly our paranoia about terrorism that has contributed to our lethargic acquiescence in such authoritarian behaviour by the state. There are still definitely limits to this acquiescence. It is cheering to see that new attempts by the government to extend the period of detention without charge from 28 to 42 days have come in for such concerted opposition from other parties and commentators of all political backgrounds.

Yet the reason there is not more forceful opposition to other aspects of the government's authoritarian behaviour is, shamingly, partly down to the fact that many of us do not feel these measures directly affect us and we cannot be bothered to think our way into the minds of those who are most affected. The most powerful arguments for civil liberties are those that remind us that "we" have the right to keep the state in line. But when those most heavily affected do not look like "us", we are less likely to protest. When the poetry and reading habits - however nasty - of a young woman can be used to secure her conviction for possessing materials "likely to be useful to a person committing or preparing an act of terrorism", then we feel the control of the state tightening. But because Samina Malik is hardly someone that most British people could sympathise with, it is fatally easy for her rights to free expression to be downgraded.

The terror suspects who have experienced the most freakish curtailments of their liberty are people such as Mustapha Taleb, one of the cleared suspects in the ricin trial, whom I met last year when he was living under the most intensely onerous bail conditions. These meant that he could only move around a small designated area and could not use mobile phones or the internet. Taleb has since had to experience continued detention while awaiting the results of secret hearings of secret evidence to which his own lawyer does not have access. If few people have bothered to listen to his frustration with the maze in which he finds himself, I think that's because we are reluctant to extend to alien immigrants - even though he had legal status here as a refugee - the same right to justice we believe is due to us.

The rights to liberty and a fair legal hearing are being routinely denied not just to terror suspects. Our asylum system, so tortuous and Kafkaesque that it defeats trained lawyers, let alone frightened victims of persecution who have just arrived on our shores, condemns many genuine asylum seekers to a paperless limbo or to arbitrary detention. While MPs and journalists debate whether suspected terrorists should be detained for more than 28 days without charge, for instance, a young girl of 13 can be held in Yarl's Wood detention centre for three months.

Meltem Avcil, a Turkish Kurdish girl whom I met during her lengthy incarceration, briefly became a cause celebre among campaigners when she and her mother resisted deportation. An innocent, bright teenager who has been living in this country for six years was held in prison conditions that were a terrifying ordeal for her and her mother. "I wish I had died and never seen that day," she said of the day she was taken into detention.

But she is hardly alone. Immigration detention centres in the UK are stuffed with people whose detention has been arbitrary and prolonged. Many of them have been unable to access competent - or sometimes any - legal representation. This is only set to worsen with the cuts to legal aid now coming into force. Although the government says it only detains children or victims of torture under exceptional circumstances, today I could take you to meet children who have been locked up for weeks, and women who bear the marks of torture on their bodies and in their minds - with no idea when they will be released or deported - sitting in fear in these detention centres. Such people are silenced. They are pushed to the margins of all our debates, and when we talk about "our" civil liberties and our human rights these people, who are not "us", hardly seem to figure.

Once upon a time people protested vehemently that the government should not act secretly and repressively against its own people. It is vital to continue that tradition, but it is just as vital to state clearly that our government should not act secretly and repressively against any individual in this country. Civil liberties must be for everyone, or we will find one day that they are for nobody.

natasha.walter@theguardian.com