The following is an edited extract from a new book by Johan Lidberg and Denis Muller (eds), In the name of security: Secrecy, surveillance and journalism, published by Anthem Press.

The dramatic increase in national security laws has confronted journalism with threats and challenges so great as to weaken its fourth-estate capacity and unnecessarily curtail its ability to inform the public about the largest political issues of our time: terrorism, and what governments are doing in their people’s names to respond.

These threats and challenges take many forms. The most ubiquitous and potentially harmful is the threat to the anonymity of confidential sources. Technological developments that facilitate surveillance by the state security apparatus make it increasingly difficult for journalists to protect confidential sources from agencies that may wish to prosecute them for breaches of laws criminalising specific disclosures of information. It also shows that the laws regulating surveillance in the Five Eyes countries – US, UK, Canada, Australia and New Zealand – lack adequate safeguards to protect journalists’ sources.

The technological capacity to track individuals and their communications surpasses anything previously seen in human history. This, combined with the plethora of laws passed since 9/11 creates a suffocating blanket of surveillance backed by legal powers of suppression, some of which go so far as to obtrude on the principle of habeas corpus.

Read more: New bill would make Australia worst in the free world for criminalising journalism

The seriousness of these consequences imposes new and weighty ethical challenges on journalists. The ethical principles involved are universal among democracies and so the challenges are the same for journalists in all the countries studied. In addition to source-protection, these challenges include deciding how far, and in what circumstances, a journalist may ethically break the law in order to inform the public. This is made especially acute by the fact that it is rare in any of the jurisdictions studied to have a public-interest test against which disclosure may be tested and excused.

AAP/Lukas Coch

A further ethical complication is that it is a central responsibility of journalists who are prepared to receive information from confidential sources to satisfy themselves, so far as possible, that the source is acting in good faith and not prosecuting a hidden agenda.

Secrecy surrounding security services and their operations is more entrenched in Australia and the UK than in the other countries studied, making it even more difficult for journalists in those countries to obtain information about what is being done by the security agencies in those countries’ names.

Read more: New foreign interference laws will compound risks to whistleblowers and journalists

Even though it is incontrovertible that there has been a dramatic shift towards protection of national security at the price of some encroachment on civil liberties, this is a price voters in those countries are prepared to pay in order that their sense of safety may be enhanced. This is clearly shown by an analysis of public opinion polls. The polls also show that voters only become concerned about impingements on privacy or civil liberties when they can see that they personally might get entangled in the security machine.

There are some differences between the Five Eyes countries in their responses to terrorism and in the constraints on government. In the US, the first amendment to the Constitution explicitly supporting a free press still provides protection for journalism when it comes to reporting security matters. Canada and New Zealand appear similar to the US in this respect.

The two real stand-outs among the Five Eyes are the UK and Australia. This can probably be explained by the UK Official Secrets Act and incorporation of the spirit of this act into Australian laws. This is most clearly manifest by the fact that access to information laws do not apply to security and intelligence agencies in these two countries.

Add to this the fact that the Australian parliament has passed more anti-terror laws than any other liberal democracy since 2001, and a continuum emerges where journalism in Canada and New Zealand appear least affected by the shift in the security and openness see-saw, with the US somewhere in the middle of the continuum and the UK and Australian journalism most affected by the rebalancing.

Journalism in the other countries of study, Brazil, India, South Africa, Denmark and Germany, appears to be less affected by legal changes and more troubled by the wide and unclear definitions of national security. This makes it possible for governments to classify large tranches of information under the wide umbrella of national security and block access to this information.

The relationship between journalism and national security in Brazil, India and South Africa is examined by looking in particular at information-access regimes. The study shows that while in each country laws promoting open government are passed in a flush of idealism, gradually they get whittled down as governments find openness not so attractive in practice.

This is not so different from the life cycle of information-access regimes in more mature democracies. While the EU exerts a moderating influence on the secrecy inclinations of member states, Denmark and Germany both languish well below the mid-point on the 30-point Global Right to Information (RTI) index of openness.

Until recently, there was little international guidance on how to balance national security and access to information. However, in 2013 at Tshwane in South Africa, representatives from 70 countries reached agreement on 50 principles offering comprehensive guidance on how to strike the balance between national security and public access to information.

The principles most relevant to the issues canvassed in this book say:

The public has a right of access to government information, including information from private entities that perform public functions or receive public funds (Principle 1).

It is up to the government to prove the necessity of restrictions on the right to information (Principle 4).

Governments may legitimately withhold information in narrowly defined areas, such as defence plans, weapons development, and the operations and sources used by intelligence services. Also, they may withhold confidential information supplied by foreign governments that is linked to national security matters (Principle 9).

But governments should never withhold information concerning violations of international human rights and humanitarian law, including information about the circumstances and perpetrators of torture and crimes against humanity, and the location of secret prisons. This includes information about past abuses under previous regimes, and any information they hold regarding violations committed by their own agents or by others (Principle 10A).

The public has a right to know about systems of surveillance, and the procedures for authorising them (Principle 10E).

No government entity may be exempt from disclosure requirements—including security sector and intelligence authorities. The public also has a right to know about the existence of all security sector entities, the laws and regulations that govern them, and their budgets (Principles 5 and 10C).

Whistleblowers in the public sector should not face retaliation if the public interest in the information disclosed outweighs the public interest in secrecy. But they should have first made a reasonable effort to address the issue through official complaint mechanisms, provided that an effective mechanism exists (Principles 40, 41, and 43).

Criminal action against those who leak information should be considered only if the information poses a real and identifiable risk of causing significant harm that overrides the public interest in disclosure (Principles 43 and 46).

Journalists and others who do not work for the government should not be prosecuted for receiving, possessing or disclosing classified information to the public, or for conspiracy or other crimes based on their seeking or accessing classified information (Principle 47).

Journalists and others who do not work for the government should not be forced to reveal a confidential source or other unpublished information in a leak investigation (Principle 48).

Public access to judicial processes is essential: invocation of national security may not be relied upon to undermine the fundamental right of the public to access judicial processes. Media and the public should be permitted to challenge any limitation on public access to judicial processes (Principle 28).

Governments should not be permitted to keep state secrets or other information confidential that prevents victims of human rights violations from seeking or obtaining a remedy for their violation (Principle 30).

There should be independent oversight bodies for the security sector, and the bodies should be able to access all information needed for effective oversight (Principles 6, 31–33).

Information should be classified only as long as necessary, and never indefinitely. Laws should govern the maximum permissible period of classification (Principle 16).

There should be clear procedures for requesting declassification, with priority procedures for the declassification of information of public interest (Principle 17).

The implementation of the Tshwane principles has become more important than ever as the phenomenon of fake news has developed in the wake of the 2016 election of Donald Trump as president of the US. This is of course not a new occurrence. Disinformation has been used by security and intelligence agencies for many decades. What is new is the scope and scale, to a large extent magnified by social media.

AAP/EPA/Larry W. Smith

What is also new is the extent to which professional media and investigative journalists struggle to get through to the public with their verified accounts, showing what was fake and what was real news. In this environment, access to raw information becomes more important than ever. This is made all the more complex by the fact that many intelligence agencies are active players in the misinformation and disinformation game. The above points make reporting security and intelligence issues more challenging than ever.

Democracies need to reassess their fundamental values and the price in civil liberties they are prepared to pay for national security. This confronts the citizens of those countries with a starkly uncomfortable question: At what point does the trade-off mean that the terrorists have won? If these powers, developed in times of emergency, are not rolled back in times of lower threat levels, they will increase from a higher level next time there is a real or perceived security emergency. This simply entrenches the erosion of civil liberties and the concomitant weakening of democratic principles.

Read more: Government needs to slow down on changes to spying and foreign interference laws

The war on terror has no single identifiable enemy. Indeed, a lot of the real and perceived threats come from within nation-states, as well as from outside. In these circumstances, there is no single entity with whom governments can negotiate the terms of peace. That leaves the prospect of an open-ended period during which the state of exception increasingly becomes the state of normality, supported by ever-growing mass surveillance capabilities as described by whistleblowers interviewed in the BBC documentary Weapons of Mass Surveillance. The whistleblowers’ testimonies point to a future where security agencies can record all digital traffic within a country in real time, store it and conduct retrospective searches of suspect activity.

Who do we entrust with these truly awesome surveillance powers? How will they be used by authoritarian regimes?

How far are we prepared to go in the name of security?