A secret US cybersecurity report warned that government and private computers were being left vulnerable to online attacks from Russia, China and criminal gangs because encryption technologies were not being implemented fast enough.

The advice, in a newly uncovered five-year forecast written in 2009, contrasts with the pledge made by David Cameron this week to crack down on encryption use by technology companies.

In the wake of the Paris terror attacks, the prime minister said there should be no “safe spaces for terrorists to communicate” or that British authorites could not access.

Cameron, who landed in the US on Thursday night, is expected to urge Barack Obama to apply more pressure to tech giants, such as Apple, Google and Facebook, which have been expanding encrypted messaging for their millions of users since the revelations of mass NSA surveillance by the whistleblower Edward Snowden.

Cameron said the companies “need to work with us. They need also to demonstrate, which they do, that they have a social responsibility to fight the battle against terrorism. We shouldn’t allow safe spaces for terrorists to communicate. That’s a huge challenge but that’s certainly the right principle”.

But the document from the US National Intelligence Council, which reports directly to the US director of national intelligence, made clear that encryption was the “best defence” for computer users to protect private data.

Part of the cache given to the Guardian by Snowden was published in 2009 and gives a five-year forecast on the “global cyber threat to the US information infrastructure”. It covers communications, commercial and financial networks, and government and critical infrastructure systems. It was shared with GCHQ and made available to the agency’s staff through its intranet.

One of the biggest issues in protecting businesses and citizens from espionage, sabotage and crime – hacking attacks are estimated to cost the global economy up to $400bn a year – was a clear imbalance between the development of offensive versus defensive capabilities, “due to the slower than expected adoption … of encryption and other technologies”, it said.

An unclassified table accompanying the report states that encryption is the “[b]est defense to protect data”, especially if made particularly strong through “multi-factor authentication” – similar to two-step verification used by Google and others for email – or biometrics. These measures remain all but impossible to crack, even for GCHQ and the NSA.

The report warned: “Almost all current and potential adversaries – nations, criminal groups, terrorists, and individual hackers – now have the capability to exploit, and in some cases attack, unclassified access-controlled US and allied information systems.”

It further noted that the “scale of detected compromises indicates organisations should assume that any controlled but unclassified networks of intelligence, operational or commercial value directly accessible from the internet are already potentially compromised by foreign adversaries”.

The primary adversaries included Russia, whose “robust” operations teams had “proven access and tradecraft”, it said. By 2009, China was “the most active foreign sponsor of computer network intrusion activity discovered against US networks”, but lacked the sophistication or range of capabilities of Russia. “Cyber criminals” were another of the major threats, having “capabilities significantly beyond those of all but a few nation states”.

The report had some cause for optimism, especially in the light of Google and other US tech giants having in the months prior greatly increased their use of encryption efforts. “We assess with high confidence that security best practices applied to target networks would prevent the vast majority of intrusions,” it concluded.

Official UK government security advice still recommends encryption among a range of other tools for effective network and information defence. However, end-to-end encryption – which means only the two people communicating with each other, and not the company carrying the message, can decode it – is problematic for intelligence agencies as it makes even warranted collection much more difficult.

The latest versions of Apple and Google’s mobile operating systems are encrypted by default, while other popular messaging services, such as WhatsApp and Snapchat, also use encryption. This has prompted calls for action against such strong encryption from ministers and officials. Speaking on Monday, Cameron asked: “In our country, do we want to allow a means of communication between people which we cannot read?”

The previous week, a day after the attack on the Charlie Hebdo office in Paris, the MI5 chief, Andrew Parker, called for new powers and warned that new technologies were making it harder to track extremists.

In November, the head of GCHQ, Robert Hannigan, said US social media giants had become the “networks of choice” for terrorists. Chris Soghoian, principal senior policy analyst at the American Civil Liberties Union, said attempts by the British government to force US companies to weaken encryption faced many hurdles.

“The trouble is these services are already being used by hundreds of millions of people. I guess you could try to force tech companies to be less secure but then they would be less secure against attacks for anyone,” he said.

GCHQ and the NSA are responsible for cybersecurity in the UK and US respectively. This includes working with technology companies to audit software and hardware for use by governments and critical infrastructure sectors.

Such audits uncover numerous vulnerabilities which are then shared privately with technology companies to fix issues that could otherwise have caused serious damage to users and networks. However, both agencies also have intelligence-gathering responsibilities under which they exploit vulnerabilities in technology to monitor targets. As a result of these dual missions, they are faced with weighing up whether to exploit or fix a vulnerability when a product is used both by targets and innocent users.

The Guardian, New York Times and ProPublica have previously reported the intelligence agencies’ broad efforts to undermine encryption and exploit rather than reveal vulnerabilities. This prompted Obama’s NSA review panel to warn that the agency’s conflicting missions caused problems, and so recommend that its cyber-security responsibilities be removed to prevent future issues.

Another newly discovered document shows GCHQ acting in a similarly conflicted manner, despite the agencies’ private acknowledgement that encryption is an essential part of protecting citizens against cyber-attacks.

The 2008 memo was addressed to the then foreign secretary, David Miliband, and classified with one of the UK’s very highest restrictive markings: “TOP SECRET STRAP 2 EYES ONLY”. It is unclear why such a document was posted to the agency’s intranet, which is available to all agency staff, NSA workers, and even outside contractors.

The memo requested a renewal of the legal warrant allowing GCHQ to “modify” commercial software in violation of licensing agreements. The document cites examples of software the agency had hacked, including commonly used software to run web forums, and website administration tools. Such software are widely used by companies and individuals around the world.

The document also said the agency had developed “capability against Cisco routers”, which would “allow us to re-route selected traffic across international links towards GCHQ’s passive collection systems”.

GCHQ had also been working to “exploit” the anti-virus software Kaspersky, the document said. The report contained no information on the nature of the vulnerabilities found by the agency.

Security experts regularly say that keeping software up to date and being aware of vulnerabilities is vital for businesses to protect themselves and their customers from being hacked. Failing to fix vulnerabilities leaves open the risk that other governments or criminal hackers will find the same security gaps and exploit them to damage systems or steal data, raising questions about whether GCHQ and the NSA neglected their duty to protect internet systems in their quest for more intelligence.

A GCHQ spokesman said: “It is long-standing policy that we do not comment on intelligence matters. Furthermore, all of GCHQ’s work is carried out in accordance with a strict legal and policy framework, which ensures that our activities are authorised, necessary and proportionate, and that there is rigorous oversight, including from the secretary of state, the interception and intelligence services commissioners and the parliamentary intelligence and security committee.“All our operational processes rigorously support this position. In addition, the UK’s interception regime is entirely compatible with the European convention on human rights.”

Michael Beckerman, president and CEO of the Internet Association, a lobby group that represents Facebook, Google, Reddit, Twitter, Yahoo and other tech companies, said: “Just as governments have a duty to protect to the public from threats, internet services have a duty to our users to ensure the security and privacy of their data. That’s why internet services have been increasing encryption security.”