The rapidly changing nature of internet-based communications has left the security agencies and the police unable to legally track the online activities of terrorists and serious criminals in 25% of cases, the Home Office says.

Security chiefs say that the current law requiring European-based communications service providers, such as BT and Virgin Media, to collect and store monitoring data on everyone's email and internet use fails to cover major overseas-based players, including Gmail and Hotmail.

"In some cases it is no longer possible to obtain data about the sender and recipient of an email," said Charles Farr, the head of the Office for Security and Counter-Terrorism. "Communications data from internet-based services is not always available; for some internet-based services it is not generated, collected and stored by the internet service provider. Many service providers are based overseas," he said.

"About 25% of requests for communication data by the police and agencies can no longer be met. This has a direct impact on the investigation of crime in this country and our ability to identify and then prosecute criminals and terrorists."

Home Office sources say that the "traffic data" – who was in contact with whom from what location and at what time – now provides evidence in 95% of serious organised crime investigations and has "played a significant role in every major security service counterterrorist investigation over the past decade".

The new internet surveillance law will not give the police the power to retain the content of any emails, tweets or other social media use or their use in real time. Instead it will build on an existing 2009 European Union directive which requires internet service providers to collect and store for 12 months the mobile phone and internet use data they keep for the purposes of billing customers.

The Home Office says it will foot the bill for the costs of the industry keeping the 25% extra data that is required. Earlier estimates of a £2bn bill applied to the creation of a central Home Office database that is no longer being proposed.

The bill will restrict access to the most sensitive types of data to the police, emergency services and intelligence agencies. Local authorities, which were responsible for 1,800 out of the 500,000 requests for such data last year, will need the approval of a magistrate in future for such requests.

Senior police officer Peter Davies, of the Child Exploitation and Online Protection Centre, said the trail went cold in 4,000 out of the 12,000 suspected child abuse cases it investigated each year because it lacked access to the records of the suspect's email traffic. He cited a recent operation by the Lincolnshire police which uncovered a child sex abuse ring and identified 211 suspects. "We believe there were another 70 suspects we could not identify and investigate because we could not follow the email trail," said Davies.

The surveillance measures will form the most controversial element in a wider law and order package that will also include a separate crime and court bill that will feature in Wednesday's Queen's speech.

The bill will also include legislation to establish the National Crime Agency from next April, to speed up immigration appeals and to strengthen the powers of UK Border Force officers.

The crime and courts bill will also include radical Ministry of Justice proposals to reform judicial appointments and a radical scheme to allow magistrates sitting on their own to operate from community centres and police stations to deal with low-level uncontested cases within days or even hours of arrest.

But it is the new stand-alone bill on internet surveillance that has already proved the most controversial within the coalition, with the Liberal Democrats insisting that it must be accompanied by the "strongest possible safeguards", including independent oversight, and measures to protect the security of everyone's data. It is now expected that the legislation will include a promise to publish a privacy impact statement of the new powers and the access regime will be overseen by the interception of communications commissioner.

Nick Clegg's office has been quoting with approval the Liberal Democrat president, Tim Farron, who has insisted that there "must be absolutely no question of universal internet surveillance" and warned that the party will oppose the bill if it believes it amounts to "a threat to a free and liberal society".

Clegg has won a promise that the draft clauses of the bill will not be "rammed through parliament" and instead will be subject to open parliamentary hearings, with evidence taken from civil liberty campaigners as well as the police. But the stand-alone bill has won a slot in the government's legislative timetable. Claims that the Liberal Democrats had ensured it would only be published as a "draft bill" without a place in the Queen's speech programme have proved wide of the mark.