The home secretary, Jacqui Smith, today ruled out building a single state "super-database" to track everybody's use of email, internet, text messages and social networking sites such as Facebook and Twitter.

Smith said creating a single database run by the state to hold such personal data would amount to an extreme solution representing an unwarranted intrusion of personal privacy.

Instead the Home Office is looking at a £2bn solution that would involve requiring communications companies such as BT, Virgin Media, O2 and others to retain such personal data for up to 12 months.

The decision to abandon a state central database is a setback for the police and security services who wanted rapid access to the data while conducting counter-terror and crime investigations. Instead they will have to apply for the data to be released to them on a case-by-case basis to each individual telecoms and internet company.

Smith, publishing a consultation paper detailing the private sector solution, said: "Advances in communications mean that there are ever more sophisticated ways to communicate and we need to ensure that we keep up with the technology being used by those who would seek to do us harm."

She said it was essential that the police and other crime-fighting agencies had the tools they needed to do their job. "However, to be clear there are absolutely no plans for a single central store. We recognise that there is a delicate balance between privacy and security, but to do nothing is not an option as we would be failing in our duty to protect the public."

She argued that more privacy concerns were raised by the proposal to store all the personal communications data in a state-run single database than individual private companies retaining the information for access by the police and security services as and when they needed it.

The home secretary said the cost of the project was around £2bn over 10 years, far short of speculation that it could be as high as £12bn.

The Home Office consultation paper published today was originally expected in the early new year but has been delayed by an intense cabinet debate over whether or not to go ahead with the state database.

The paper makes clear that the Home Office regards it as technically the best option, describing it as "the least challenging technically to implement and the cheapest to build and run".

But it adds that the government "recognises the privacy implications in holding all communications data from the UK in a 12-month period in a single store. The government therefore does not propose to pursue this approach."

Instead communications companies are to be required by legislation to ensure that all traffic data – who sent a text to whom at what time and from where – is collected and kept in Britain. They will also be asked to store additional third-party data crossing their networks including phone calls and internet use from outside Europe.

This goes far beyond the current data collected for billing purposes. The companies will also be asked to organise the data – for example, matching it where it relates to the same person so that the authorities can access it in a form that is immediately usable.

Smith said that while the new system could record a visit to a social network such as Facebook, it would not record personal and private information such as photos or messages posted to a page.

"What we are talking about is who is at one end [of a communication] and who is at the other – and how they are communicating," she said.

Shami Chakrabarti, the director of Liberty, said the details of the policy remained unclear. "The Home Office has been sending out conflicting signals on the superdatabasefor some months, frantically briefing journalists in private but failing to publish any detailed proposals," she said. "If this statement from the home secretary marks a genuine change of direction on privacy policy, we will welcome it. However, it might be wise to read the small print first," she said.