Citizens will have to sacrifice their right to privacy in the fight against terrorism, a former senior security official warned today.

Sir David Omand, the Cabinet Office's former security and intelligence co-ordinator, said in future the security services would need access to a wide range of personal data, including phone records, emails and travel information.

In a research paper on national security strategy, Sir David wrote: "Finding out other people's secrets is going to involve breaking everyday moral rules."

Download the new Independent Premium app Sharing the full story, not just the headlines

The document for the Institute for Public Policy think tank outlines plans to track terrorist groups through a state database which would also contain the details of innocent people.

He wrote: "Modern intelligence access will often involve intrusive methods of surveillance and investigation, accepting that, in some respects this may have to be at the expense of some aspects of privacy rights.

"This is a hard choice, and goes against current calls to curb the so-called surveillance society - but it is greatly preferable to tinkering with the rule of law, or derogating from fundamental human rights.

"Being able to demonstrate proper legal authorisation and appropriate oversight of the use of such intrusive intelligence activity may become a major future issue for the intelligence community, if the public at large is to be convinced of the desirability of such intelligence capability."

The "intrusive" surveillance techniques would involve mining databases for information on airline bookings and other travel data, passport and biometric data, immigration, identity and border records, criminal records and other government and private sector data, including financial and telephone and other communications records.

Sir David said such information may be held in national records, covered by Data Protection legislation, but it might also be held offshore by other nations or by global companies, and may or may not be subject to international agreements.

"Access to such information, and in some cases to the ability to apply data mining and pattern recognition software to databases, might well be the key to effective pre-emption in future terrorist cases," he wrote.