Privacy rights of innocent people will have to be sacrificed to give the security services access to a sweeping range of personal data, one of the architects of the government's national security strategy has warned.

Sir David Omand, the former Whitehall security and intelligence co-ordinator, sets out a blueprint for the way the state will mine data - including travel information, phone records and emails - held by public and private bodies and admits: "Finding out other people's secrets is going to involve breaking everyday moral rules."

His paper provides the most candid assessment yet of the scale of Whitehall's ambitions for a state database to track terrrorist groups. It argues that while the measures are essential, public trust will be maintained only if such intrusive surveillance is carried out within a strong framework of morality and human rights.

"Modern intelligence access will often involve intrusive methods of surveillance and investigation, accepting that, in some respects this may have to be at the expense of some aspects of privacy rights," he writes in a newly published Institute for Public Policy research paper.

"This is a hard choice, and goes against current calls to curb the so-called surveillance society - but it is greatly preferable to tinkering with the rule of law, or derogating from fundamental human rights. Being able to demonstrate proper legal authorisation and appropriate oversight of the use of such intrusive intelligence activity may become a major future issue for the intelligence community, if the public at large is to be convinced of the desirability of such intelligence capability."

Although Omand left the senior civil service in 2005, his views continue to reflect thinking at the highest levels of Whitehall. In the paper - National Security Strategy, Implications for the UK Intelligence Community - he says that a growing amount of intelligence to remove extremists and disrupt potential terrorist attacks is now derived from "protected information" - known in intelligence circles as "Protint".

"This is personal information about individuals that resides in databases such as advanced passenger information, airline bookings, and other travel data, passport and biometric data, immigration, identity and border records, criminal records and other governmental and private sector data, including financial and telephone and other communications records.

"Such information may be held in national records, covered by data protection legislation, but it might also be held offshore by other nations, or by global companies, and may or may not be subject to international agreements. Access to such information, and in some cases to the ability to apply data mining and pattern recognition software to databases, might well be the key to effective pre-emption in future terrorist cases."

Omand says such sources have always been accessible to the police and security services against a named suspect, but the use of modern data mining and processing methods involves the examination of the innocent as well as the suspect to identify patterns of interest for further investigation.

Privacy issues do arise, he says, over the use of sources of information on the movements and activities of individuals revealed by technology such as CCTV and automatic number plate readers - again complicated by the use of "smart recognition software" such as data-mining programmes.

He says that the significant challenge facing the intelligence community is how to access the full range of personal data in a way that is timely, accurate, proportionate, legal and acceptable in a democratic and free society.

Safeguards must include appropriate oversight and means of independent investigation and redress in cases of alleged abuses of power.