A revolution in medical research in Britain is to give academics and the life sciences industry unparalleled access to the cradle-to-grave health records of about 52 million people in England.

Studies of NHS records first revealed the dangers of thalidomide; established an association between power lines and childhood leukaemia; showed autism had nothing to do with the MMR vaccine; and more recently highlighted England's poor cancer survival rates, a problem that lay mostly in late diagnosis. The result was a government campaign to raise awareness of early symptoms, such as a cough that lasts more than three weeks, with the aim of saving 5,000 more lives by 2015.

Now ministers have overseen the creation of new systems to encourage a surge in studies that draw on public health records. From September GP practices will be encouraged to take part in the scheme, through the Clinical Practice Research Datalink (CPRD).

An uneasy question hung in the air when Britain banned smoking in enclosed public places. There was room for unintended consequences. Deprived of their usual haunts, smokers might light up more at home.

In Scotland, where the ban first came into force, nearly half of five-year-olds lived with a smoker. The fear was that a ban intended to reduce smoking might increase exposure among children. But the enforced ban seems to have sparked voluntary smoking bans at home.

Reassurance came when researchers trawled electronic hospital records for cases of children admitted with serious asthma attacks from 2000 to 2009, a period that straddled the ban's introduction in Scotland. Linking the electronic records together, all 21,415 of them, gave the first clear picture of child health after the ban. The number of pre-school children hospitalised with asthma began to fall after yearly rises of 9% before the law changed.

The government is now proposing to make this sort of study much easier to carry out. The CPRD will link health records on a scale never seen before, connecting patient information from GPs and hospitals to other records, such as disease and mental health registries and genetics databases. The combined datasets can be searched to answer medical research questions, and the results shared with scientists once they have been anonymised to protect patients' privacy. In time CPRD will add other records, such as pollution data, social care records and data from UK Biobank – which is charting the health of 500,000 people – and hook up with similar systems that cover smaller populations in Scotland and Wales.

The ambitious project redefines what it means to be an NHS patient. There is, though, nothing new in using health records for research. For the past 40 years the UK has kept histories of all NHS patients, a feat no other country has achieved, and the resource has proved invaluable.

Buried in the nation's health records is information that will save many more lives, and improve quality of life for others. Study of the records can reveal failing health services, rare drug side-effects, and pills that protect against diseases they were never prescribed for. But while some studies using health records go ahead, many never happen because the rules about sharing information are complex and applied haphazardly across the country. The obstacles can make researchers give up.

"The current system can be characterised by inconsistency, confusion and caution," said Prof Andrew Morris, chief scientist to the Scottish government. "If we want the UK to be the home of world class science, we need to sort these issues out."

Under the new system anyone who uses the health service will become a research subject by default, their anonymised records being made available for study, unless they opt out. Ministers will rewrite the NHS constitution to presume patient consent.

Advocates see a virtuous circle, with greater scrutiny of health records driving up quality of healthcare. But some critics argue that the collective good comes at a price to individual privacy, even when health records are shared anonymously.

A consultation on changing the NHS constitution will run through the autumn, alongside a review of patient record confidentiality led by Dame Fiona Caldicott. "The CPRD has the potential to revolutionise research, make a real difference for patients, and allow us to become world leaders in the field," said Dame Sally Davies, the chief medical officer and chief scientist at the Department of Health. What she described as the largest and most ambitious scheme of its kind would allow researchers access to data on an unprecedented level.

Nick Black, professor of health services research at the London School of Hygiene and Tropical Medicine, says there is an ethical imperative to use NHS records, which are gathered and maintained at taxpayers' expense. "It's irresponsible and misguided not to maximise the use of these data, subject to appropriate confidentiality. These databases are dreadfully under-exploited and the public could benefit enormously if they were used more. It's an incredibly cost-effective way of doing research, and knowing what the quality of care is within the NHS," he said.

Under the scheme scientists who want to study information held in patient records must first ask their local ethics committee, then send their research plan to the Medicines and Healthcare Products Regulatory Agency (MHRA), which runs CPRD. If the study is approved by an independent scientific advisory committee the service links the required records electronically, performs the scientists' search, and pulls out the results anonymously.

The first of two trials is underway. One study is comparing the long-term use of two drugs, simvastatin and atrovastatin, taken for high cholesterol. Unlike a clinical trial, the study will tests the drugs in the real world, where patients might be on more than one medicine and be less than consistent over taking their pills. The other study hopes to tease out whether antibiotics benefit people who suffer acute flare-ups of obstructive airway disease. Over the next three to four years the MHRA expects to run 500 to 600 studies annually through CPRD.

Alf Game, the acting science director at the Biotechnology and Biological Sciences Research Council (BBSRC), said the system has uses far beyond medical research. "It is not just about medicine. Anonymised data sets of basic health and medical information about the population will be absolutely invaluable in increasing our understanding of successful ageing," he said. "This kind of data would provide the whole picture, about absence of disease as well as the presence of it."

While the majority of research will use only anonymised data, some studies will need information that can potentially identify patients, such as dates of birth and postcodes. These projects can only go ahead if patients give explicit consent, or when the health secretary grants an exception, for example, to protect people in the event of a large-scale danger to public health.

Still, there are theoretical risks to patient privacy. The MHRA says it would, for example, give out only broad geographical information, far less precise than the first two letters of a postcode. But any health record that includes a handful of key episodes in a patient's life potentially puts their confidentiality at risk. A study of men over 60 in Scotland who had a detached retina in 1967 and retinal tears in 2009 might reveal other health details of the former prime minister, Gordon Brown.

Ross Anderson, professor of security engineering at Cambridge University, has used the Freedom of Information Act to try to force the MHRA to reveal how it plans to keep patient data secure, but received only partial answers. He says that until the agency fully discloses its methods – a move he says will allow them to be scrutinised and tested – their privacy claims amount to "smoke and mirrors".

"The fundamental problem is this: If your GP record is sent to the MHRA, then even if they strip off all the personal ID and give you a random number, then somebody who knows about some episodes in your past, such as an appendix operation, or knee surgery, can zoom in on you. You cannot give out longitudinal records except with quite extraordinary amounts of care," Anderson said.

On privacy, Davies said: "Protecting the confidentiality of patients remains a priority, so all data is anonymised and people can opt out."

How real the risk is hard to know. Only accredited scientists can use CPRD. They must specify and justify the information they need, and have a legal agreement with the MHRA to act responsibly. Data is anonymised and encrypted before it is sent, and no researcher can put CPRD data on a computer without it first having a certificate from the agency. The data can only be accessed by named researchers who have passwords and remote security access keys that generate a new code every 30 seconds. If foul play is suspected, researchers can be audited and whole institutions barred if bad behaviour is uncovered.

Black said fears over privacy should not trump the "huge collective benefit" that sharing patient records brings. "There has yet to be a single incident of anybody being harmed through the use of any healthcare databases. That's not to say there isn't somebody out there who is aggrieved because they somehow feel their personal information was used in a way they are not happy about. But this argument that terrible things might happen would be a lot stronger if people could come up with examples where somebody has suffered," he said.

Though sold to the public as a means to improve healthcare, there is more to CPRD than that. The government wants to make money from Britain's NHS records by charging for access to the data. Within four years the system is expected to turn a profit, recouping the £60m set-up costs in seven years.

The MHRA will charge industry double the academic rate, with private companies paying about £50,000 to £100,000 a study, depending on how much record linking needs to be done. The system can only be used for medical research, so drug companies cannot use CPRD for marketing, for example, by tracking down GP surgeries that are not prescribing their medicines. All projects that draw on CPRD data must be made public, usually through published journals.

At the heart of the government's plan is a desire to entice global pharma back to Britain, particularly to conduct lucrative, large-scale randomised clinical trials, which can involve thousands of volunteers and attract fees of more than £8,000 a participant. In recent years the UK has haemorrhaged such trials to countries where they are simpler, faster and cheaper to run. In Britain recruitment to trials can be frustratingly slow, because patients often need to give consent twice before they can enrol in a trial: once so their records can be checked to see if they are eligible, and again to invite them to take part. This "consent for consent" problem may be solved by another change to the NHS constitution to assume people are happy to be approached unless they opt out.

Other systems will run alongside CPRD. This month a group of 10 UK funders, led by the Medical Research Council, announced four new centres, in London, Manchester, Dundee and Swansea, which will carry out research on linked clinical, social and research data, to investigate treatments and drug safety around diabetes, obesity, cardiovascular disease, cancer and child and maternal health.

Sir Mark Walport, the director of the Wellcome Trust, who in June was named as the government's next chief scientist, is one of the most vocal supporters of research on NHS medical records. "We can only have the best healthcare if we advance medicine through research, and part of that is about measuring the quality of healthcare, in other words knowing how well the NHS delivers," he said.

Walport described the consent for consent situation as ridiculous. "Most people would think that when there was research that was relevant to them, there would be a mechanism to approach them to take part. The question is then who makes the approach, and that has to be handled sensitively and securely," he said.

Dr Janet Valentine, the head of public health and ageing at the Medical Research Council, said no other country had such a large and rich source of data on the health of its people as the UK.

"While research using patient data has been carried out for some time, the advent of electronic health records is enabling a step change in the type and sheer scale of research possible, especially when different large health data sets are linked together," she said.

"It's like the choice of taking a Kindle on holiday or a couple of books. We are already beginning to identify more effective treatments through the stratification of patients into those who will respond to treatments."