The NHS body charged with safeguarding confidential health data has announced a rolling programme of spot checks on companies, charities, universities and government bodies that have received medical records after an investigation uncovered "significant lapses" in protecting patient confidentiality.

The investigation had been prompted by concerns raised by the health select committee in February, where MPs had been unconvinced by the arguments put forward by the Health and Social Care Information Centre, the newly created central database for health data, that patient privacy had been safeguarded.

Such was the public anger over plans to produce a single English medical database – harvested from GP and hospital records and containing data on mental health conditions and such diseases as cancer, as well as smoking and drinking habits – that ministers first put on hold the proposals before scaling back the programme. It will now begin with a 100 or so GP surgeries in the autumn.

The audit, led by former Terrence Higgins Trust chief executive Sir Nick Partridge, found that 3,059 data releases had taken place between 2005 and 2013 – with a detailed examination of 10% of these. It found "lapses in the strict arrangements that were supposed to be in place to ensure that people's personal data would never be used improperly".

Of those examined in depth, it was found that one research programme had no legal authority to get patient-identifiable data but was still accessing NHS records in 2014. And a further eight were still getting mortality data – which could potentially pinpoint individual patients – without approval. In all nine cases medical researchers have suspended their work.

Partridge, who is a non-executive director of HSCIC, also pointed out that there were "data sharing agreements" made with three reinsurance companies that allowed those reinsurers to continue to use the data until the agreements expired in 2015 and 2016. All three companies have been asked to delete these medical records, the review said.

One set of records – which included a decade's worth of hospital data with patients' partial postcodes and partial date of birth, including month and year as well as gender, dates of admission, diagnosis, speciality, and treatment – went to French multinational reinsurer Scor. Another similar data set went to the UK subsidiary of the Reinsurance Group of America. Both were used to set "reinsurance premiums" for insuring critical illness conditions.

Another reinsurer Millman obtained two years of patient care data – detailing NHS number, age at start of hospital spell, gender, partial postcode, dates of admission, diagnosis, speciality, and treatment – to be used as part of a product sold to customers.

Partridge said that not only would the law be changed this year to restrict the flow of data "solely to the purposes of benefit to health and social care systems" but that he was well aware of "public concern about insurance companies holding data drawn from health sources".

"I ensured that the HSCIC's chief executive wrote to the three companies concerned asking them to delete the data ahead of this legislation coming into force," he said.

There was also apparent confirmation that a centralised database could be accessed by police to locate individuals. There were 12,733 "accepted and approved" approaches to the NHS, which led to 3,104 leads for officers. The HSCIC has said it will now report every quarter on requests from law enforcement, stressing that the data only reveals the location of the nearest GP and would only be given for investigations into serious crimes.

Phil Booth, coordinator at patient pressure group medConfidential, said: "This is clearly system failure over a period of years. Patient data is out there and the public don't know where it is. Companies receiving it did not know their duties as data controllers and they have not been able to distinguish between data sharing and reuse. This means we will never know who got hold of the data."

Partridge said HSCIC must learn the lessons of the past. "The public simply will not tolerate vagueness about medical records that may be intensely private to them. We exist to guard their data and we have to earn their trust by demonstrating scrupulous care with which we handle their personal information.

"We can now make sure we conform to recent legislative changes, so that data is released when it will benefit the health and social care system."