MI5 has lost control of its data storage operations and has been obtaining surveillance warrants on the basis of information it knows to be false, the high court has heard.

The security agency has been accused of “extraordinary and persistent illegality” in a legal challenge brought by the human rights organisation Liberty.

The failures have been identified by the official watchdog, the investigatory powers commissioner, Lord Justice Fulford, and admitted in outline by the home secretary, Sajid Javid.

The full extent of the problems within MI5 became apparent in disclosures made public at the hearing on Tuesday. The revelations relate to bulk interceptions of data acquired through surveillance and hacking programmes and downloaded to its computers.

The agency has a duty to ensure such material is not held longer than required or copied more often than needed.

Ben Jaffey QC, for Liberty, said there were “ungoverned spaces” in MI5’s operations where it did not know what it held.

In written submissions, Jaffey said: “Fulford’s generic warrant decision notes that warrants were issued to MI5 on a basis that MI5 knew to be incorrect and the judicial commissioners [the watchdogs] were given false information.”

An MI5 letter sent to Fulford and released to the court showed the agency did not know what material it held.

The letter said: “We are about to commence further scanning of [its computers] to ensure we have a full understanding of the data.

“The full scan had been challenging to action … We have also been seeking to understand working practices … so that we can take comprehensive action to improve assurance of our compliance with relevant safeguards.”

Julian Milford, counsel for the Home Office and Foreign Office, told the court: “We accept that this is material that discloses compliance risks with MI5.”

Correspondence between Fulford and MI5 released at the hearing showed he concluded the way the agency was holding and handling people’s data was “undoubtedly unlawful”.

He said: “Without seeking to be emotive, I consider that MI5’s use of warranted data … is currently, in effect, in ‘special measures’ and the historical lack of compliance … is of such gravity that [the watchdog Investigatory Powers Commissioner’s Office] IPCO will need to be satisfied to a greater degree than usual that it is ‘fit for purpose’.”

Awareness of the problem within MI5 dated back to January 2016 but ministers were not informed until earlier this year. Fulford said MI5’s description of the problem as “compliance difficulties” was a “misleading euphemism”.

In the absence of improvements, he said, future applications by MI5 for interception warrants “will not be approved by the judicial commissioners”.

A response sent by MI5 to Fulford earlier this year explained: “In 2016, as part of a wider review of legal compliance in anticipation of new legislation, the … problem led the team conducting the compliance review to identify, at a high level, that data might be being held in ungoverned spaces in contravention of our policies.”

Megan Goulding, a lawyer with Liberty, said: “These shocking revelations expose how MI5 has been illegally mishandling our data for years, storing it when they have no legal basis to do so. This could include our most deeply sensitive information – our calls and messages, our location data, our web browsing history.

“In addition to showing a flagrant disregard for our rights, MI5 has attempted to hide its mistakes by providing misinformation to the investigatory powers commissioner, who oversees the government’s surveillance regime.”

The Green party peer, Jenny Jones, said: “This report [on MI5] has confirmed what we’ve known for some time, that the security services, as well as the police, are operating their surveillance systems without accepting that they have to operate lawfully, and in a properly targeted manner.

“I know from my own experience of being put on the domestic extremism database that valuable security time and energy are wasted on the wrong targets.”

The shadow home secretary, Diane Abbott, said the case “highlights the failure of government legislation, which only facilitates more and more public bodies having access to and gathering information. Without proper scrutiny and oversight, the unauthorised handling and retention of data puts all our information at risk.”

Some of the material is thought to include exchanges between lawyers and clients that should have been protected by legal privilege. The National Union of Journalists has joined the legal challenge, which is due to be heard in full at the high court next week.

The hearing continues.