A Federal Court judge's ruling this week that CSIS has been illegally storing Canadians' communications data for more than a decade has shed new light on the agency's secretive data analysis program.

In his redacted ruling, Justice Simon Noël found the domestic spy agency held on to what it calls "associated data," vast amounts of data about private electronic communications obtained under warrant, long after it had decided the information was not related to a security threat.

CSIS has been storing that information for an indefinite period in an Operational Data Analysis Centre (ODAC), a shadowy part of the agency that essentially maintains a database of metadata, which includes phone numbers, email addresses and geographic coordinates.

Noël found that when it came to the ODAC, the intelligence agency did not fulfil its "duty of candour," which obligates the agency to be forthcoming with the Federal Court about its investigatory practices when seeking warrants.

The judge found that CSIS was concealing these efforts — to collect and store vast amounts of data — from the court, something the agency director now says is "regrettable."

MPs Michel Picard, Michael Cooper and Murray Rankin discuss the ruling that CSIS broke the law by keeping metadata for too long 9:36

Is CSIS keeping the content of my communications?

At issue in this landmark ruling is not the actual content of a "communication event," which can largely be defined as a phone call or an email, but rather associated information like phone numbers, call times, date stamps and GPS data.

But that doesn't mean it's useless information, Noël writes in his judgment. Far from it.

"Metadata, on its own and processed through aggregation and analysis, can provide intimate insights into the lifestyle and personal choices of individuals; it is not an innocuous kernel of information."

What is the Operational Data Analysis Centre?

It's hard to say, exactly, because Noël, who had been issuing warrants to CSIS to allow them to collect this data through "intrusive methods," had no idea there was such a thing.

CSIS set up this centre in 2006 to house "non-threat, third-party information," that it obtained from service providers (largely mobile carriers and internet companies).

This, Noël ruled, is hugely problematic because warrants stipulated that information unrelated to a threat must be destroyed.

Despite those legal limits, the agency felt that information it collected was "under-utilized," because it was not sufficiently analyzed. Internally, the ODAC was billed as a "centre for excellence for the exploitation and analysis" of databases.

Most of the judge's description of the ODAC in his ruling is blacked out, but he discovered that the centre processes metadata and spits out an "end product of intelligence which reveals specific, intimate details on the life and environment of the persons CSIS investigates."

A Federal Court judge says Canada's spy agency illegally kept potentially revealing electronic data about people over a 10-year period. (Sean Kilpatrick/The Canadian Press)

This practice, Noël wrote, goes beyond CSIS's limited mandate, which restricts its activities to those that are "strictly necessary for the purpose of protecting the security of Canada."

CSIS legally acquired the data and thought it was allowed to retain it, the agency's director, Michel Coulombe, said in a press conference Thursday.

"It is now clear that the Federal Court disagrees with this interpretation."

He did concede Thursday that existing data collection practices "have proven to be an effective tool."

Coulombe also said its policies had the full backing of lawyers in the Department of Justice

"We've heard the court loud and clear," Robert Frater, chief general counsel for the Department of Justice, who attended the press conference with Coulombe, said. "We are taking steps to improve our practices and we will meet that standard."

Who knew about this operation?

CSIS told the court that it had informed Stockwell Day, who was serving minister of public safety, about the associated data collection program.

But Day refuted that claim in an interview with CBC News Network's Power & Politics. "If he is suggesting, or anybody, that they thought they told me something inappropriate was going on, yeah, I would suggest they did not."

Goodale says CSIS must be forthcoming with the Federal Court 2:33

The current public safety minister, Ralph Goodale, said Friday the Security Intelligence Review Committee (SIRC) initially flagged possible problems with the collection and storage of associated data in a report tabled earlier this year.

Goodale said he only became aware of the full extent of the problem recently, when Noël filed a preliminary report "a couple of weeks ago."

"I took the immediate step of informing SIRC of the issue and asking SIRC to intervene in the situation to supervise the management of the data and to make sure that there was full compliance with the federal court's judgment," Goodale told reporters.

CSIS does not need a warrant to review Revenue Canada files

In addition to collecting and storing metadata for years, Noël also documents in his ruling how the spy agency can now obtain information from the taxman without a warrant.

"The Court no longer adjudicates applications for warrants to obtain information from the Canada Revenue Agency," he wrote.

This is part of a new trend, expanded in the former Harper government's anti-terrorism legislation, Bill C-51, which dramatically expanded information sharing among government departments. It allows CSIS to operate internationally and gives it the ability to take action to actually prevent a threat.

What happens now?

Coulombe said CSIS has halted all access to, and analysis of, the data in order to assess the "operational" impact of the ruling and to determine the way forward.

"We are working closely with the Department of Justice to make sure that we meet our obligations with the court," he added. "The trust of Canadians is essential in the fulfillment of our mandate."

Goodale said the Federal Court ruling is "timely" as it comes in the midst of the government's public consultations over the future of national security.

He has already promised to launch a parliamentary oversight committee of CSIS, a promise the Liberals made after it tacitly endorsed parts of the controversial Bill C-51.

Noël said, in the context of this ruling, "it may be time for Canadians to renew a debate regarding the mandate and functions of our domestic intelligence agency.

"It is my opinion that the CSIS Act is showing its age," the judge said of the 30-year-old legislation.

Daniel Therrien, the country's privacy commissioner, told CBC News Friday that CSIS has already reached out to his agency to propose next steps.

"At this point I can tell you that we welcome discussions on changes to CSIS policies required by the judgement. We also welcome a proposal from CSIS to meet to discuss their Privacy Impact Assessment for the Operational Data Analysis Centre and how it should be updated following the Court's decision."