A week ago, most Canadians were unlikely to have heard of Communications Security Establishment Canada (CSEC) and its program gathering "metadata" on untold numbers of global phone calls and online messages. But on Monday, The Globe and Mail reported that the agency's operations, meant to collect foreign intelligence, also at least "incidentally" intercept the communications of Canadians. Many people were left to wonder: Is Ottawa invading our privacy?

Now, it turns out that some officials in Ottawa had the same worry – as long ago as 2008, according to a report obtained by The Globe. Marked "Top Secret" and "CEO" (Canadian Eyes Only), the document sounds alarms over surveillance activities of great "complexity and breadth."

"Some of CSEC's metadata activities raise issues that make us question whether CSEC is always in compliance with the limits," reads the report from the office of then-CSEC watchdog Charles Gonthier, a former Supreme Court justice. It flags questionable activities and the possibility that Canadians' private information had been compromised.

Story continues below advertisement

The government seems to have acted pre-emptively ahead of the report: Other records show that some surveillance activities by CSEC ("see-seck") were put on hiatus from April, 2007, until October, 2008, when they were resumed with new rules under Defence Minister Peter MacKay.

That may seem reassuring. But the suspension showed that this shadowy program had gone awry – to the point that the government was willing to give up information in the fight against terrorism and other crimes for more than a year.

The Globe unearthed these facts via access-to-information legislation. The courts, Parliament and the wider public have never been informed of them.

This week's revelations have made it clearer to the public that Canada, like other governments, is voraciously scouring the globe for telecommunications data trails – phone logs, Internet protocols and other "routing" information.

The idea is that this "metadata" will help them map out social networks that could point to security threats.

The sharpest image has come from revelations about CSEC's U.S. counterpart, the National Security Agency (NSA). Leaks have shown that the NSA has compelled telephone and Internet companies to hand over mass quantities of records on Americans' communications, in apparent contradiction of its mandate to leave U.S. citizens alone.

The 29-year-old whistleblower, Edward Snowden, has been hiding out in a Hong Kong hotel room and calling government surveillance programs "existential threats to democracy."

Story continues below advertisement

CSEC is a close ally of the NSA and borrows some of its approach and language, as well as quite a bit of its technology, although CSEC has never adopted the U.S. agency's methods or software wholesale.

And where Washington has demonstrated its commitment to the NSA with the construction in Utah of an immense $2-billion base said to be capable of processing a zettabyte of data, the Canadian government is building CSEC a gleaming new $900-million, 72,000-square-metre compound in Ottawa – even as it has relocated military and RCMP operations to older, cheaper offices on the outskirts of the nation's capital, in buildings vacated by fallen technology companies.

While officials will not answer questions about how often CSEC's technical skills are used to advance domestic investigations, by all accounts the collaboration is ongoing – the new headquarters will include a glass bridge leading to the Canadian Security Intelligence Service building next door.

"There is close co-operation" between CSEC and CSIS as well as the RCMP, said a former domestic-security official, who spoke unauthorized on condition of anonymity.

"But there are walls," he also stressed.

He maintained that the strictures surrounding the sharing of surveillance intelligence in Canada can make CSEC less like Big Brother and more "Boy Scouty."

Story continues below advertisement

The records obtained by The Globe show that these distinctions have been debated vigorously within the security bureaucracy over the past decade.

Though heavily redacted, these hundreds of pages of documents together yield a glimpse into a subterranean security world and its actors, controversies and, perhaps, corrective actions.

CSEC's roots are in the Cold War. During the Second World War, the Canadian government's scientific-research agency, the National Research Council, had been commandeered for military purposes. Afterward, it reverted to a mostly civilian role, but continued to include a communications branch as a workplace for civilian engineers listening in on communist countries. The branch would suck "foreign signals-intelligence" out of the sky, from ocean cables and from listening posts in embassies abroad.

This continued until the CBC-TV program the fifth estate aired an exposé of the agency's activities (and links to U.S. intelligence services) in 1974. The next year, it was transferred to the Department of National Defence and renamed the Communications Security Establishment.

Its role shrank after the fall of the Berlin Wall. But the Sept. 11, 2001, terrorist attacks on the United States would reverse that trend. That December, Parliament passed Canada's Anti-Terrorism Act, which effectively enlisted CSEC's powers in the global fight against al-Qaeda. While the legislation better clarified the powers and limits of the surveillance agency, it also opened some chinks in the ironclad strictures against spying on Canadian citizens: Subject to checks and balances, the Minister of National Defence was given broad latitude to put in play certain programs that had been regarded as borderline.

"CSEC has significantly increased its ability to identify and collect communications," says the agency's own online account of its evolution. "Under the authority of several ministerial authorizations, [CSEC] has strengthened its most promising and technologically advanced collection activities."

Story continues below advertisement

This brings us to the Canadian metadata program. The basic tradecraft of agencies such as CSEC or the NSA is simple enough: Suck up enough phone logs and Internet protocol addresses and the like, sort through them with enough technicians and computers and algorithms, and you can divine how data move among people. It's in this way that metadata yield insights into patterns of suspicious activity.

But what to do when a citizen's digital communication gets sucked up into the great big metadata-mulching machines? For decades, the consensus has been that democratic governments can eavesdrop on foreign conversations so long as they cover their ears when their own citizens enter the exchange.

Does metadata mining force a reconsideration of such practices? In Ottawa, the records show, that debate has been going on for a decade.

On March 9, 2005, Bill Graham, then defence minister, signed a top-secret directive that explicitly allowed CSEC to contemplate collecting some Canadian telecommunication trails, so long as they pointed to threats out in the wider world.

"I obviously signed the decree authorizing the collection of metadata because I was satisfied that was an essential thing they had to do," Mr. Graham, now retired from Parliament, reflected in an interview with The Globe. He didn't see the program as a fundamental change. "I saw it as one more tool in the toolbox."

Mr. Graham said politicians like him must place great faith in their bureaucrats, because signals intelligence makes no more inherent sense to a cabinet minister than it does to the rest of us. "Unless you're an expert yourself, it's tough for you to grasp the total significance of what could seem a small, incremental change," he said.

Story continues below advertisement

Besides allowing CSEC to fulfill its foreign-intelligence mission, the "incidental" collection of Canadian metadata was meant to better advise CSEC "clients" – namely the RCMP and CSIS. However, these domestic agencies need warrants to spy on Canadian telecommunications. They are also supposed to need warrants to get CSEC to divulge what it knows about specific Canadian telecommunications. But the ministerial directive gave CSEC broader legal cover.

In this period, the affable former Rhodes Scholar then in charge of CSEC had a nifty way of demonstrating how his agency was staying within the letter of the law.

"Metadata is the envelope information," surveillance czar John Adams would tell his counterparts during briefings, sources say. Sometimes, the major-general-turned-mandarin would even wave a paper envelope to make his point that the metadata program is far from the digital equivalent of steaming open letters. Citizens' communications contents were, are and would forever be sacrosanct inside the envelope, and off-limits. CSEC merely wanted a better glimpse at the address, return address and other routing information.

In this alluring metaphor, CSEC was nothing more than a third party who happens to see the outside of a mailed letter while it's in transit. Metadata could be observed, mapped and shared without running afoul of laws precluding domestic surveillance.

The trick, of course, is for observers to get into a position to see all the envelopes whizzing around: They have to have the capacity to be inside every mailbox.

Even before Mr. Snowden, the NSA has had to deal with several leakers over the years, but CSEC technicians never tell – well, almost never.

Story continues below advertisement

The Canadian agency's one whistleblower to date has alleged that, during the 1970s, he spied on Margaret Trudeau's car phone after the RCMP Security Service asked him to figure out if the prime minister's wife was smoking dope.

"There was no watchdog. They just did whatever they felt like doing," claims Mike Frost's 1994 tell-all book, Spyworld.

Not long after it was published, CSEC got a watchdog.

Since 1996, a small office led by a series of retired judges known as commissioners has inspected the agency's activities to ensure that they are legal. In 2006, the role was given to Charles Gonthier, who had stepped down three years earlier from a 13-year stint as a justice on the Supreme Court of Canada.

"He has to be one of the most low-profile judges we have had," University of Alberta law professor Sanjeev Anand told The Globe when the judge retired. "A lot of people remember him for being very quiet."

Those qualities made him perfect for a new job, which depended on secrecy. But on Jan. 9, 2008, Mr. Gonthier spoke up, with his top-secret missive criticizing the metadata program.

Released only in part under access-to-information legislation, the judge's report is so heavily redacted that reading it is a real exercise in interpretation. He had concerns about overbroad metadata definitions and lax record keeping. But his real gripe was how the program contemplated handing over "foreign" intelligence to domestic agencies: Could this CSEC program really be said to be not "directed at" Canadians?

To Mr. Gonthier, it appeared that CSIS and the RCMP might get to hear things about Canadians without obtaining the usual warrants. He asked whether the fruits of a "foreign-intelligence collection" program ought to be used "in the context of a criminal or national security investigation of a Canadian in Canada."

The report had been slow in coming, but Mr. Gonthier had begun raising these issues earlier, which seemed to lead to the April, 2007, temporary halt on some operations. "The CSEC commissioner asked questions about some activities related to metadata," CSEC spokesman Ryan Foreman confirmed to The Globe. "While those questions were being addressed, the chief of CSEC voluntarily suspended those activities until clarifications were made."

A 2011 memo advised Mr. Adams on how to brief the government now that the old metadata operations manual had been ripped up and a new, improved one put in its place. Eventually, senior security bureau- crats would ask new Defence Minister Peter MacKay to make these changes official: Would the minister sign a new ministerial directive, with slightly altered language and altered privacy protections?

The new directive Mr. MacKay signed on Nov. 21, 2011, looks remarkably like the old one Mr. Graham signed, though the redactions make it impossible to say for sure.

It held to the same line: "Metadata is information associated with a telecommunication … and not a communication," as another briefing note put it to Mr. MacKay before he signed off on the renewed powers.

Mr. Adams left CSEC in early 2012 and was replaced by career bureaucrat John Forster. The broad program continues today under the new rules, though the details remain unclear.

After The Globe's first report this week on the metadata program, Opposition MPs grilled Mr. MacKay during Question Period: What was this thing? Why hadn't the minister told anyone about it? And was Ottawa spying on Canadians' e-mails – yes or no?

Mr. MacKay defended and deflected. Canadians, he said, can trust that the program he signed off on really protects privacy. It did not allow "looking at the information of Canadians" and "does not target communications of Canadians."

It's difficult to talk about these matters, of course: South of the border, the U.S. Director of National Intelligence has recently admitted that he had been answering questions about the NSA in "the least untruthful manner" that he could.

Mr. Gonthier died at 81 in 2009, while still serving as CSEC commissioner. His successor, Robert Décary, took the unusual step of releasing a public statement late this week.

It said in part: "I have reviewed CSEC metadata activities and have found them to be in compliance with the law and to be subject to comprehensive and satisfactory measures to protect the privacy of Canadians. However, given that these activities may impact the privacy of Canadians, I had already approved, prior to recent events, the start of a specific review relating to these activities."

As for CSEC, "it's like a phoenix rising," said one federal security official, remarking on the new building, with its glass walkway to CSIS headquarters – a literal bridge of the divide between foreign and domestic intelligence in the future.