Concerns have been raised that neither of the bodies tasked with overseeing the UK's spy agencies were aware that data they collected was shared with the private sector.

According to documents released as part of an ongoing court case between the UK government and Privacy International, GCHQ and MI5 didn't tell watchdogs they shared bulk communications data (BCD) and bulk personal datasets (BPD) with third parties.

The long-running case, which is being heard by the Investigatory Powers Tribunal, is considering the legality of mass data collection and storage, and the transfer of that data to other bodies.

The tribunal has previously ruled that bulk collection of personal data by GCHQ and MI5 between 1998 and 2015 was illegal, and has now moved on to consider transfer of data to other bodies.

The latest hearing, running October 17 to 19 at Southwark Crown Court, in London, has shed light on a batch of previously unseen documents.

The trove includes letters from the Investigatory Powers Commissioner's Office (IPCO), which started work last month and replaced the offices of the Intelligence Services Commissioner (ISCom) and the Interception of Communications Commissioner (IOCCO).

They reveal that "neither ISCom nor IOCCO were previously informed by GCHQ that the sharing of BPD/BCD datasets with industry partners... had occurred", which only came to light as part of GCHQ's witness statement during the trial.

The IPCO said that on being notified it ordered an "immediate inspection" of such data sharing, and a summary of the results of the first phase of this audit are published as part of yesterday's document drop.

As well as offering a strong indication that spy agencies are pulling in social media data, it reveals concerns about contractors' access to datasets held by GHCQ.

Writing in the results summary, the IPCO said that the audit had "identified a concern in relation to non-UKIC [intelligence community] staff", which includes contractors, industry partners and academics.

GCHQ told the IPCO that contractors and academics do not give BPD in full or in part, and do not get access to the search interface.

However, the IPCO noted that "this would not preclude a contractor with system access rights going into the system, extracting data and then covering their tracks".

It added that contractors may be involved in the design and build of systems that hold personal data, and for some systems they may gain administrator rights.

When asked if any GCHQ data was held by industry off-site, the commissioner's office was told that it would "need to make enquiries". GCHQ said that if it was the case, "it might be on a small scale for desktop development".

The second phase of the review will look at access of non-UKIC staff in more detail.

Unstructured databases are complicated, m'kay?

A further concern raised in the audit is that GCHQ might not be correctly defining BPDs after the agency told the IPCO that they had "taken a cautious approach" to identifying them.

"Uncertainty with the list provided to IPCO by GCHQ led to some concern that GCHQ may not be adequately identifying BPDs," the summary said.

On top of this, GCHQ had trouble describing the nature of its database to IPCO – it told the body that the size and scale can be difficult to set out "in a way that makes straightforward to understand during independent, non-technical, oversight" and that unstructured databases are "complicated to quantify".

On the nature of the databases, IPCO said:

Typically the agencies tick boxes to denote certain types of data, such as sensitive medical data or financial details. In such cases we are advised that it is expected that the authorisation itself will also include further details regarding the sensitive nature of material within the dataset, it can be the case with less structured databases, such as social media data, that it is possible to obtain any type of data, which renders the deals a gauge of likelihood rather than a description of data.

University of East Anglia law professor Paul Bernal said it's not clear whether intelligence agencies' alleged use of data from social media is appropriate.

"Despite what it may seem, people do have an expectation of privacy to some degree on their social media – and the information derivable from the public data is potentially very sensitive," he told The Reg.

"The Information Commissioner told the Samaritans that this kind of data could count as sensitive personal data – it really isn't just fair game. We need to be very clear about that – it's not always OK for our social media data to be scrutinised without consent."

Millie Graham Wood, solicitor at Privacy International, said that the "risks associated with these activities are painfully obvious".

The group is arguing that the latest revelations make the government's position – that there are effective safeguards in place for data collection – is "untenable", adding that the case would raise questions "as to the reliability of government's evidence".

More teeth for IPCO?

Bernal also noted that the situation posed a "big test" for the IPCO.

"Oversight is perhaps the single most important element of the new Investigatory Powers Act – whether it actually works or merely acts as a rubber stamp will only be shown by what happens in practice. This is the first test of that. Will the commissioner show that oversight means something?"

Sir Adrian Fulford PC QC, the new Investigatory Powers Commissioner, has hinted at plans to address information sharing.

Writing to Privacy International as part of a separate campaign by the group, Fulford noted (PDF) that intelligence agencies have a statutory duty to provide the IPCO with "all information necessary to enable us to conduct our oversight function".

He added: "We are also considering how any potential duty of candour upon the applicant will facilitate our oversight in this area. This is a matter we are currently working on."

In addition, it seems that the IPCO plans to better equip itself to challenge the spy agencies' more technical responses.

In response to a question posed by tribunal president Michael Burton about the technical understanding of data-processing techniques, the IPCO said: "Neither IOCCO nor ISCom had any technical understanding of industry partners' processing techniques. IPCO, in contrast, is acquiring these resources."

The tribunal hearing continues. ®