The good news in the telephone records revelations that blew up on June 5 and 6 when the Guardian newspaper revealed that Verizon was handing over records of every call made by every customer to the National Security Agency is that they are just the basic call records.

This means that the phone numbers on each end of a call, plus other data such as the call duration, are transmitted to the agency, where they're stored in a database for processing. The contents of the calls are not included in the information that's sent to the NSA.

That doesn't sound like good news, you say? It is, compared to the revelations during the administration of George W. Bush when the content of the calls was also being recorded. What the NSA is looking for are global patterns in communications that may involve terrorism. The agency isn't looking to see how many times you called the pizzeria in a given month, or even to see how often you called your extra-marital partner.

The reason they don't care about personal phone calls for most Americans is that in the context of big data, individual events are meaningless. It's the patterns that matter, and it's the patterns of calls in the U.S. that appear to happen in conjunction with patterns of known or potential terrorists in other countries that matter most. This means that if a terrorist event such as a bombing in (for example) India elicits a series of calls in the U.S., that's something the NSA is interested in seeing.

But to be an effective use of big data, the NSA needs as much of that data as it can get, so it's a safe bet that all or at least most calls in the U.S. are being collected, not just those at Verizon. It's also a safe bet that the NSA is collecting calls outside of the U.S. as well. The reason you haven't seen a court order for those calls is that the NSA is required by law to get a court order to spy on Americans in the U.S. Outside the U.S., well that's a different story. After all, it IS a spy agency, right?

The study of vast collections of data is quickly becoming a favorite method of extracting intelligence from seemingly random data, as I found out recently at a seminar on cyber-security and big data. But the only way these methods work is by sorting through such a huge volume of data that analysis can start to discern patterns. Those patterns can then show a trained observer what's happening or even about to happen very early in the process.