Researchers at UC Berkeley have developed new attacks that analyze HTTPS traffic and can accurately determine what pages you’ve visited during an encrypted session.

One thing that’s been made abundantly clear by mathematicians and cryptographers alike is that despite the NSA’s dragnet surveillance of phone calls and Internet traffic, the spy agency has not been able to crack the math holding up encryption technology.

Those who wish to spy and steal on the Internet continuously hit a wall when it comes to crypto algorithms, leaving no alternative but to find a way to subvert the technology in order to reach their targets.

In response, security and privacy experts, as well as cryptographers, have urged companies to turn HTTPS on by default for web-based services such as email and social networking. A group of researchers from UC Berkeley, however this week published a paper, that explains new attacks that aid in the analysis of encrypted traffic to learn personal details about the user, right down to possible health issues, financial affairs and even sexual orientation.

The paper “I Know Why You Went to the Clinic: Risks and Realization of HTTPS Traffic Analysis” builds on previously successful research on SSL traffic analysis, Tor and SSH tunneling exposing vulnerabilities in HTTPS leading to precise attacks on the protocol that expose sensitive personal information.

The researchers—Brad Miller, Ling Huang, A.D. Joseph and J.D. Tygar—developed new attack techniques they tested against 600 leading healthcare, finance, legal services and streaming video sites, including Netflix. Their attack, they said in the paper, reduced errors from previous methodologies more than 3 ½ times. They also demonstrate a defense against this attack that reduces the accuracy of attacks by 27 percent by increasing the effectiveness of packet level defenses in HTTPS, the paper said.

“We design our attack to distinguish minor variations in HTTPS traffic from significant variations which indicate distinct traffic contents,” the paper said. “Minor traffic variations may be caused by caching, dynamically generated content, or user-specific content including cookies. Our attack applies clustering techniques to identify patterns in traffic.”

Using the techniques presented in the paper, an attacker could learn much more about a user’s activity only than just the IP address of the website they’re visiting; specific pages on the site can now be deduced with greater accuracy than previous work, the researcher said.

The paper points out a number of privacy consequences as well beyond government surveillance. For example, enhanced SSL traffic analysis by an ISP can lead to be enhanced customer data mining and intrusive targeted advertising. Employers can also more effectively monitor employees’ traffic and the techniques can also improve the censorship efforts by oppressive regimes, putting the liberties of privacy advocates at risk.

The attacks were tested on a number of heavily visited websites, including the Mayo Clinic, Kaiser Permanente, Planned Parenthood, Wells Fargo, Bank of America, Vanguard, Legal Zoom, the ACLU, Netflix and YouTube. The researchers established a baseline by visiting webpages on the respective sites and recording subtle changes to the URLs, especially those brought upon by browser cookies and caching that affect packet sizes for internal pages compared to homepages that are much more highly trafficked.

The researchers said that their techniques, conducted against more than 6,000 webpages, were able to accurately identify internal pages and information 89 percent of the time on average.

The paper also presents a possible defense against these attacks, which the researchers called Burst, which they demonstrate reduces attack accuracy by 27 percent. The paper said the technique operates between the application and TCP layers and is able to obscure high level features of traffic.

“The Burst defense outperforms defenses which operate solely at the packet level by obscuring features aggregated over entire TCP streams,” the paper said. “Simultaneously, the Burst defense offers deployability advantages over techniques such as HTTPS since the Burst defense is implemented between the TCP and application layers.”