Technology development seems to gallop a little faster each year. But there's always one laggard: encryption. Why the deliberate pace? Because a single, small mistake can cut off communications or shut down businesses.

Yet there are times when you take stock—only to discover the encryption landscape seems to have transformed overnight. Now is that time. Although the changes have been incremental over several years, the net effect is dramatic.

[ Give yourself a technology career advantage with InfoWorld's Deep Dive technology reports and Computerworld's career trends reports. GET A 15% DISCOUNT through Jan.15, 2017: Use code 8TIISZ4Z. ]

Some of those changes began shortly after Edward Snowden's disclosures of the U.S. government’s extensive surveillance apparatus. Others are the natural result of cryptographic ideas reaching the marketplace, says Brent Waters, an associate professor at the University of Texas at Austin and the recipient of the Association for Computing Machinery’s 2015 Grace Murray Hopper Award.

“Many of the new tools and applications available are based on research innovations from 2005 and 2006,” Waters says. “We are just realizing what type of crypto functionality is possible.”

A step closer to an encrypted world

Encrypted web traffic is the first step toward a more secure online world where attackers cannot intercept private communications, financial transactions, or general online activity. Many sites, including Google and Facebook, have turned HTTPS on by default for all users. But for most domain owners, buying and deploying SSL/TLS certificates in order to secure traffic to their sites has been a costly and complicated endeavor.

Fortunately, Let’s Encrypt and its free SSL/TLS certificates have transformed the landscape, giving domain owners the tools to turn on HTTPS for their websites easily. A nonprofit certificate authority run by the Internet Security Research Group, Let’s Encrypt is backed by such internet heavyweights as Mozilla, the Electronic Frontier Foundation, Cisco, and Akamai.

How ubiquitous has HTTPS become? In October, Josh Aas, head of Let’s Encrypt and former Mozilla employee, posted a graph from Mozilla Telemetry showing that 50 percent of pages loaded that day used HTTPS, not HTTP. While the graph showed only Firefox users, the figure is still significant, because for the first time, the number of encrypted pages outnumbered unencrypted pages. NSS Labs expects the trend to continue, predicting that 75 percent of all Web traffic will be encrypted by 2019.

Free certificate offerings will further accelerate adoption. By next year, the number of publicly trusted free certificates issued will likely outnumber those that are paid for, says Kevin Bocek, vice president of security strategy and threat intelligence at key-management company Venafi. Many enterprises will also start using free services. With certificate cost no longer a consideration, certificate authorities will focus on better tools to securely manage certificates and protect their keys.

Speaking of certificate management, after years of warnings that SHA-1 certificates were weak and vulnerable to attack, enterprises are making steady progress toward upgrading to certificates that use SHA-2, the set of cryptographic hash functions succeeding the obsolete SHA-1 algorithm. Major browser makers, including Google, Mozilla, and Microsoft, have pledged to deprecate SHA-1 by the beginning of the year and to start blocking sites still using the older certificates. Facebook stopped serving SHA-1 connections and saw “no measurable impact,” wrote Facebook production engineer Wojciech Wojtyniak.

From May to October 2016, the use of SHA-1 on the web fell from 3.5 percent to less than 1 percent, as measured by Firefox Telemetry. Enterprises can’t be complacent, though, since recent estimates from Venafi suggest approximately 60 million websites still rely on the insecure encryption algorithm.

“We look forward to the industry's movement toward greater use of stronger certificates like SHA-256,” Wojtyniak said.

Crypto is still king

Cryptography has taken quite a beating over the past few months, with researchers developing cryptographic attacks such as Drown, which can be used to decrypt TLS connections between a user and a server if the server supports SSLv2, and Sweet32, a way to attack encrypted web connections by generating huge amounts of web traffic.

Nation-state actors also have encryption in their crosshairs. Late last year, Juniper Networks uncovered spying code implanted in specific models of its firewall and Virtual Private Network appliances. Many experts believe the NSA was involved.

Shortly after the cache of hacking tools allegedly belonging to the NSA made its way to underground markets this summer, Cisco discovered a vulnerability in its IOS, IOS XE, and IOS XR software that powers many of its networking devices. The flaw, which could be used to extract sensitive information from device memory, was similar to the vulnerability exploited by the tools and was related to how the operating system processed the key exchange protocol for VPNs, Cisco said.

Even Apple’s iMessage app, the poster child for how companies can bring end-to-end encryption to the masses, had its share of issues. Cryptography professor Matthew Green and his team of students at Johns Hopkins University were able to develop a practical adaptive chosen ciphertext attack that could decrypt iMessage payloads and attachments under specific circumstances. The team also found that iMessage lacked the forward secrecy mechanism, meaning attackers could decrypt previously encrypted messages, such as those stored in iCloud. Forward secrecy works by generating a new key after a set period of time so that even if the attackers obtained the original key, the previously encrypted messages can’t be cracked.

One thing remains clear despite all the bad news: Cryptography is not broken. The mathematics behind cryptographic calculations remain strong, and encryption is still the best way to protect information.

“The latest attacks have not been on the math, but on the implementation,” Waters says.

In fact, encryption works so well that attackers rely on it, too. Criminals are equally as capable of obtaining keys and certificates to hide their activities inside encrypted traffic. The fact that this attack vector is fast becoming default behavior for cybercriminals “almost counteracts the whole purpose of adding more encryption,” Bocek says.

Cybercriminals are using encryption to great effect in ransomware. Once the files are encrypted, victims have to either pay up to obtain a key or wipe their systems and start over. Just as attackers target flawed implementations, security researchers have successfully developed decryption tools for ransomware variants that contained mistakes in their encryption code.

Government backs down on backdoors

Technology firms have always had to balance security and privacy concerns with law enforcement requests for user information. FBI Director James Comey had been pushing hard for backdoors in technology products using encryption, claiming that increased use of encryption was hindering criminal investigations. While companies frequently quietly cooperate with law enforcement and intelligence requests, the unprecedented public showdown between the FBI and Apple showed that in recent years, enterprises are beginning to push back.

The FBI backed down in that fight, and a bipartisan Congressional working group—with members of both House Judiciary and Energy & Commerce Committees—was formed to study the encryption problem. The House Judiciary Committee’s Encryption Working Group unequivocally rejected Comey's calls for backdoors and advised the United States to explore other solutions.

“Any measure that weakens encryption works against the national interest,” the working group wrote in its report. “Congress cannot stop bad actors—at home or overseas—from adopting encryption. Therefore, the Committees should explore other strategies to address the needs of the law enforcement community.”

Weakening encryption so that police can break into encrypted devices would speed up criminal investigations, but it would be a short-term win "against the long-term impacts to the national interest," the working group warned. Alternative strategies include giving law enforcement legal methods to compel suspects to unlock their devices and improving metadata collection and analysis.

While the working group report indicates Congress will not pursue legal backdoors, other encryption-related battles are looming on the horizon. The report seemed to support letting police use "legal hacking" to break into products using software vulnerabilities that only law enforcement and intelligence authorities know about, which poses its own security implications. The technology industry has an interest in learning about vulnerabilities as soon as they are found, and not letting the government stockpile them with no oversight.

As for Comey's "going dark" claim, the working group said “the challenge appears to be more akin to ‘going spotty.’”

Adding to the enterprise tech stack

Governments have been trotting out the terrorists “going dark” argument for years and will always play on those fears, says Mike Janke, co-founder and chairman of encrypted communications company Silent Circle. What's changing is that the enterprises are becoming more serious about securing their communications stack and are less willing to compromise on those features.

Many organizations were shocked at the extent of government surveillance exposed by former NSA contractor Edward Snowden. They reacted by integrating secure video and text messaging tools along with encrypted voice calls into the enterprise communications stack, Janke says. Encryption is now a bigger part of the technology conversation, as enterprises ask about what features and capabilities are available. IT no longer treats encryption as an added feature to pay extra for, but as a must-have for every product and platform they work with.

Consumers were outraged by the surveillance programs, and anecdotal evidence indicates many have signed up for encrypted messaging apps such as WhatsApp and Signal. But for the most part, they aren't paying for secure products or changing their behaviors to make privacy a bigger part of their daily lives.

The change is coming from CSOs, vice presidents of engineering, and other technical enterprise leaders, because they're at the forefront of making security and privacy decisions for their products and services. With Tesla now digitally signing firmware for every single one of its internal components with a cryptographic key, it's easier to ask TV manufacturers or toymakers, "Why aren't you doing that?" says Janke.

Consumers are the ones who will benefit from encryption built in by default as enterprises change their mindset about the importance of encryption.

Riding the innovation wave

Cryptography tends to go in waves, with important innovations and research from 2005 to 2006 finally coming out as practical applications. Researchers are currently looking at improving the "precision of encrpytion," instead of the current model of all or nothing, where if something is exposed, everything gets leaked. "Encrpytion can be precise like a scalpel, giving fine-grained control over the information," Waters says.

Google has looked at cryptography in its experiments with neural networks. Recently, its Google Brain team created two artificial intelligence systems that was able to create their own cryptographic algorithm in order to keep their messages a secret from a third AI instance that was trying to actively decrypt the algorithms.

The dawn of quantum computing will also spur new avenues of research. “If large-scale quantum computers are ever built, they will be able to break many of the public-key cryptosystems currently in use,” wrote the National Institute of Standards and Technology in a public notice. Once such machines become widely available, “this would seriously compromise the confidentiality and integrity of digital communications on the Internet and elsewhere."

To prepare for that eventuality, NIST is soliciting work on "new public-key cryptography standards," which will "specify one or more additional unclassified, publicly disclosed digital signature, public-key encryption, and key-establishment algorithms that are capable of protecting sensitive government information well into the foreseeable future, including after the advent of quantum computers.” The submission deadline is Nov. 30, 2017, but NIST acknowledges the work will take years to be tested and available, noting that "historically, it has taken almost two decades to deploy our modern public key cryptography infrastructure."

“Regardless of whether we can estimate the exact time of the arrival of the quantum computing era, we must begin now to prepare our information security systems to be able to resist quantum computing,” NIST said.

There have been a number of intriguing advances in cryptography, but it will likely be years before they become available to enterprise IT departments, and who knows what form they will take. The future of cryptography promises even more security. The good news is we are already experiencing some of the benefits now.

This story, "Encryption in 2016: Small victories add up" was originally published by InfoWorld .