Red Hat has a sterling reputation in Linux security circles. That means the company has a workable process for preventing problems and responding to them. Even if you don't use Linux, the Red Hat security approach has a lot going for it, and some of its practices might be worth adopting in your own shop.

It's an IT truism that Linux is the most secure mainstream operating system. That doesn't mean the operating system doesn't have security holes. Of course it does. However, when those holes show up, Red Hat is usually the first Linux company to deliver security patches.

For example, Red Hat was the first Linux distributor to deliver Meltdown and Spectre patches. During a presentation at the 2018 Red Hat Summit, Mark Thacker, Red Hat's principal technical product manager for Red Hat Enterprise Linux (RHEL) security, observed, "When the Spectre and Meltdown news broke early, Red Hat was one of the few operating system companies ready with patches."

It's all fun and games until somebody gets hacked. Christopher RobinsonRed Hat product security program manager

That didn't surprise Richard Morrell, Falanx Group CTO, Cloud Security Alliance CISO, and former Red Hat security strategist. "Being in the security team at Red Hat means more than just ensuring you are able to roll patches out to users who could be consuming any one of a number of major release variants,” Morrell said.

Responding appropriately also means playing the diplomat in a fast-moving political drama. A lot goes on behind the scenes during a security embargo, Morrell explained. Someone has to educate and politely brief maintainers and security stakeholders in aligned commercial businesses and listen to their concerns. “Red Hat since day one has always been out there ensuring that the playing field was level for the entire community,” he said.

The security consciousness at Red Hat applies to more than Linux security, given the company’s involvement in virtualization and cloud. There were 50 sessions with security content at the summit, underscoring the company’s commitment to securing its software supply chain, said Thacker. Among those addressed below are Red Hat’s overall security attitude, Linux-specific security (along with related packages), and use of hardware-enabled encryption, identity management, and automated security.

Red Hat’s security approach

As Christopher Robinson, manager of the Red Hat Product Security Program, put it in his conference session, "It's all fun and games until somebody gets hacked."

He explained, "All software is created by people. People make mistakes, and therefore there are flaws in software. Sometimes good people find the mistakes; sometimes they don’t.”

Drawing on an executive summary of the 2016 Verizon’s Data Breach Investigations Report, Robinson emphasized that breaches are not just a problem for high-value targets or the result of a clever evil-doer. In fact, 63 percent of all attacks in 2016 were the result of password guessing or reusing credentials. Worse still, Robinson cited, "84 percent of breaches take months to years to discover."

Robinson outlined the basic steps of handling a breach, using the acronym PICERL:

Prepare

Identify

Contain

Eradicate

Recover

Lessons learned

Being serious about security takes dedicated attention, said Thacker. "There are a gazillion Linux software packages, but we choose only those that are useful for our enterprise customers." Once a Linux package is in the RHEL distribution, Red Hat makes sure its kept secure, even if its original developers abandon it.

Ultimately, Red Hat supports "everything we've shipped.” For example, Thacker asked how many people in the audience were still using the long-out-of-date RHEL 5 with its decade-old 2.6.18 Linux kernel. Quite a few hands were raised. That’s why "Red Hat went back to that old Linux kernel and patched the Meltdown and Spectre security holes, even though the upstream patches couldn't be ported back to it," explained Thacker.

Locking down Linux

When it comes to Linux in general, Red Hat starts by setting up a secure foundation in the operating system. "We apply security throughout the stack," said Thacker.

Red Hat does this by enabling SELinux, the 20-year-old Linux security system. It sets access control security policies using mandatory access controls (MAC) across the RHEL family, from bare metal to containers to virtual machines.

Thacker added that containers owe their security to SELinux. The company is also locking in methods to set systems so they execute only white-listed utilities and applications. This will leverage software identification (SWID) tags across the Red Hat software portfolio.

Part of the Red Hat security remit is evaluating security tools, processes, and packages. One newer security feature in which Red Hat has an active role is USBGuard, a software framework that prevents mounting of questionable USB devices. USBGuard implements basic allowing and denying capabilities based on device attributes. Thacker said, "With USBGuard, an admin can control when and where and what USB devices can be plugged in. For example, you can make sure keyboards are OK, but cameras are never OK. Or keyboards on servers are OK, but only if they use the front port during work hours."

Red Hat is also keeping an eye on encryption standards. For instance, the company is depreciating older, less secure standards.

Red Hat also now supports Trusted Platform Module (TPM) 2.0 encryption chips, starting with RHEL 7.5. TPM stores encryption keys for a specific host system for hardware authentication. This enables a hardware root of trust. It's never been popular in Linux circles, but Thacker noted that Microsoft has taken advantage of TPM with BitLocker, its full disk encryption feature. That’s increased interest in TPM encryption among Red Hat customers.

In particular, customers are interested in using TPM with PKCS #11, an open standard for cryptographic tokens controlling authentication information such as personal identity, cryptographic keys, digital signatures, and biometric data. But this won't be coming soon. "There's lots of work to be done," Thacker said.

Storage encryption on the disk and over the network

In the nearer term, Red Hat is combining Linux disk encryption standards such as Linux Unified Key Setup (LUKS) and Network-Bound Disk Encryption (NBDE) with TPM. This protects against disk thieves and other direct assaults on storage devices.

Brian Atkisson, a Red Hat senior principal systems engineer, commented during the conference, “We want all data to be encrypted both at rest and in transport. That’s easier said than to do. We want to use LUKS to accomplish this."

Multicloud Storage for Dummies: Want to get a handle on how to better use multiple clouds but don’t know where to start? This guide explains how. Download now

"Eventually," Atkisson continued, "we want to deploy encrypted disks without needing someone to manually enter a password.” That requires an automated system across the entire platform, but nobody wants to implement a solution for every platform. “We want a universal encryption system," he added.

But that universal system won't be escrow servers. "Escrow servers are horrible," snarled Atkisson. "Avoid escrow. We looked, and we couldn't find anything that worked."

So how will Red Hat create this universal data encryption system? The system under development uses Clevis, which resides on a client with a LUKS mount. Clevis makes a remote call to a decryption key server, Tang, explained Nathaniel McCallum, a Red Hat principal software engineer. Clevis and Tang use public-key infrastructure (PKI) to generate a unique, cryptographically strong encryption key. If Clevis and Tang's keys match, the mount and decryption happen without human interaction. That makes life much easier for sysadmins who would otherwise need to manually enter a password to decrypt and mount the encrypted drive.

Clevis is small, fast, memory-resident, stateless, and doesn't store keys. It consists of only 600 lines of C. "But with it, you can do everything you can do with an escrow server,” McCallum said.

He concluded, "Escrow needs a lot of infrastructure under it, and you end up with chicken-and-egg security problems. With this, we avoid those and the entire class of Heartbleed-type attacks."

Red Hat isn't just advancing the current state of cryptography. The company is also keeping a close eye on advanced cryptographic algorithms. Indeed, it’s even looking at secure computers in the rapidly approaching post-quantum computing cryptographic world.

Identity management

Identity management is a core part of the Red Hat stack, but the company is conscious of interoperability issues. Red Hat has its own Red Hat Identity Manager, but as Thacker observed, most of its customers use Active Directory (AD).

So integrating its identity services with AD is a top job for Red Hat, which it accomplishes securely by using PKI and X.509v3 certificates. For those shops using open Red Hat identity services, the Lightweight Directory Access Protocol (LDAP) is still the foundation.

Moving ahead, Red Hat is working to integrate its identity and authentication services with TPM. It also plans to provide automatic enrollment into identity services as they're deployed, using Ansible and other management tools.

The company is also working on tighter integration with middleware and developer tools. Specifically, Red Hat wants to extend identity management into containers and do a better job of integrating PKI across the whole container, server, and cloud stack.

Automating security compliance

Red Hat is using OpenSCAP to assist administrators and auditors with assessment, measurement, and enforcement of security baselines in its programs. These include the management programs Ansible, Red Hat Insights, and Red Hat CloudForms, which identify and prevent out-of-compliance components from executing.

What's out of compliance? That's determined by Red Hat's security team, which provides rapid responses to known vulnerabilities. Users can also check Red Hat's vulnerability API for program versions listed in the company's remediation database.

“Just because a security issue has a logo does not mean it is critical,” Thacker noted. “And just because a security hole doesn't have a trendy name or logo doesn't mean that it's trivial." Looking ahead, the Red Hat security team is working on increased common logging across platforms, more OpenSCAP profiles, and patching more products on day one as security monitoring and automation become more common.

In short, while Red Hat may not be a "security" company, it sure is acting like one in the ways that count the most: securing customers and their users for the long run.

The security lifestyle: Lessons for leaders