How to keep on top of security updates

By David Mytton,

CEO & Founder of Server Density.

Published on the 22nd May, 2014.

Every computer user needs to stay on top of updates for their apps, browsers, and OSs. As a consumer, it’s easy to do this – the best browsers like Chrome auto update in the background and over the air updates on iOS make sure most people get updates quickly, and easily. These kind of mass market updates tend to be well tested so you can update without fear of bricking.

The life of a sysadmin is a little more complex though. Although the mainstream OSs provide mature package management features, there are often a lot of frequent updates across the OS, kernel updates which require reboots, application (e.g. databases) updates which need testing and library release which need recompiling or updating codebases.

I’ve written before about our canary concept for rolling out updates because we deploy them manually rather than letting the OS auto update whenever it likes, but how do you discover new releases in the first place?

Critical security update notifications

The first step is to separate general OS/library/app updates with critical security releases. Like the recent Heartbleed bug, these need to be deployed as quickly as possible.

The first point of call is the OS security announcements. We use Ubuntu and so have the choice of subscribing to the Ubuntu Security Announcements list, or via RSS. In these announcements I look out for keywords like “remote” or “denial of service” because these mean that there’s an external risk, contrasted to an exploit which requires a local user. This is more difficult to exploit because it first requires access to our servers, which is restricted to our ops team.

Debian has a similar mailing list as do other Linux distros like Red Hat. Windows has an update mechanism built in which makes it easier than with multiple Linux distros.

Updating software packages

Most of the software products and libraries we use are installed through system packages using apt. This is our preferred method of installing because it makes it easy to manage via Puppet, is standardised across all our servers, offers signing of packages and importantly, gives us a single place to apply upgrades. This is one of the reasons why our monitoring agent recommended installation method is via our OS packages.

A lot of packages are provided as part of the OS release from Ubuntu package maintainers, so they get rolled into the security notifications.

The disadvantage of this approach is that you often don’t get the very latest version through the OS vendor packages. To work around this, vendors will usually provide their own official repositories (e.g. our MongoDB installations are through the MongoDB, Inc repos).

Everything else is manual

A few things are installed manually, such as PHP pecl packages for some of our legacy apps. We can update them using pecl but have to keep an eye on the release mailing lists for those specific packages.

And where we run “on-premise” versions of products we also subscribe to the relevant mailing lists. Our policy is to prefer SaaS where possible but some things aren’t available like this – we run our own Puppet Enterprise server (with an announce mailing list) and have the MongoDB Backup Service running within our infrastructure.

We also keep an eye on pages like the MongoDB Alerts and mailing list.

What about general security mailing lists?

You could subscribe to mailing lists like BUGTRAQ or the CVE lists but these are really high volume and aren’t specific enough to our environment. We can get what we need from the vendors, mostly from Ubuntu.

There are also commercial products like the Ubuntu Landscape product which we used for a while but it was too expensive to maintain a support contract for so many servers, with limited value. So long as we can stay up to date with the most important security releases, we can be sure that we have a secure infrastructure as regards to software updates.