Introduction

Security patch management is difficult field – Complex, highly critical, and hard to get right. For an organization, loss or theft of data can be catastrophic. Infrastructure outages, customer data leaks and infiltration by malicious software or hostile actors are among the worst of worst-case scenarios.

No single set of steps will ever be complete. No tool will ever cover every possible application. As security solutions evolve, so too does the treat landscape. As engineers improve their process and tooling, so do malicious outsiders. To keep up with the evolving dangers we face, it is time to move beyond the traditional approach.

What is the problem with patching?

The Traditional Approach

In order to talk about improving on tradition, we need a workable definition of ‘traditional’.

In most organizations, security patches have been managed – traditionally – by system administrators.

Reports are generated by a patch management tool, or a system management console.

When a patch comes in, it is evaluated, scheduled, and then deployed.

As we will see, this is far from enough.

Reporting

One of the chief problems with most tools is information – Either lack, overflow, or inaccuracy. Incomplete or misleading reporting could lead to an over- or underreaction. It can waste time and resources or worse, lead to a security breach.

If standard updates are mixed with security patches when generating reports, it becomes easier to overlook things, and prioritization of severe and high-impact flaws becomes harder.

If reports are generated only occasionally, covering only parts of the threat surface, risks may end up lost or ignored.

Reports must be timely, comprehensive and accurate. Otherwise, we run the risk of creating a Potemkin village of security.

Software coverage

One of the easiest mistakes to make when managing security patches is to overlook things, either because they don’t seem important, or because current tooling is incapable of discovering them. If the majority of the network runs Microsoft Windows, it is easy and almost natural to think of yourself as a “Windows Shop”, ignoring any Unix systems, macOS laptops or alternative platforms like smartphones and other wearables.

More alarmingly, recent years have seen a spike in actively exploited security flaws for devices like network routers, hardware firewalls, building or industrial control systems, and so forth. Many such compromised devices are able to spy on or attack the rest of your network.

Further, it is a simple fact that no commercial or freely available offering will cover any and all software on the network. To ensure we get as close to complete coverage as possible, we must ask ourselves the following questions:

What operating systems are present on our network (including on devices like routers, firewalls or smartphones), and do you have enough information from those platforms?

How good is our application coverage, and what are the risks faced if vulnerable programs are overlooked?

Are application dependencies like software libraries and binary dependencies being considered? Is software installed under compatibility layers like Cygwin, Gnu on Windows or MSYS included?

Is there coverage for open source software present only as source code? Whether it is compiled into a released or internal product, or used for internal management and administration tasks, such programs – like any others – are part of the potential threat surface.

Some software like Docker images or virtualized systems are hard to scan and catalog. They may not be detectable by any traditional means. Is the risk of using outdated images, even if enabled only on demand, part of your strategy?

Has any consideration been given to things like browser plugins, development tool extensions, or app configuration? Depending on version and the program in question, certain settings in an otherwise security application can result in a compromise. While such risk is not traditionally considered part of the domain of patch management, it still deserves thought when creating a comprehensive approach to continuing security.

What and when to patch

Not all security flaws are equally dangerous, and this reality must be considered when creating a patch management strategy.

Necessary outages for upgrades can be very expensive, and in the worst cases, may create risk of SLA breaches.

Some patches are very easy to apply, while others can be extremely time consuming. In some cases, there may be no solution available at all – Such as when dealing with unsupported or end-of-life applications, or recently discovered security flaws with no resolution implemented. Some vendors have an unfortunate tendency to avoid or delay the implementation of solutions, while others prefer to hide the existence of a flaw entirely. For this reason, vulnerability intelligence tools are as indispensable as the patch manager itself.

When considering the severity and impact of a software vulnerability, as well as the risks it might expose you to, there are several factors to take into account.

The industry standard rating system for vulnerabilities is the Common Vulnerability Scoring System, or CVSS. While it has been criticized in some circles, it’s core metrics add a lot of value over the traditional patch-or-don’t approach to evaluating danger.

For a more in-depth study of the topic of vulnerability severity, the CVSS version 3 specification document is a good starting point: https://www.first.org/cvss/specification-document

Some of the most key things to consider, distilled, are as follows:

1) How easy is it to exploit the vulnerability?

For example, a vulnerability in a a server hosting the company website is more severe than a flaw in a program running on an isolated subnet.

2) How many conditions must be fulfilled for an attack to be executed?

Is the exploit always applicable, or does it only pose a problem under certain conditions such as when enabling optional features?

3) Who can exploit the vulnerability?

If a vulnerability is exposed, can it be exploited by anyone, or only by users with a certain level of access? Is it usable over a network, or only from the local machine?

4) How easy is the attack to execute?

For example, can it be done by anyone using an automated tool, or does the attacker need a high level of skill?

5) What is its impact?

That is, what will happen if an attack does succeed? Will you loose data? Will confidential information be leaked, or will a production system grind to a halt?

Knowing what will happen if the exploit is used to attack you is key when deciding how to act.

How to do it right

So far, we have talked only about problems, and the ways things can be done wrong (or not done at all). In this section, we will discuss some elements of a strategy better suited to keeping us safe in the long run.

Please keep in mind that an article with the rough length of this post – Less than two thousand words – can never convey the full complexity of the situation, and that software security is a complicated and ever-evolving field.

Secure management of a software infrastructure relies on continues education, diligence, and a frankly generous helping of paranoia. There is no fast way to become an expert, and no single person with expert knowledge of each and every subdomain.

Collaboration, knowledge sharing and humility is – as always – the path to a better approach.

Prioritize

Some problems are more pressing than others. Just as a community should respond differently to hateful messages spread by vandalism and an actively organizing group of dangerous extremists, an organization must act first on those things that are more important.

If there is a low severity problem with a key piece of software – Can it wait?

If there is an important piece of work in progress when a highly severe security flaw is discovered – Can you afford the risk of delaying updates until the work is completed?

When our tasks are time consuming and prone to error, what is most important to finish first?

Be consistent

Once a set of guidelines and practices have been selected, they must be followed. If you have created a formula for deciding when to delay a release or another important task, you should avoid the temptation to make exceptions. If a severe enough vulnerability is discovered, you must resist the plea to avoid interruptions.

A single slip-up can be enough to damage your reputation, compromise important information, or in some cases – like in hospitals, industrial control systems or infrastructure – cause irreparable damage or even a loss of life.

Security is a zero-sum game. You either resist attack – Or you don’t. There is no middle ground.

Cover a wide surface

As discussed in the section on software coverage, it is important to look at your network as complex above all. Do not be deceived by the reassuring sterility of aggregated information. The devil, as they say, truly is in the details, and it takes no more than a single oversight to render pointless all of your hard work.

Relying on any one tool, process or team is a dangerous bet, and the best process is always one that takes human error into account.

Unless security is an organizational priority, it is no priority at all. If you cannot rely on a team to accurately report the software it uses, implement a patch for an internally discovered flaw in a timely manner, or if you put blind faith in a single tool or vendor, you are leaving yourself open to attack.

Security is in many ways all but impossible to achieve, and all efforts towards it must be treated as part of a greater whole. Taking the issue lightly will, eventually, lead to disaster. Murphy’s law all but guarantees it.

In Conclusion

Security patch management is difficult, expensive, resource and time intensive, and – above all – impossible to ignore.

To get it right, we must treat the domain with the attention it deserves. Continuing learning, expanding our application coverage, evaluation of risk and proper reporting are among the keys to avoiding disaster.

As technology continues to evolve, so too does patch management. New platforms and devices, and the ever-increasing amount of open source software dependencies & third-party applications pose new risks. Meanwhile, we must resist the temptation to believe the promises of any vendor who say they can provide complete or bulletproof security – A thing which simply does not exist.

No approach is flawless. But, compared with naive reliance on proprietary tools, an approach respecting the complexity of the challenges faced will take you much further.

Security patching is just like driving a car – You only have to get it wrong once.

Stay updated.