There’s a lot of debate/conversation taking place right now around in the information security community regarding the merits of penetration testing vs. bug bounties, pentesting vs. vulnerability assessments, bug bounties or a red team engagement, and the role of trusted advisors in all of it.

In my grumpy estimation, far too few understand the differences between these enough to decided which to use, and when. The nicest thing to do would be to blame sales, but the problem is deeper than that.

I cover these definitions in a lot of depth here, but let me give the super-abridged version:

A vulnerability assessment is designed to find as many flaws as possible in order to make a prioritized list of remediation items

is designed to find as many flaws as possible in order to make a prioritized list of remediation items A penetration test is designed to see if a mature defense can stop an attacker from achieving one or more specific goals

is designed to see if a mature defense can stop an attacker from achieving one or more specific goals A bug bounty is designed to use the benefits of a crowd to discover as many vulnerabilities as possible

is designed to use the benefits of a crowd to discover as many vulnerabilities as possible A red team engagement is designed to continuously test and improve the effectiveness of a company’s blue team by mimicking real-world attackers

is designed to continuously test and improve the effectiveness of a company’s blue team by mimicking real-world attackers A trusted advisor (in the security context) is someone who should be able to tell a company, based on their maturity and goals, which approach to take at any given point in time

Maturity matching

In my opinion, the maturity of the organization in question is the most important factor in the decision of what type of assessment to use, and when.

What do I mean by maturity?

Is the company patched?

Does the company have a list of everything it owns, who has access to it, and when that access was last reviewed?

Does it know where its data is and how its protected?

Does it know how that data moves around the environment during business processes?

If the answer to these questions are no, then they’re likely low-maturity. Meaning, they should stop thinking about hybrid-cloud, crowd-red-team ninja assessments and focus on basics. Make a list of where your stuff is and get patched.

But let’s assume we’re at medium maturity—whatever that means—and we’re ready to start doing some security assessments. Where do you start?

It’s never a bad time to have a trusted advisor. You should always have one. Whether they’re internal or external doesn’t matter that much, but you need to be able to quickly ask questions like these and get answers. So, first priority is to have someone who can help you build and stay on track with a strategy. The first type of security assessment to do, i.e., the assessment type for lowest-maturity companies, is the vulnerability assessment. This is because it’s not for testing mature defenses, per say, but rather giving you a list of all the stuff you need to fix to have a good defenses. It’s a prioritized list of everything you should fix, and in what order. That’s first. After you’ve done one or more vulnerability assessments and you’ve fixed everything that’s been found (that’s an important part, otherwise you’re wasting a lot of money) it’s time to move on to penetration testing. This is probably best done with a trusted vendor or even an internal group (if you have one). You probably don’t have one at that point, though, otherwise you wouldn’t be in this position. So a trusted external group is likely best because what is likely to be found at this stage could be pretty bad, and pretty embarrassing. After you’ve done lots of vulnerability assessment and lots of pentesting, and you’re now coming back with few results whenever you do them, you’re ready for the next stage, which can go in multiple directions. If you want to continue finding more vulnerabilities, and the systems you’re testing are not overly sensitive (source code reviews, private networks, crown jewels, etc.), then you should start thinking about doing a bug bounty. A properly sized and managed crowd can often find additional vulnerabilties that can only be caught by “many eyes”, but you want to be sure that you’re picking the right things for them to look at, and managing it closely. Finally, either after or at the same time as pentesting, you should start looking into red team options. Red teams are permanent, continuous campaigns designed to emulate real-world attackers. Done right, they’re the highest maturity testing type as they must remain mostly independent from the organization they’re testing, must remain extremely current and sharp, and must constantly evolve their tools and techniques to stay effective. The one key for the red team is that its purpose is to improve the blue team, which is a very different goal than that of a vulnerability assessment, a penetration test, or a bounty.

Summary

Have a trusted advisor (a person or company) who can steer your security assessment efforts throughout your maturity levels. Start with vulnerability assessments, and don’t do any of the other types of testing until you have cleaned up your environment through remediation enough that it’s hard to find things. Then move to pentesting with a trusted company. If you’re getting no results anymore from trusted advisor pentesting, consider adding a bounty program to leverage the “many eyes” of a crowd. After the vulnerability assessment stage is complete, when you’re moving into the pentesting phase, you should feel free to start looking into red team options as well, if not just because it will take some time to research and find good options. Red teams are not “better” pentests; they’re a different kind of assessment altogether, with different goals. Remember to have your trusted advisor(s) guiding you through all of this; there are lots of variables that determine when, and how, to do what. This is just a guide to get you started.

Notes