Augmenting Security Engineers

So what kind of “tedious work” does a pentest usually entail? Regardless of the outcome, there’s always a lot of follow-up work needed for pentests. It usually involves: filing bug reports, opening cases, writing and formatting documents with some boilerplate text, emailing collaborators, etc. None of this is rocket science, but it tends to be very time-consuming (and boring) work.

For a real-world example: we’ve done thousands of pentests on apps released via the Salesforce AppExchange. When we give those reports, we try to make it very actionable; for any vulnerability we might uncover, we like to educate people, not just correct the vulnerability. We usually say: “Here’s why this is bad. Here’s how to fix this instance, but more importantly, here’s where to learn about best practices, so you can prevent similar issues in the future”.

That means that each one of these vulnerabilities might have 3–5 paragraphs of text. It might involve filing reports and sending emails, or logging bugs against the developers who created the software. It’s extremely time consuming — not individually, perhaps, but certainly in aggregate.

So the flow of Vulnreport is pretty simple, and built around exactly how pentesters work: as you’re performing a test, whenever you encounter a vulnerability (say, Stored XSS), you just click to select it from a list, and enter a little data about it:

A sample pentest in Vulnreport

A sample vulnerability finding in Vulnreport

At the end of a test, you select “pass” or “fail”, and it will generate a nice report. Boom. (OK, there’s a bit more to it than that, but not that much; you can find all the gory details on our documentation page.)

If you’ve been a part of Salesforce’s partner ecosystem and gone through the AppExchange Security Review process in the past couple of years, you’ve likely seen what the generated report looks like. The most important thing to us is that although our engineers are only inputting the technical vulnerability data, we’ve already spent time once, up-front, to write explanations for each class of vulnerability (“Vulntype” in Vulnreport). That means that when a report is generated, it will pull that data together to explain what XSS means:

An example of a report generated by Vulnreport

Integration into Workflow

For reporting, the system supports what we call “linked objects”. This allows you to perform any type of external action (reporting, sending results emails, filing bugs, toggling record states, etc. — the possibilities are practically limitless), by implementing a simple interface. For us, that means updating records in our internal ticket-tracking system, and sending emails. For you, that might mean logging a bug in JIRA, posting to a channel on Slack, or opening a case in your service desk app, or … well, really, anything.

As an example, I implemented a sample connector to post information in a case in Salesforce via the API (you can see it in our Github repo, here). Doing so took me under an hour! (If you have ideas for other connectors you’d like to create and contribute, we’d love to include them.)

Sharing Vulnreport

At Salesforce, Vulnreport is now deployed across the Trust team. While v.1 ran the AppExchange security review process, in its current iteration Vulnreport is also used to manage our vendor assessment program, infrastructure security program, M&A security audits, and Open Source Software security approvals. This system has saved us a TON of time — we estimate it to be about 1 full year of a security engineer’s full-time work. (Not to mention probably raising their quality of life, too! Nobody likes doing boring, repetitive tasks.)

As we began talking about it to our partners and other security consulting groups we work with, we heard the same thing over and over again: “Hey, this would be useful for us too!” They could have built something similar, of course, but we happened to do so first.

And, since we’d already spent the time to create it, it only made sense to contribute it as an open source project, so everyone can get the benefit. In fact, among the security consulting groups we’ve shared it with, a couple are already planning implementations within the next few weeks. And, we’ll be showing some demos about Vulnreport at Black Hat USA’s Arsenal this week, to unveil it to the masses.