How to Evaluate Web Application Security Scanners

Ask 20 penetration testers which web application security scanner they prefer to use and you will get 20 different answers, if not more. Every web vulnerability scanner has its own pros and cons and what works for Mr. X does not necessarily work for Mr. Y, almost like in the case of everything else. Therefore, you shouldn't simply base your purchasing decision or build a web application security program on what your colleagues say or think or on the aspects such as whether it's an open-source application, whether it's available for Windows or Linux, how nice the user interface is, etc. You have to do some testing and make your own decision.

This guide will explain how to evaluate vulnerability scanning tools and help you choose the right vulnerability detection tool that fits your requirements. Note that the list focuses on dynamic application security testing – DAST tools, not static application security testing – SAST tools that only scan the source code.

Make a List of Requirements

Before getting your hands dirty with web application scanning, compile a list of your current web security requirements. The most typical ones are:

Automating tasks: There are several advantages to automating the identification of web application security vulnerabilities. The most common reason that the management can think of is to save time but it is not just about that. Read more in the following article: Why Web Vulnerability Testing Needs to be Automated.

There are several advantages to automating the identification of web application security vulnerabilities. The most common reason that the management can think of is to save time but it is not just about that. Read more in the following article: Why Web Vulnerability Testing Needs to be Automated. Lowering the costs of web application security by doing in-house scanning rather than hiring a seasoned expensive penetration tester or service.

by doing in-house scanning rather than hiring a seasoned expensive penetration tester or service. Increasing the coverage: As opposed to penetration testers, an automated web application security scanner has an extensive set of heuristic web vulnerability checks that are frequently updated by a number of researchers and security experts. This allows users to identify all type of web application vulnerabilities in custom-made web applications. Some scanners also have a vulnerability database for known web applications such as WordPress and Joomla that also comes in handy if your business or customers are using such web applications.

In short, automated web application security scanners are mostly required to save time during application security testing and to ensure that all technical web vulnerabilities are identified. Once you listed down your requirements, proceed to gather as much information as you can about the websites and web applications that you will be scanning.

Document All the Web Applications to Be Scanned

The next step in the selection process is to document the web applications that you will be scanning using the automated web application vulnerability scanner. During this stage, it is important to identify the most common factors of web applications. For example, if something is implemented in 2% of the web applications that you are scanning, then you can test such parts manually. But if something is implemented in 60% of web applications or more, then its testing should definitely be automated. The following list of questions will help you build a list of web applications common factors:

Which development frameworks are used to build web applications? Is there any type of authentication mechanism (access control) used in web applications? What type of database backend server is used? Are URL rewrite rules being used? Do you need a client-side certificate to access web applications? Are there any client-side scripts that need to executed and tested? Do you use custom 404 error pages? Are there any types of protection mechanisms implemented in web applications, such as Anti-CSRF mechanisms?

Once you have the list of common factors, you can shortlist the web application security tools that you would like to evaluate. For example:

If all your web applications are developed with PHP and .NET, you do not need to test automated tools that cannot scan and identify vulnerabilities in such web technologies. For example, some web application security scanners cannot automatically crawl, scan and identify vulnerabilities in web applications built with .NET, hence there is no need to test such scanners. If authentication is implemented, automated web security scanners that you want to test must support the type of authentication mechanism that is being used and you must be able to configure them to automatically login to the website during a web security scan. If URL rewrite rules are being used for search engine friendly URLs, the scanner should be able to identify such URL structure and scan the website properly without getting stuck in some sort of infinite loop or without requiring extensive configuration changes. If web applications that you want to scan have some sort of a mechanism such as anti-CSRF, the scanner should still be able to automatically scan the web application without missing any parts of it. Note: Such mechanisms are typically barriers for most automated tools.

Testing Web Application Security Scanners

Once you have completed all the paperwork, it is time to get your hands dirty and start scanning websites. Download the trial or demo version of each web vulnerability scanner that you would like to evaluate. If the trial is not available for download, contact the software vendor.

How Web Application Security Scanners Work

If you are new to automated tools such as web application security scanners, before doing anything else, you need to understand how such tools work. First, the scanner crawls the target website or web application and identify all possible web application attack entry points and parameters. During this stage, the crawler accesses every link that it discovers, including links in client-side scripts, etc. During the scanning stage, the scanner sends specially crafted HTTP requests, which include a payload simulating a known vulnerability that is used to test if a website is vulnerable or not.

Although automated scanners are designed not to be intrusive, there is still a small probability that such attacks might tamper your web application or database, depending on how secure your web application already is. In light of this, we move to the next point.

Always Scan Realistic Test and Staging Web Applications

First and foremost, it is important to scan realistic web applications, i.e. web applications that are similar or ideally identical to the real live ones that you will be scanning. Unfortunately, many people evaluate web application security scanners against vulnerable web applications such as DVWA (Damn Vulnerable Web Application) and OWASP WebGoat developed by the Open Web Application Security Project. These type of web applications have been built for educational purposes and might be completely different than the web applications that your business or customers use.

Therefore, do not base your findings on any of such demo sites. Don't forget that a web app security scanner can easily cover thousands of different vulnerabilities but these simple test systems will only show how the scanner behaves according to 10 of these thousands of vulnerability cases. Therefore, you should conduct your tests against a test or staging website if you would like to see how the web application security scanner actually performs.

It might take some time until you get used to the automated tools that you are using, so stay safe and scan test websites. What's for sure is that with time, you will get used to the tools that you are using and soon should be able to scan live websites. In fact, it is recommended to scan both staging and live websites because some vulnerabilities might only be introduced when switching from the staging server to the live server.

Evaluating Web Application Security Scanners and the Results

Now that you know what you need and how to evaluate the software, it is time to fire up the scanners. Below are some points, which, if followed, should help you determine the best automated web application security scanner that fits your requirements.

Web Application Coverage

During a scan, check the list of all crawled objects. Ensure that the sitemap representation of your web application includes all the files and their variations, scripts, client scripts, input parameters, directories, etc. that are on your website. If not all objects are listed, it means that the crawler is not able to automatically crawl all of the web application, thus might not identify all vulnerabilities.

Web Vulnerability Reporting

Once the automated website security scans are finished, start comparing the findings of both scanners.

Which web application security scanner detected the most vulnerabilities?

Which web application security scanner reported less false positives?

Are there any vulnerabilities that were not detected by any of the scanners (false negatives)?

By finding the best ratio between all of the three points mentioned above, you should find the ideal automated web security scanner. Definitely, you do not want a scanner that reports hundreds, if not thousands of vulnerabilities but most of them are false positives. You might ignore false positives or mark them manually. However, this mechanism might not scale well because when you actually start using your scanner regularly on large web applications, and there are bigger problems with false positives, such as a lot of extra time being spent on this process (i.e. verifying false positives), you and your users will start losing faith in the scanner. To learn more about false positives and the impact they have on a penetration test, refer to the article The Problem of False Positives in Web Application Security and How to Tackle Them.

On the other hand, you do not want a scanner that does not detect much of the most commonly exploited vulnerabilities. The best automated scanners are typically those scanners that can report all of the most commonly exploitable web application vulnerabilities, such as SQL injection and Cross-site scripting (XSS) (but also less common ones that may be dangerous, for example command injection or path traversal), report the least false positives, and require the least configuration.

Each automated scan should always be accompanied by manual penetration testing, so those one-off or completely new vulnerabilities, which are rarely or never exploited in a real-life attack, can be identified manually. Community whitehat hackers can also help in that with the right bounty program. But at least, the bulk of the work is done and developers can already start concentrating on remediating the reported vulnerabilities.

Integration with Other Web Security and Development Tools

Another important feature or set of features to look for in web application security scanners is how easy they can be integrated. For example, does it output scan results in standard formats that can be parsed by other tools, such as XML? Can an automated web security scan be launched via the command line or batch files? The more such features it has, the easier it is to integrate web application security scanning in the SDLC process of web applications.

And for those penetration testers who always like to dig deeper into web applications, can the scanner import the output of other web security tools typically used during manual security audits, such as other proxy servers, or HTTP analyzers? Support for such tools is an important factor to consider as well, since the more integration capabilities a scanner has, the more of the process can be automated.

Choosing Your Web Application Security Scanner

As we have seen, there are several criteria that you should look into when choosing a web application security scanner. What is important is that you should focus on your needs rather than relying on the opinions of other people. Ensure that the automated tool you are about to choose can scan web applications that you want to audit and allows you to automate as much as possible, to save time, money, and other resources.