In this high-level comparison of Nessus, Nexpose, and OpenVAS, I have not attempted a detailed metric based analysis. The reason being it would be time-consuming and difficult to get a conclusive result due to the large differences in detection and the categorization of vulnerabilities by the different solutions.

I have chosen to target the 3 different vulnerability scanners in a "black box" test against a Metasploitable version 2 Virtualbox.

Background Info

The testing deliberately focuses on network vulnerability scanning capabilities rather than looking at the web application vulnerability detection in detail. I believe that a network vulnerability scanner should be capable of identifying poorly configured services, default services that have poor security and software with known security vulnerabilities.

Notes on the Vulnerability Scanner Testing

External tools, apart from Nmap, that OpenVAS can use have not been installed. These external tools are mostly web application vulnerability detection tools, including wapiti, Arachni, Nikto and Dirb.

OpenVAS version 5 has been tested with the full scan profile. Ports were all TCP ports scanned with Nmap and top 100 UDP ports.

Nessus version 5 was launched using the External network scan profile. It was also tested with Internal Network Scan however, results were similar.

The Nexpose scanner was executed with the Full audit profile.

No tweaking of default scan profiles was undertaken.

No credentials were used during the scan. It was an external network service focused scan.

These results are only a quick overview. I have not followed up every discovered vulnerability to determine false positives and false negatives.

Edit 1st of September 2012 (clarification of scanner versions and plugins used)

Nessus : The home feed was used for the Nessus testing. According to the Tenable website The Nessus HomeFeed gives you the ability to scan your personal home network (up to 16 IP addresses) with the same high-speed, in-depth assessments and agentless scanning convenience that ProfessionalFeed subscribers enjoy.. Note when using the Nessus scanner with the home feed it cannot be used in a professional or commercial environment.

OpenVAS : The default OpenVAS 5 open source signatures and software was used. This is free to use under the GNU General Public License (GNU GPL).

Nexpose : The community version of Nexpose was tested. According to the Rapid7 website " Nexpose Community Edition is powered by the same scan engine as award-winning Nexpose Enterprise Edition and offers many of the same features." With this version you can scan up to 32 IP addresses.

And now for the results.....

Nessus 5

External Network Profile Critical 3

High 6

Medium 22

Low 8

Info 137 OpenVAS 5

Full Audit Scan Profile High 38

Medium 24

Low 36

Log 44 Nexpose

Full Audit Scan Profile Critical 49

Severe 103

Moderate 18

These total numbers, without any context around the categorization of findings or the accuracy of the results, provides us little value, except to highlight the wide variation in results from the different scanners.

Analysing a specific sample of Security Issues

In order to look at some more meaningful results, I have examined a sample set of exploitable and mis-configured services on the Metasploitable system.

This is only a sample of exploitable services on the target host. There are many more vulnerabilities present on the system; both network services and web application security holes.

At the last minute I decided to include Nmap with its NSE scripts against the Metasploitable host. The results were interesting to say the least, while not a full blown vulnerability scanner the development of the NSE scripting ability in Nmap makes this powerful tool even more capable.

the numbers get interesting...

These are the numbers of vulnerabilities correctly discovered and rated by each vulnerability scanner from the sample set of exploitable services.

Nessus OpenVAS NexPose Nmap 7 7 7 6

7 out of 15 security holes identified

Security Issue Nessus OpenVAS Nexpose Nmap FTP 21

Anonymous FTP Access FTP 21

VsFTPd Smiley Face Backdoor FTP 2121

ProFTPD Vulnerabilities SSH 22

Weak Host Keys PHP-CGI

Query String Parameter Injection CIFS

Null Sessions INGRESLOCK 1524

known backdoor drops to root shell NFS 2049

/* exported and writable MYSQL 3306

weak auth (root with no password) RMI REGISTRY 1099

Insecure Default Config DISTCCd 3632

distributed compiler POSTGRESQL 5432

weak auth (postgresql) VNC 5900

weak auth (password) IRC 6667

Unreal IRCd Backdoor Tomcat 8180

weak auth (tomcat/tomcat)

Notes about the sample set of tests

All the above vulnerabilities and mis-configurations, except for Anonymous FTP, can be exploited to gain shells on the system (in most cases with root privileges) using Metasploit or other methods.

(in most cases with root privileges) using Metasploit or other methods. There are a number of examples where the scanners do not detect weak or default credentials. While not specifically testing passwords, if MySQL is being checked for weak credentials why not other services?

Items such as the INGRESLOCK backdoor and the Unreal IRCd vulnerability are fairly obscure, however, this makes them good examples for testing overall capability.

The Metasploitable version 2 release page has good examples of exploiting many of the mis-configurations in this list. This highlights not only how a poorly configured service can lead to a root shell but also the fact that vulnerability scanners need to be able to detect these types of security related mis-configurations.

These scans were conducted in a black box manner, when running internal scans it is recommended to perform credential supplied scanning. This means providing the vulnerability scanning tool with valid Windows domain, SSH, or other valid authorisation so it can perform checks against the local system. This is of most value when looking for missing patches in an operating system or third party software and detecting installed applications.

Conclusion

Vulnerability scanning is an important security control that should be implemented by any organisation wishing to secure their IT infrastructure. It is recommended by the SANS Institute as a Critical Control and by the US-based NIST as a Security Management Control.

The results show significant variation in discovered security vulnerabilities by the different tools. It may be helpful to compare vulnerability scanners to anti-virus solutions; they are both an important security control that can enhance an organisation's security posture. However, as with anti-virus, a vulnerability scanner will not find all the bad things.

This will be common knowledge for most in the security industry who have performed network vulnerability testing. When performing vulnerability scanning, it is necessary to check the results for accuracy (false positives) and to actively look for things that were missed (false negatives).

A recommended approach to vulnerability scanning

Tune the vulnerability scan profiles to suit your requirements

Perform a detailed analysis of the results

Run secondary tools such as Nmap, a secondary vulnerability scanning solution and/or specialised tools. The use of multiple tools will provide a greater level of coverage and assist in confirming discovered vulnerabilities.

Performing internal focused testing in conjunction with external facing vulnerability scans adds value when working to secure Internet connected networks or servers.