During testing of the Cisco Nexus 7000 series switch I identified a high impact (CVSS8.8) vulnerability within the OS which also formed the basis of my talk at Bsides Manchester.

Note that the views are my own and don’t represent my employer.

TL:DR

I identified a vulnerability in the Cisco NX-OS data centre switch range that as executed is a CVSS 8.8 against the Cisco NX-OS data centre switches.

I have worked with Cisco between Feb and Oct 2017 to get the vulnerabilities resolved and followed a coordinated disclosure approach. However Cisco have decided that the vulnerabilities identified are not as severe, a point on which we disagree. This post outlines the details of the exploit now that is had a published fix and how the chained set of vulnerabilities identified has an impact.

The nature of the vulnerabilities discovered in February 2017 were so fundamental to the way the software operates that the fix required a major code re-write. It took the vendor 222 days from the vulnerability being reported to it being fixed and updated, with software being published on the version 8 code branch. A recommended mitigation was shared:

Administrators should ensure that all methods that could potentially allow access to the underlying CLI such as the Python scripting parser are restricted to trusted users only.

Bursting the VDC bubbles in the Cisco NX-OS

Timeline of disclosure

20 Feb 17 — Reported to Cisco PSIRT and provided a video demonstrating the vulnerability. I was initially told that the vulnerability that I had disclosed was already known and therefore a duplicate. Upon challenging the response and describing that it was possible to take over other VDC functionality it was then passed on for further investigation.

28 Feb 17 — CISCO PSIRT acknowledged the the vulnerability as demonstrated scores 8.8 against CVSS v3

Hi Greg, We understand that you must report your findings to the management in your organization. We have determined this vulnerability scores an 8.8: CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H. We consider this is a High severity security defect. According to our security vulnerability policy, critical and high severity vulnerabilities must be fixed in all vulnerable products and in all shipping versions of code before we publicly disclose them. Depending on the number of affected products and releases, this could be a considerable task. This vulnerability was also reported by one of our internal testing teams and we already have started integrating the fixes. While I cannot give you a definitive answer, because of the extent of the changes, I estimate that we will be issuing a security advisory in the late fall. A CVE will get assigned once the investigation is complete and fixed code is available. This usually occurs about 5 weeks prior to the advisory being released. Do you have plans for public disclosure?

6 March 2017 —I notified Cisco that I do plan to disclose publicly at a later date once the vulnerability is fixed by Cisco, but that keen for Cisco should work to a similar timeline that it imposes on non Cisco vendor when vulnerabilities are discovered. http://www.cisco.com/c/en/us/about/security-center/vendor-vulnerability-policy.html. The view from CISCO was that this policy will be fed back to their research team.

17 March 2017 — Cisco confirmed that one of the vulnerabilities that I had discovered during testing had previously been reported and had been fixed in a later release of software. https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/Cisco-SA-20150630-CVE-2015-4231

The vulnerability as published suggested that this vulnerability has a lower CVSS score, but I had demonstrated that this goes beyond being able to delete a file.

14 April 2017 — Over the previous few weeks I had worked with Cisco to demonstrate the issue and how it is realised. At this point Cisco broke the vulnerability down into three parts.

CSCur08416 = Python parser escape, is public (has been disclosed) and is fixed in 7.3.x code branch and it will be fixed in 8.1.1 that will be released by early May 2017.

CSCvd78353 = Privilege escalation through ‘export’. This was a new defect logged for the issue I disclosed. Cisco said that there is no fix for this currently and won’t make the 8.1.1 release.

25 July 2017 — Followed up with Cisco on progress for the fix (5 months since initially disclosed). CSCvd78353 will now be fixed in September 2017.

30 August 2017 — Cisco confirmed that the software team are committed to fixing the issue before 29th September 2017. The vulnerability I had identified was fundamental in the software design for the NX-OS as environment variables had been widely used as a method for managing security controls and configuration. A recommended mitigation was shared:

Administrators should ensure that all methods that could potentially allow access to the underlying CLI such as the Python scripting parser are restricted to trusted users only.

28 September 2017 — Cisco published a fixed version of the code on the version 8 code branch, but not all fixes are in place in the 6 and 7 code.

From initial disclosure to a first code fix took 222 days

18 October 2017 — Cisco release CVE-2017–12301 https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20171018-ppe relating to Python parser escapes that I have also demonstrated to Cisco PSIRT.

Cisco Nexus Switches*

*https://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/Data_Sheet_C78-437762.html

Critical vulnerability to a low risk vulnerability

Over the following weeks I had contacted Cisco to discuss how the scoring is calculated and the significance of the vulnerability discovered. This is in part due to the way that CVSS scores are calculated and do not take account of vulnerability chaining. The CVSSv3 specification allows for analysis of chained vulnerabilities, but this is optional. NCC Group have written about this problem previously too in “When CVSS fits and when it doesn’t”

Vulnerability Chaining CVSS is designed to classify and rate individual vulnerabilities. However, it is important to support the needs of the vulnerability analysis community by accommodating situations where multiple vulnerabilities are exploited in the course of a single attack to compromise a host or application. The scoring of multiple vulnerabilities in this manner is termed Vulnerability Chaining. Note that this is not a formal metric, but is included as guidance for analysts when scoring these kinds of attacks.

Unfortunately, despite a fix for the first bug being available on new code branches, Cisco software site still recommended using a vulnerable version 6.2(16). As a system administrator you always want to follow the vendor’s guidance on which versions of software to use, but advice from vendors is not always consistent in terms of security vulnerability fixes versus stable working code. This specific problem been reported, and the team at Cisco are working to update their advice.

Not only are the recommend versions vulnerable to the vulnerabilities described in the post, but also a number of others that have Cisco have resolved in later versions. I informed Cisco of the guidance they provided and to update their guidance in Oct 17, it is now 21 Dec 17 and the same recommendations remain.

Suggested software release for Nexus 7000 — recommends a vulnerable versions

Suggested software release for Nexus 7000 — recommends a vulnerable versions

The attack surface

The NX-OS ‘Nexus’ series switches have a virtualized switch function. This enables you to run multiple isolated switches on one physical device. The feature is discussed in more detail later.

As part of security testing of a Cisco Nexus 7000 series switch, we wanted to assess the isolation of the Virtual Device Contexts (VDC). The goal was to see if we could affect the operation of a VDC, outside of the attackers control. VDCs are analogous to virtual machines on a hypervisor, and provide isolation through multiple virtual switches. The Cisco whitepaper on VDC functionality includes the two images below, which show the functions in a single switch and when using multiple VDC’s.