The Path for Testing Path Traversal Vulnerabilities with Python

I have noticed an odd behavior in requests module in Python, which uses urllib3. I inspected the root cause via regression testing and I found that the root cause of the issue is a change that was introduced in urllib3.

import requests requests . get ( "http://127.0.0.1/../../../../doing/certain/check" )

Which should typically send the request to the following path: /../../../doing/certain/check . Instead, it’s requesting /doing/certain/check and modifying the specified input.

The reason is, dot segments escaping, following RFC3986.

The change to normalize URIs was introduced in this commit[1] to follow RFC3986[2]. The change is based on the “Remove Dot Segments” at the RFC.

This breaks a lot of security tools that are performing fuzzing for path traversal vulnerabilities in web-applications.

I have compiled a number of solutions for this issue.





Solutions

Option 1: urllib.request

url = "https://example.com/../../../etc/passwd" urllib . request . urlopen ( url )

Option 2: requests.Request

Credit: Rich Warren - [3] and [4].

url = "https://example.com/../../../etc/passwd" with requests . Session () as s : r = requests . Request ( method = 'GET' , url = url , headers = headers ) prep = r . prepare () prep . url = url req = s . send ( prep , verify = False )

And

url = "https://example.com/../../../etc/passwd" s = requests . Session () r = requests . Request ( method = 'GET' , url = url ) prep = r . prepare () prep . url = url

Option 3: Using urllib3.HTTPConnectionPool

Credit: Quentin Pradet

import urllib3 pool = urllib3 . HTTPConnectionPool ( "localhost" , 8000 ) r = pool . urlopen ( "GET" , "/../../../../doing/certain/check" ) print ( r . status )

Option 4: Downgrading urllib3

It’s possible to use previous versions of requests or urllib3 .

The option for current tools that use requests or urllib3 is to revert to urllib3==1.24.3 . However, this may not be the best option from an engineering perspective.





Final Additions

I started a discussion on Python Urllib3 Issue tracker on Github: https://github.com/urllib3/urllib3/issues/1790

I wanted to thank Seth Michael Larson, Quentin Pradet, and https://github.com/hodbn, for all their work on urllib3 and with the Python community!





References

Commit to support the removal of dot-segments: https://github.com/urllib3/urllib3/commit/5b047b645f5f93900d5e2fc31230848c25eb1f5f RFC3986 - Section 5.2.4 - Remove Dot Segments: http://tools.ietf.org/html/rfc3986#section-5.2.4 https://github.com/trustedsec/cve-2019-19781/issues/13 https://github.com/trustedsec/cve-2019-19781/blob/master/citrixmash.py#L140