URL Rewrite Rules

Web application developers use URL Rewrite Rules to hide parameters in the URL path structure. This makes it easier for search engines to index all the pages on a website, while web browsers are supplied the URL in a format they understand and makes them easy for users to remember.

For example, when you browse a hardware store online library, the URL typically looks something like this: http://www.example.com/tools/hammer/. Using a URL rewrite rule, the web server converts this URL to a specific format, so it can retrieve the data from the back end database and display the tool details to the website visitor:

http://www.example.com/library.php?tools=hammer.

In this example, we can determine that the subdirectory ('/tools') in the first URL is actually a parameter in the library.php file that accepts inputs, which in this case is the tool name ('hammer').

During the scan, Netsparker sends normal HTTP requests to the web application to mimic an attacker, ensure that such requests are accepted by the web application, and all parameters in the URLs are properly scanned. It is also possible to scan pages which have more than one parameter in the URL.

Problems with URL Rewrite Rules

This table lists and explains the problems that can occur when automated web vulnerability scanners attempt to scan websites that employ URL rewrite technology and rules.





Problem Description Parameters in URLs are not scanned A common problem web vulnerability scanners have when scanning web applications that use URL rewriting technology is that scanners are unable to identify parameters in the URLs. Scanners assume that the URLs are directories rather than parameter names or values, and leave them unscanned. For example, when scanning the URL http://www.example.com/tools/hammer/ the scanner would think that both tools and hammer are directories, while in reality tools is a parameter and hammer is a value. Prolonged vulnerability scans This problem can lead to prolonged scans and incorrect scan results. For example, if the web vulnerability scanner is scanning a tool database that contains 100,000 tools, since the scanner is unable to identify that there is a parameter and a value in the URL, it would think that they are all different pages, so it will try to crawl them and scan them all. If memory problems and other exceptions are not handled properly by your scanner, this could also lead to your software crashing, leaving you with no results and a number of wasted hours. If you do not configure URL rewrite rules in Netsparker, it will heuristically identify the pattern and will limit the scan to avoid having prolonged scans and incorrect results. Configuring URL rewrite rules is a difficult process Since URL rewrite technology has become really popular in web applications, many commercial web vulnerability scanners allow users to configure the scanner so it can identify the parameters in the URLs and scan them. But even though web vulnerability scanners can be configured to scan websites using URL Rewrite Rules, there are several other problems users can face: Configuring of URL rewrite rules support is very difficult

User must know how to write regular expressions

User should have access to web server configuration files Unless you are the developer of the web application itself or have a deep understanding of the web application, and unless you have direct access to the configuration files, it is impossible to configure URL rewrite rules on the scanner. And, even if you do know how to do it, configuring rewrite rules is a very difficult and time consuming task. Web applications are not properly scanned for vulnerabilities Assuming you manage to configure URL rewrite rules in your web vulnerability scanner, there are further problems, or at least, there are a number of limitations to how the scanners scan the web application. As a security precaution, web applications do not accept HTTP requests which are already ‘translated’, such as http://www.example.com/library.php?tools=hammer. By default .NET web applications do not accept such HTTP requests. The problem is worsened when scanning MVC web applications because such applications use a different approach to URL rewriting. While Netsparker can scan MVC web applications, many other web vulnerability scanners cannot, even when URL rewrite rules are configured. Once you configure the URL rewrite rules in your scanner, it sends a type of HTTP requests called translated queries. Even though the web application security scanner reports that the scan ran successfully, most of the HTTP requests were denied and the parameters in URLs were not scanned, providing you with a false sense of security.

By default, Netsparker can heuristically identify URL rewrites in the target website. Netsparker can also automatically configure the settings. However, if you can configure the URL rewrite rules manually as explained here, the scan will be more efficient.

Specifying the Parameter Type

Specifying the correct parameter type makes the web application security scan more accurate.

For example, imagine specifying the following pattern: ‘/{PARAM}/{ID}’, where ‘{PARAM}’ is the parameter that will be scanned and ‘{ID}’ is its value. If the parameter type is not specified, leaving the default 'Any', this means that both the below URLs will match this URL Rewrite configuration:

http://www.example.com/products/18

http://www.example.com/products/date.js

The above matching is too generic and might lead to incorrect scan results. For example Netsparker might not scan some files because after collecting enough samples (i.e. /product/1, /product/2, /product/3 etc) it will assume that /products/date.js is just another value of the product parameter, due to the generic matching.

is just another value of the product parameter, due to the generic matching. In this case, if you set the parameter type to 'Integer', you avoid this problem, since Netsparker will only expect integers when scanning such parameters. Then, when Netsparker detects something else, such as /product/date.js , it will recognize it is a file.

Encoded URLs

If you are manually configuring URL Rewrite Rules, and your website URLs use encoded values, always specify the decoded value.

For example, suppose the Website URL is:

http://www.example.com/user/john%2dDoe.

The correct URL rewrite rule should contain the decoded character as follows:

http://www.example.com/user/{firstname}-{lastname}

This rewrite rule is incorrect because it contains the encoded character:

http://www.example.com/user/{firstname}%2d{lastname}

URL Rewrite Fields

This table lists and describes the fields in the URL Rewrite tab.

Field Description Root Path Max Dynamic Signatures If a URL block in the root path contains more items than this limit, it will be identified as a URL rewrite parameter. It must be between 1 and 10,000. This field is displayed only in the Heuristic tab. Sub Path Dynamic Signatures If a URL block in the sub path contains more items than this limit, it will be identified as a URL rewrite parameter. It must be between 1 and 10,000. This field is displayed only in the Heuristic tab. Block Separators Enter separators to use to split the URL into blocks. This field is displayed only in the Heuristic tab. Analyzable Extensions If the URL contains a file extension, it will be analyzed only if the respective extension is in this list. This field is displayed only in the Heuristic tab. Enable Heuristic URL Rewrite detection Netsparker will try to automatically detect other URL rewrite rules if this option is set. This field is displayed only in the Custom tab. Placeholder Pattern This contains the relative path with placeholders for URL rewrite parameters. This field is displayed only in the Custom tab. RegEx Pattern This is a regular expression used for matching the URL rewrite parameters. This field is displayed only in the Custom tab.

From the main menu, select Scan, then New Scan . The New Scan window is displayed. In the Scan Options section, select the URL Rewrite tab. Select an option: None, Heuristic or Custom:

If None is selected, then no rules will be applied

If you keep Heuristic as the default, then:

Netsparker Enterprise will automatically populate the fields:

Root Path Maximum Dynamic Signatures

Sub Path Maximum Dynamic Signatures

Block Separators

Analyzable Extensions

If you select Custom :

Select Enable Heuristic URL Rewrite Detection so Netsparker will attempt to automatically determine other URL Rewrite rules. Both Custom rules and Heuristic rules will be in effect. Or disable, so that only the Custom rules will be in effect.

In the Placeholder Pattern and RegEx Pattern fields, enter the relevant information

and fields, enter the relevant information Click New to enter further rules

Click Launch .

Open Netsparker Standard. From the Start a New Scan dialog, in the Scan Settings menu, click URL Rewrite .

Heuristic URL Rewrite Support is the default option. Select Custom from the dropdown.

The Custom fields are displayed.

Check Enable Heuristic Rule Detection , so the scanner still tries to automatically detect any additional URL Rewrites on the target website and use them in conjunction with the ones you configured. Click Copy Rules to copy custom URL rewrite rules listed below. Click Exclusions to exclude certain URLs from URL Rewrite Detection. The Excluded Paths dialog is displayed. Complete as required and click Save on the dialog box.

Click New to launch the URL Rewrite Rules wizard.

In the first step of the wizard, specify a URL that matches the URL rewrite rule you want to add, such as ‘http://www.example.com/tools/hammer/’

If you wish to configure the URL Rewrite rules manually in Netsparker, without using the wizard you can simply click on the Placeholder Pattern and RegEx Pattern input fields to populate them manually.

In the second step of the wizard, you have to specify which of the path segments is a parameter and its type, by following the below procedure:

Check the URL path segment that contains a parameter value

Specify the parameter name

Specify the parameter type from the drop down menu in the Parameter Type column

If there are multiple parameters in the URL, specify all of them. An example of this is (http://www.example.com/departments/equipment/tools/hammer/), as illustrated.

Click Finish so the placeholder pattern and regular expression are automatically generated. Click on any of the values to manually modify them, for example to manually write a regular expression.