Exploit kit creators have been inventing increasingly interesting methods of masking their exploits, shellcodes, and payloads so that it is harder for analysts to define the type of the exploit and know what actions they may perform. Several days ago analysts found the usage of the Diffie-Hellman cryptographic protocol in the Angler Exploit Kit, which is one of the most popular exploit kits at the moment.

In Angler, threat actors used the Diffie-Hellman protocol to get a structure with the shellcode of one of the recent exploits for the CVE-2015-2419 vulnerability for the Internet Explorer 11 browser and then for the CVE-2015-5560 vulnerability exploit for Adobe Flash. Most likely, the goal of the threat actors was creating difficulties in firewall detection of the exploit (as firewalls cannot decipher a shellcode and exploit by the means of the intercepted traffic analysis) and also making it harder for the analysts to get the exploit code. However, the experts from Kaspersky Lab managed to perform a successful attack against Diffie-Hellman protocol implementation and decipher the shellcode.

Angler vs. Analysts

To make matters worse for analysts, JavaScript code and ActionScript code multiple obfuscation and a user IP ban upon sending the encrypted structure with a shellcode to the user were used in addition to the Diffie-Hellman protocol. After getting the structure with the shellcode by that means (encrypted with a one-time key by using the Diffie-Hellman protocol), the exploit kit sample becomes unusable after one processing: the analyst is unable to understand what a specific file does, reproduce the attack, and, quite often, identify the exploit and vulnerability at all.

A key exchange and getting a shellcode for the CVE-2015-2419 exploit

There is a key exchange request in the picture above. As a response, a browser gets from the threat actors’ server an encrypted array that contains a shellcode to exploit the vulnerability. The same traffic request has been used to download the Flash vulnerability exploit.

As the secret for key generation is new each time, an analyst is unable to send it to the browser once more, reproduce the attack, and identify the vulnerability, even if he has the recorded traffic.

Diffie-Hellman Protocol Implementation Features

The used implementation of the Diffie-Hellman protocol includes the following:

The server generates a random number g (16 bytes) and sends the HTML page with the number g and JavaScript implementation of the Diffie-Hellman algorithm to the user’s browser. JavaScript generates a random modulo p (16 bytes) and a random private key Ka (16 bytes) in the user’s browser, and then JavaScript calculates the public key A = gKa mod p and sends the three numbers (g, A, p) to the server as a JSON object along with the Internet browser version. {"g":"40a262b1360a6a16612ca8251161a9a5","A":"5eff90f1c48342f5d519cd02b5dfd8b","p":"1b0b5c6e6b0b5b7e6c6d0b1b0a8c3c7e","v":"17923"} The server generates its own random private key Kb and its random encryption key Kx (16 bytes) and finds the Diffie-Hellman shared secret Kdh = AKb mod p. After that, the server encrypts the shellcode by using the XTEA algorithm and the key Kx, then base64_encode and urlencode, getting the string b as a result. Then, the key Kx is also encrypted by XTEA with the key Kdh, base64_encode, and urlencode, getting the string k as a result. And finally, the server calculates its public key B = gKb mod p and sends Base64-encrypted JSON object that contains B, k, and b to the browser: eyJCIjoiMDJhYTY1MjZlNmVkYzAwNDIzOTRiN2VhODFlYzViNzUiLCJrIj…1k1dnVNYWY1UlVXZjYxSSUzRCJ9 After Base64 decoding: {"B":"02aa6526e6edc0042394b7ea81ec5b75","k":"I5nkiFBk3LALF%2BnfkR7%2FYQ%3D%3D","b":"to0ShZH…3Y5vuMaf5RUWf61I%3D"} A user’s browser calculates the Diffie-Hellman shared secret Kdh = BKa mod p, decrypts k urldecode, base64_decode, and XTEA by using the key Kdh, getting the key Kx, and eventually decrypts the urldecode, base64_decode, and XTEA shellcode by using the key Kx.

It is safe to assume that the aim of using the given sophisticated cryptographic system is shellcode interception prevention by listening to the Internet traffic between the server with the exploit kit and the user’s browser. We managed to perform a successful attack against the implementation of the encryption protocol and decrypt the shellcode. We used the modified Pohlig-Hellman algorithm for the attack (a deterministic algorithm of discrete logarithm-finding in the residue ring modulo a prime number).

According to the original algorithm, for the case when the Euler function expansion of the modulo p into prime factors q i is known (coprime factors Q i )

the complexity of finding the private key Ka and the Diffie-Hellman shared secret Kdh by using intercepted public keys A and B is

We used an optimized algorithm of finding the discrete logarithm in the residue ring modulo a prime number, taking into account the infinitesimality of logp with respect to q i , and low probability of occurrence of large prime factors raised to the power of greater than one in the Euler function φ(p); i.e., α i will equal one for large q i with a high probability. Owing to that, the complexity of the modified algorithm is

which allows us to perform a successful attack in case if all q i < 1018. The experiment has shown that the given condition is observed in more than a half of cases of using the aforementioned Diffie-Hellman protocol implementation (the case of randomly generated g, p, Ka, and Kb without their extra security checks).

Description of the Modified Pohlig-Hellman Algorithm

Let us find the expansion of the number p into prime factors (the factorization can be easily done with Cryptool): p = 0x1b0b5c6e6b0b5b7e6c6d0b1b0a8c3c7e = 35948145881546650497425055363061529726 = 2 * 101 * 521 * 195197 * 7138079603 * 245150552958961933 Let us find the Euler function for the number p: φ(p) = (2-1) * (101-1) * (521-1) * (195197-1) * (7138079603-1) * (245150552958961933-1) = 17761863220777184249809368812124288000 Let us find the expansion of the Euler function into prime factors: φ(p) = 2^10 * 3^2 * 5^3 * 13 * 19 * 79 * 167 * 383 * 48799 * 45177719 * 5603527793 In order to find the browser’s private key Ka, it is necessary to find a discrete logarithm: A = gKa mod p

A = 0x5eff90f1c48342f5d519cd02b5dfd8b = 7892150445281019518426774740123123083

g = 0x40a262b1360a6a16612ca8251161a9a5 = 14017453774474660607531272629759062185 (mod p) As immediately finding Ka modulo φ(p) is quite time-consuming, let us find Ka by turns for each of the coprime factors Q i of the Euler function φ(p) [1024, 9, 125, 13, 19, 79, 167, 383, 48799, 45177719, 5603527793], and, by using the obtained results and the Chinese remainder theorem, let us immediately find Ka modulo φ(p). In order to find Ka modulo Q i , it is necessary to find a discrete logarithm To do that, we shall 5.1. take the number H=⌊√(Q i )⌋+1;

5.2. calculate D c =D a H mod p;

5.3. make a sorted table of values D c u mod p for 1 ≤ u ≤ H;

5.4. find such a value of 0 ≤ v ≤H, that the element D b ∙ D a v mod p is in the table;

5.5. The value of Ka modulo Q i equals Hu-v. The implementation of the described algorithm in Java is given in the Appendix A. As in the reviewed example the maximum value of Q i is only several billions, the program execution time did not exceed several seconds. For some of the Q i factors of the Euler function φ(p), there are several possible Ka values (there are possible Ka modulo Q i values in the row number i): [834, 898, 962, 2, 842, 906, 970, 10, 850, 914, 978, 18, 858, 922, 986, 26, 866, 930, 994, 34, 874, 938, 1002, 42, 882, 946, 1010, 50, 890, 954, 1018, 58, 826] [4] [18, 68, 118, 43, 93] [9] [12] [42] [6] [21] [11929] [24277014] [2536644002] 1 2 3 4 5 6 7 8 9 10 11 [ 834 , 898 , 962 , 2 , 842 , 906 , 970 , 10 , 850 , 914 , 978 , 18 , 858 , 922 , 986 , 26 , 866 , 930 , 994 , 34 , 874 , 938 , 1002 , 42 , 882 , 946 , 1010 , 50 , 890 , 954 , 1018 , 58 , 826 ] [ 4 ] [ 18 , 68 , 118 , 43 , 93 ] [ 9 ] [ 12 ] [ 42 ] [ 6 ] [ 21 ] [ 11929 ] [ 24277014 ] [ 2536644002 ] By going over all of the possible combinations of obtained Ka values by using the Chinese remainder theorem, we find several tens of possible Ka modulo φ(p) values: 0x8ae47b27ebdbcbe1b78c4a67de5b78a

0x5ef6ad7b83c6e7e0442ac5f5dc7f9a

0x1ed2c9a202ac327647ba12cf06ac3a

…

0x1dfce04948a67285c2ecef8dedf73da

0x3509c62b730c0bb7d9a56fefe2cf342

0xb5518dde7541768bd286d63d8e75f42

0x60776871627621379c91be922e40fd2

0x9e44a7fc4adbdd59bbce55db999dfda

0x98ec54ff8019a390e6c4f1985d21b5a All of the obtained values of the private key Ka lead to the same value of the Diffie-Hellman shared secret Kdh = BKa mod p: 0x0eb034f99e33e17df058de5b448b7241 By knowing Kdh, it is possible to decrypt the encryption key Kx from k and the shellcode by using Kx. The PHP script for decrypting the intercepted shellcode by using the known Diffie-Hellman shared secret is given in the Appendix B. The decrypted shellcode is given in the Appendix C.

Testing of the Diffie-Hellman Protocol Implementation Attack in the Angler Exploit Kit

To test the effectiveness and functionality of the attack, several tests were conducted.