Two days ago VMware published new update for Workstation version 10 (release notes). According to the changelog it should fix some security issues reported in OpenSSL. That’s nice, however there is a small problem with this update. VMware.exe (the GUI part of VMmare) started crashing immediately after update. This was quite a learning experience, do not update critical software if you have something important to do, as the new version can be worse than the one you are using. Since I didn’t have the previous installer at hand, I had to somehow resolve this issue differently (yeah sure, I just wanted to debug it and see, why it is crashing).

The crash on my machine is 100% repeatable, and it crashes on security cookie check inside libcurl.dll. As most of you know libcurl is an open source library, so the bug should be hidden somewhere inside libcurl source code, or in the additional/changed code that guys from VMware possibly touched. In the latter case, there should be available modified source code of the libcurl on the VMware page. It is not the case this time, so (in theory) it is fault of the libcurl itself. Crash happens inside the function that seems to enumerate DNS servers. Libcurl can be compiled with many different options and can use many different third party libraries. VMware’s libcurl version is compiled with static version of zlib and c-ares, it is also dynamically linked with OpenSSL. It is worth to note which versions of mentioned libraries are used. Most of open source libraries includes version number inside compiled binaries so it is rather easy to gather those information:

VMWare 10.0.5 VMWare 10.0.6 Current version libcurl 7.19.5 7.24.0 7.42.1 zlib 1.2.3 1.2.3 1.2.8 c-ares 1.5.1 1.7.5 1.10.0

Particulary interesting for me was libcurl and c-ares, as zlib probably doesn’t enumerate DNSes :) I’ve tried to localize the crashing function inside the source code of mentioned libs, but I was missing one detail (obviously!). In the newest version of c-ares, the function similar to the one that is crashing VMware is named get_DNS_NetworkParams(). This function doesn’t crash and it looks fine. After a few minutes of looking at the code and wondering what’s wrong I realized that I’ve freshly cloned repository with the latest changes and I should probably go back in time and check some older version. Fortunately c-ares repository is properly tagged with release version numbers (at least since version 1.7.0). After checking out proper revision I’ve easily identified crashing function: get_iphlpapi_dns_info(). Simplified code, just to show the problem:

#define INET_ADDRSTRLEN 22 int get_iphlpapi_dns_info ( char * ret_buf, size_t ret_size ) { const size_t ipv4_size = INET_ADDRSTRLEN + 1 ; /* +1 for ',' at end */ size_t left = ret_size ; char * ret = ret_buf ; //... for ( /*...*/ ) { //... if ( left > ipv4_size ) { some_function_that_fills_given_buffer ( ret, ipv4_size - 1 ) ; size_t stringlen = strlen ( ret ) ; ret [ stringlen ] = ',' ; ret [ stringlen + 1 ] = ' \0 ' ; ret + = stringlen + 1 ; left - = ret - ret_buf ; // <- !!! BUG BUG !!! } //... } //... } //... // function call char buf [ 512 ] ; get_iphlpapi_dns_info ( buf, sizeof ( buf ) ) ; //... #define INET_ADDRSTRLEN 22 int get_iphlpapi_dns_info(char *ret_buf, size_t ret_size) { const size_t ipv4_size = INET_ADDRSTRLEN + 1; /* +1 for ',' at end */ size_t left = ret_size; char *ret = ret_buf; //... for (/*...*/) { //... if (left > ipv4_size) { some_function_that_fills_given_buffer(ret, ipv4_size - 1); size_t stringlen = strlen(ret); ret[stringlen] = ','; ret[stringlen + 1] = '\0'; ret += stringlen + 1; left -= ret - ret_buf; // <- !!! BUG BUG !!! } //... } //... } //... // function call char buf[512]; get_iphlpapi_dns_info(buf,sizeof(buf)); //...

Variable left should be updated on each iteration with the length of the written data, so it will always contain number of bytes left in the ret_buf. As you may already noticed, ret_buf doesn’t change throughout the function, so left is updated with the length of all strings written since the first iteration. It goes below 0 very fast and since it is declared as size_t which is unsigned it will be interpreted as such in the if (left > ipv4_size) comparison and at the end it will overrun ret_buf (only if there is enough records to process by the for loop).

Described bug was present in c-ares library since the very begining (2004-06-10). There were some changes regarding IPv6 (2011-05-17) that were modifying this function, but bug persisted probably due to copy&paste from IPv4 handler. Someone fixed this bug almost a year later (2012-02-25) and the other person rewrote whole code few months later. Unfortunately VMware chose to use version 1.7.5 which was released before this bug was fixed (2011-08-16). Blast from the past.

If you are experiencing crashes in VMware Workstation 10.0.6 just use libcurl.dll from the previous build, at least it will stop crashing.

UPDATE: Someone at VMware forum pointed out that it is sufficient to get libcurl.dll from the \OVFTool\ directory and it will solve the issue. Indeed it is true, as libcurl from mentioned directory uses c-ares version 1.9.1 (libcurl itself is also newer as it has version 7.30.0).

UPDATE 2 (2015-07-10): This issue was fixed with v10.0.7, libcurl was updated to v7.32.0. I’ve also received free copy of VMware Workstation v11. Thanks!: