A week ago, somebody sent me an e-mail about a memory leak in Hiawatha. I was a bit surprised that someone experienced a memory leak in Hiawatha, because I'm always very keen on memory allocation and use Valgrind during development to make sure I made no mistakes. I haven't experienced memory leaks on my own server and I use most of Hiawatha's features. But later I realized that I often restart Hiawatha, because I do a lot of testing and experimenting. So, if there was a memory leak in Hiawatha, it wouldn't get the chance to grow big on my server. However, on some other servers that I maintain, I saw that Hiawatha was indeed using a lot more memory than wat seemed to be normal. This showed that there is indeed something wrong with Hiawatha's memory usage.

For several days, I've been digging through my code and did severe testing with Valgrind. I even wrote a memory debugger which maps all malloc()-related systems calls to own functions to keep track of memory allocation and freeing. This proved that all memory allocation and freeing in Hiawatha is done correctly. Hiawatha itself is not leaking memory! That left me with the conclusion that one or more libraries that Hiawatha uses is leaking memory, due to a bug in that library or due to incorrect usage by Hiawatha. After a lot of reading and testing, I was almost sure that Hiawatha is using al the libraries correctly. I had done all I could, but still had no idea of the cause of the memory leak.

I asked a good friend of mine who helped me before with Hiawatha (pthreads and CMake) to take a look at this issue. He told me to give the Xcode tool Instruments a try. (Everybody doing programming and owning a Mac, give that tool a try! Really!) We used Instruments to keep track of every memory allocation done by Hiawatha or one the the libraries it uses. It showed us that Hiawatha is not leaking memory. However, at the same time we used the command line tool 'ps' to see how much memory Hiawatha uses. It told us that the total amount of Hiawatha's memory usage was growing with every request. Hiawatha started with 3 MB and after about hundred requests it used about 10 MB. But with every test we did, the amount of memory usage at the end was different. So Hiawatha's memory usage was growing while at the same time it was not leaking memory. We were both flabbergasted.

After doing a lot of searching on the Internet, we both found something that most likely explains what's going on here. It seems that when an application deallocates memory via free(), the free'd memory is not returned to the OS, but keeps being assigned to the appliation. Uh... wot? If this is true, and it seems it is, than this means that as a developer you have no control over the amount of memory usage of your application. We found this hard to believe, so we searched some more for how to actually release free'd memory, to give free'd memory back to the operating system. We couldn't find anything. I already tried memory_trim(), but that doesn't make any difference.

Anybody out there who has experience with this? Is this something that can be solved or should it be ignored? Any advice or tips are welcome!

Update: there is a discussion at Reddit about this article.