One of the most common things I hear both inside and outside of Microsoft is that people have a difficult time reproducing IronPython performance statistics. For example, in our performance lab suppose we measure benchmark XYZ as being N% faster or slower under IronPython than CPython, and other IronPython users might observe our measurement of XYZ as being off by an order of magnitude. There are tons of reasons user A might not see the same performance out of IronPython as user B (e.g., different hardware, same hardware with a different OS, different background processes consuming hardware resources, etc). In this post forget about differences in hardware, OSes, background processes and let's just focus on the CLR version.



Will we get consistent results comparing IronPython performance to CPython under different .NET releases? The answer to this is pretty obvious - no. While our team compiles IronPython to run under .NET 2.0, there were performance improvements that went into later versions of .NET which will directly impact IronPython performance. That is, simply by upgrading to a newer version of .NET you'll see various performance improvements (and potentially degrades) in IronPython without making any other changes to your system.



What if instead of comparing performance on different major .NET versions, we instead looked at performance under .NET X.Y both with and without security patches installed. Would IronPython's performance be the same? Somewhat counter-intuitively, the answer is also no. As a concrete example, the following measurements were taken from a single 32-bit Vista machine under .NET 3.5 and .NET 3.5 SP1:

float comparisons were 37% faster under .NET 3.5 SP1

nested for loops were 30% faster under .NET 3.5 SP1

method invocations slowed down 26% under .NET 3.5 SP1