In Ukraine we have a proverb stating “Trust but verify”, and that is one of the reasons why I tend to be sceptical, when someone mentions performance related statement that I cannot easily verify.

Unfortunately most of people tend trust such statements, when they are done by a respected people in community, because they don’t expect any unverified information being distributed by them.

The Famous Benchmark Story

As all of us know there was a huge tsunami in Magento community this February after MageCore Inc did release a performance benchmark stating that Magento 2.0 is requiring 7x time more resources than Magento 1.x in order to properly operate in production.

Unfortunately, they didn’t release Gatling simulations at the same time as their benchmark results, and it made it incredibly hard to check if their conclusions are legitimate. Also what felt incredibly strange for me, is that no-one else in the community did ask them to provide it with their setup settings in order to make a proper benchmark verification.

It took me one and a half month to get the full reproducible environment, that they were using, published by them. Then it took me another three months to finalise proper scenarios, as it is hard to dedicate a full week for fixing a benchmark, when you are working as independent consultant.

I must admit, that I appreciate the amount of efforts MageCore Inc team has put into this load test. It is a huge amount of work, but there is something in it, that leads to wrong conclusions.

So, which findings did I come with after these 3 months? It is quite an interesting read, so grab a bucket of popcorn and enjoy it!