A few months ago we 've been celebrating a big thing in Evolution Gaming — the release of a new game. Hooray! Unique, first of its kind 3D roulette game built with React and WebGL went live. We 've been looking for recognition and success…

Development of the game was lasting around a year. This was the first game delivered by 3d development department in the company. Finally, we start using 3d graphics instead of video streamed games. Finally, ̶a̶ ̶r̶o̶b̶o̶t̶ WebGL is going to replace a croupier. Finally hi-tech is going to be celebrated in such a conservative segment as B2B solutions for big casinos in the Vegas and Macao!

Two days after the release we got some complaints from users. It was very little of them — less than 1% percent of the overall number of players but they were devastating.

“You are cheating on us! I bet on red, the ball is on red and the game says I lost!”

Few teams started to investigate the bug which was a disaster for the company image of a top solutions supplier. The appearance of a bug (as it normally happens with bugs) was quite a surprise. First of all, backend passes a very strict certification in a special bureau for every jurisdiction we run a game. Then frontend is being tested in four levels: by developers, by automated testing, manual testing and manual testing in a separate “products quality” department.

Evolution Gaming QA pipeline

How this multi-layered machine missed through that serious bug as lost of sync between a real game result and a visual representation?

And yeah it was less than 1% of players… In our team, it was basically not reproducible. We required a few stressful hours before got the first catch: ”Bug is reproducible with older Android phones.” Next hour — next news: ”Bug is reproducible better under the heavy CPU load.”. After that report we started to throttle CPU to the maximum so got the bug reproducible 30% of runs. Which is better than 0% but still fixing a bug in that way is crazy hard. Every time you need to prove or break your hypothesis you need 3 runs to reproduce and still you can never be sure…

The bug was caused by dropped frames and the approach we played roulette ball animation (keyframe animations). Costs of having the bug were huge for the company and team was exhausted fixing this. Since recalling the game from clients was impossible fixing last from the midday until 4 am.

How to test against dropped frames?

Your application uses animations or any kind of logic being executed on requestAnimationFrame? Especially for you, I did a little Chrome extension called FPS Control which allows to drop frames (25% / 50% / 90%) and limit FPS (2 / 10 / 30). How does it work? Basically, monkey patches the requestAnimationFrame so it works as expected and if we have this extension earlier…

Conclusion

Learn from others, stress your apps and games with dropped frames and low FPS to prevent critical bugs. And remember… It is or you stress app with testing or app stresses you with bugs.