Every developer is familiar with the if(browsertype) statement. Different browsers render applications differently, so web applications need to be able to detect on which browser they are running and adjust their app code accordingly. Successfully testing all browsers and all versions are no small feat which is exactly why Sauce Labs built their solution on Selenium. Sauce Labs enables QA teams to execute Selenium based automation suites on multiple permutations, operating systems, and versions, for multiple browsers and browser versions.

At first glance, this seems like we’re done and this is the perfect solution to achieve complete application matrix coverage. Unfortunately, nothing is that simple, and upon digging deeper, it is apparent that not all environments are available for certification. You will have some critical use case gaps, there’s no way around it. So what are the best practices to improve Selenium on Sauce Labs?

Mobile: Emulators vs. Real Devices

Appium, which is basically Selenium for mobile, is the most common library that allows Selenium to run on mobile, including mobile browsers. However, when running Appium mobile browser tests on Sauce Labs, tests do not run on actual devices but only on device emulators. This means that not only might the device not behave in the same manner when integrated with the browser (such as integrating with other device applications or battery consumption), but also that the browser might behave differently inside the emulator than it would on a real mobile device. For example, in a web app with a contact form that includes a phone number, with the planned functionality that a click on the number will open a dialing app, emulator testing is problematic because redirecting from a browser to a different app is not supported. So when the test runs, but nothing happens, we will receive a false negative. Moreover, when we publish the app’s support matrix, we won’t be able to say for sure which devices are supported.

Now that we understand where the issue is, what’s the work around? There is no external tool or code fix that can be applied to solve this just good old fashion planning. The key is to differentiate tests covering UI responsiveness from functionality and user experience. The first set has no problem running on Sauce Labs the other two must be identified in advance and allocated real mobile devices for testing. There are several solutions for real device testing, such as Amazon Device Farm, and many on-premise solutions are available as well.

WebDriver Consistency

WebDriver is a constant challenge for Selenium developers. To run Selenium tests on a browser, a special driver for each and every browser you need to test must be located on the machine where the tests are executed. The problem is that browsers themselves release new versions almost on a monthly basis, and therefore the WebDriver must also be updated to the newest version or sometimes to a completely new WebKit driver, like GeckoDriver for Firefox 47 and above. Problems begin to arise when you want to run the same test on multiple browser versions.

To combat this Sauce Labs supplies a considerable matrix that covers most of the common browser types out there. But if your tests were designed on one browser version with a specific WebDriver version (most times the driver will be added to the project as a reference or a dependency), these tests might not run on most of the machines in the matrix Sauce Labs supplies. For example, the below pom file, with Firefox webdriver dependency, will run on Firefox version up to 47, but as we explained, the 47 and above version requires the new Marionette driver.