We rely heavily on running all of our unit tests and UI tests on each pull request to ensure that they are constantly green and no tests are broken.

While running unit tests from a terminal is quite simple, if you use tools like robolectric, different architecture patterns and run them solely on the JVM, it’s quite a complex problem to run your espresso or other tests (like calabash or appium) that require a device to run from a command line interface.

Running tests on real devices

One solution would be to run these tests on real devices that are connected to the CI instance directly, but depending on your CI solution (we use Jenkins) this might be a problem. Also, this will not scale well if you have multiple people creating or updating pull requests at the same time. Another problem is, that you probably would want to run on real devices that most of your users use. For XING that would be:

Most used devices for the XING Android App (July 2018)

As you can tell from the chart above you want to connect Samsung devices to your CI server and run the tests on there. Good luck with that. It will be tough just to unlock all of these devices before running the tests. We had a similar set up in the past using a nice tool called OpenSTF that we integrated with our Jenkins but it was causing nothing but pure, sweet pain, when using it there. Another problem of the many we had was that we need to maintain potential adb failures that would occur regularly. In addition to that we started to see problems with power supply on our device farm as soon as we started adding more devices to it. Suddenly, devices would switch off or discharge at random.

We saw the failures, weren’t happy about them and decided we need something else because this approach just wouldn’t work and introduced way too much flakiness, where tests would randomly fail or not even start because the phone crashed.