In the previous part (Running Android UI Tests — Part 1) we explained how we run Android UI tests and could easily clear data, clear notifications, pass params by adding simple Java annotations to our test methods. In addition, we collect logs, dump local db and preferences xml file, record and extract mp4 video and more. Read part 1 and check our open source project for step by step tutorials and examples.

In this part we will show how to run many tests in parallel on multiple devices. We will also show how to run group of sequencing tests on the same device when running in parallel.

We improved execution UI tests time by 40% 💫

Parallel — The regular approach vs. push approach

Before diving into optimizations and how we improve parallel execution test times, we explain the current sharding option that is provided by the Android testing framework.

The regular approach - Sharding

The Android Instrumentation runner supports test sharding using the numShards and shardIndex arguments (documentation).

Run:

adb -s {device} shell am instrument -w -e numShards 4 -e shardIndex 1

numShards = number of groups we ask Android to divide all our tests.

shardIndex = the group index we want to run on specific {device}.

This means, that if we want to run in parallel on 4 devices, we need to execute the same shell command with shardIndex starting from 1 to 4.

The main issues of this approach are:

Execution time — Android will split the tests to the groups without taking into consideration the execution time of each group. It might happen that the group of test in shard #1 will run for 1 hour, while the other shard will finish in 20 minutes. Grouping following tests — Android will split the tests to groups randomly. So if you decided to split a very big scenario to sequential tests in the same class, sharding might break your logic by putting the tests in different shards. And you have no control over this.

The push approach

The idea is simple. Instead of splitting all tests into equal groups, we push the next test to a free device. If all devices are running, we wait until one of the devices completes the test execution.

For example: We took 16 UI tests and ran them by using 2 different approaches. You can see an improvement of 40% when pushing tests to free devices, instead of allocating an equal number of tests per device.