Writing reliable automated Web UI tests can be challenge. This article describes a few of the common problems and offers some solutions to help make your tests robust.

The example code provided is mostly for the LoadTestGo Scripting API, but should apply to any Web UI automation framework, such as Selenium or Watir.

As we see it, there are two common problems that cause false positives in automated Web UI tests:

Brittle element selectors (be that finding element by XPATH, CSS or custom selectors) Timing issues / Incorrect waits

We’ll go into into how you can find elements reliably, and how to avoid timing related issues.

Reliable element selectors make reliable tests. To automate a Web UI you generally need to find the HTML elements on the page to act on. There are many ways to choose the elements on the page, some examples:

Find by text

Find by id

Find by class

Find by tag

Find by element hierarchy

Find by some combination of the above

What makes a selector reliable? The answer largely depends on the website you are testing, and what’s likely to change in future. We give some general guidelines below.

You can use Chrome DevTools to inspect the page content (the page source is not as useful, many pages generate the DOM on the fly). This will show you the DOM layout, element ids, and element class names used on the page. You can right click on an element and select ‘inspect’ to show the element in DevTools Elements panel. Most other browsers have really good support for this built-in too.

// Click <button> element with text 'Sign In' b.click("button:contains(Sign In)");

Finding elements by their text contents can be very useful, especially for buttons/links. The text is what the user sees, so even if the developers change element ids / class names / tags / etc, the text will often remain the name. Also it has the advantage of being easy to read/understand as well, whereas element ids and class names can be quite cryptic when reading the script or when errors occur.

On the minus side, when a UI is localized to different languages, you may not want to use text as obviously it will be different per language.

The other pitfall is using text strings that are too common. There’s a danger another element with the same text will be added, potentially breaking your test. The remedy to this is identifying the container for the element, so that the scope for mistakenly referring to the wrong element is limited.

// Click element with id 'foo' b.click("#foo");

Elements ids are used to uniquely identify important elements on the page.

Developers give ids to elements so that the elements can easily be found from JavaScript code. This is perfect for us when automating the page. Developers/Designers can make many change to the page layout/presentation can be made without changing the element id. By using element ids, your tests are somewhat isolated from the page layout, and don’t have to be updated for small changes to the DOM.

Sometimes higher level components are uniquely identified by id, but you need to automate sub elements. In these cases it’s best to find the component first, then find the sub element using a class name / tag name.

While element ids are supposed to uniquely identify only one element per page this is not enforced, and pages ‘in-the-wild’ may have more than one element with the same id. Luckily this is pretty rare.

// Click element with class 'foo' b.click(".foo");

Class names are used for CSS styling, so they are typically used on many different elements. They can still be good at picking out elements you are interested in however. It all depends on how the page was built. Often a page will use class names for the duel purpose of CSS styling and to allow JavaScript to find the elements.

// Click <div> with ancestor element of id 'foo' b.click("#foo div");

Finding by hierarchy is typically the most susceptible to changes. You want to keep the hierarchy based queries to a minimum. They are most powerful when used in conjunction with more reliable selectors. If the element you want doesn’t have a good way to identify it, you can often find an easier to locate element that’s close to it in the hierarchy and then use css selectors to find the element you really want.

// Click <button> with class 'foo' b.click("button.foo");

Some tags can change often and can be arbitrarily nested for visual changes, e.g. div or span . When used in conjunction with other selectors they can be useful, but typically it’s not a good idea to rely on them.

On the other hand, tags that specify an interactive component on screen can be really handy, e.g. button , select , form , input , a , etc. These are more unique on the page, and often help clearly indicate intent in the test script.

// Type 'hoo' into input form element with name 'foo' b.type("input[name=foo]", 'hoo');

The ‘name’ attribute on input elements is particularly useful, typically this doesn’t change often as it is tied to backend code and is usually named something descriptive.

Some frameworks add their own custom attribute fields as well e.g. data-* . These can also be useful as selection criteria.

By default LoadTestGo scripts only consider elements that are visible. Some other test frameworks, typically ignore visibility information when finding elements. It’s common to have several buttons / links that match the same selectors, especially with drop down navigation / context menus or highly dynamic pages.

Every time to use a selector, make sure it is referring to the right item! You’d be surprised how often the wrong element gets selected, but the test appears to work anyway. Running in interactive mode also has the advantage of allowing you to tighten up your selectors, experimenting with different selectors and learning the different selector possibilities.

The LoadTestGo Script Editor has a console mode, that allows you to interactively type JavaScript commands and get feedback. You can test out different selectors using Browser.highlight(), a text box will show up in the browser, showing the boundary of the element that was selected. Browser.query() / Browser.queryVisible() will give a list of elements that match the given selector. Browser.query() will even give you a reason why an element is not visible.

The Script Editor can also auto generate selectors for you! In the browser, click the magnifying icon then the item you want to generate a selector for. A selector for this element will then be written to the console.

A major part of writing good tests is waiting on the right thing. This is tricky as often tests will work with incorrect waits by chance, and then fail later.

A lot of factors can change the waits between steps in your script:

More content being added to the page

Optimizations to the page

Tests running on a faster/slower machine

Different workloads on the test machine

Browser updates

Test framework updates

If any one of these change and you are not correctly waiting in your script, then you may start seeing failures. Failures due to waits can be hard to track down, it can be hard to tell if there is a real regression on the site you are testing or if it’s just a problem with the script. Tackling these problems can be time consuming and generally cause a fuss, making folks lose confidence in the validity of testing via automated UI tests. By improving the script waits you can go a long way to avoiding this from happening.

With load test scripts, often we run a small sanity load test first exercising the scripts over and over to at least make sure the scripts are reasonably reliable, before running a full test. You might want to do something similar.

Waits/sleeps for a fixed time period should be a last resort, there’s usually a better way to wait on the thing you need. With fixed waits the intention is not as clear, what exactly are you waiting on?

That said sometimes you need to add waits to simulate ‘think time’, or make a test more like what a real user would do. Generally I would try to get the test working without the fixed waits first, then after everything is working add the waits.

The option to turn on/off the waits when debugging can be useful too. At the very least the debugging cycle is faster.

This is probably the most important bit.

In LoadTestGo you can wait on an element to appear, before clicking on it:

b.waitForVisible("#foo");

This will actually wait until the element is really visible (i.e. renders somewhere on the page).

With Selenium/WebDriver the equivalent would be:

WebDriverWait wait = new WebDriverWait(driver, 30); wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("foo")))

Ok, this is really just the previous point, but it’s so important, it’s worth stating twice! Even if your test works without an explicit wait when you are writing it, in the wild or later down the round it may fail. It’s safest to put in a wait for the element to be visible before interacting with it.

So even with the above, there is still a problem. The page can make the element visible, then hide/remove it briefly, then make it visible/re-add it again. If your interaction with the element happens while the element is hidden, the test may fail. You can get around this by putting the whole block in a loop and catching any element hidden exceptions.

For example, using LoadTestGo scripting this would be:

var notClicked = true; while (notClicked) { try { b.waitForVisible("#foo"); b.click("#foo"); notClicked = false; } catch (e) { pizza.sleep(100); } }

In Selenium/WebDriver this would be:

boolean notClicked = true; while (notClicked) { try { WebElement element = driver.findElement(By.id("foo")); if (element.isDisplayed()) { element.click(); notClicked = false; } } catch (StaleElementReferenceException|NoSuchElementException e) { // DOM changed: element was removed Thread.sleep(100); } }

NOTE: When you find an element in Selenium/WebDriver and then try to use it after it has been removed/made invisible you will get the infamous org.openqa.selenium.StaleElementReferenceException: Element is no longer attached to the DOM exception. The above code catches this and tries again.

I don’t recommend doing this every time you write a test, as the resulting code can be unwieldy, but if you do get random failures due to this, then it may be worth employing the above technique as a rule.

I’m hoping to make this easier to do in LoadTestGo scripting in future, maybe by having any functions that interact with the DOM have an automatic retry.

It’s good programming practice in general, to only add fixes for things you have confirmed you run into. Code that you haven’t confirmed fixes a problem, will likely just mislead you and likely doesn’t fix the problem in the first place. Just the nature of the beast :)

You will see in a lot of Selenium tests sprinkle waits and timeout adjustments:

selenium.setTimeout(SomeNumberNeverConfirmedToFixAnything); ... selenium.waitForPageToLoad(); ...

While these can be useful and necessary, often they are more like a placebo. Take the time to find to the thing you really should be waiting on, and wait on that!

I hope that this gives you some ideas. Let us know in the comments what you think makes for good reliable tests!