Case Study: Poorly written Cucumber tests.

Today I had the pleasure of debugging a Selenium test written with Cucumber. The test would fail every 10 or so runs, and because of that it was disabled for several months. When I tried to fix it, it became pretty obvious why it was not a happy test, and why it got disabled in the first place.

Let’s do a quick run through of things NOT to do with cucumber.

1) If your feature file is 240 lines but only describes 8 features, you are not using cucumber as intended. Your feature file is probably filled with a lot of steps that look like this “And I fill in ‘Hello’ into ‘#this_is_some_id’”

The idea behind Cucumber is to describe the features of your application, and not the implementation of them. Leave the nitty-gritty implementation of the test for a step definition file. The rule of the thumb is, if you cannot understand what the test is supposed to check for in 30 seconds or less, your test needs a re-write.

2) Don’t Repeat Yourself (or DRY) principle applies to cucumber features as in any other programming situation. You can call any step definition from within any other step definition in Cucumber. Make small, one or two line step definitions and compile them into larger ones to complete the task at hand.

3) Feature descriptions and scenario names have to be very clear and educational. This is especially true if your feature is 84 lines long. Just like any comment for a complicated function, make the scenario name very descriptive. When a person has given up trying to understand the steps that you have written down, they will always jump up to scenario name to understand what is going on.

As for the Feature description, think of this area as an open text field. You do not have to limit yourself to one line description. Write several sentences; describe what is going on in the feature file. It’s far too common to see only 1-word feature descriptions.

4) Tags are a great way to group your tests. However, some teams use them for more purposes than just grouping. For example, adding @javascript or @selenium tag to a feature will force the feature to run in a web browser via selenium instead of a much faster headless browser.

Furthermore, some tags are used to start up a service that the feature is using. Some of these services take a long time to start up, and add a significant overhead to overall test run. Lesson being, do not just copy paste the tags from other features, or put a tag at the top of the feature file. Instead, start writing a feature without any tags, and slowly add the tags that apply.

5) Many times I have seen a feature file that is 300+ lines. My personal record has been a feature file with 1,200 lines. This is actually really bad for test performance and speed. For example, test parallelization gem “parallel_tests” reads in all of them feature files that you have, and segments them into groups. Often this means that one of the segments will contain the 800 line feature file, which means that after all the other segments have finished running we are still waiting for the big feature file to complete.

Try to keep your feature file down to 10 or less features, and find better way to group the feature files into smaller groups.