The Cucumber Parallel Testing project provides a solution to allow Cucumber based tests to run in parallel in order to speed up test execution time. This is particularly important for integration tests, which are typically slower than unit tests and can often account for a significant portion of build time.

The project is hosted on the Maven Central Repository and can be included using the following Maven dependency.

<dependency>
  <groupId>uk.co.hmtt</groupId>
  <artifactId>cucumber-parallel-testing-core</artifactId>
  <version>1.0.0</version>
</dependency>

The source code is hosted on GitHub and can be found HERE. GitHub contains more information regarding using and configuring the solution.

Why use this solution for parallel testing?

The Cucumber Parallel Testing project has the capability to run feature files in parallel. It can lead to a significant reduction in build time where scenarios are spread relatively evenly across multiple feature files. It is not an appropriate solution where all scenarios are located in a single feature file.

The solution takes advantage of the failsafe plugins multi thread capability, which creates multiple threads under which to run JUnit tests.

The primary aim of the Cucumber Parallel Testing project is to manage these threads and ensure that each feature file is only run once, by the first thread that accesses it. Subsequent threads skip over the processed feature file and attempt to access the next. In this manner each thread processes a sub-set of all features, and because they run in parallel the execution time is significantly reduced. 

There are a number of other approaches to parallel testing, but they typically rely on either splitting feature files into multiple directories, or tagging tests. There are two issues with this approach -

  • It can be difficult to ensure an even split of tests at the directory or tag level. It relies heavily on the developer determining what a fair split of resources is. 
  • While an optimal split of resources may be achieved at project initiation, it is often difficult to ensure that this is continued for the lifetime of the project as the test suite expands and new developers are introduced who may not be familiar with the approach.

The Cucumber Parallel Testing solution does not suffer these disadvantages as it simply processes feature files in turn, and does not require a particular structure or developer intervention in order to achieve an optimal split. 

Acceptance Test Use Case 

The acceptance tests used within the CI build for the project illustrate the time saving advantage of the project.

The acceptance tests consist of five tests which are each hard-coded to include a five second sleep. The build time of the acceptance test project (cucumber-parallel-testing-acceptance) should therefore be at least 25 seconds. However build time also includes a number of other steps, including compilation, so the total time expected should exceed 25 seconds.

The build produces a Cucumber report which illustrates the time taken to run all steps -

As you can see in the bottom right of the report, the total time taken is 25 seconds and 323 ms for the execution of the Cucumber scenarios alone (this excludes compilation etc).

However, the Maven reactor summary tells a different story.

The Maven reactor summary reports that the time taken to run the cucumber-parallel-testing-acceptance project is 14.759 seconds. Over 10 seconds quicker than the Cucumber report suggests.

The reason for this is that the Cucumber report simply sums the time taken for each feature and doesn't take into account that features may have been run in parallel. So 25.323 seconds is the total time taken for all of the features to execute, and would have been the elapsed time if they had been run sequentially.

However, the scenarios were run in parallel using three threads. There were five features files, so two of the threads processed two features, and one thread processed one feature. Therefore the slowest running threads were the two that ran two features; taking at least ten seconds. Ten seconds is approximately the time it took to run all scenarios, and not the 25.323 reported in the Cucumber report. The additional time is made up of non related steps, such as downloading artefacts, compilation etc.

Real World Use Case 

Hmtt Ltd implemented a similar solution for one of our clients. The client had a Selenium web UI test suite that took approximately 25 minutes to run, and was executed as part of an over-night scheduled test script.

Hmtt Ltd introduced a Selenium Grid on a Docker cluster and re-configured the tests to run in parallel. We chose to run three parallel threads and configured three Selenium nodes (defaulting to Chrome, but configurable). The result of this change was to reduce build time to approximately 9 minutes. This meant that they could include the test suite within the CI build which gave them immediate visibility of any issues.

Add comment


Security code
Refresh