Executing Parallel Test Sessions with TestNG and Selenium WebDriver

In today’s fast-paced world, the primary goal of every business is to release their websites or mobile applications to the end users as early as possible. As a result, businesses constantly search for ways to test, measure, and improve their products. With the increase in competition, faster time to market has become an important aspect for any business to survive in today’s competitive market. However, one of the possible challenges many businesses face is the release cycle time, which usually gets extended for various reasons.

Different techniques and approaches are tried to check the business requirement, development, quality, and overall progress of the delivery required to overcome the challenges. All of this ensures releasing the software smoothly to the market with best quality.

Following are some of the reasons which I have faced in my experience while working with many projects:

• Coding and business requirement issues.

• Automation pipeline taking longer than expected to run, which eventually slows down the delivery time to production.

• Build generation and deployment takes a lot of time, which causes a delay in running automated regression tests and eventually performing manual exploratory testing at the end.

• Flakiness of the automated tests.

• Prioritization of the features at the last moment when the release is near which pushes the engineering team ultimately risking the release.

And likewise, there might be much more to list. However, I have listed a few things in common with many teams.

From the reasons listed above, let’s talk about why the Automation pipeline takes longer than expected to run, which eventually slows down the delivery time to production, and also take a look at the possible solution of how the pipeline can be optimized and time to production could be minimized.

Consider a scenario of an e-commerce website which is built to run on Chrome, Firefox, Microsoft Edge and Safari browsers. You need to run the automation regression test suites on all the browsers before you release it to Production. Let’s say, each test suite has around 50+ test cases to run. So considering we have 4 browsers, Chrome, Firefox,

Microsoft Edge and Safari we need to run 200 tests, after that only we would be able to provide feedback for the builds.

Consider a single test taking around 10 seconds to run, so considering 50 test cases, it would take 500 seconds to run a single suite while performing testing on Chrome browser alone and again 500 seconds for Firefox and then for Edge and so on. So, considering all the 4 browsers it would roughly take around 2000 seconds to run all the tests.

This is where parallel testing comes to the rescue!

In further sections of this Parallel testing tutorial, we will discuss parallel testing and show how to perform Parallel testing using TestNG and Selenium WebDriver.

So get ready to take this journey of running automation tests in parallel with Selenium WebDriver and TestNG.

Table of Contents

• What is Parallel Testing?

• What are the advantages of Parallel Testing?

• How to perform Parallel Execution of Tests on the pCloudy platform using Selenium and TestNG?

What is Parallel Testing?

Parallel testing is the way to execute multiple automation tests at the same time, where tests are run on different threads and consume different resources. We can run tests on different platforms and browsers. So, in the example of the e-commerce application above, it took 500 seconds to run tests on Chrome browser and again 500 seconds to run on Firefox browser.

So, in total we would need 1000 seconds to run the tests on both Chrome as well as Firefox browsers. We can save 50% of our time by running the tests parallel.

Let’s consider that there are multiple versions for Chrome and Firefox browsers where the website needs to be tested upon, the time will be multiplied by the number of versions we add to run the tests on plus if we consider different OS’s and their respective multiple versions. By running the tests in parallel, we can save 50% of our time as all the tests would run at the same time providing us with faster feedback.

Let’s discuss in detail about how to run the tests in parallel with Selenium WebDriver using cloud hosted mobile and web browser platform pCloudy and how it can help you save time involved in running the tests.

If you look closely at Figure 1 above, it shows the orchestration of the tests of how they are designed to get executed on Browser 1 and Browser 2 respectively, when they are run serially.

Consider “Browser 1” as Chrome browser with the latest version and “Browser 2” as Firefox browser with latest version or maybe another Chrome browser with a different version, it all depends on where you want to run your tests. Once the tests get executed, “Browser 1” will be set up and Test 1 to Test 4 will get executed on it.

Once the execution finishes on “Browser 1”, then only execution of another set of same tests will begin on “Browser 2” . Here, the time taken to run tests on “Browser 2” may be more depending upon the waits, locator strategy used,etc. And also the browser and platform, by platform, I mean the OS and its respective version, used for running the tests.

Now, let’s talk about running the browser tests in parallel. In Figure 2, all the tests are running parallel on multiple browsers. Here, as soon as you execute the tests, all the 4 browsers in the respective OS and OS Versions are set up at the same time.

These 4 browsers may be Chrome, or combination of Chrome, Firefox, Edge, Safari, etc., depending on the test strategy chosen to run the tests, and could be run on OS like Windows or may belong to different OS’s like one browser running on “Windows” and another one on “Mac”, the goal is to execute all the tests at once, so we save time and get quick feedback on the tests.

Advantages of running the tests in Parallel

I think by now, you have a fair idea about what parallel testing means and why we use it. Now, let’s discuss some advantages of Parallel testing.

The following are some of the advantages of running tests in Parallel:

1. Reduces the overall execution time of the tests

2. Provides leverage to run tests on multiple browsers with different browser versions, different OS’s and OS’s versions.

3. Accelerated feedback on the builds/deployment.

4. Fast feedback helps the team to fix the issue early and remove the bottlenecks.

5. Achieve faster and frequent release while keeping up pace with the Continuous Integration and Continuous Development.

How to perform Parallel Execution of Tests using Selenium WebDriver on pCloudy with TestNG?

The following programming language and tools have been used in writing and running the tests:

• Programming Language: Java 11

• Web Automation Framework: Selenium WebDriver (Version 4.5.3)

• Test Runner: TestNG

• Build Tool: Maven

• Cloud Platform for test execution: pCloudy.com

Parallel Execution Strategy:

As mentioned in the initial part of this blog, we would be running all the tests in parallel as follows:

We would be creating a json file and reading all the above mentioned configuration params from that file. This would help us in adding/removing the configurations easily without having to modify the code every time. The only change that would be required would be to the json file.

Testing Strategy

Let’s discuss the testing strategy for the pCloudy website on what and how we automate it?

1. We would be navigating to https://pcloudy.com

2. Next, get text displayed in the center of the screen and assert it. Following is the text that would be checking:

Continuous Testing Cloud

Deliver Flawless Digital Experience

through our Device and Browser Cloud

Here is the screenshot of the page we would be checking the text from:

1. Next, we would hover the mouse over the “Resources” menu and after it opens the menu lists, we would find the “Blogs” menu and click on it.

2. Once the “Blogs” page opens we would be checking the page title -> “pCloudy Blogs” on the Blogs page.

Here is the screenshot of the page we would be checking the text from:

Getting Started

As discussed earlier, this project has been created using Maven. TestNG is used as a test runner. Once the project is created, we need to add the dependency for Selenium WebDriver, TestNG and jackson-databind in the `pom.xml` file.

pom.xml

Versions of the dependencies are set in a separate properties block. This is done for maintainability, so if we need to update the versions, we can do it easily without having to search the dependency throughout the pom.xml file.

For reading JSON configuration file, we are using the JsonNode class from the jacksondatabind dependency. It has some rich features and allows us to read the JSON file seamlessly with less code to write. Appropriate approach and logic needs to be applied to get the respective values.

Selenium Java and TestNG dependencies are added which will actually help us writing the web automation tests and running them respectively.

What is TestNG?

TestNG is a test automation framework which helps in running the tests, “NG” stands for “Next Generation”. It is designed to make end to end testing easy for the testers.

How does TestNG help in running tests in parallel?

TestNG allows running the tests in parallel by setting the “parallel” attribute in the “<suite>” tag respectively to “tests”, “methods”, “classes”, “instances”.

As per the setting provided to run the tests, it will start executing the tests in separate threads.

TestNG provides the following options to run the tests in parallel as per their documentation:

parallel=”tests”: Providing this value for the parallel attribute in the <suite> tag will run all the methods in the same <test> tag in the same thread, but each <test> tag will be in a separate thread.

parallel=”methods”: Providing this value for the parallel attribute in the <suite> tag will run all your test methods in separate threads. Dependent methods will also run in separate threads but they will respect the order that you specified.

parallel=”classes”: Providing this value for the parallel attribute in the <suite> tag will run all the methods in the same class in the same thread, but each class will be run in a separate thread.

parallel=”instances”: Providing this value for the parallel attribute in the <suite> tag will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads.

As we saw, there are multiple options provided by TestNG, however it all depends upon your test design and orchestration to run your tests in parallel. [Source]

Automation Tests

I won’t be discussing finding the locators using Selenium WebDriver and using the Page Object model. You can check the A Comprehensive Guide To Locators In Selenium blog for getting details on locator strategies in Selenium WebDriver . You can also check the Everything you need to know about Page Object model and Page Factory in Selenium blog for more details page object model and page factory in Selenium.

Code Repository

All of the code which is showcased in this blog can be found in this Github repository

Getting Started

Let’s discuss the strategy we defined for running the tests in parallel.

To start with, a configuration file in JSON format has been created and placed in the src/test/resources folder from which we will be reading the Desired Capabilities required to run the tests on real devices on the pCloudy platform.

Configuration for parallel run (FileName – parallel.config.json)

JSON Array

“oSAndBrowserCaps” is created in the parallel.config.json file for setting OS, OS Version, Browser Name and Browser Version. Here, “id” field is a unique param for each block and this field is used in the TestNG parameter for running the tests as based on this id, the capabilities for OS, OS version, Browser Name and Browser Version will be set and tests will be run accordingly. This file is stored in the src/test/resources folder of the project.

Code Walkthrough

Reading the JSON config file and setting the Desired Capabilities

Configuration (FileName – DriverManager)

As we are reading the desired capabilities from the JSON file it is necessary to parse the configuration JSON file and get all the details from it and set those values in the desired capabilities that would be required to run the tests. Here, JsonNode class from jacksondatabind dependency is used for reading the JSON file.

The JSON configuration file has only one JSON Array “osAndBrowserCaps” which holds different set of values as per the devices we need to run the test on.

A method parameter Id is created in the startDriver method() which will take the value as per the Parameter value set in the testng.xml file. Once we have the value, we would be comparing this value with the value provided in the id field of the parallel.config.json file.

This is done so we have the proper block of desired capabilities assigned to the respective tests we are planning to execute. So, if we plan to run the tests on Chrome browser on

Windows OS, the correct block of desired capabilities should be picked from the configuration file and mapped to the desired capabilities to run the tests successfully.

We would be reading the Json file using Java’s FileInputStream class. JsonNode class will help us in reading the Json Object Values from the file. Once we read the osAndBrowserCaps node we would be checking if the id value matches with the value we provided in the testng.xml parameter. If it matches, we would be fetching each field and its respective value and setting it in the MutableCapabilities.

After the release of Selenium 4, as per the Selenium WebDriver’s documentation, it is recommended to use respective browsers BrowserOption class like for Chrome – ChromeOptions class, for Firefox – FirefoxOptions class and so on instead of using DesiredCapabilities class, and these BrowserOption classes extends MutableCapabilities and while running the remote driver since we are supplying the OS and browser names from json config file hence MutableCapabilities class comes handy to set the capabilities like browsername, OS, browser version, etc..) as required by RemoteWebDriver to run the tests.

After the capabilities for the OS and the browser and their respective versions are set , we would be setting the common capabilities which are required as per the pCloudy platform for running the tests and enabling the video to view live tests on pCloudy.

The following capabilities are mandatorily required to be provided else tests won’t run:

Client Name, email and API Key can be copied from the Profile page once you login into the pCloudy website with valid credentials.

API Key value should be taken from the “Browser Cloud API Access Key” field as we are automating tests on Browsers.

As these are secret values which are specific to particular users, we would be passing these values on runtime.

To pass the values on runtime we are using the System.getProperty() method. This helps us in passing the actual values on run time and saves us from not hard coding the secret values in the code. We will be discussing how to pass these values on runtime in the later section when we discuss running the tests.

Finally, these capabilities would be set in the RemoteWebDriver along with the GRID_URL which will eventually make use of it to run the tests respectively. The value for GRID_URL can also be found on the profile page after logging in with valid credentials on the pCloudy platform.

Once the driver is set, we would be setting up the browser timeouts by implementing implicit wait, page load timeout and script timeout

An Implicit wait asks the WebDriver to wait for the specified time when finding an element or elements before throwing No Such Element Exception.

A Page Load timeout lets Selenium to wait for the specified time for the full web page to load before throwing any exception.

A Script timeout is set to allow WebDriver to wait for a specific amount of time and let the async script to finish execution on the page before throwing an error.

Next, we would be maximizing the window of the browser.

ThreadLocal class is used for setting the driver for running the tests as it is threaded safe and works very well when tests are run in parallel. The main reason behind using ThreadLocal is that two threads cannot see each other’s ThreadLocal variables even if two threads set different values on the same ThreadLocal object.

It will help us in running the parallel tests safely. So, there is no chance of one session overlapping another when we run the tests in parallel.

Base Test

Base Test is created so it could be extended and used in other tests, so we don’t repeat ourselves in setting the same configuration repeatedly. Single Base Test is created to help us set the base for all the tests we write.

As you can see, Id is captured as a part of TestNG annotation @Parameters which will be set using testng.xml file and based on this field value, OS and browser configurations will be read from “parallel.config.json” file and accordingly OS and Browsers will be allocated to the tests.

Running the Tests

Considering the parallel tests strategy we had set earlier, the following testng.xml has been generated which will help us in running the tests in parallel in Windows 10 OS, 1 test will be running in Chrome browser 102 version and second test will be running in Firefox browser 105 version.

Running the tests (FileName – testng.xml)

Check out the <suite> tag in the testng.xml file

We have set the values for attribute as parallel=”tests” and thread-count to “2”. So, once we execute this, it will start running the tests in parallel in 2 different threads.

The following table explains the devices and platform details that would be used in running the tests in parallel.

Running the Tests in Parallel:

There are 2 ways to run the tests, which are as follows:

1. Running through testng.xml

2. Running using Maven.

Running the tests using testng.xml

Before we execute the test we need to set the “username’ and ”api key” in the VMOptions field in the TestNG configuration window:

These values needs to be prefixed with “-D” and will be passed as:

-Dusername=<pCloudy username> -Dapikey=<pCloudy browser cloud api access key>

To run the tests using testng.xml, we need to right click on textng.xml and select Run ‘…\testng.xml’, it should start executing the tests in parallel.

Checkout the Screenshot below from IntelliJ, which shows the execution status of the tests by running it using testng.xml:

Running the tests using Maven:

Trigger the following command on the terminal to run the tests using Maven:

mvn test -Dusername=<pCloudy username> -Dapikey=<pCloudy browser cloud api access key>

Following is the screenshot from IntelliJ, which shows the execution status of the tests using Maven:

Once the tests are run successfully, we can check out the pCloudy Reports and view all the test details of the test run. Check out the Screenshots below, which will give you a fair idea of how the report for Selenium tests looks like.

All Reports Dashboard

Test 1 details running on Chrome browser

Test 2 details running on Firefox browser

We can also view the active session by navigating to the Reports >> My Active Sessions page.

The below page shows all the ongoing active sessions. Currently we are running tests in parallel on Chrome and Firefox browsers and hence there are two active sessions shown on this page.

By clicking on the “plug icon” in the “Action” column we can view the actual test execution inside the cloud machine, check the below screenshot which gives a fair idea of what is happening inside the cloud machine. Since we have clicked on the session with Windows 10 OS and Firefox browser, we see the tests running on it successfully.

Conclusion

In this blog we discussed what parallel testing is, what are its advantages and disadvantages. We also touched on some points related to understanding TestNG and how it can help us in running tests in parallel.

We also discussed how to run parallel tests using TestNG on pCloudy platform which is a cloud hosted platform for running tests on mobile devices and browsers and how we can view the details of the test run using its reports. Hope you got a better idea of working with Selenium and TestNG to run parallel tests.