How to Run Parallel Tests Using Pabot With Robot Framework

Python is one of the widely preferred languages for front-end automated testing. One of the primary reasons for its adoption is the support for frameworks that offer TDD and BDD. Robot Framework is one such Python automation framework that makes a cut above the rest.

You can also leverage SeleniumLibrary in the Robot Framework to automate interactions with WebElements in the Document Object Model (DOM). Apart from automating tests, you can also accelerate test execution by harnessing Pabot with Robot Framework for parallel test execution. Pabot is a parallel executor that helps parallelize test execution at different levels – test suite, test case, and resource file.

In this blog, we will look at how to perform parallel test execution using Pabot with Robot Framework.

What Is Pabot?

Pabot is a parallel executor for Robot test suites. It helps execute multiple test suites concurrently, thereby reducing the overall test execution time. Since it offers parallelization at different levels (e.g., test suite, test case, and resource file), parallel execution with Pabot must be leveraged if you have a significantly large test suite.

By default, Pabot does parallelization at the test suite level. Each process has its own memory space and runs a single test suite. In this case, individual tests within the test suite run sequentially, whereas all the test suites (e.g., test1.robot, test2.robot, etc.) run in parallel.

Resource files normally contain reusable components such as high-level keywords, variables, libraries, and other settings commonly used across test suites.

Now that we have covered all the essentials of Pabot, let’s look at using the Robot Framework for parallel test execution on the local grid and the LambdaTest cloud grid.

You can also check out our detailed Robot Framework tutorial to dig deeper into the ins and outs of the Robot Framework.

Parallel Testing Using Pabot With Robot Framework

The primary prerequisite for the demonstration is Python, you can refer to getting started with Selenium Python to get a quick recap of Python for Selenium automated testing.

To demonstrate parallel testing with Pabot and Robot Framework, we will run the same test scenarios across the local grid and LambdaTest cloud grid.

Project Prerequisites

Note: Please replace pip3 with pip depending on the pip version installed on your machine.

Here is the basic setup requirement for running parallel tests with Pabot and Robot Framework:

1. Install the VS Code (preferred) or PyCharm Community Edition.
2. Install the IntelliBot plugin in case you opt for PyCharm Community Edition.
3. Install Robot Framework, SeleniumLibrary, Pabot, and other external libraries required for automated web testing.
4. To better manage dependencies and environments, it is recommended to use a virtual environment (venv). To create the virtual environment, run the commands virtualenv venv and source venv/bin/activate on the terminal.
5. Install Pabot by triggering pip3 install robotframework-pabot on the terminal. For our example project, we will add all the required libraries in the requirements.txt file.
6. Invoke the pip3 install -r requirements.txt command to install the project dependencies (or packages).
7. Run the pip3 show robotframework-pabot command to check whether the Pabot installation was successful. At the time of writing this blog, the latest version of the Pabot executor is 4.0.6. However, we are using the 2.18.0 version in this blog on parallel testing with Pabot and Robot Framework.
8. Apart from Robot and SeleniumLibrary, install the webdriver-manager and robotframework-pabot libraries.

Test execution with the Robot Framework and SeleniumLibrary will be performed with frameworks installed on the local machine and the cloud grid like LambdaTest. We will touch upon the integral nuances of cloud execution in further sections of the demo.

LambdaTest is an AI-powered test execution platform that lets you run Python automated tests in parallel using Pabot with Robot Framework on an online Selenium Grid of real browsers and operating systems.

Project Structure

Shown below is the project structure where the tests demonstrating the usage of the Pabot and SeleniumLibrary with Robot Framework are located in the following locations:

• Tests/LocalGrid: Implementation of Robot Selenium tests executed on local machine.
• Tests/CloudGrid: Implementation of Robot Selenium tests executed on LambdaTest cloud grid.

Let’s do a deep dive into the project structure:

• Resources/PageObject: Consists of three sub-directories mentioned below:
  • Common: Contains the file LambdaTestStatus.py that helps in updating test execution status on the LambdaTest Web Automation dashboard.
  • KeyDefs/Common.robot: Common.robot houses custom keywords for realizing actions like opening/closing browsers, clicking menu items, and more
  • KeyDefs/SeleniumDrivers.robot: This solution (inspired by this Robot Community Thread) is required only if you are using Selenium (v 4.9 and above).

  SeleniumDrivers.robot houses custom keywords for managing WebDrivers for Chrome, Firefox, and Edge browsers. The management is done using the webdriver-manager library.

  • Locators/Locators.py: As the name indicates, this file contains all the locators used to locate elements on the document under test. In the following sections of this blog, we will discuss a few of these locators.
• pyproject.toml: Configuration file for build management purposes.
• requirements.txt: File that contains dependencies (or external libraries) to be installed for the project.

After creating a virtual environment (venv), run the commands poetry install –no-root and pip3 install -r requirements.txt to install the required dependencies.

Lastly, all the tests of this Robot project can be executed on the local machine and LambdaTest cloud grid.

• Execution on the local machine: export EXEC_PLATFORM=local
• Execution on the LambdaTest cloud grid: export EXEC_PLATFORM=cloud

When executing tests on the LambdaTest cloud grid, you also need to export environment variables LT_USERNAME and LT_ACCESS_KEY, which can be obtained from your LambdaTest Security page. The combination of LT_USERNAME and LT_ACCESS_KEY are used for authentication when tests are run on LambdaTest.

The Common. robot file contains all the user-defined keywords and resources. A separate file named Locators.py separates the locators from the core test logic. This helps realize the benefits offered by the Page Object Model to a certain extent.

Implementation

To demonstrate parallel execution of the Pabot and Robot framework, we will execute 3 tests in 2 test suites simultaneously. The test implementation will remain unchanged, regardless of whether the tests are executed on the local machine or LambdaTest cloud grid.

Test Scenario 1:

Navigate to the LambdaTest Selenium Playground website. Click on Input Form Submit. Enter all the required details in the newly opened form. Submit the items.

Implementation:

Filename: test_sel_playground.robot

Code Walkthrough:

To get started, we first use the Resource keyword to include the external resources (e.g., libraries, user-defined keywords, etc.) in the test suite. The Common.robot file contains all the user-defined keywords. Also, the Variables keyword is used to define the variables that will be used throughout the course of tests and test suites.

The value of the environment variable EXEC_PLATFORM is assigned to a local variable that is accessible across the test. To execute on the LambdaTest cloud grid, we first create a local dictionary variable named lt_cloud_options, which consists of browser options (or capabilities) generated with the LambdaTest Automation Capabilities Generator.

Next up, we assign the value of &{lt_options_cloud} to the LT:Options key within the &{CAPABILITIES_CLOUD} dictionary. Hence, &{lt_options_cloud} is nested inside &{CAPABILITIES_CLOUD} under the key LT:Options.

The user-defined Test Teardown keyword calls the other user-defined keyword – Close local test browser or Close test browser depending on whether the test is executed on a local machine or LambdaTest cloud grid.

All the test cases (or test scenarios) are added in the Test Cases section. If the tests are executed on the local machine, the first step is to set the path of the respective browser driver (i.e., ChromeDriver, EdgeDriver, or GeckoDriver). As stated earlier, this step is only necessary if you are using Selenium 4.9 (and above).

Depending on the point of execution (i.e., Robot Framework on the local machine or LambdaTest cloud), the respective browser is instantiated so that automated tests can be performed on the URL under test.

Two user-defined Open test browser and Open local test browser keywords are created to instantiate respective browser(s) on the local machine and LambdaTest cloud grid respectively.

The Open test browser keyword takes three input arguments – test URL, browser name, and additional options ${lt_options}. The line ${options}= Evaluate sys.modules[‘selenium.webdriver’].${BROWSER}Options() sys, selenium.webdriver helps in creating an instance of browser-specific options in Robot Framework dynamically. The Evaluate keyword of the BuiltIn Robot library helps in executing Python code directly within a Robot test.

In order to set the capabilities (or browser-specific options), the set_capability method of the selenium.webdriver package is used in the Robot Framework. The method is invoked on the ${options} object (created in the earlier step), and it sets the custom capability LT:Options with the value ${lt_options}.

Lastly, the Open browser method of SeleniumLibrary is finally invoked for instantiating the corresponding browser on the required platform (or operating system). The remote_url option in the Open browser keyword is set to LambdaTest hub URL (i.e., @hub.lambdatest.com/wd/hub) when tests have to be executed on the LambdaTest cloud grid.

The Open local test browser keyword also takes three input arguments: TEST_URL, BROWSER, and DRIVER_PATH. To get the corresponding driver path, we have created keywords like Update Chrome Webdriver, Update Firefox Webdriver, etc., to read and return the path to the required browser driver.

The webdriver-manager library helps in the automatic management of drivers for different browsers. For example, ${driverpath}= Evaluate ${CHROME_DRIVER_MANAGER}().install() modules=webdriver_manager.chrome uses the webdriver_manager for automatically downloading and installing the latest ChromeDriver.

Once the Selenium Playground website is opened, a check is performed to validate if Input Form Submit is present on the page. The WebElement is located using the XPath property, and its presence is validated using the Page should contain element keyword of SeleniumLibrary.

After the click is performed, the Input Form Demo page opens up. Here, entries in the text boxes (e.g., Name, Email, Password, etc.) are populated using the Input text keyword in SeleniumLibrary.

Lastly, the select from list by value keyword is used to select the US from the country drop-down. Once the input data is submitted by clicking the Submit button, the Execute JavaScript keyword in SeleniumLibrary triggers a window scroll to the start of the page.

Finally, the presence of the text Thanks for contacting us, we will get back to you shortly. is checked to verify if the page submission was successful.

Test Scenario 2:

Navigate to the LambdaTest Sample ToDo App. Select the first, second, and fifth checkboxes. Send Yey Let’s add it to list in the textbox with id = sampletodotext. Click the Add Button and verify whether the text has been added or not.

Implementation:

Filename: test_todo_app.robot

Code Walkthrough:

Except for the browser and OS combinations, both the tests in the test suite have similar implementations. The Page should contain element keyword of SeleniumLibrary is used for checking the presence of the checkboxes (for the first and second items) on the page. The items are checked using the Click button keyword in SeleniumLibrary.

Once the LambdaTest ToDo App is opened, The Name locator is used to locate the elements FirstItem and SecondItem.

On similar lines, the ID locator is used for locating the Add button on the page. The Input text keyword is used for inputting Yey Let’s add it to list as the new item in the ToDo app. The Add button is clicked to add the said item to the list.

Lastly, once the new item is added to the ToDo list, it is located using the XPath locator. The Get Text keyword in SeleniumLibrary returns the text value of the element identified by the XPath property.

The Should Be Equal As Strings keyword in the BuiltIn library compares the string value assigned to ${response} with the expected value. The test fails if both the string objects are unequal.

Important: SeleniumLibrary is not thread-safe. Hence, there is no safety-net for concurrent access to browser instances or parallel test execution across multiple threads.

The Pabot runner eliminates the thread-safety issue, as parallelism is performed at the process level (not the thread level). Each process in Pabot has its own memory space, which allows parallel processes to run in isolation without interfering with each other.

Test Execution

Before using Pabot and Robot Framework for parallel test execution, let’s run the same tests sequentially. This will allow us to benchmark the test execution time (or performance) when the same test scenarios are run in parallel using the Pabot runner.

After completing the steps mentioned in the Project Prerequisites section of this blog on using the Pabot and Robot framework for parallel test execution.

Serial Test Execution:

Invoke the following command to run the test scenarios located in the Tests/CloudGrid folder serially.

It is essential to note that the time taken for execution also depends on factors like machine performance, browser performance, network connectivity, and more. Hence, the numbers being showcased might vary depending on the factors mentioned above.

Run the robot Tests/CloudGrid/ command on the terminal to trigger the serial execution of Robot Framework tests on the LambdaTest cloud grid. As seen below, one test failure was added to demonstrate the usage of logging LambdaTest exceptions for handling Selenium exceptions in Robot Framework tests.

The total execution time is 1.52 seconds, where three tests (in two test suites) pass, whereas a test expectedly fails.

Shown below is the execution snapshot from the LambdaTest Web Automation dashboard. All four tests spread across two test suites execute successfully.

Test Execution Snapshot (Result – Passed):

Test Execution Snapshot (Result – Failed):

Let’s see what happens when the tests and/or test suites are run in parallel using Pabot runner.

Parallel Test Execution (Default – Suite-Level):

By default, Pabot splits the test execution at the test-suite level, with each process running a separate suite. Tests inside a test-suite (i.e., *.robot files) run in series, whereas the suites run in parallel.

In our case, we have two test suites where test_todo_app.robot file consists of three test scenarios, whereas test_sel_playground.robot file consists of a single test scenario.

Invoke the following command on the terminal:

Shown below is the execution snapshot, which indicates the test execution was successful:

Here are some interesting things to notice from the test execution:

• Test suites are executing in parallel, with each process (PID: 68622 and PID:68623) running a separate suite.
• During test execution, a .pabotsuitenames file is automatically generated. It contains information about test suites when they are executed in parallel.

Shown below is the execution snapshot from the LambdaTest Web Automation dashboard. The total test execution time has come down from 1.52 seconds (with serial execution) to 1.27 seconds (with suite-level parallelism).

Parallel Test Execution (Test-Level):

Pabot lets you achieve parallel execution at the test-level through the —testlevelsplit option. In case the .pabotsuitenames file contains both tests and test suites, then the –testlevelsplit option will only affect new suites and split them.

Here, each process runs a single test case. For instance, if the test suite(s) have four tests (e.g. T1, T2, T3, T4), then four separate processes (P1, P2, P3, P4) will run each test in parallel.

Test-level parallelism must be preferred over suite-level parallelism in cases where suite(s) have a large number of independent tests that can run concurrently. In our case, four test scenarios can run in parallel.

Invoke the following command on the terminal to run individual tests in parallel using Pabot.

Since there are three tests in total, three parallel processes are invoked with each running a single test. As seen in the test execution screenshot, four parallel processes (PID: 73648, 73649,73650, and 73651) handle one test scenario each.

As seen from the LambdaTest Web Automation dashboard, all four tests are executed in parallel:

Test execution time when tests are parallelized at the test level is 1.24 seconds, whereas suite-level parallelized tests are also executed in 1.27 seconds.

Since the execution time might vary, it is important to choose a parallelism mechanism that yields ideal savings over subsequent runs.

There are scenarios where you might want to invoke more/less parallel executors to achieve more out of parallelism with the Robot Framework. This is where the –processes option lets you customize the number of parallel executors for test case execution.

When customizing the number of parallel executors, it is recommended to consider the available resources (physical CPU cores, memory, etc.). For this, you can invoke the below command in the terminal:

Let’s do a test split execution with 2 parallel processes and see the difference. Invoke the following command on the terminal:

As seen from the execution snapshot, at a time, only two tests (in different test suites) are executed in parallel across two processes (PID: 77111 and 77112).

The total execution time is 1.31 seconds, which is slightly reduced (as expected) when the same tests are parallelized across 4 processes.

Setting the environment variable EXEC_PLATFORM to local helps in executing tests located in the Tests/LocalGrid folder with Robot Framework on the local machine.

To summarize, both thread-level and suite-level parallelism can potentially accelerate test execution time. However, you must keep a watchful eye on test dependencies and available resources (i.e., CPU cores, memory, etc.) to avoid conflicts or flaky tests (or unexpected results).

Conclusion

Thanks for getting this far! Though the Robot Framework supports sequential test execution (out-of-the-box), tests can be run in parallel by leveraging the Pabot with Robot Framework. Pabot lets you parallelize test-suites as well as tests, that too without major code modifications.

As we all know, excessive parallelism can cause test flakiness if the tests are not designed to handle concurrent execution. It becomes essential to follow Selenium’s best practices when opting for parallel test execution using Pabot and Robot Framework. You can also leverage the benefits of a cloud grid like LambdaTest in case you are looking to capitalize on the benefits of cloud testing and continuous integration.

Until next time, happy testing!

Citations

• Robot Framework: https://robotframework.org/
• Pabot GitHub Repository: https://github.com/mkorpela/pabot