Test Execution Time – why is it important and how can we speed it up?

As the complexity of our system under test increases, it will likely mean the number of our tests will keep growing exponentially. Large test suites mean longer test execution times. We can come into situations where we have many tests (especially for regression) and it can take many hours for all those tests to run. Not to mention the time and resources it takes to analyze the test results, investigate the failing tests, debug and fix the flaky ones, etc. So without further ado let’s see what we can do to make our automated tests execute faster.

Combine API tests with UI tests

Sometimes we will have to deal with tests that can’t be executed without certain preconditions. At times we might need a specific set of test data before we can test what we need to test. For such situations, if we have access to an API it can make our lives a bit easier. APIs aren’t just better for automating business logic, they are also great for test preparation. The reasoning is the same as with authenticating via API, instead of the UI. We can cut some corners and save a lot of time, improving our test execution times considerably. This is especially true for preconditions which are frequently used in different many tests. A lot of tools have support for this, some even have native, out-of-the-box support for making API calls from your UI testing framework. Furthermore, API tests can help us avoid malicious code, and execute tests faster, they are less dependent on any technology stack, as we are relying mainly on the universal HTTP protocol. And the benefit that is the most important to the business, API tests are a lot cheaper compared to UI tests.

Note

You can use LambdaTest Cloud to test on a variety of simulators, emulators and real devices. No need to install anything, just sign up for a free account and start testing.

Try LambdaTest Today!

Favor smaller tests whenever possible

Small tests are sometimes referred to as Atomic Tests – each test is very small without having a dependency on other tests. A test that fails or passes independently of other tests makes it more reliable, faster to execute, and much easier to debug when it fails. Also, adding new tests is much easier when those tests are smaller. Granted, it’s not always possible to have very short tests – sometimes we need to emulate a real-world user journey, but as a general rule, we should favor many smaller tests over a few very lengthy detailed tests.

Only automate dynamic UI sections if it can’t be avoided

Sometimes it’s better not to automate certain elements of a web page or an app. Automating elements with a lot of animation, or highly interactive elements, can be very demanding. This usually involves adding a lot of waits in your tests – and such tests are difficult to maintain. Dynamic elements should be tested manually, as automating them can be very flaky and demanding.

For example, let’s imagine a relatable CSS animation. When a user adds an item to the shopping cart, the cart icon might move and update to show the number of items in the cart. Now let’s imagine, that we have a very slow network, we would need to tell out the test to wait until the animation occurs, and until the updated element is visible. The result is that this test can fail for a wide number of reasons, making it hard to debug, and also, such a test would be pretty slow to execute. So in the end, think about the economics of automating a dynamic element and determine if it’s worth the investment – usually, it’s not.

Make sure to mock 3rd party dependencies

When our app needs to connect to 3rd party systems, that adds another layer of complexity to our tests. We come into situations where we end up testing code that we are not responsible for. It is considered a given that we assume that the 3rd party code we are using has already been tested. This is especially important to pay attention to when dealing with sensitive data, like when using payment plugins. In such cases, the security of the integration is of utmost importance. These external dependencies should either be mocked (use a dummy version to simulate the real 3rd party extension) or bypassed on an API level. Especially avoid testing someone else’s UI, and use their API for that instead if you must. For mocking purposes, you can use public mocking libraries to generate credible-looking test data. If you are unfamiliar with mocking, you can always reach out to your developers for help. Developers are accustomed to using many mocking libraries, which they use mostly for unit testing – to test small chunks of the code in isolation. In the end, mocking dependencies allows us to focus on testing what’s crucial for our stakeholders and not spend time on too many details.

Use direct URLs instead of button clicks

We need to think about what is happening behind the scenes when we are automating a user journey. Sometimes translating manual test cases into automated tests is not the best way and most optimal way to go. Let us go with a simple navigation example: if we have a navigation menu, we do not automate clicking on it. It is faster to navigate directly to the URL to which that menu item was originally linked. This has the positive effect of making our test a bit less prone to flakiness, be it a false positive or a false negative test execution result. By avoiding clicking on the nav button, we reduce the number of actions, and page load times improve.

Prefer API authentication over UI login

Having dedicated tests with login forms can be pretty costly, and a big bottleneck. Many testers have probably seen methods that call a UI login test a precondition for many other tests. This is a huge time waster and an unnecessary overhead. So, if you have an API you should use that instead. Authentication over API is way faster than UI-based login and it is a lot more reusable. The best thing is that this kind of API test is one of the simplest ones – you would usually need an email/username, password, and maybe a cookie consent. Make your POST request and bam! You are fully authenticated in a fraction of the time it would take to do the same by clicking on the UI. Just make sure that you are not doing this blindly, first ensure that you have a thorough understanding of the login options, to get an idea of all the parameters that you can send to the backend via an API call.

Avoid automation cookie consent banners

Cookie consent banners are the bane of any automation engineer’s existence. Automation is often redundant and they should be bypassed – as they need to be tested separately, from more of a legal-acceptance point of view. Also, another thing is that they can really easily be bypassed, giving us a significant speed boost to our test execution time. Most of the time you can easily get rid of these consent banners by using the developer tools from the browser. All we need to do is to give the required value and place the cookie before our page loads. That will allow us to not even see the cookie consent popup when we load up our test.

Run your tests in parallel

Eventually, the number of tests you need to execute will grow over time, making sequential execution very time-consuming. Just imagine (this is especially applicable to UI tests) having hundreds, or thousands of tests – each one running only once the previous test has been executed! Running those tests in parallel will enable you to run multiple tests at the same time. You can run your tests in parallel on multiple machines, like virtual ones, to significantly reduce the test execution time.

Another, even more convenient, approach that you can try is using a cloud service, such as HyperExecute for your tests. This way you save additional time by not having to the setup test environment and provide resources on your own. Such an approach will need to fall under the testing budget you have allocated, as these are mostly commercial-based services. When running tests in parallel we should have in mind that we should avoid hard-coding any values, and instead make sure that our test data is well organized. If we play things “right” and use smart parallelization within our test automation strategy we will gain many benefits. Our tests will be faster, more economical to run, have better coverage, integrate better into the pipeline (our CI/CD process), enable us to have better testing practices, and give us the opportunity to move into a direction of continuous testing.

Wrap-up

So, let us briefly sum up what can we do to improve our test execution:

• Combine API tests with UI tests – we do this to gain faster execution, write less automation code, and have cheaper execution as well.
• Favor smaller tests whenever possible – smaller tests are easier to debug and maintain, have a single point of failure, and should pass or fail independently of other tests.
• Only automate dynamic UI sections if it can’t be avoided – avoid automating highly dynamic content, as it is very demanding, instead check them manually.
• Make sure to mock 3rd party dependencies – do not invest in testing other people’s code, mock 3rd party dependencies to save time and to make your tests more stable.
• Use direct URLs instead of button clicks – opening a URL directly instead of clicking on the button will save us time, also, it will make the test simpler. Our tests will be less flaky and faster to execute and the pages will load faster with fewer waits.
• Prefer API authentication over UI login – authenticating via API is way faster and far more reusable than using a UI test as some kind of shared step, which other tests are dependent on. Tests should fail or pass separately.
• Avoid automation cookie consent banners – these banners are more and more present and dealing with them directly from the UI can be very time-consuming, especially in the long run. We should access the cookie directly without clicking on buttons, and our page will thank us for being able to load faster!
• Run your tests in parallel – parallel execution of our tests allows us to execute more tests in less time, compared to executing our tests in a sequence. We can utilize virtualization and cloud technologies to make our parallelization efforts even more effective.

There are a number of other smaller things we can do to make our tests execute faster. If we are using tools like Selenium, we can speed up our tests by removing redundant Selenium commands, we should minimize the number used to get a speed benefit. Another thing is to favor explicit over implicit waits, explicit waits are a bit more demanding to implement but it’s worth the extra effort.

From the UI side of things, there are also small tweaks we can do, such as avoiding the use of Xpath, when possible, and using tags and data attributes to manipulate elements directly. Running our tests in a headless browser mode allows up to run UI tests directly from memory, without interacting with the browser’s GUI, most modern testing tools have built-in support for headless mode. Also, we need to pick times when our tests are being run, and at what frequency; a smaller set of sanity tests can be scheduled to run several times a day whereas a bigger test set should be triggered less frequently.

We should also invest time in exploring different tools to find the ones best for our context. Our tests should also be deterministic, meaning that they should be resilient and not fail due to environmental and data issues. Another good practice is to do a database and test environment clean-up, so we start with a clean slate when we run our tests. Make sure your tests are documented (it doesn’t have to be overly detailed, as long as it provides value), and favor lower-level tests over more complex ones.

Hopefully, these tips will be helpful, Happy testing!

Mirza Sisictester

Mirza has always been a technology geek, helping friends and family with computer-related issues. Started in tech support and moved to software testing. Mirza worked as a freelance web developer for a while as well. When he’s not sharing memes online Mirza is usually learning new things, writing posts for his blog, and being an active member of the testing community. His site: https://juniorlearnstocode.blogspot.com/

See author's profile