How To Stop Test Suite after N Test Failures in Pytest?

An exhaustive test-suite comprises of many test cases that test different features on various combinations of browsers, platforms, and devices Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing test automation.

The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).

Also, check out this video tutorial on How to Skip or Stop Tests in pytest.

In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.

Using @pytest.mark.incremental Decorator To Stop Test Suite After N Test Failures

The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.

PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.

conftest.py (to skip test in Python with pytest)

The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) in order to skip the test in Python with pytest.

This PyTest Tutorial for beginners and professionals will help you learn how to use PyTest framework with Selenium and Python for performing Selenium automation testing.

How To Skip Test In Python With pytest Using @pytest.mark.incremental?

To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an automated browser testing example that contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.

Fixture with scope as the class is used to insatiate the Chrome browser instance. Implementation to skip test in Python with pytest, is also included in conftest.py.

Conftest.py

test_step.py

As seen in the implementation, incremental testing is applied to class Test_Scenario_2 and Test_Scenario_2 via the @pytest.mark.incremental decorator.

In test case test_2(), Google search for LambdaTest is performed. Once the search results are ready, a click on the first search result is done and the page title is compared with the expected title. For testing, we have deliberately introduced an error in the code and the Selenium test automation case fails as the titles do not match.

Test case test_3() which follows after test_2() will be skipped from execution and marked as xfail as the previous test [i.e. test_2()] in the same class has failed. Selenium test automation case test_4() which is a part of class Test_Scenario_2() is also marked for incremental testing.

Here is the command for execution:

Shown below is the execution snapshot:

Here is the final status of the tests performed in the example:

Test_2 executes and fails whereas test_3 execution is skipped due to failure of test_2. Hence, it is marked as xfail. Though the test_4 () passes here, even if it were to fail, the test would be marked as fail (and not xfail) as there is no dependency on the test from the other class.

In case, incremental testing is removed from the above implementation, all the tests would be executed. Only test_2() will fail whereas the remaining three tests would pass as the @pytest.mark.incremental decorator is removed from the implementation.

Pytest Tutorial: Executing Multiple Test Cases From Single File

This certification is for professionals looking to develop advanced, hands-on expertise in Selenium automation testing with Python and take their career to the next level.

Here’s a short glimpse of the Selenium Python 101 certification from LambdaTest:

Using maxfail Command Line Option To Stop Test Suite After N Test Failures

PyTest offers a command-line option called maxfail which is used to stop test suite after n test failures. For example, to stop the test suite after n test failures, where the value of n is 1. Once a failure is encountered, all the subsequent tests are skipped.

Though it is not a good practice to develop inter-dependent Selenium test automation cases, in some scenarios, you may have to develop such tests. The maxfail option comes handy in such cases as unnecessary execution of dependent tests can be avoided. Shown below is the syntax of maxfail:

How To Skip Test In Python With pytest Using maxfail?

To demonstrate the usage of maxfail, we take the automated browser testing example which was shown in the section How To Skip Test In Python With pytest Using @pytest.mark.incremental. The implementation specific to skip the test in Python with pytest will be removed, remaining code remains the same.

conftest.py

test_step.py

The example shown below is executed where we stop the test suite after n test failures and the maximum number of failures set to 2. Here is the execution command:

As incremental testing is not enabled, all the four Selenium test automation cases will be executed and the execution stops once the number of failures becomes 2. Test case test_2() fails as the window title does not match the expected one, rest all the three test cases are executed. Out of 4 test cases, 1 test case fails and 3 test cases pass. Here is the execution snapshot:

In this pytest Tutorial, learn how to use pytest fixtures with Selenium and how to set up your test using the pytest.fixture() decorator.

All In All

In this Python tutorial, we looked at how to stop test suite after n test failures. Skip tests in Python with pytest can be followed in Selenium test automation scenarios where tests are inter-dependent and execution of dependent tests needs to be skipped in case the primary test case fails.

The @pytest.mark.incremental decorator is used to skip tests in Python with PyTest. The other mechanism to stop test suite after n test failures by using maxfail command-line option.

This brings an end to this Python tutorial. I hope this article was informative, and you now know skip tests in python or how to stop test suite after n test failures. In case you still have any doubts, do reach out to us in the comment section below, and we’d be happy to help you with your problem. Happy Testing!!!?

Himanshu Sheth

Himanshu Sheth is a seasoned technologist and blogger with more than 15+ years of diverse working experience. He currently works as the ‘Lead Developer Evangelist’ and ‘Senior Manager [Technical Content Marketing]’ at LambdaTest. He is very active with the startup community in Bengaluru (and down South) and loves interacting with passionate founders on his personal blog (which he has been maintaining since last 15+ years).

See author's profile