Handling Flaky Tests: Strategies for Stable Continuous Testing

An essential characteristic of automated tests is determinism. This means a test should consistently yield the same result if the system under test (SUT) remains unchanged. However, in test automation, the term “flaky test” refers to a test that produces inconsistent outcomes, alternating between passing and failing without any changes to the underlying code or environment.

Such non-determinism sends confusing signals to the team, making it difficult to interpret test results. This erosion of trust in the test suite leads to disregarding signals, integrating features with real failures, and nullifying the purpose of testing. Instead of speeding up progress, flaky tests slow it down, obscure actual bugs, and increase costs. As a result, automation in testing becomes a double-edged sword if not properly maintained.

Why Flaky Tests Are Problematic

Flaky tests pose significant challenges, especially in environments heavily reliant on automated testing, such as CI/CD pipelines. Unreliable tests can:

• Erode trust in test results.
• Delay releases.
• Consume additional resources for troubleshooting.
• Conceal genuine issues, leading to faulty application releases.

Flaky tests can severely impact cross-browser testing. Ensure your automation runs consistently across 5000+ real browsers and devices with LambdaTest.

The Pitfalls of Ignoring Test Automation Context

There are many ways to approach test automation: by context, by data, by domain. However, what often happens is the rush to jump in, write test cases, and create a design pattern abstraction (like page objects or app actions) without deeper thought.

This inertia and lack of understanding can affect how code is written and what is validated, leading to suboptimal results. Here’s an exercise to illustrate the issue:

• Open your test suite and check how many duplicate assertions exist across different specs.
• Identify test cases with multiple assertions that don’t directly relate to the core case.
• Measure how long your test interface takes to complete.

Now ask yourself:

• Do you spend more time “fixing” or “improving” the suite than actually testing the software?
• Are you automating because someone told you to, or are you creating valuable tests with clear objectives?

If this resonates with your situation, your automation efforts might be counterproductive, introducing more work rather than reducing it. Understanding the context and purpose behind automation is key to avoiding this loop.

Is Your Test Automation Adding Value or Creating Chaos?

It may sound absurd, but if you’re in this situation, automating testing is a waste of time, introducing more work than reducing it. Once you understand that this situation is not ideal, there are ways to get out of this loop. Reflect on how you’re designing tests, and consider:

• Are you writing meaningful tests with a clear purpose?
• Are you following principles that reduce maintenance and increase reliability?

A post I wrote on four simple steps for creating meaningful automated tests provides actionable tips for improving your approach. Remembering these principles while designing a test will significantly reduce the maintenance effort later.

Reflecting on Test Automation Practices

Before we deep dive into handling flaky tests, I want you to reflect on these questions:

• How much time do you spend debugging your test script errors?
• How long do you wait to have it run again because the network failed?
• How much time do you spend arguing with DEV because, on a rerun, it works?
• How much time do you spend refactoring your test code because you had an unreported change?
• How often did you take the first test run result outcome and refer to it as the absolute truth of a passed test?
• How long do you spend rerunning the tests to get a positive result?

These questions highlight systemic issues in test automation practices, often stemming from a lack of understanding of the context in which the test suite operates.

What causes Test Flakiness?

Many of you work in environments where sprints are crammed with work; regression coverage is lagging, etc. Working on automation seems very tempting and may simplify things initially when building/extending automated test suites. This easiness of thought leads us to overengineer the solution and write tests impractically. There could be many reasons a test is flaky: concurrency, undefined behaviors, third-party dependencies, infrastructural problems, etc. There is an insightful article on how Google handles flaky tests, which I recommend you to read.

Need to detect and fix flaky tests quickly? Try LambdaTest Test Analytics—a real-time debugging and analytics solution to spot flaky patterns before they become bottlenecks.

Identifying Flaky Tests

Detecting flaky tests in automated testing requires a combination of strategic thinking and specifics. I have tried to summarise the main criteria together with strategies employed in the diagram below:

You will gain more insights and embrace a more precise methodology to identify and subsequently mitigate the inconsistency in the collection of automated tests over time. Often, flakiness could be caused by an issue in one of your test or staging environments that won’t be carried over to production. However, issues such as network errors, load times, or problems with their parties could eventually affect the end user. If you ignore flaky tests, you may overlook the possibility of user impact.

What to do with Flaky tests?

While a common tendency is to ignore flaky tests, which is not a great idea, it’s also impracticable to spend time and effort troubleshooting inconsistent test results. There are several options to mitigate flaky tests, and they may be helpful in different situations. Finding the balance between fixing the tests and giving them room to grow without the noise of failing due to false negatives should be the mantra of every QA Engineer.

While ignoring flaky tests may seem tempting, it’s neither practical nor advisable. Here are actionable strategies to address flaky tests:

1. Fix the Root Cause

If a flaky test covers a crucial path, invest the time to identify and resolve the root cause.

2. Enable Controlled Reruns

Rerun failed tests with specific criteria. For instance:

• Allow reruns for network delays or environmental issues.
• Limit reruns to a predefined number.
• Quarantine the test if it remains flaky after reruns.

3. Temporarily Pause or Remove Tests

If a flaky test blocks the cycle, pause or remove it temporarily. However, this adds technical debt, so plan to address the inconsistencies later.

4. Isolate Flaky Tests

Separate flaky tests into a dedicated suite. This allows the main test suite to maintain confidence while the flaky tests are reviewed and fixed individually.

5. Delete Unnecessary Tests

If a test has never found critical bugs or doesn’t cover important user paths, consider deleting it.

6. Establish Testing Guidelines

Prevent flaky tests by emphasizing testability during development. Incorporate guidelines into code reviews, static analysis, and other processes.

7. Leverage Reporting Tools

Utilize tools that:

• Detect the flakiness level per test run.
• Trace changes causing flakiness.

8. Document Flaky Tests

Maintain records of how flaky tests were handled. Documenting reasons and resolutions can:

• Help teams understand past actions.
• Identify recurring patterns.

Mitigation Strategies and Effectiveness

The following mitigation strategies are based on ideas from my experience handling flaky tests and formulated from various sources adhering to the best practices.

1. Improving Test Isolation

Implementation: Test cases are redesigned to be self-contained, with mock objects or stubs used for external dependencies.

Effectiveness: Enhances test reliability by removing external factors but requires careful management of mock objects to ensure they accurately represent real-world scenarios.

2. Enhancing Test Enviornment Stability

Implementation: Containerization tools are used to create standardised environments.

Effectiveness: Offers high consistency across test runs, reducing environmental flakiness; however, it may introduce complexity in managing containerised environments.

3. Addressing Timing & Concurrency Issues

Implementation: Incorporate explicit waits and synchronisation mechanisms in tests.

Effectiveness: Reduces flakiness due to timing issues, but it may increase test complexity and execution time.

4. Test Data Management

Implementation: Use dedicated or separate datasets or databases for tests and test runs.

Effectiveness: Prevents data-related flakiness but requires additional setup to manage isolated data sets and environments.

5. Rerun Flaky tests with Analysis

Implementation: Automatically rerun failed tests and analyse the outcomes.

Effectiveness: Useful for immediate identification of flaky tests, but doesn’t address the root cause and could lead to ignoring real issues.

6. Design Patterns & Code Quality

Implementation: Regular refactoring and adopting patterns like the Page Object Model in UI tests.

Effectiveness: Improves the maintainability and readability of tests but requires ongoing effort and adherence to best practices.

7. Use of Advanced Tools & AI

Implementation: Using specialised tools & machine learning algorithms/AI to detect patterns and have predictive analytics.

Effectiveness: Can significantly enhance the detection process; however, the accuracy depends on the tool/model capability.

8. Comprehensive Logging & Monitoring

Implementation: Thorough logging for every test run along with observation of testing trends.

Effectiveness: Enables thorough analysis of flaky tests but may produce substantial amounts of data to review.

9. Team Collaboration

Implementation: Regular meetings, knowledge-sharing sessions, and the collective responsibility for ensuring quality.

Effectiveness: Encourages a proactive approach to test maintenance while significantly depending on team culture, mindset and collaboration.

10. Regular Reviews

Implementation: Regular assessments of the testing suite and evaluations of the code with an emphasis on test script coverage.

Effectiveness: Helps in early identification and rectification of potential flakiness but requires dedicated time and resources.

Four Simple Steps to Meaningful Tests

Creating meaningful automated tests reduces maintenance efforts. Keep in mind these 4R3I’s:

• Revisit, Review, Revise, Reframe
• Ideate, Innovate, Iterate

By following these steps, you can proactively minimize test flakiness.

Conclusion

Effectively addressing flaky tests requires a combination of technical solutions, process improvements, and collaborative teamwork. While no approach completely eliminates flakiness, adopting these strategies ensures greater consistency and dependability in your test automation suite. As your test suite grows, remain vigilant and adapt to evolving contexts to maintain reliability and trust.

Optimize your test runs with AI-driven insights, real-time debugging, and a scalable cloud grid. Start testing smarter with LambdaTest.

Nithin SS

Nithin is the founder of Synapse QA, a community space for test automation professionals and software quality advocates. He possesses over a decade of experience in the IT field, with a strong emphasis on building high-performing teams and ensuring excellence in Quality Engineering across projects. In addition to his role as Head of QA at Lodgify, Nithin conducts test automation workshops and empowers professionals to share their insights through Synapse QA, fostering collaboration and innovation within the testing community.

See author's profile