Webinar: Beyond Convenience: Regression Testing in its Real Avatar [Voices Of Community]

In the dynamic and ever-evolving landscape of software development, testing is pivotal in assuring the quality and dependability of applications. Developers and testers face numerous challenges to deliver flawless and efficient code.

Among these challenges, regression testing is one of the most significant yet often underestimated. This crucial aspect of testing, typically associated with repeating existing tests in response to changes, is far more nuanced than commonly perceived.

Despite its complexity and critical importance, regression testing needs to be more concise in industry discussions and practices. It’s commonly seen as an innate skill that every tester claims proficiency in from the early stages of their careers. However, the true essence of regression testing extends beyond the capabilities of an individual tester or even a dedicated testing team.

Our esteemed guest, Rahul Verma, a Consulting Tester and Coach from Trendig GmbH, has teamed up with Manoj Kumar, VP – Developer Relations & OSPO, LambdaTest, to delve into what regression testing genuinely entails and uncover its multifaceted nature. We explore the intricate relationship between changes in software and their impact on quality dimensions, introducing the ‘3 Lake Theory of Regression’ to illustrate this concept. This session also challenges the conventional approach of blind repetition in regression testing, advocating for a more profound and collaborative effort.

This article aims to demystify regression testing, presenting a clear and comprehensive understanding of its complexities. Whether you are new to the field or an experienced tester, this presentation will enrich your knowledge and approach to regression testing, providing valuable insights and guiding principles for effective practice.

Nuances of Regression Testing

Rahul starts the session with a thought-provoking question to the audience:
“Regression testing. When you hear this term, you may instinctively deem it a basic tenet of the software testing realm. But isn’t it curious how we tend to oversimplify things when observed from a distance?”

Rahul further explains that viewing a painting from afar might make you believe it’s a simple task, requiring only a few brush strokes. Yet, when you approach it closely, you begin to discern the intricate details, precision, and effort of every masterpiece.

Upon diving into regression testing, you soon realize that what seemed like a quick task is anything but. There’s an art to it; like any art, it requires attention to detail, methodical precision, and an understanding of the broader picture. So, where are we erring? What crucial aspects of regression testing are we overlooking in the name of convenience or simplicity?

The Reality of Basics

Rahul addressed a concerning industry trend that tends to undermine regression testing. For instance, phrases like “Regression testing doesn’t find bugs,” or the inappropriate application of the ‘Pesticide Paradox’ principle to discredit regression testing have become alarmingly common. This principle, well-articulated in Boris Beizer’s book, is now being taken out of context, leading to skewed perceptions.

Test automation is also a major talking point. It’s an assumption that if tests are automated, they are considered the essence of the regression testing cycle. But, as many seasoned testers know, automated tests can’t capture the intricacies and edge cases inherent in exploratory testing.

Further, the positioning of exploratory testing as a superior alternative and the diminishing value of regression testing over time have added to the confusion. Often, there’s a mistaken belief that regression testing is merely the repetition of known information or written test cases.

The bias toward UI or functionality is a significant issue in the current testing narrative. In many circles, regression testing is seen merely as an extension of functional testing or, at most, some elements of UX testing. This reductionist view needs to consider the multi-faceted nature of regression testing.

In its true essence, regression testing is a layered and nuanced domain. It’s not merely about repeating tests or checking functionality. It’s about understanding the system’s behavior, identifying potential risk areas, and ensuring that new changes have kept existing functionality intact. As we continue to navigate the evolving landscape of software testing, we must revisit our understanding of “basic” concepts and appreciate their depth and importance.

Misconceptions about Regression Testing

Fundamental Confusion

The terms “regression testing” and “regression” are often used interchangeably. This is more than just a gap in foundational understanding; it represents a complete absence of it. Misusing these terms reflects a profound misunderstanding of testing fundamentals.

Oversimplification & Blind Repetition

Regression testing is frequently misconstrued as mere repetition without more profound thought or analysis. This mistaken belief leads to the idea that by simply investing in platforms, tools, and technologies, one can solve all issues related to regression testing. Whether technologically aided or done manually, the essence of regression testing isn’t mere repetition.

Neglect of Innovation

A prevalent misconception is that regression testing solely involves repeating existing tests. The creative aspect, which includes crafting new tests based on evolving requirements, often needs to be noticed or discussed.

Bias Towards Functionality and UI

A reductive view has taken hold where regression testing is often equated only with functional or UI testing. This narrow perspective does a disservice to the comprehensive nature of regression testing.

Misrepresentation

By presenting regression testing as a weak straw man, it’s made to appear inferior, enabling other methods or tools to be marketed as superior alternatives. This tactic oversimplifies regression testing, reducing it to a surface-level activity.

Artificial Distinctions

Another prevalent fallacy is the separation of regression testing from exploratory testing as if they share no common ground. Some even go to lengths to position exploratory testing as a superior alternative, further muddying the waters.

Point of Direct Contact and Ripple Effect

Rahul drew a parallel between an object (like a stone) hitting the water, and as it plunges, there’s an evident point of impact, causing ripples that extend outward. The central impact is apparent, but what about the surrounding ripples? How far do they reach? And in a philosophical sense, how much of the water is genuinely affected by the stone’s descent?

Theoretically, every drop of water feels the stone’s influence. However, real-world constraints make us ask the practical question: which ripples matter the most?

Now, think of these ripples in the context of regression testing. Here, the stone’s impact represents a change or update, and the ripples symbolize the areas of potential influence or disruption. Just as we can’t feasibly track every ripple in a pond, we’re bound by constraints when it comes to regression testing. There’s a cost to consider, time limitations, the availability of skilled personnel, and the right tools.

The Constraints

• Cost and Time: We often need more resources and schedules that restrict the extent of our regression testing.
• Skills and Knowledge: Do our teams genuinely understand the ‘ripples’ of the software world? Do they know which ones to prioritize and how to measure them?
• Tools: Having the correct tools is essential. Even with the best intentions, addressing certain ripples might be possible with the appropriate tools.
• Processes: Even if you have the tools and personnel, the desired results won’t be achieved if the process isn’t aligned.

When any of these elements are missing or inadequate, the approach to regression testing often becomes a fallback to convenience. People begin settling, labeling their makeshift methods “regression testing.” Unfortunately, this diluted understanding and application is widespread in many organizations.

The Bias of UI Lens

In software testing, there lies a subtle bias. This bias, often unrecognized, centers around viewing the test objects through a singular UI lens. This perspective, while pervasive, can limit our understanding and effectiveness in testing.

Many view the test objects – whether APIs, services, classes, or modules – as monolithic entities. This “black box” viewpoint narrows our interaction to just the interface. We communicate, observe, and then deduce. But is this lens the most effective?

While it’s easy to box a test object into the “black box” category, the reality is multifaceted. When viewed closely, even the smallest code is a composite of various components. Modules have classes; classes have methods, which are intertwined with dependencies.

A piece of software doesn’t operate in a vacuum. It’s installed on an operating system, perhaps communicates over a network, and might rely on third-party services. The software might also need to coexist or interact with other third-party applications. These elements aren’t mere add-ons; they’re integral to the functioning of the software.

Given this intricate web of components and external dependencies, regression testing becomes even more crucial. It’s not just about assessing the changes in the software but understanding how these shifts might influence or be influenced by the larger ecosystem.

Back to Basics: What is a Regression?

Rahul further explains regression can be understood as a bug that is either transient or non-transient. This means it could be a permanent bug or a temporary one. It might resolve on its own, or it could persist until specifically addressed. While not overly technical or filled with fancy wording, this definition captures the essence of regression in the context of regression testing more effectively than most explanations out there.

The Nature of Regression Bugs

• Unmasked Problems: A problem might already exist, but it remains unnoticed due to how it was previously used or the absence of further changes. Due to specific changes, this problem can become more prominent and get unmasked.
• New Problems at the Point of Change: It’s a common misconception that the regression will only occur elsewhere if a change is made at point A. However, it’s also possible to have regressions at the same point of change.

What is Regression Testing?

Regression testing is essential in software testing, focusing on detecting regressions – bugs caused by changes in the software. It’s crucial regardless of the tools, technology, or organization size. This testing ensures that new modifications don’t negatively impact existing functions by identifying bugs resulting from any change anywhere in the software.

Beyond Functional Regressions

The common bias towards functional testing in regression testing limits its true potential. There is no hard and fast rule that functional changes, such as code modifications, configuration alterations, or new features, will only lead to functional regressions. A functional change could also result in performance, security, or interoperability regressions. This means there’s a many-to-many relationship between the changes made and the potential quality dimensions they can affect.

It’s important to ask yourself whether your regression testing includes performance and security tests. Often, there is an incorrect assumption that existing tests will cover all potential regressions. However, new changes usually require the creation of new tests, as repeating the same tests can be a flawed approach driven more by convenience than thoroughness.

In the agile world, there’s a tendency to prioritize speed, but to incorporate true flexibility, it’s sometimes necessary to slow down. This allows for proper adaptation to changes before accelerating again. Unfortunately, the desire to be fast often compromises the depth and thoroughness of testing.

3 Lake Model of Regression

Regression testing in software development is a complex and multifaceted process. It encompasses not just the identification of bugs due to technical changes but also considers the broader impacts of ecosystem changes and business decisions. Understanding these diverse influences is crucial for effective regression testing.

Technical Changes

Technical changes include modifications in code, configuration, or the addition of new features. While these changes are often expected to lead to functional regressions, they can also cause unexpected performance, security, and interoperability issues. This aspect highlights the intricate relationship between various changes and their potential impacts across quality dimensions.

Business Decisions

Business decisions, such as a new marketing campaign, can have unforeseen impacts on software performance. For instance, a significant discount campaign might increase an eCommerce website’s traffic, leading to performance regressions. These scenarios demonstrate how business-driven changes, although not technical, can necessitate different approaches to regression testing.

Ecosystem Influences

The ecosystem in which the software operates plays a significant role in regression testing. Legal changes, compliance requirements, or advancements in competing technologies can impact the software’s relevance and functionality. While not directly related to the software’s code, these external changes can lead to regression, making the product less effective or obsolete.

Practical Regression Testing: Go Beyond Convenience

Regression testing is acknowledged as a highly complex task in software development. It requires much more than mere repetition of existing tests; it demands the creation of new testing approaches and strategies. The challenge lies in balancing the need for exhaustive coverage with the practical limitations of time and resources.

The Complexity of Regression Testing

• Beyond Blind Repetition: Emphasizes the need to create new testing methods rather than just repeat existing ones.
• Accepting the Challenge: Acknowledges the complexity and essential nature of regression testing in software development.

Coverage and Practical Limitations

• The Paradox of Coverage: As test coverage increases, the feasibility of running all tests decreases, leading to a need for selective testing.
• Educated System vs. Chaos: Highlights the importance of making choices based on a well-informed system to avoid disorder.

The Role of Testers and Team Dynamics

Effective Utilization of Advanced Technologies

• Beyond Surface-Level Solutions: Concerns about using advanced technologies like AI for superficial problems rather than complex testing challenges.
• Potential of AI in Regression Testing: Advocates for the application of AI to build systems that can diagnose interdependencies and construct risk models.
• Reframing Automation Goals: Suggests a shift from using automation for speed and cost reduction to enhancing test coverage and depth.

The session ended with an interactive Q&A session.

QnA Session

Q: How can we anticipate and manage regression risks when delivering partial features to production in a mini waterfall SDLC model?

A: In managing regression risks in a mini waterfall SDLC with partial feature delivery, it’s crucial to prioritize product quality over strict adherence to process models. Rather than rushing deployments, slowing down can enable more effective testing and quality assurance. This approach may not align with rapid deployment cycles, but it’s more beneficial for delivering a quality product. The focus should be on comprehensive testing and quality, not just fulfilling process requirements like frequent deployments or daily stand-ups. Ultimately, advocating for a quality-first approach, regardless of the specific SDLC model, is key to addressing regression risks effectively.

Q: How can regression testing be completed on time amidst frequent config changes and performance alterations?

A: Addressing the challenge of timely regression testing amidst frequent changes requires a shift in perspective. As highlighted in a recent presentation, approaching this issue is not about accumulating numerous tests but understanding the essence of regression testing. Factors like cost, time, and technology should maintain this understanding.

Key Steps:

1. Understanding Over Constraints: First, understand what regression testing entails without being limited by constraints. Knowledge about regression testing is fundamental, and merely providing resources like time and money won’t automatically address knowledge gaps.
2. Building a Risk Model: Take the time to build a risk model. This involves analyzing many-to-many relationships between quality dimensions and changes. Use historical data to inform this model, and consider pausing current activities to focus on this crucial task.
3. Importance of a Cross-Functional Team: Forming a cross-functional team is critical. This team should include members from different departments, reflecting Agile’s foundational principle of cross-functionality. Such a team is essential for regression testing, performance, and security engineering.
4. Knowledge and Decision Making: Knowing when and what to test is as important as conducting the tests. Discussing costs and resources becomes irrelevant if you need to be made aware of the need for specific tests like performance or security testing.
5. Start Small and Build: Begin with a small, dedicated team consisting of a tester, developer, architect, and occasionally a manager. This team can slowly expand its scope and influence. The change may be slow, but it will likely be effective and sustainable.
6. Methodology and Adaptability: The approach to regression testing can vary. You might pause other activities to focus on testing or integrate testing into ongoing work. The key is to start making changes and be serious about the importance of comprehensive regression testing.

Q: How can we balance risk-based testing with thorough regression testing in a fast-paced development environment, avoiding convenience-based testing?

A: Risk-based testing is integral to effective software testing, not just as a separate methodology but as a core aspect of all testing practices. Every tester should inherently consider risks in their approach.

• Risk-Based Testing as a Core Concept: Contrary to risk-based testing as a specialized method, it should be viewed as a fundamental part of all testing activities. All testing, in essence, should be risk-based. There’s no such thing as non-risk-based testing because assessing risks is integral to the testing process.
• Application in Testing Techniques: For example, in equivalence class partitioning, a standard testing technique, considering risk changes the approach. In functional testing, this might mean focusing on specific partitions. In security testing it expands to a detailed examination of special characters and their sequences.
• Adapting to Time Constraints: The ability to adjust testing strategies based on time constraints is crucial. Testers must prioritize and cover the most significant risks within the given timeframe. This approach emphasizes the importance of understanding what value can be delivered in different time frames – from days to weeks.
• Balancing Skills and Time Resources: When given more time, like six months, the question shifts from time management to skills and knowledge. Testers must ask themselves if they can create unique and effective tests over an extended period. It’s about more than just having time but also having the expertise to utilize it effectively.
• Continuous Knowledge Enhancement: Testers should continually enhance their knowledge and skills. This involves revisiting and deepening understanding of fundamental testing concepts to identify gaps and improve testing strategies.
• Finding the Right Balance: Good testers find a balance in their approach, adapting their testing strategies to knowledge limitations and practical constraints. As testers grow into management roles, they can extend this balanced approach to their teams, helping them understand and navigate organizational constraints while fostering a culture of quality and thoroughness.

Hope You Enjoyed The Webinar!

We hope you liked the webinar. In case you missed it, please find the webinar recording above. Share this webinar with anyone who wants to learn more about clean coding practices. You can also subscribe to our newsletter Coding Jag, to stay on top of everything testing and more! Stay tuned for more exciting LambdaTest Webinars.

That’s all for now. Happy testing!

Sparsh Kesari

Sparsh Kesari is a software developer specializing in Full Stack Development. He is an Open Source enthusiast and has been part of various programs including GitHub Externship. He has also interned at organizations like GitHub India, RedHat, MeitY, Government of India. He is actively involved in the testing and QA community work as well. In his current position, he works as a Developer Relations Engineer at LambdaTest, exploring and contributing to the testing world.

See author's profile