Common Challenges in Continuous Testing and How to Overcome Them

Today’s software industry is much more complicated than it was in the past. It is evident in almost every aspect of customer demands, fierce corporate rivalry, and, of course, global changes that directly affect this market. Because of this, businesses today need to have a competitive edge over rivals, which means they must be able to release products quickly and of exceptional quality.

Therefore, it won’t come as a huge surprise if I say that one of the most crucial objectives of every company—from startups to enterprises—will be to secure their ability to offer frequent releases to their customers. And is it easy? No, it might take several years (especially for large corporations) due to a variety of factors, including culture, technological constraints, and, of course, the testing process, which is most likely to take up the majority of the time in each project.

So, how companies can handle it? Here is where Continuous Testing comes to help. Continuous Testing, according to Wikipedia, is “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.”

Simply put, the main objective of continuous testing is to build quality into the product from the very beginning of the software development life cycle. Using this method, we push our teams to make both culture, mindset, and technical adjustments to make sure they can test at all stages of development (Unit testing, integration testing, Security Testing, system testing, and acceptance testing) and accelerate the overall development process to achieve better results for their customers.

Continuous testing, as I mentioned above, is essential for businesses to stay one step ahead of their competition, but it also comes with inherent complexities and challenges that are worth mentioning before you dive right in.

Challenge 1: Implement a new mindset in an existing methodology

As I mentioned earlier, before continuous testing became incorporated into the DevOps terminologies, development teams had to adopt a completely different mindset. I observed how some organizations suffered serious delays or failed to make the transition as a result of poor team communication. People dislike change, especially drastic changes like that brought on by the adoption of continuous testing.

Make it possible for your teams to comprehend the “Why” and the advantages of the change when communicating it to them. To improve the product for their customers, they will, for instance, learn new technologies and tools. Additionally, it’s crucial to give your teams time to adapt to the changes; avoid pressuring them before you’re sure they are willing to take the journey with you.

Challenge 2: Are we automating enough? Is no longer the question

Before continuous testing becoming the new norm in cloud-based developments (SaaS products), organizations’ primary objectives were to cut back on testing efforts, particularly manual testing. Therefore, the primary question that persisted throughout was, “Are we automating enough?” In the past, this was an effective strategy, but in the era of continuous testing, it needs to be reevaluated. Massive unstructured test suites that are the result of the “Automating Everything” effort become unmanageable in a matter of months and take far longer to maintain than the testing itself. Furthermore, organizations and their development teams cannot run them continuously as part of a structured process per fix per build and thus face the entire motivation of automating.

What has changed in the modern era, then? We must act quickly and concentrate on the tests that will yield the highest return on investment. This is a result of realizing that automating everything is just not practical. For instance, I used to have a simple checklist for my teams to use before deciding what to automate and what not to do. (After a few projects it was no longer necessary as they embrace it as an ongoing process).

Let’s check a few items from this checklist:

1. What is the value of the test from a business perspective?
2. What is the risk of automating/not automating the test?
3. Is it a repetitive test that should run per build/project?
4. Will it save Engineering time?

Challenge 3: Continuous testing is not just for testing

The idea that continuous testing is solely focused on testing is one of the most common misconceptions. This is not the case. Even though testing is at its core, there are several other areas involved in the process that we must concentrate on to make the most of continuous testing.

For example:

• Creating health dashboards related to each test suite.
• Executing test suites on multiple versions, platforms, and frameworks.
• The ability to re-run tests in case of failure (While keeping the failures logs, Dumps, and any other information relevant for debugging).
• Generating auto reports to relevant stakeholders using.

Challenge 4: False positives and False negatives

Be aware of false negatives and false positives. They are more frequent than you would ever expect! When you run a test automation script, it may report an execution failure even if the system is functioning properly, or it may report the test execution as successful even though the system experienced failures. While both are harmful, false positives can be more destructive since you assume everything is OK until there is an issue.

Consider your rollbacks wisely. Even if you follow all of the finest continuous testing and CI/CD standards, there is no assurance that the build will not crash in production. Be always prepared for the worst-case situation. Before you deploy changes to production, make sure your data is backed up. If something goes wrong, you can immediately roll back to the previous production version and do another round of testing to validate the application works properly again.

Beware that if you roll back from the production environment, you can’t simply confirm your pre-production state and commit the customizations immediately. You must examine the whole pipeline and all test environments to ensure that there are no loose ends since you do not want to delay the very same release cycle multiple times.

Let’s delve into the real-world experiences of testing professionals. In a recent social media poll we conducted, we asked the question, “What is your biggest challenge with test execution?” The responses from the testing community shed light on the common hurdles faced during test execution, which align with the need to optimize resource utilization.

This insights into challenges with test execution provide valuable guidance for overcoming obstacles in continuous testing, ensuring that resources are used efficiently.

Challenge 5: Inadequate utilization of resources

Organizations must make investments in resources, infrastructures, frameworks, and manual labor if they want a Continuous testing process that is effective and scalable. If it is not managed and tracked over time, this interment could be very expensive. For instance, a daily CD process utilizes a variety of servers (both physical and virtual), each of which consumes resources that, when added up over time, can be quite expensive. (This is relevant even more if you use cloud providers such as Azure, AWS, and more).

Engineering teams will be expected to ensure that they are not wasting these valuable resources on pointless job executions that fail due to inadequate coding and testing. This can be a major issue for startups or low-budget projects that cannot afford to waste money on unutilized resources. As a result, it is critical to keep a constant eye on the CD execution pipeline to identify and address areas where resources are being wasted.

Challenge 6: Security Vulnerabilities

Continuous testing has plenty of advantages for delivering products with great quality. Unneutrally, it is also a great potential for hackers to target your CD pipelines because there are always some security flaws that are not addressed well enough by the business. If it was not clear by now, the CD pipeline contains a massive amount of sensitive information that is related directly to the product which makes it even more attractive for attackers to try and get this data.

Today, we have some excellent ways to decrease this threat by utilizing various tactics that could be applied as a full line of defense:

1. Hiring external contractors to handle all aspects of the security testing process. For businesses that don’t have the internal expertise to handle this job, this solution is incredible.
2. Incorporate monitoring systems that track the entire pipeline to ensure that the basic security level is maintained at all times.
3. As a rule, we can reduce the amount of sensitive and restricted information that is transferred through code.
4. Secure code repositories by using different methods such as Two-factor authentication (to authenticate the author) or Secure local backup (in case of a complete hijack of the main repository.)

Challenge 7: The role and responsibilities of testers

The majority of problems we have encountered so far have been primarily technical or procedural, but what about software testers? What makes them a part of this world? What are their new responsibilities and roles? So, unless you plan to eliminate testers from your development pipeline, I advise you to take the appropriate steps to neutralize this potential time bomb. Let me explain, software testers are the ones that usually (less or more depending on the SDLC methodology used) make the tests or at least the test designs, so now in a continuous testing approach, we push more and more test activities to developers who test in stages (unit, integration, and so on) that testers simple cannot.

I frequently observed how testers struggled to grasp their new roles and what was required of them, which resulted in extreme frustration and fear for their jobs. To get around it, this change needs to be clarified in a way that lets testers know how they fit into the new SDLC and what kind of RNR is expected of them. For instance:

1. Track, Monitor, and debug (based on technical skills) automation results and communicate potential problems.
2. Perform Ad-Hoc tests using Risk-based testing and Exploratory sessions.
3. Managing defects during the release cycle.
4. Design test scenarios for major business flows.
5. And more.

Conclusion

Since I have been working in this field for almost 20 years, I have witnessed at least one or two major occurrences that have advanced our sector. Continuous testing, in my opinion, is one of them today because it interacts with other important changes like the rising use of cloud services at all levels (IaaS, PaaS, and SaaS), as well as the fast-paced society we now live in. Therefore, continuous testing ought to be adopted in some fashion, whether through technical means or by using particular elements of it.

David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile