Agile Development Best Practices for Determining What to Automate

Agile teams will not prosper if they don’t use frameworks to automate various development processes, including tests, builds, test environments, etc. Understanding that the primary goal of all agile teams is to successfully deliver real value to the customer ,which can be interpreted as providing functional software after the sprint, is necessary to comprehend why a lack of automation results in failure.

In this article, I will address the central questions of “What can and what we shouldn’t automate in agile projects”.

What can we automate in an agile environment?

Most types of testing benefit from automation, starting from basic unit tests through system tests. But, as you know, unit tests don’t supply enough coverage to catch system regression failures. Running a set of manual tests before every check-in, which could be dozens of times a day, isn’t practical. When developers can’t run tests by pushing a button, they will not be motivated.

Let’s talk about the kinds of tests that are most suitable for automation. Automation starts with an automated framework that allows developers to check their code often and receive quick feedback about its impact. So let’s start with this first.

Continous Integration (CI) system

This is where we create the most important, albeit logical and straightforward, ground-rule for automation. Due to the nature of agile software development, which is faster than any other approach, agile teams should focus on automating any repetitive or tedious work involved in developing software.

And no candidate is better for this than automating build creation as part of an agile development process. Due to the fast nature of agile development, the team should create numerous builds per day, especially to test newly added code.

CI systems are crucial in an agile environment. Continuous integration and build processes are the two systems that give the most significant ROI of any automation effort:

• CI allows for immediate feedback at the unit-test level (if you have the relevant unit tests to support it).
• It reduces many of the risks involved in adding new code without testing it.
• It allows the team to create and deploy numerous (stable) builds and provides for multiple check-ins per day.
• It improves communication because team members can receive a notification once the build is ready without checking the status.
• CI systems speed up testing time by reducing the number of errors at the unit level before these errors become apparent in advanced phases of the testing process.

Based on the above, agile teams must implement continuous integration and build the framework as soon as possible. Although it requires continual maintenance, it’s the only option for agile teams to succeed and reduce technical debt in large complex projects.

Based on these simple facts, you may see that agile teams must implement continuous integration and build the framework as soon as possible. Although it requires continual maintenance, it’s the only option for Agile teams to succeed and reduce technical debt in large complex projects.

Development and Test Environments

Agile teams need to test and develop in a fast-changing environment; as a result, there is less room for the creation and maintenance of work environments. Agile teams can use the automated deployment of their environments without multiple hours of manual work. In addition, the team can use automation to handle many other areas related to their work environment:

• Creation and cleaning of the testing data and configuration.
• Setup of specific topologies and architectures.
• Simulating a particular scenario for reproducing a bug.

Testing of the User Interface (UI)

The agile development process embraces the approach that the team must deliver an incremental working functionality at the end of each iteration; as a result, the team will usually execute basic automated regression tests at the GUI level.

As I mentioned earlier, I’m a great believer in automated testing. Still, in some cases, we need to consider whether we want to use it, especially when we want to test the user interface of an application whose GUI changes.

To overcome the challenges of GUI testing, there is of great importance in selecting the most suitable tool for the job, one that’s easy to maintain and flexible enough to absorb changes. This is probably the most critical key to successful GUI automation.

Testing all layers of the application

I’m a great believer in automated solutions that can reduce manual testing efforts to the bare minimum necessary. It starts at the first layer of the application by running unit tests that we all agree are crucial in reducing problems that won’t become more significant problems later when found in that layer of testing.

Next, we have the second layer of component tests. Programmers test components as a whole module by using a set of inputs and outputs that the module produces. The third and, for me, the most crucial part of the testing strategy is integration tests, where modules are tested together as one suite. And if that is not enough, why not test the whole system by running the fourth layer of system tests, which test the entire application as an entire system.

Performance, Load, and Stress tests

Suppose you’ve ever been involved in the testing process that included one of the testing mentioned above types. In that case, you probably know that it’s almost impossible and undoubtedly ineffective to use manual testing methods as the preferred way to run them. Furthermore, there is a wide range of tests that you cannot run without automation tools. In addition, using manual tests will not provide the accurate test results we can achieve by using dedicated automation tools that can simulate the exact scenario without any human interference that may affect the testing process and, therefore, the results.

What shouldn’t we automate..?

If you are familiar with how I approach testing, you are probably already aware of my strong support for automation testing frameworks that enable the team to develop a more effective, efficient, and dependable coding and testing process. However, some parts of the testing procedure still require the intelligence, common sense, and eyesight of humans.

Usability Testing

Usability testing is very different from other test types that determine the quality of the software. It cannot be automated because it requires someone to work with the software to determine how it was experienced and where the gaps are in the user experience.

GUI Testing (Is it worth the ROI?)

GUI testing is one of the most challenging areas to automate. I’ve seen just too many organizations that invested thousands of human hours in automating the GUI of their products but, in the end, found it was a waste of time that didn’t provide the expected ROI. Some GUI tests can be used to ensure there are no unexpected changes in the GUI. Still, you should ask yourself whether it’s worth the costs and investment instead of improving other quality issues that will provide a better ROI and reduce the risks in that area.

Tests that are not worth the investment

I used to be involved in an automation project where the test team automated a thousand provided tests (at least on paper). So what is wrong here? They automated almost all the available test scenarios without really thinking about the ROI.

The team invested weeks in automating tests marked as low risk, tests that would never fail, and tests whose failure had a meager chance of impacting the software. The entire automation process was based on the spirit of “let’s automate everything” instead of asking the simple question of the ROI that this automation project provides.

In some cases, some tests are written without real thought as to whether they are essential or not. Once the automation project starts, the team will automate these tests just because they never want to rerun them (because they know there is a 0% chance that it will make a difference).

Tests that need to be executed only once

The main goal of automated testing is to allow the team to focus on essential things in the software development lifecycle. Automating test scenarios that will run only once are not worth the team’s time to invest in the design, creation, and execution of these tests.

Exploratory Testing

In my opinion, exploratory testing is the best and most efficient method that agile teams use in any testing process. Exploratory testing can be used for learning purposes (you learn more about the software when testing it) or to provide a fast way to evaluate the overall quality of the software. However, when it comes to real testing effort, a skilled tester must design and execute tests.

Exploratory testing should be done by humans and not by automated scripts because automated scripts will not let the tester take in new information he generated from the exploratory session and use it to improve future testing and development processes.

In addition, although exploratory testing is a great testing approach, there is a real need for automated tests that will allow the team to focus on their exploratory sessions without worrying about any regressions that automated tests should cover.

David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile