Agile Test Planning – The Levels of Precision

A frequent misperception about Agile development is that it requires less planning or no planning; this misperception is mainly related to one of the core items in the Agile manifesto stating that we need to “Respond to change over following a plan.” While this sounds good in theory, it can’t hold water in complex software development projects (we can use it as a mindset and try reducing planning but cannot ignore it).

Agile teams tend to plan all the time in the different phases of the project; yes, it’s different from other software development methodologies where the planning is mainly done at the beginning of the project, while in Agile development, it’s done in smaller chunks, with focusing on what is needed to complete a feature, story or task. If you’re looking to improve your Agile interview skills, check out our curated list of Agile interview questions and answers.

In addition, the whole planning process is done with a clear goal to achieve more than just the planning plan; we use the planning sessions to generate fast feedback to learn and adapt, to make future planning sessions more effective and less time-consuming. These Agile planning efforts and strategies are relevant to all Agile activities and therefore used in test planning; we plan our tests in small chunks and short feedback loops. As the team plans to test, the focus is on what they need to know now.

A worth mentioning point for test planning is the size and complexity of the project. In small, experienced teams, planning for testing can be simple. They have the tools and knowledge to consider the type of testing, their automation requirements, and how much effort the team will take to complete the testing. On the other hand, test planning can get complicated in large organisations where each project involves multiple teams working on the same product. However, the basic principles apply regardless of the project complexity and size.

In Agile projects, we don’t know all the details about each Epic and Feature at the beginning of the sprint. Some may be predictable and straightforward because we’ve done similar work before, allowing us to understand the complexity, effort, and testing needed. Others may be more complicated, or they may be so new and unknown to the team that it’s challenging to come up with the information to handle the testing challenge. In Agile Development, we should use a different mindset from the one that assumes that we will have all the requirements upfront and where planning is based on that information.

Using different Agile frameworks, we can do exactly this by using the principles of iterative development and simplicity; we look at levels of planning and ask what information the team needs to know at each level. Different organizations will have different contexts, so you need to differentiate between what is essential to your team and organization and what is not.

We discuss this approach within the boundaries of sprints used in the scrum framework, but we can also apply the concepts to flow-based methods such as Kanban. Team using flow-based methods may have the possibility to release a working increment after the completion of each story, or they may accumulate the stories until a feature is complete and then released to production.

In the Scrum framework, most likely the most commonly used Agile framework in the industry, the organization, and its teams will have four primary levels of details that they will have to consider when planning while at the same time keeping the whole product in mind. I use the following terms for these different levels of precision for test planning:

Epic: One or more teams working on a single project releasing at specific intervals or on defined dates. An Epic is a collection of many features.

Feature: A feature is a collection of user stories representing the business capability of a piece of functionality that is useful to the business; A feature completion time can require multiple iterations to complete. A feature delivery may involve many teams working on different parts.

User Story: A small testable chunk of functionality created by the team (Non-Functional requirements) or by the product owner (Business Requirements). Regardless of the type of the story, it should be completed within a single sprint (in my teams, I set a limitation of 2-3 days). A story is a group of many tasks.

Task: A piece of work that’s part of a story, defined only by the development teams and taking less than one day to complete.

Let’s explore how you may adapt your test planning to each level, as well as discuss the level of documentation, level of risks, and artifacts that might be expected at each of the four levels I shared above. For connivances, let’s assume we have a three-month product release (a yearly quarter) with two-week iterations using cross-organization. During this time, the teams are expected to deliver multiple features that each team is responsible for delivering in the release cycles.

At the Epic level, teams should already have a good idea of the product vision and its goals to answer the questions of “Are We Building the Right Thing?” The high-level test approach should cover the main testing aspects of the product because this is the time when you are planning what will be in the release scope. Due to the importance of planning at this level, I always ensure that my teams will have a simple checklist of questions that they can follow to ensure they run an adequate test planning session, for example;

• What are the test environments you will need?
• Are there any new technologies that the teams will have to learn before starting the project?
• Are there any new tools that we should purchase?
• What types of test types you will use.
• What is the planned coverage (Automated Vs. Manual)
• What will be tested during the different phases of the release pipeline?

These activities may involve the help of more departments and groups outside your delivery team, such as marketing, Support, operations, and more. For example, you may need the system team to run endurance and security tests that the team will not cover during development or ask the finance teams for a budget to purchase new hardware/software that the teams will use during the testing process.

Epic

A product delivery cycle of a Single-Team
When your team is working on a feature/story, they are responsible for completing all testing activities (Design, Documentation, Execution, and Analysis). The team may still need external help, such as security experts, but the team is accountable. When you have a single responsible team, the meaning is that the project is usually tiny and is limited to only a few iterations.

A product delivery cycle of multiple-Teams

Having multiple teams work on the same project can (and usually will) add complexity. Therefore, an overall test approach should consider dependencies between code parts developed by different teams, testing only their components or features associated with the project. As you may guess, test planning at this level is crucial for achieving good results once the teams will start to develop and test the features and stories associated with this project.

One practice I usually do in my projects is to create project “maps” that help all stakeholders in the project; these maps are typically created in a “Mind map” tool which helps visualise the different aspects involved in the testing effort. Also, when creating these “High-Level” maps, I involve vital stakeholders who understand the big picture and how this release integrates into the system.

Below are a few examples of the lists you can use at the Epic level:

• Map of owners and responsibilities per area of testing.
• Map of all dependencies outside the product teams or between features.
• Map for all risks related to each testing effort and mitigation plan.
• Map of Test environments and Tools that the team will have to use in the project.

Creating these maps will contain all the teams’ information to work on their features. It continually allows them to look at the big picture and aim to find possible integration issues early and before the testing effort t is started. This will increase communication and reduce the risks.

Consider the investment return for all the time you can save by helping your teams start working on their features while knowing what challenges they may have and consider it part of their test planning efforts. For example, teams with many dependencies with other teams will know that they will need to work together on their code and test the integration between the different components.

If it’s possible and you have enough information with the right group of people, you may think about creating a product release test plan before the start of delivery cycles; I recommended keeping it as lean as possible and straightforward as you can. Focus only on the high-level aspects of the test plan (the teams will do the detailed test planning). Consider who the audience is and what is essential to know before creating their test strategy and detailed test plans. Also, at the Epic level, test planning should include specific topics I already described above, such as: Determining the test strategy and identifying testing risks.

Before moving forward to the feature level, the last thing worth mentioning is that teams may not have enough information to create a solid test plan. Therefore they may decide to defer test planning until after any necessary investigation (primarily by using spike solutions) and more about complex features.

Feature level

Features are work units driven from their Epic representing some business-value capability broken up into user stories. We need to make sure that these stories (functional and sometimes non-functional stories) Are testable so that the team involves in the project gets into a rhythm of coding and testing each story until the complete feature is “done.” Remember, a feature is a set of stories, and sometimes there can be many of them. Each delivery team concentrates on a particular feature without always knowing what other teams are doing; this uncertainty increases when multiple teams are involved (I used to work on a project with more than 50 teams working in parallel). There may be dependencies Between the teams.

If we take Scrum, for example, each scrum team works with its product owner (PO) to determine how much work the team can complete during the delivery cycle base on their capabilities, capacity, etc. in a scrum, the team will hold a planning session so the team can understand the goals they need to achieve and working with the PO to understand what are the stories they need to deliver base on their priority (set by the PO). And what about the testing activities? This is where testers usually add their value by asking clarifying questions that may help the team determine the size of the feature or story to understand the actual effort involved in delivering it.

Testers can also help the team consider the impact on other system components and issues while integrating the code with other parts of the system. Also, they promote a testing mindset to what security or performance concerns there might be. As I mentioned at the Epic level, feature-level testing involves different assumptions and risks that need to be monitored and documented in a dedicated repository (Wiki, mind maps, Confluence, SPO, etc.) visible to engineering and business stakeholders relevant to the project. Watch how the teams use this information, and ensure you keep improving it using retrospectives and other dedicated meetings to ensure you are using the correct format. If there are communication gaps, look for the best fit for your situation.

Sometimes, Mangers or clients will demand a formal test document outlining what your team will test in the delivery cycle. Before you create a formal document, I recommended that you write it in a way that allows other non-technical stakeholders to understand it and keep it as simple as possible (One or two pages without getting into details of scope unless explicitly requested). You must think about who will use it and why they want it. In my projects, if this document is not asked for, I still try to ensure that any testing risks will be written and shared with anyone interested (my preferred option is to create a mind map capturing those risks and any other information I think is essential).

When it is time to break the feature into stories, I work with my team’s product owner to create high-level acceptance tests, using specific examples of expected behaviors and off-course misbehaviors for the feature. Following this approach, you can help your team focus on the proper scope of work and keep the business value as high as possible. Try to use a mind map as much as possible (it will be worth it, this I can promise you) for creating a testing strategy for the feature with your whole team before they will, break it into smaller parts that will be executed as part of the iteration’s stories. It is one way to start a discussion and allow all team members to contribute and reveal hidden gaps or potential issues before they arise.

As you probably know, planning testing activities for extensive features containing multiple user stories that cross iterations can be complicated. To help the team, I instructed them to add a dedicated “Features testing” story for these. The story will contain tasks related to testing activities, such as “Creating automation Strategy,” “Set Test Environments,” “Explore the feature,” “Define testing efforts,” and more. This type of story ensures that the feature testing is done and calculated as a team effort in a given iteration.

Story Level

Once we set the test plan at the feature level, the team will start working on the story level, moving from the “Macro” to the “Micro.” At this level of precision, we need (per story) to set high-level acceptance tests: an example of expected requirements and at least one example of misbehavior that defines the story’s scope and the necessary tests the team will have to run to validate it. The planning for the story levels happens during story reediness sessions (sometimes called refrainment or backlog grooming), which help the team create tests for the story and can be expended during the iteration planning sessions.

So if want to simplify my suggested process, we will use the reediness sessions to start writing test ideas based on the information and knowledge you already have at the feature level, continue during iteration planning, and then expand it to other tests that will guarantee that the required functionality works as expected.

Task Level

When thinking about testing activities at the task level, there are fewer planning and execution-related activities, such as creating test data, setting up environments, writing unit tests, and more. Some teams estimate tasks during iteration planning, but I find this a waste of time as the estimation should be only on the story level. However, this can be useful for teams that are new to Agile. When the iteration starts and tests are executed, make sure to call attention to it during daily standups by reporting progress or asking for help if the testing task takes significantly longer than anticipated.

David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile