Rethinking Software Testing Metrics: An In-Depth Analysis

There is just one area where each member of the software testing community has a distinct point of view! Metrics! This contentious issue sparks intense disputes, and most conversations finish with no definitive conclusion. It covers a wide range of topics: How can testing efforts be measured? What is the most effective technique to assess effectiveness? Which of the many components should be quantified? How can we measure the quality of our testing performance, among other things?

As we strive to find out the optimal protocol, the metrics question has always concerned me. I researched professional literature, participated in research conferences and spoke with professionals in the field. I’ve even asked consumers to submit feedback on how they view its execution and what metrics they’d like to see. However, I have yet to propose the final metrics solution.

The truth is that metrics are subjective

As it turns out, there is no such thing as a perfect practice. The harsh reality is that testing metrics are subjective and must be established on a per-company or even per-project basis. When considering testing metrics, keep the following considerations in mind:

• What frameworks, tools, and processes will be utilized to collect data for metrics?
• Is there any prior metrics data to compare? Do we want to make a comparison?
• Are we more interested in the process than in the ultimate result?
• Who is requesting the metrics? And what does he want to accomplish?
• How many departments and teams are working on this project?
• What metrics should you use (test, project, quality, and so on.)?
• What, if any, software development process (SDLC) is used?
• How many customers are waiting for the product?
• What is the scope and complexity of the project?
• What are the quality standards for this delivery?
• Is it a stand-alone project or a sequel?
• Is there a business measure of success for the project, and how does it impact the testing stage?

The list goes on and on. And the answers vary from one business to the next. I have no intention of covering various measurement alternatives or dealing with computations, data collection, application methods, and so on. My main goal is to promote awareness of one topic that all software professionals should aim to answer honestly: Are you leveraging metrics to enhance software testing, team efficiency, and performance? Or have measurements become a target in and of themselves, influencing the way you work?

Metrics might cause you to chase your tail

I am certain that the act of evaluating—collecting particular data pieces and developing measurements and criteria—frequently influences people’s behavior and, as a result, the ultimate results. When something is measured, people instinctively think it is important and work to improve the outcomes. It is human to desire to succeed, and we will go to great lengths to ensure that our figures are correct.

This underlying objective is frequently counterproductive and has a detrimental influence on the whole operational process. Some team members may focus only on their individual goals, ignoring, even unintentionally, the broader demands of the team and project. As an example, a security force that is rewarded for increasing revenue may focus its efforts on issues as many parking fines as possible at the expense of pursuing criminals.

This also holds true in the field of testing. Unfortunately, many businesses use a basic technique that measures a tester by the number of defects identified. However, the major reason that businesses choose this strategy is that it is usually assumed that the tester’s duty is to detect defects. Managers are persuaded to believe that quantifying mistakes will offer a good overall view of task quality since more defects found early would result in earlier corrections and a better final outcome.

In practice, though, the opposite may happen. This measurement may motivate testers to try to raise the number of defects simply by submitting more minor defects, breaking possible issues into many categories, and bypassing the effort required to uncover bugs that require a thorough and time-consuming investigation. In this kind of circumstance, the tester may devote less time to reporting the defects in depth and instead focus on detecting new defects. This might have an impact on development or product release. As a result, developers will have to deal with more problems, many of which may turn out to be non-issues or minor ones.

How to decrease the human factor by selecting the right metrics

Instead of measuring people, measure a process or a result.

• Metrics are useful decision-making and management tools, but they should not be used in place of human judgment or gut instinct. Not numbers, but your instincts and team comments should be trusted.
• There are several approaches to dealing with the “human factor”:
• Measure things towards the conclusion of the process (it is ideal to measure process results), not at the beginning.
• Beware from employing metrics to follow up on individuals or, worse, to measure them.
• Confirm that the group understands the reasoning underlying the measured data and obtain their support.

After you’ve decided on your measurements, you may put them to the test against a set of basic conditions. These factors, I feel, can contribute in deciding if the metrics are effective.

• What effect does the measuring team have on meeting the goals?
• Can the team change the process in such a manner that it affects the index?
• Do you have a strategy for what you’ll do after you obtain the metrics results?
• Does this metric capture significant business data?
• Is this statistic used to assess the outcome of a process?

Let’s look at an example. Setting business objectives, which assess the efficiency of the testing team; will be established by the number of defects detected in the product within the first quarter post-release, which is one viable way for a testing team. The logic behind it:

• To attain these measurements, the testing team cannot alter the process.
• Plans to enhance certain aspects of the project process, including testing, may be quickly devised as necessary.
• Generally, having fewer production defects is a key aspect of the overall success of the company.
• It assesses the outcome of the testing process.
• The testing team is not the only entity accountable for this outcome, but it has a significant effect on it.

Of course, this metric overlooks factors like testing inefficiency, testing duration, and even the effect of tester quality on the number of defects. Other forms of metrics, if properly constructed, can encompass other similar aspects. I feel that measuring work is vital because it provides tools for evaluating the quality of the work done and the output, as well as for improving procedures and outcomes. It should, however, be properly examined.



David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile