Accelerate Test Velocity: Key Strategies for Faster Time-to-Market

In today’s fast-paced and competitive business landscape, slow time-to-market goes beyond just a delayed product release. When an organization takes longer to introduce its offerings to the market, it not only loses valuable opportunities but also exposes itself to the risk of being outshone by competitors who excel at swift delivery.

However, focusing on faster time-to-market alone is no longer sufficient to stay ahead of the competition. It is essential to recognize that the true challenge lies in outpacing competitors and delivering value to customers before others can seize the opportunity.

As a result, companies have turned their attention to accelerating test velocity as a means to expedite their delivery cycles. While many organizations have made strides in improving their testing processes by focusing on running more tests faster, they often reach a plateau where further enhancements are necessary to achieve true acceleration and stay ahead in the market

Quality Engineering transcends the notion of simply running more tests faster and delves into building a robust and productive software production system. It recognizes that speed is not an isolated objective but rather the outcome of a holistic approach that prioritizes quality throughout the entire development lifecycle.

Start by defining success on three perspectives

When striving to accelerate test velocity and improve time-to-market, it is common to focus solely on optimizing test execution and infrastructure. However, this approach can lead to time wastage if the desired goals and outcomes are not clearly defined from the beginning.

To avoid this, it is essential to begin by defining what success looks like in the context of the project or initiative. To achieve success in accelerating test velocity and achieving faster time-to-market, it is crucial to start by defining success from three perspectives: outcomes, KPIs, and metrics:

• Outcomes: These are the tangible business values generated by specific software outputs, such as the introduction of new features. It’s important to note that outcomes are not solely measured in numbers, but they encompass the overall value and impact on the business.
• KPIs (Key Performance Indicators): KPIs are measurable numeric indicators that provide insights into the business value from various angles, including customer experience, financial performance, and operational efficiency. Examples of KPIs can be NPS (Net Promoter Score), churn rate, or contact per order.
• Metrics: Metrics are supporting measures that help track the progress towards the desired direction. They serve as benchmarks for implementing new processes and ensuring that the ecosystem is moving forward. For instance, a metric could be the execution of test campaigns within five minutes after each code build or the lead time from defect identification to resolution.

In our paradigm of Quality Engineering where “Quality brings Speed”, the two main areas of success are:

1. Speed outcomes and KPIs: These metrics will measure the true acceleration of cycle time, delivery time, and overall stability to prevent rework and ensure a smooth development process.
2. Quality KPIs and metrics: These metrics assess the extent to which the focus on building quality is progressing and how it contributes to speed indicators simultaneously.

That stage of success definition is the foundation to align your effort to accelerate test velocity and time-to-market. The next step is to identify the key issues that limit the speed and efficiency of software development in regards to our definition of success.

Identify more than software limiting factors

Software limiting factors enable us to gain insights into the specific areas that hinder performance and impact outcomes. They are fundamental to the concepts of value-stream and lean techniques to unlock the performance of a given system.

This stage is crucial to prevent scattered priorities and requires effective analysis, critical thinking, and problem-solving techniques. Employing methodologies like impact mapping can be valuable in achieving this objective, as they provide a structured approach to identifying and addressing the key factors that limit software performance.

Limiting factors help us filter and prioritize problems by identifying the specific factors that impact the overall performance of a system. By focusing our efforts on these key issues, we can effectively address the root causes of performance challenges, especially in complex software delivery processes where multiple factors may contribute to certain symptoms. This approach allows us to allocate resources efficiently and tackle the most significant issues that are hindering the system’s performance.

Gaining a holistic understanding of the entire system is necessary to direct our efforts towards the most impactful areas. Focusing solely on the test process, such as the time taken for execution is not the priority if defining acceptance criteria takes twice as long, surpassing purely technical concerns.

Design not only the test, but the supporting system

A well-designed supporting system is essential for effective test automation, execution, and maintainability. Neglecting upfront design often leads to software that is overly complex, inflexible, and costly to maintain. Investing in test design is necessary for accelerating test velocity and time-to-market, but it goes beyond designing the tests themselves.

The true investment in test design following our philosophy of Quality Engineering is that the design is not limited to the test: we also need to address the supporting forces of the software production system.

The system can be seen in five domains to inject quality:

• Methods focusing on the collaboration and processes
• Architecture including technologies and infrastructures
• Management to animate the team and drive forces
• Organization to structure investments, teams and interactions
• Skills to make sure the expertises is acquired.

One example is to systematize new methodologies like BDD as part of the definition of user stories, aligning the roles and organizational responsibilities between the product owner, developer, and quality roles for such tasks to stick in the long run.

Another one for test environments management is to consider technology to provide repeatability and automation, but combining it with Devops and FinOps methods to make them scale only when needed as part of the delivery pipeline, instead of keeping all non-production environments always on.

Automate what may stay, or is staying

The temptation to automate everything during iterations is strong, as it eliminates the need for future consideration and becomes a part of recurring executions. However, test automation comes with costs for development, execution, and maintenance, requiring careful consideration before implementation.

A practical approach to assess the need for automation is to assess the:

• Value per automation usage.
• Cost per automation usage.

The value per automation usage extends beyond cost reduction and includes benefits such as process reliability, enabling 24/7 execution, or stability even with team turnover. On the other hand, the cost per use helps determine the frequency at which a particular process will realistically occur, allowing for an initial assessment and continuous review of whether automation should be implemented or maintained, as some tests may be heavily utilized at a given time, their relevance may diminish as the product evolves.

If the need for automation is confirmed, automation should only be initiated only after the process has been executed manually or in a hybrid automated manner a few times. This approach enables rapid improvements based on the lessons learned from initial trials. As Bill Gates famously stated, “automating an inefficient process will only amplify its inefficiencies.”

Streamline collaboration by minimizing hands-off

Minimizing hands-off is a crucial requirement for achieving speed, as it aligns with the principles of lean methodology. Each hands-off point in the value chain introduces disruptions such as information loss, changes in personnel, and potential errors, all of which hinder test velocity and time-to-market.

Tests often experience these disruptions, whether it’s between product owners, developers, and quality professionals in terms of acceptance criteria or defect management, or between operations and quality professionals regarding observability and monitoring.

Three key areas require focus:

1. Collaborative test management ensures comprehensive test definition.
2. Integrating a continuous testing infrastructure into the software delivery pipeline and processes enables efficient test execution.
3. Collaborative portals that combine requirements, execution, and defect management capabilities are essential for comprehensive test reporting.

Shared way of working and an enabling automation mindset for the definition of success are fundamental to that collaboration streamline.

Build system capabilities over project activities

Large projects pose challenges in terms of uncertainty, risk, and adaptability, which is why breaking them into incremental pieces of delivery is recommended. However, this approach can result in higher setup costs for each increment, hindering fast time-to-market.

To overcome this challenge, a shift in the mindset of Quality Engineering is essential. It requires investing in the software production system beyond individual projects and incremental deliveries, treating test design, environment setup, observability, and other test velocity requirements as system capabilities rather than project-specific activities.

Embracing the paradigm of platform engineering and quality architects enables teams to deliver and learn at a rapid pace, leveraging system capabilities for iterative and efficient development with a focus on quality and speed.

Implement tests at the lower execution cost

Failure to consider the costs of test execution from both a test and business perspective can lead to significant challenges. It is important to understand why, in relation to the test pyramid, prioritizing lower-level execution tests is preferred.

First, there is the notion of test frequency with 4 options:

1. No test executed
2. Unit test as part of the build
3. Other tests as part of CI/CD
4. Other tests on a continuous basis like monitoring.

Then the notion of costs which is not only what to pay to execute the tests but the overall costs to build, run, and maintain as an organization including technology costs. Think of your tests at the road toll that you can take to go faster in some cases, and sometimes only for comfort when you can even be slower. Your business doesn’t need that.

Time-to-market and Test Velocity with Quality

Accelerating time-to-market and test velocity with quality requires to go beyond a narrow focus on tests alone. Organizations must actively build a software production system that enables fast iteration and embraces quality as a core principle.

By investing in test design, identifying software limiting factors, devising effective automation strategies, and minimizing hands-off points, companies can optimize their processes and achieve faster time-to-market. They should also consider the comprehensive concept of test execution costs, including direct expenses, organizational implications, and technological considerations, to make informed decisions about testing strategies.

Time-to-market and test velocity requires an active approach where organizations must shift their mindset to embrace Quality Engineering. The success is sustainable speed powered by Quality in the overall software production system.

Antoine Craske

https://qeunit.com

Passionate about digital, architecture, transformation with more than ten years of experience in the software industry in different positions as engineer, project director, engineering director, IT transformation, management and architecture.

Founder and organizer of the QE Unit, the Quality Engineering community. Director of Architecture & Technology at La Redoute, Co-Founder of Cerberus Testing, Partner at atale.io. Meetup organizer at TICE.Leiria, Ministry of Testing Leiria.

See author's profile