What critical technical skills must CXOs drive in their organizations to successfully implement continuous quality testing?

Continuous Testing is the process of executing testing as part of the software delivery pipeline to obtain feedback on the business risks associated with a software release. From a CXO perspective, continuous testing is simply about ensuring the product can provide a delightful customer experience.

The crucial role of a CXO in driving continuous testing across the organization is to define a vision for continuous testing tailored to its customers’ success through building a quality product. Vision statements sometimes can become very generic and high level resulting in the broader organization failing to connect and resonate with it; continuous testing vision is no exception to this. The result of such disconnect needs to be better implemented through continuous testing that fails to unlock the value initially envisioned by the vision statement. A typical vision for continuous testing should articulate how the end customer will be able to derive value out of the product being built continuously and how testing can help with that value creation. A typical vision statement that CXOs can formulate would be as given below.

To continuously build quality into our Software Development Lifecycle and ensure an exciting customer experience always

Evaluating the current state of continuous testing

A recent study by Forrester indicated that only 26% of non-CXO respondents reported that their organization was following all the best practices of continuous testing, whereas 57% of CXO believed that they were following Continuous Testing best guidelines already. This highlighted that CXOs had a highly inflated estimation of their continuous testing maturity. This study further reveals the necessity for CXOs to assess their organization’s continuous testing maturity independently.

As a CXO charted with the mandate of driving continuous testing skills within their organization, one of the key prerequisite actions is to evaluate the current state of testing and continuous quality practices that are prevalent within the teams. In a typical agile implementation, teams tend to become siloed, and this results in inconsistencies in the maturity of the continuous testing practices that are currently existing. A thorough assessment of the current state of continuous testing can help CXO understand the as-is maturity and then, based on the To-Be state, comprehend the gap that needs to be bridged. This cognizance of the gap between the as-is and to-be states can help CXO enable their hiring managers to bring in the right skills to bridge this gap. Some of the key parameters to evaluate on a continuous testing assessment would be.

• Skills for continuous testing
• Hiring Strategy – Tech skills
• Training plan
• Test strategy and plans
• Test Estimation
• Test management
• Test automation
• Test tools
• Test Data Management
• Test infrastructure management
• Test Reporting
• Levels of Testing
• Unit Testing
• System Testing (ST)
• System Integration Testing (SIT)
• User Acceptance Testing (UAT)
• Test in production
• Leadership
• Culture

The continuous testing assessment is a questionnaire structured across these different parameters. The assessment questionnaire is surveyed across multiple stakeholders, and based on their responses, we assign a score of 1 to 4 across each of the questions. The results of such an assessment can objectively point to CXOs on the current state of continuous testing, where the gap is, and what needs to be improved.

Skills for Continuous Testing

Skills required for continuous testing can be categorized into three broad groups:

• Technical skills

The technical skills required for a professional to implement continuous testing for an organization are these foundational skills that aren’t tied to any specific tool but more to ground-level architectural capabilities expected of engineers implementing continuous testing.

• Test Data Management
• Test Environment Management
• Service Virtualization
• CI/CD fundamentals
• Test Automation Architecture
• The architecture of the Software under the Test
• Soft Skills

While this blog is about the technical skills required for continuous testing, it is imperative to note that any discussion on continuous testing would be incomplete without mentioning the soft skills required for a successful implementation of continuous testing.

• Strong analytical and problem-solving skills
• Communication
• Being curious
• Exploration mindset
• Tooling Skills
• Jenkins/TeamCity/AzureDevOps

These are some of the popular CICD tools that test professionals need to master to implement a good continuous testing strategy.

• Jmeter

Apache Jmeter is an open-source load testing tool. It is designed to measure the performance of websites. This load-testing tool can be used in DevOps methodology.

• Selenium/Playwright/Specflow

Functional UI test automation is a key aspect of implementing a continuous testing strategy, and these are some tools that can aid in automating the functional flow of applications built in Java & .Net.

• Postman/SoapUI

API tests are lightweight and much more effective in providing quick feedback compared to the bulkier E2E UI tests. Some good API testing tools that automation engineers can utilize would be Postman and SoapUI.

• Appium

Appium is an open-source automation tool for mobile applications that allow users to test native, mobile, web, and hybrid apps. It also supports automated tests on emulators and simulators.

• Vagrant

Vagrant is a DevOps testing tool. It is used for building and managing virtual machine environments in a single workflow. It offers an easy-to-use workflow and focuses on automation. It also reduces development environment setup time and increases production parity.

• PagerDuty

PagerDuty is a DevOps tool that supports incident management and provides real-time alerts, visibility, and monitoring for business-critical software systems.

• Docker

Docker allows DevOps teams to build, ship, and run distributed applications. This tool enables users to assemble apps from components and work collaboratively.

• Appverify

AppVerify is one of the best tools for a continuous testing solution for business applications. AppVerify allows you to test the entire business flow, including all third-party apps, without adding plugins or writing a single line of code. It facilitates continuous testing for Agile and DevOps Teams.

Pitfalls in hiring and building technical skills for Continuous Testing

The most common pitfalls teams encounter while building and hiring skills for continuous testing are

• Hire for specific tools, thereby building a team of tool operators
• Hire before you have a continuous test strategy
• Hire only developers for continuous test implementation roles
• Focus less on core testing skills and more on tool-based skills
• Not having a structured skilling framework to upskill internal talent

Knowledge Management framework

An efficient knowledge management framework is key to implementing a robust continuous testing strategy within the organization. Reskilling and Upskilling are essential tenets of the knowledge management framework. Reskilling is needed for the professional recycling of skills. Specific to the testing world and in the context of continuous testing, this is getting the traditional QA engineers to be skilled in continuous testing. Upskilling is the process of learning new tools and skills as they emerge.

The 5-step skilling framework that CXOs can leverage to train their teams

1. Business Goals. Based on your organization’s strategy and vision, identify key business goals that will help the organization to reach that vision.
2. Identify Skills. Based on the goals identified in step 1, identify skills needed by the teams to achieve the goal.
3. Measure skills. Create assessments to assess the existing skills of employees.
4. Identify Skill gaps. Assign scores based on the assessments and identify gaps based on the scoring pattern.
5. Create a plan. Develop a skilling plan based on the need to upskill or reskill, identify specific learning tasks for specific skills, and attach milestones to these tasks.

Building a Knowledge Matrix

A knowledge Matrix is a tool that CXO can use to map the technical skills of employees to their level of expertise in those skills. It can be used to manage, plan and monitor existing and desired skills required for continuous testing within an organization.

Advantage of Knowledge Matrix

• Help to hire managers to identify gaps in skills and hire accordingly
• Ensure that the team has a good mix of people with the right technical talent
• Help to promote employee development
• Track high performers and employees with key skill dependencies within the team

Steps to Create a Knowledge Matrix

• Identify the skills required for continuous testing

Identify the technical and soft skills required for continuous testing. A good way to do this is to ask more experienced employees about it.

• Create a scoring framework

A scoring framework ranks the user’s competence and skill level. An example scoring system would be to score on a scale of 1 – 4. Each of these scoring levels is attributed to a scoring descriptor to identify the level of skill that the person is being graded on.

1 – A Person with basic knowledge and understanding of the skill but still needs to be applied to his work.

2 – The person might have applied the skill but might still need support

3 – The person has experience utilizing the skill and can work independently. solve problems proactively

4 – A person with lots of experience in the respective skill can coach and mentor junior people in the team on this skill.

• Evaluate Skills

Skills can be evaluated by having teams undergo skill assessments and certification programs through which we could asses skills and assess employees. CXOs must be cognizant that a specific budget must be planned and allocated for the skilling and certification program.

• Reporting and Insights

Based on the data collected through the knowledge index, we could draw insights and value such as the most skilled person within a team and those that need additional training and support to bring up their skill level. The reporting insights help identify skills that might be lost if specific employees leave the company, and these are cases in the knowledge matrix in which only one person within the team is available on a particular skill.

A sample Knowledge matrix is given below.

The balance between Tools Vs Skills

There is always a false sense of focus on the tools required for continuous testing, whereas the core technical and functional testing skills are overlooked. There needs to be a good balance that CXO need to drive in their skilling program between the need for training on tools and the training required for engineers to nurture and grow their core technical and functional skills. CXOs need to understand that any amount of powerful tools in the hands of an engineer with poor technical knowledge becomes powerless and, in effect, counterproductive.

Technical Skills – Capacity planning for continuous quality

When faced with the need for staffing the team with continuous testing skills CXOs are often posed with the dilemma of whether to find/hire this talent in-house or outsource to external vendors. There are pros and cons to both of these approaches and a CXO need to weigh the pro and cons and take an optimal approach. The below list can be used as a guiding principle for CXOs to make an informed decision on what activities of continuous testing should be done in-house and what needs to be outsourced to vendors.

In-house Continuous Testing Activities

• Strategy for Continuous Testing
• Tool selection & consolidation
• Continuous Testing Architecture – how it integrates into global DevOps strategy
• Test Automation Framework Design
• Coding Standards and Guidelines

Outsourcing Activities

• Test Automation script creation
• Test data creation/extraction
• CI Integration
• Reporting
• Implementation of Monitoring and alerting

An ideal ratio between In-house and Outsourcing capacity can be 60:40 but this can vary based on the specific context and complexity of the continuous testing implementation.

Metrics to Measure the Success of Continuous Testing

The ability to measure the progress of a continuous testing process at each stage of the application delivery life cycle is vital to launch quality software with minimal business risks. Some of the key technical metrics that can be used to measure progress for continuous testing are as below.

In addition to the above metrics, there are also the metrics around customer success which, when measured along with the above technical metrics, can provide valuable insights to CXO on the success of the organization’s continuous testing implementation.

Roles in Continuous Testing

Some of the key roles in continuous testing that CXOs need to plan for are listed below. However, these roles often overlap with each other for example, a Test Architect can also function as a tool smith.

• Test Architect
• Toolsmith
• Test Automation Engineer
• Non-functional test expert (security/performance)
• Test Data Management (TDM)

C-level executives need to appreciate and understand the value of the skills that each of these roles brings to the team.

Conclusion

In conclusion, continuous testing has become a critical component of modern software development, and CXOs play a crucial role in driving technical skills to implement and maintain it successfully. By understanding the importance of continuous testing and its impact on overall product quality, CXOs can ensure that their teams have the necessary technical skills and resources to implement continuous testing effectively. This includes investing in training programs, fostering a culture of continuous learning and improvement, and providing access to the latest tools and technologies. By doing so, CXOs can not only improve their organization’s technical capabilities, increase their competitive advantage, and drive business growth. As the demand for faster, more reliable software continues to grow, CXOS must prioritize continuous testing and support its teams in developing the technical skills needed to succeed in this critical area.

Ashok Kumar

Engineering Leader responsible for quality within agile teams, QA delivery, innovation, and value Quality evangelist and test architect focused on shifting left quality. Coached teams on the agile way of working and the significance of quality becoming a team responsibility. Successfully Delivered & Executed Quality Engineering Programs for major Fortune 100 Clients. Built high-performing teams and coached them on a culture of quality.

See author's profile