Extra! Extra! Automation Declared Software! [TestMu 2024]

Automation is an essential part of software development, even when using drag-and-drop interfaces or record-and-playback tools. Behind these seemingly simple interfaces lies a complex layer of code, created by user actions, that drives the automation process. This complexity remains true for AI-driven tools, where the software still relies on prompts, responses, and careful management.

In this session, Paul Grizzaffi, Senior Principal Automation Architect at Nerdery, emphasizes that automation should be approached as a full-fledged software development initiative, not a secondary task. Underestimating the effort required can lead to delays, maintenance challenges, and budget overruns. Treating automation with the same rigor as any software project ensures it remains effective and sustainable.

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the LambdaTest YouTube Channel.

Getting Started with the Session

Paul emphasized that automation is undeniably software, regardless of the tools or methods used to create it. Whether it’s AI-driven solutions, record-and-playback tools, or drag-and-drop interfaces, automation involves software at its core.

The code may be hidden under the hood or behind the scenes, but it’s there, governed by the principles of software development. He stressed that recognizing automation as software is crucial to understanding its complexities and ensuring its successful implementation.

Risks Involved in Software Test Automation

Paul shared insights from a situation where a client’s automation, developed by another company, exhibited several critical issues:

• The automation was only functional on a single laptop, failing to work on any other machine.
• This highlighted a significant flaw in the development process, as automation should be robust and adaptable across different environments.
• Poorly built automation can result in expensive maintenance and unsustainable practices.
• Paul also addressed the “lottery problem” or the “hit by a bus problem,” where crucial knowledge is concentrated in one individual, creating a major risk if that person leaves the project.

This highlighted the importance of building automation that runs reliably across different environments, the potential high costs associated with poorly constructed automation, and the significant risks of relying too heavily on one person’s knowledge for project sustainability.

Approaching Test Automation from a Software Development Perspective

Paul suggested that to effectively manage the risks in software test automation, it should be approached from a true software development perspective. This involves considering several key factors:

• Development Considerations: Automation should be developed with the same rigor and attention to detail as any software product, ensuring it is reliable.
• Maintenance: Planning for ongoing maintenance is crucial to prevent costly fixes and to keep the automation effective over time.
• Documentation: Proper documentation is essential for ensuring clarity and ease of use, especially when team members change or new features are added.

Paul Grizzaffi mentions several key points for effective documentation: API documentation, quick start guides, programming assistance, and code comments. These elements help ensure clarity, ease of use, and better understanding of the code for both current and future team… pic.twitter.com/ycwOlXJhLG

— LambdaTest (@lambdatesting) August 21, 2024

• Stewardship: A concept Paul learned from a colleague, it involves actively managing and nurturing the automation project, ensuring it aligns with organizational goals and remains valuable.

By addressing these aspects, teams can mitigate the risks associated with software test automation and achieve sustainable success.

Leveraging AI in Test Automation

While AI can assist in creating tasks, generating tests, running them, and providing answers, it still fundamentally relies on software programming. Paul referenced Grady Booch, a notable computer scientist, to emphasize that AI requires specific instructions in a particular language, and its output must be carefully verified, which is, at its core, programming. He highlighted the key elements needed to effectively manage AI and software-based automation:

• Design and Development: It’s crucial to plan what you want to achieve, determine the code to write, and identify the problem you’re solving or the prompt you’re using for AI tools like ChatGPT.
• Documentation and Storage: All elements, including source code, prompts, and logs, need to be stored systematically.
• Testing and Review: Like any software, test automation needs to be thoroughly tested and reviewed by multiple individuals to catch errors and improve reliability.
• Delivery: The final product must be stored in a repository or delivered through a pipeline (such as Azure DevOps or Jenkins) to ensure it is accessible beyond just the author.
• Error Handling and Maintenance: Mistakes are inevitable in both the software and its use, so time must be allocated to address and fix bugs.

The ultimate goal of test automation is to produce valuable information delivered through logs, errors, and reports. He also talked about the importance of focusing on providing actionable information to ensure the value of test automation efforts.

Leveraging AI in test automation allows teams to streamline processes, reduce manual effort, and accelerate delivery cycles. AI test assistants such as KaneAI by LambdaTest takes this a step further by offering natural language-driven test authoring and management, enabling even non-technical team members to create, debug, and evolve test cases with ease.

KaneAI is an AI test assistant featuring industry-first capabilities for test authoring, management, and debugging, designed specifically for high-speed quality engineering teams. It empowers users to create and refine complex test cases using natural language, significantly lowering the time and expertise needed to begin with test automation.

Building Trust and Efficiency in Test Automation

Testing in test automation is crucial—not just for the software but also for the automation scripts themselves. While it may not be necessary to write a unit test for every script, specific steps that are prone to bugs should be tested carefully.

• Testing Exposes Risks: The purpose of testing is to identify risks that could undermine the organization’s objectives. Test automation helps make testers more efficient and effective in uncovering these risks.
• Trust in Test Automation: Automation is only valuable when trusted. Without trust, there is a risk of duplicating work, as humans may redo tasks they don’t trust automation to handle.
• Human Oversight is Essential: Testers should actively review new scripts, monitor their behavior, and investigate unexpected results. This involvement ensures alignment with organizational goals and fosters confidence in automation.

Key Considerations for Effective Test Automation

To effectively manage test automation, it’s essential to define specific work items rather than using vague terms like “testing” or “QA” in tools like Azure DevOps or Jira. Clear task breakdowns help everyone, including program managers and developers, understand the full scope of work involved.

Here are things to consider for effective management of test automation:

• Programming and Script Creation: Develop test scripts, even if using AI, by creating and refining prompts to ensure accurate results.
• Testing: Verify that the automation performs as expected and meets the necessary performance criteria.
• Documentation: Provide clear information outlining the expected behavior and outcomes of the automation.
• Review: Conduct code or script reviews and ensure rework is planned if needed.
• Rework: Allocate time for rework after reviews to address any identified issues.
• Retesting and Re-review: If substantial changes are made, plan for retesting and possibly another review.
• Additional Tasks: Consider any organization-specific requirements or processes that may require extra tasks.

These steps ensure a comprehensive approach to managing test automation, enhancing clarity and efficiency throughout the process.

Common Pitfalls in Automation and How to Avoid Them

Automation comes with its share of potential pitfalls that can quickly transform efforts into costly setbacks. Without careful planning and management, automation can result in wasted resources, increased costs, and a disappointing return on investment.

To truly unlock the value of automation, it is crucial to identify these challenges and address them proactively. By implementing strategic measures to mitigate these risks from the beginning, organizations can ensure their automation initiatives remain effective, sustainable, and aligned with their overall business objectives.

Paul Grizzaffi explains that maintenance involves fixing bugs, ensuring product parity, and upgrading infrastructure. This approach helps keep automation effective and consistent with the latest technology. pic.twitter.com/5LCqB1k143

— LambdaTest (@lambdatesting) August 21, 2024

• Focus on Maintainability: Maintenance often becomes the highest cost in the total cost of ownership for automation. Planning for low maintenance upfront is essential, as the cost of script creation is only a small part of the total expense.
• Adapt Automation to Product Changes: Every product update or feature addition requires updates in automation. The greater the product change, the greater the effort required for automation changes.
• Account for Infrastructure Upgrades: Tools like Selenium or Playwright frequently release new versions with additional capabilities or changes that might break existing automation. Being prepared for infrastructure upgrades and adapting the automation accordingly is vital.
• Regularly Audit Test Results: It’s important to review not just the failed test results but also the successful ones. Ensure that the test scripts align with the intended goals and continue to be trustworthy. Without regular audits, test scripts can become outdated or irrelevant.
• Recognize Test Expiration: Understand that some tests may have expiration dates, particularly early in the development cycle. Tests should be periodically reviewed and removed or consolidated to maintain efficiency.

Summing Up the Session

The session focuses on the need to treat automation as a crucial element of software development rather than a mere add-on. Paul highlighted that whether using traditional tools or AI-driven methods, automation must be approached with the same discipline and care as any other software project. This includes thorough planning, testing, and continuous maintenance. Ignoring these aspects can lead to unforeseen costs, delays, and diminished effectiveness.

By adopting a strategic approach, organizations can harness the true potential of automation—transforming it from a set of tools into a powerful driver of efficiency and long-term success. This session provided key insights into overcoming the challenges of automation and highlighted the importance of clear goals, processes, and continuous improvement in achieving automation success.

Time for Some Q&A

Here are some of the questions that Paul took up at the end of the session:

1. How do we push automation to companies that see it as secondary, only reflecting on our work as support?

Paul: Instead of selling automation as a concept, the focus should be on how it can achieve business objectives. Emphasize the role of automation in improving efficiency and effectiveness.

Demonstrate how automation saves time, reduces effort, and aligns with the company’s goals, such as meeting deadlines or lowering operational costs. Rather than pushing the idea of automation itself, promote the tangible benefits it can bring to the organization.

2. What AI platforms can be used to help developers with unit testing?

Paul: The answer to this question depends on the specific context and requirements of the development team. There isn’t a one-size-fits-all AI platform for unit testing. The choice of platform should depend on the challenges the team faces and the goals they aim to achieve. Before selecting a platform, it’s essential to assess the team’s needs and determine which technology can best address those needs.

3. What are the key benefits of automating software testing in today’s development environment?

Paul: The benefits of test automation are closely tied to the specific needs of the organization. For example, if the goal is to reduce time to market, automation should focus on speeding up the testing process. If the aim is to expand test coverage due to frequent bug leakage, automation should target comprehensive coverage.

Automation is a force multiplier that supports testers, making them more efficient and effective in their roles. Aligning automation efforts with the organization’s specific goals ensures maximum benefits.

4. How does automation development compare to traditional software development in terms of required skills and expertise?

Paul: Automation development and traditional software development require similar skills, especially in today’s environment, where many components are built using libraries and packages. However, the specific skills needed can vary depending on the type of automation.

For example, web development automation may require similar skills to traditional software development, while more specialized areas, like embedded systems, may require additional expertise.

5. How to automate when there are hardware limitations, and the application needs bare-metal servers?

Paul: When dealing with hardware limitations, the simplest answer is that if you don’t have the hardware, you can’t run certain tests. However, there are alternatives, such as using mock servers or virtualizing parts of the environment to simulate the conditions needed for testing.

If acquiring the necessary hardware is cost-prohibitive, consider the business case and evaluate whether the cost of not having the hardware outweighs the cost of investment. Sometimes, it may be more economical to operate without the hardware, depending on the potential business impact.

Got more questions? Drop them on the LambdaTest Community.