Generative AI for Software Testing – Hip or Hype?

Since the start of 2023, we have been bombarded by articles claiming that AI-driven chat tools like ChatGPT will take your job. Meanwhile, this chat tool from the OpenAI foundation saw usage down nearly 10% in the month of June. Speaking of the downturn on the All-In podcast, Venture Capitalist Chamath Palihapitiya said,

… it’s part of the hype cycle and we all kind of fell for it. The only winner here is [chip maker] Nvidia. I think the losers here are most of the VCs who pumped in hundreds of millions to billions of dollars in Rando stock. And consumers, they tried it. They didn’t like it. They moved on. They’re waiting for the next

So who’s right? Who’s wrong? And what does that mean for you?

Other Possible Explanations

One possibility is- instead of using AI less,it seems like more tools are popping up. For example, since ChatGPT 4’s March 2023 release,other companies have started making their own AI tools. While Google’s Bard looks for fresh information on the internet, Claude.ai is pitched at a specific niche – an administrative assistant. So, it’s not about using AI less but having more options to choose from.

Another possible reason for the drop in ChatGPT usage might also be due to the summer break of high school and college students. Many students rely on generative AI for their essays, and they make up a significant portion of the user base.It is even possible that while the software engineering usage of ChatGPT increased, the sharp decline in student-related work resulted in an overall decrease in its usage.

It seems strange that Chamath would backpedal his optimism because of a single-month decline. The root cause may be more revealing and could relate to the lack of concrete and successful use cases for generative AI. Given that Chamath’s primary role involves exploring and funding companies in this field, he should have access to examples showing that a small dip isn’t a significant concern. So, the question arises: Is the downward trend genuine, or could there be other factors at play?

As it turns out, as Chamath implied, “we’ve been trapped in the hype cycle.”

The Gartner Hype Cycle

The Hype Cycle is a graphical explanation of the process that technologies go through as they move to adoption. Starting with a trigger, you have the “peak of inflated expectations” (when the technology seems to promise to revolutionize everything), the “trough of disillusionment” (when we realize it will not), the slow slope of enlightenment, and, finally, the plateau of productivity.

Figure 1: Gartner’s Hype Cycle.
Hype seems to accompany most new innovative technologies.

Most innovative technologies often come with a lot of hype. I remember a time when XML was a big deal, with multiple paper magazines dedicated to it. People thought it was the future, and I was advised to use it for everything. However, we later found out that XML was slow and used a lot of disk space, and that databases were a better way to store data. JSON became the standard for returning API calls.

Java was once marketed as the perfect language for the internet, and Ruby on Rails was considered a magic solution for web servers. While Rails did boost productivity for some web applications, it had issues at a larger scale. Then came microservices, API calls, and JavaScript, but companies soon realized that large microservices setups brought cost and complexity problems.

The point is, technology trends keep evolving, and what’s considered the “next big thing” today may not hold that title tomorrow.

Generative AI, as it turns out, is not a magical cure-all. It might lead to great increases in productivity, but we certainly haven’t found them yet.

It is time to climb the slope of enlightenment. XML did that, for example. Today, XML has a legacy role for some Web Services Definitions, and as a place to force data in a hierarchical form that has binary, or large-object data embedded within it.Unlike hype, that leads to real value.

The Software Engineering Leverage

There are always more ideas for software projects but limited staff hours to execute them. Ed Yourdon, an early software pioneer theorized in his book Death March that unrealistic deadlines often emerged because, if they were realistic, the projects themselves would clearly be unprofitable. For a positive return on investment, the value must outweigh the cost.

If we can get more productive with generative AI, essentially reversing Yourdon’s fear – projects that were previously deemed unprofitable become viable. Individual projects cost less, so the team can do more with less people. This allows the Backlog to continue to expand, with even the less prioritized projects now being economically feasible and worth pursuing.

To provide some historical context, the term “Software Crisis” was coined in 1968 at a NATO Conference. Over the years, we’ve moved from punched cards to interactive terminals to screens, hexadecimal to assembler to compiled, object oriented and functional programming.

While each of these innovations was initially seen as a silver bullet and later left some disappointed, they collectively contributed to a significant boost in productivity compared to decades ago. Today, a proficient programmer can create a cloud-based web application over a weekend, a task that would have taken a team of programmers months in the 1990s.

The software crisis was supposed to be about too much work and not enough programmers. Since then, we’ve seen nothing but improvements in productivity. Yet the demand for software continued to do nothing but increase. In other words, even if generative AI significantly amplifies programmer efficiency, say by a factor of 10, it doesn’t imply a 90% reduction in the need for programmers. Instead, it implies a surge in what can be developed. Its important to note that this is a hypothetical scenario with significant uncertainity. It might be a 10% percent increase, and the argument will remain unchanged; it does not lead to a 10% reduction in the need for programmers either.

With all that covered, we are left with the question – “What should we do in this changing environment?”

Moving Forward

We could retreat to our day jobs and rest on our laurels but that doesn’t align with our curiosity and the ever-evolving tech landscape. These technologies are like a continuous journey- If one stops moving, they are invariably left behind. Besides that, learning is fun, so why stop now?

The alternative, of course, is to not fear the reaper, but instead embrace it. ChatGPT has a free tier. Github’s CoPilot, powered by OpenAI, assists with code suggestion in the development environment, integrates with Visual Studio, VS Code, IntelliJ, JetBrains, and a dozen more IDE’s. It is also available for a one-month trial for free, and ten dollars per month after that.

So, let’s leverage these tools, play with them and embrace the learning process. You don’t need to find some magical use case; sometimes there might not be one. But if you explore these tools (and how to use them effectively) better than others, you can take a leadership role explaining their potential to others.

As we continue this series, we’ll explore the possible uses of generative AI in testing, offering in-depth insights. Frankly, the published uses of AI in testing are for the most part in the realm of dreams and ideas. It wouldn’t take much for you to be on the edge of innovation.

Change isn’t something for us to fear.
These tools might just help us steer.

Matt Heusser

As the Managing Director of Excelon Development, Matt Heusser, consults, trains, and does software delivery while helping others do it. Probably best known for his writing, Matt is the lead editor of “How to Reduce The Cost of Software Testing” (Taylor and Francis, 2011), editor for Stickyminds.com, and recipient of the 2015 Most Popular Online Contributor to Agile at the Agile Awards. A 2014 recipient of the Most Influential Agile Test Professional Person Award (MAITPP) in Potsdam, Germany, Matt also served as the lead organizer of the Software Testing World Cup. He is a former member of the board of directors of the Association for Software Testing and creator of Lean Software Testing family of methods.

See author's profile