Creating Reliable and Scalable Test Automation Frameworks

The full transcript

Kavya (Director of Product Marketing, LambdaTest) - Hi, everyone. Welcome to another exciting session of the LambdaTest Experience (XP) Series. Through XP Series, we dive into a world of insights and innovation featuring renowned industry experts and business leaders in the testing and QA ecosystem.

I'm your host, Kavya, Director of Product Marketing at LambdaTest, and it's a pleasure to have you all with us today. Our topic for the session is creating reliable and scalable test automation frameworks.

In today's session, we'll explore best practices, architectural considerations, and key design principles to assist testers and automation engineers such as you in creating robust and reusable test automation frameworks.

Let me welcome our guest on the show, Gary Parker. Gary is currently a Senior QA Architect at Betway Group, where he's responsible for QA architecture, tooling, frameworks, and processes.

He specializes in front-end web and mobile technologies and has more than 10 years of experience in the QA industry across different domains, products, and environments, making him a perfect speaker for today's topic.

So, let's jump straight into the discussion. Gary, it would be awesome if you could share a bit about yourself and your past QA journey. The journey so far that you've taken. Yeah, over to you, Gary.

Gary Parker (Senior Test Architect, Betway Group) - Thank you, Kavya! So yeah, my kind of QA journey has spanned over quite many years, as you said. I started off at university. I studied game design and development. So I always wanted to get into kind of tech and development and design.

My first testing job was at Sega. I was testing video games, which was a lot of fun. And at that point, I just decided I really liked QA and testing. I've been working with various different companies over the years, and the company I'm at at the moment, Betway, is where I've spent almost 10 years of my career.

When I first joined the company, there was no test automation, there was no tooling for that. We were just manual testers. And it's something we've implemented over the years, these different processes, frameworks, and kinds of automation languages.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much, Gary. So, let's start from the basics, right? Based on your prior experience, let me know if you'd like to share an example of when you implemented a test automation framework effectively.

Gary Parker (Senior Test Architect, Betway Group) - Yeah, sure. From the very beginning, we had just manual test cases, and as I said, there was a lot of manual testing across different environments. There was no automation. It took us a week to deliver some features.

We implemented our own in-house framework at the beginning, which was in C-Sharp and .NET and it was supported by multiple different teams. We had to break down our system into different reusable components, stuff as login, registration, and navigation.

And we made these modules, hooks, reporting, all of the kind of stuff you want to share across teams. And we created our own Node.js package, which the teams could download, get the latest version, and anybody could contribute to the automation framework.

It's like the perfect start for somebody who hasn't worked with automation or frameworks before to understand that process. How does it actually work to build and implement a framework?

Over the years, we've moved over to frameworks that other people are building. You've heard of Playwright, WebDriverIO, and Cypress. We've done lots of POCs. We haven't just jumped into using these frameworks. We've tested them out and compared them over the years and the usage varies across the teams.

They use it based on their needs and their requirements. But yes, it's been a long journey, and there's been a lot of upskilling and in-house training for people who need to learn stuff like JavaScript or TypeScript who have never used it before.

So yeah, that's kind of how we've started from the beginning and where we've got to now, where we have automation running as part of CI and part of our release pipelines. We've gone from releasing once every two weeks to multiple times a day. So, we've come a long way.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much, Gary, because that is also an interesting journey, not just for you as a QA professional but also for the organization and the transition that you make into automation testing.

That is really insightful on how you went about it. Moving on to the next question, can you provide a decision-making framework to help us evaluate and select the most suitable framework for any tester's specific needs, perhaps?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, definitely. That's really important. My career has gone from being a QA engineer to more senior technical leadership in architecture. In the beginning, I picked whatever shiny tool kind of appealed. If there was something new on the market, I wanted to try it out.

That generally can be quite a bad way to approach it because you're using something that may not be suitable or may not fit the needs of the team. So you do need some kind of governance or decision-making framework, as you worded it. So we said about what are the requirements of the team? What is their skill set? What are the developers using?

In the past, we've chosen specific frameworks based on front-end developers using JavaScript as an example because they can cross-knowledge share, they can work on each other's code reviews, and it just works really nicely for getting developers involved in writing automation.

And yeah, we have a general checklist. Is this a new technology or are we replacing something we already have built in-house? What is the objective? How long are we going to run a POC for?

Is it a week or a month? And what kind of insights and data are we trying to get? Is this new framework going to make us more efficient? Are we going to write code faster and deliver faster?

What are we trying to achieve by implementing this new framework? And yeah, that's something we've done across the department. And it's been really important. For example, we've gone from C-sharp and .NET to Playwright and WebDriverIO over the years.

We've had a noticeable difference in test execution speed stability and collaboration between devs and QA. All of these kinds of benefits from moving to this framework.

And yeah, I just think you should measure what you're working on, analyze the requirements of the project you're trying to run, and just keep everyone involved, document it, share it with the team, kind of demo what you're achieving.

Kavya (Director of Product Marketing, LambdaTest) - Thanks for sharing that, Gary. Those are some great tips and I'm sure our audience might definitely find it useful. So moving on to some real world examples, can you list down some examples of how to break down a framework into well-defined, reusable modules for efficient maintenance?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, definitely. Before we jump into that, I think definitely review how many reusable components you need or how well you want to break down this framework. If you have a very small project, there may be no requirement for it.

But if you're building this across multiple products or projects, you do need to consider this. In the past, we've broken stuff down, such as managing cookies and headers and authentication so we can log into the application.

All of this stuff is quite complex, and if we can build a module or something we can reuse, it's beneficial to everyone. And I think, again, like I mentioned, documentation and knowledge sharing. If you are building these components, share them with your team.

From past experience, I've seen multiple teams build the same thing over and over, and it defeats the point of having a framework and having this reusability. So yeah, authentication, session management, anything that's unique, if it's a web application or mobile application, there's obviously going to be things that you don't want to share between projects.

Any kind of help classes and any interactions with APIs or third parties are generally not specific to that project. And there are benefits of having that elsewhere. So you can import those kinds of modules and, yeah, also stuff with reporting and data management.

If you're sending your test execution data to another platform, that's something you should be sharing with everyone and we collaborate and bring all of that data together in one place. And it's something that's easy to put into a framework within your organization. I'm sure there are many more. That's hopefully a few that will be helpful.

Kavya (Director of Product Marketing, LambdaTest) - Absolutely, those are some incredibly informative tips, of course. So from my understanding, what I'm able to understand is, of course, that reusable modules at the end of the day, it's super helpful. And especially if you sort of club it with content assets such as documentation, it's only going to sort of help better.

So yeah, great tip, of course. Moving on to the next question, how can software testers leverage this framework design principle to influence application development and, of course, promote testability from the beginning of the development life cycle?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, that's really important. So bringing this kind of collaboration between developers and QAs is what all teams should be kind of striving to achieve. And I think having everyone involved in automation in the same way you're involved in development has lots of benefits.

You're building test automation from the very beginning. You're considering the requirements alongside development planning. So there are no gaps; you're not missing test coverage because QAs have written their test code in isolation from the developers.

And yeah, I've seen product teams directly put their QA automation code within their development projects as well. And there are tons of benefits to that. They're writing code reviews and reviewing them alongside development code. They're getting insights from developers who may have a bit more experience in that language or that technology.

And you get the added benefits of being able to influence development. In the past, we've had issues with locators or selectors, so we haven't been able to hook into certain parts of the website, as an example.

But now this opens it all up, so you can say you can either directly do it yourself or you can speak to the developer who's working on that functionality or that feature and ask them to add pieces of code to make your life easier. We've done that quite a few times, adding labels or hooks within the web application.

It also gives you insight into what they're writing. If they're building unit tests or API tests, and there's an overlap that you may not have been aware of, you can reduce the amount of UI automation you're building because it's obviously going to be a bit slower, a bit more flaky compared to those automated tests and you can kind of balance out that test coverage.

I have just seen lots of benefits for delivery and getting stuff into production. Everybody is on the same page. It is not a case of development finishes and QA starts. It is from start to finish that we get the feature to production.

It has automation built into it. We have full confidence in what is going on in our live environment. I just think that's something we need to promote, not just in our organization, but the whole kind of industry should be kind of working towards this.

Kavya (Director of Product Marketing, LambdaTest) - Thanks so much, Gary. So interestingly, there have been two recurring, I would say, themes that I've heard in my conversations with leaders like you. One is, of course, the collaboration between developers and testers.

And the other bit is how quality is everyone's responsibility. So, while you were sharing these insights, it sort of closely resonated with these two themes or, I would say, topics that I've been hearing coming from the QA leaders.

And from your team's end, how have you sort of worked on building that collaboration between developers and testers? Is there an example that you'd probably like to share?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, that's definitely a hot topic, and it can be quite controversial as well. I won't get into too many specifics, but yeah, trying to influence people to be responsible for quality can be quite difficult, and there are definitely challenges that come with that.

Kavya (Director of Product Marketing, LambdaTest) - We will avoid the controversies.

Gary Parker (Senior Test Architect, Betway Group) - You can work with people who are 100% on board and you don't really need to influence at all. They want to help you support and build quality and automation.

Others, you need to present it and show what are the benefits of us working closer together? What are the benefits of us, or let's say developers, writing UI and API tests when they haven't done that in the past?

So, yeah, it's a lot of presenting and demoing what we could potentially achieve, how we could get stuff to production faster, and how it benefits everybody on the team. If everybody is responsible for quality, there's no doubt.

In the past, there may have been some doubts at the beginning of the lifecycle or at the end of the lifecycle because there's uncertainty about who is responsible for when something goes to production, and who is responsible for monitoring.

That's been quite a gray area in the past. And now, as you said, it's kind of everybody's responsibility. We have QA and developers on call checking post-release if our application is performing as intended.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much. That is a really good insight, of course. So, moving on to the next question, Gary, as a technical leader responsible for leading QA architecture, how do you go about onboarding your team for a new framework, free work without disrupting the environment?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, I think that's really important for multiple reasons. It may be a case that you're onboarding a team that hasn't run any automation in the past, as an example, or we're going from their experience of working with C-sharp to JavaScript, something completely new to them.

They may not quite understand what the benefits are. And to them, it's very disruptive. If you're a QA engineer and your focus is testing automation and getting a feature out. That disruption can cause a lot of unnecessary hassle and they're going to have to put time aside to learn a new project, learn a new framework.

So you have to manage it quite carefully. In the past, we've kind of handled it as an actual project. So it's accounted for in our overall delivery. It's put in their backlog alongside the product. And what we do at the very beginning is some kind of demonstration or presentation, just outlining the why. Why are we doing this? What are we trying to achieve? What are the benefits?

So, yeah, like the benefits of implementing as a technology and as a project. Usually, we've already done some background with the architects and leads. So, the technical leaders are all on board and they're ready to support this project. It is just supporting the teams. You have to consider the different skill sets.

You may have junior engineers up to senior level, and they are all going to need different support. We run stuff like workshops and 1:1 sessions. We usually group people in small chats where we can have discussions. People can give their honest feedback and just raise any concerns as we go along.

It's not something that's going to happen overnight. It's going to probably be a three to six-month transition, as an example. And there's a lot of considerations. We may have a lot of test coverage in our previous framework that we now have to migrate over. So, how are we going to transition between these two projects? And how are we going to make sure quality doesn't suffer during that time?

Kavya (Director of Product Marketing, LambdaTest) - Thank you. That is a great approach to onboarding a team without actually causing a lot of interruptions, I suppose, in their existing projects aspect because it has to be really seamless while moving from one framework to another.

And, you know, I just wanted to double-click on the movement part of it, right? So, what strategies and resources would you recommend for any QA professional who's looking to master a new test automation framework?

Gary Parker (Senior Test Architect, Betway Group) - Yeah. So, I would first go to their official website and their documentation and read up on that. Generally, all of the popular test frameworks have very good documentation or tutorials that go along with that.

Personally, I think you just need to get on; you just need to do it and kind of get your hands dirty and write code. That's the easiest way to start learning. And then, when you're struggling a bit, seek support from your more kind of senior members of the team.

But yeah, don't get too caught up in not knowing everything. You just need to start writing tests, learn as you go along, and things will start to make sense. Once you start seeing your tests executing and things are failing, you have to start debugging and you understand how to kind of work your way through the code, it starts to make more sense.

And I think generally, in the industry, there's a lot of really good support for the frameworks that are out there. There are a lot of kind of leaders in the industry who are creating blogs, documentation, and tutorials. So there's a lot of resources for everyone.

Kavya (Director of Product Marketing, LambdaTest) - Awesome, thank you so much. Moving on to the next question, how can QA teams design the framework to incorporate monitoring and alerting mechanisms to identify issues during the test execution proactively?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, so I think this kind of depends on what your architecture looks like. So, most frameworks come with their own reporting tools, or you may have to install a third party. So that's one part of it. You have your CI reporting, so this may grab the data from your test reports and it will display it in your pipelines.

And then the last part is more of where that data is getting sent to. You may use, I won't name-drop too many tools, but there are a lot of data management tools and visualization tools, which are quite important for more architects and QA managers.

If they want to see it from a high level, like across the department, how successful is our test execution? How long is it taking us to execute tests? And just meant going back to our point of transitioning between frameworks. You need to see that data.

So are we getting faster, or has this transition to another framework made us slower or is test execution actually getting worse than we haven't realized? So having that data and monitoring is really important. And then you've got your production and maybe even your staging or test environment monitoring, what is the health of your websites.

So having that in a central location is really important. And then sending alerts to whichever kind of Slack or chat teams channel you have, getting that instant feedback is really important. You don't want to find out the next day that some test has failed and there's an issue in your production environment.

So, yeah, it's about this constant feedback loop throughout your development lifecycle. So, as much kind of monitoring and alerting as you can put in, the better. And I know in some teams, we have developers using their own technologies, and they've kind of collaborated and put them into one. So, you get stuff like response times alongside test execution, which is really useful. So, you can kind of pair up this data.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much again. So what I'm taking away from it is creating that constant feedback loop, as well as incorporating the monitoring and alerting systems into your entire test execution approach itself to begin with, should definitely help QA teams.

Moving to the last question of the day, considering the ever-evolving testing landscape, what are some key considerations for building test automation frameworks that adapt to future technologies and testing needs?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, I think, like, as you said, the testing landscape is changing. You don't want to lock yourself into one tool or framework. So I think being as kind of flexible and open as possible, having very good documentation in place for your actual product and your test coverage so that, let's say, tomorrow you do need to move to a new framework.

It's a smoother transition than it would be. It's still going to be quite difficult, but you'll have that foundation in place. I think the tools that are popular today in five to ten years they may not even exist anymore. You need to be able to switch and transition quite easily. I think building your frameworks in a way that reflects that is always beneficial.

Don't build lots of very customized code that only works in one application. Try to make your framework as flexible and reusable as possible. That you can kind of chop and change components. If you're using a reporting or a monitoring tool, it's just one file change if you move to another product. Same with stuff like log-in or navigation.

You have small helper classes, which, if tomorrow you have to support a new project it's easy; it's just a one-line code change, so from the very beginning, kind of build with that in mind that you may need to change it tomorrow.

Kavya (Director of Product Marketing, LambdaTest) - Keeping future-proofing in mind right from day one, I suppose, is the key to it. Gary, thank you so much. That has been a lot of great insights. Before we wrap, I have a quick question as well.

Is there anything else that you might want to share with the QA community given that there are so many new technologies and, as you said, right, as we discuss testing needs that are evolving? So, are there any parting words that you might want to share with the QA community at large?

Gary Parker (Senior Test Architect, Betway Group) - Yeah, I think, as I said, keep an open mind and don't jump into every new tool or technology just because someone has said it's great. Do your own research and your own analysis and give an opinion from your own point of view. That's always beneficial because not everybody who speaks on the internet is correct. There's always going to be a different opinion.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much. That was really helpful. And as we wrap up today's show, I would like to thank Gary for the insightful session. Your expertise has definitely given our audience a really deep understanding of creating reliable and scalable test automation frameworks.

I really appreciate your time and contributions to the session. To our audience, thank you for joining us today. We hope you find this session informative and engaging. Stay tuned for more episodes of the LambdaTest Experience (XP) Series, where we continue to bring you cutting edge insights from our industry experts.

And if you'd like to get in touch with Gary for more questions, you can reach out to him on LinkedIn, as well as on Twitter. We would be able to add his handles to the description itself. Thank you once again, Gary, for joining us today. It's been a pleasure hosting you!

Gary Parker (Senior Test Architect, Betway Group) - Thanks a lot. Bye-bye!