Revolutionizing Testing with Test Automation as a Service (TaaS)

The full transcript

Kavya (Director of Product Marketing, LambdaTest) - Hi everyone! Welcome to another exciting session of the LambdaTest Experience (XP) Series. Through XP Series, we dive into the world of insights and innovation, featuring renowned industry experts and business leaders in the testing and QA ecosystem. I'm your host, Kavya, Director of Product Marketing at LambdaTest, and it's a pleasure to have you with us today.

Before we dive into the heart of today's discussion, let me introduce our distinguished guest speaker. Joining us today is Murat Mercan, Senior Expert Software Development at Turkish Airlines. With a wealth of experience driving digital transformation, Murat has played a pivotal role in aligning software testing strategies with overarching business objectives. His journey, marked by a collaboration between coding and quality assurance, brings a unique perspective to today's discussion.

So, what can you expect from today's webinar? We’re delving into the intricacies of Test Automation as a Service (TaaS). From managing test environments to the nitty-gritty of test data preparation, we will explore the nuances that make it a game changer. Whether you are a seasoned developer or just dipping your toes into testing, there’s valuable information tailored just for you. Now, it's time to hand the virtual mic to Murat. Murat, over to you.

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Dear Kavya, thank you for the warm introduction. It is a pleasure to be here today. And I am truly grateful for the opportunity to share insights and experiences with this esteemed audience. Throughout my career as a software developer in test, I had the privilege of immersing myself in dynamic intersectional software development, digital transformation, and software quality management.

Today, I'm going to talk about revolutionizing testing with test automation as a service, abbreviation TaaS. So, let me share my screen with you. Here is one of my favorite quotations from Charles Kettring. He said - “A Problem well stated is a problem half-solved”. So, let's define the problem first.

In today's digital transformation process, the need for modern technologies and infrastructures have increased even more. With new software development tools, application frameworks, and various infrastructure components, developers need a more advanced tool set, such as flexible infrastructure architecture, where you can easily configure and operate system components. That's why you need to transform legacy service-oriented architecture to microservices.

The duplicated data in a data warehouse is another issue within legacy systems. So, you need to come up with a modern data pipeline—the need to catch up with rapidly changing technology with your development speed. However, the waterfall does not meet your demands. So, you need to start working with an agile mindset.

And, most of you know that Agile is an iterative approach to project management and software development that helps teams deliver value to their customers faster and with fewer headaches. Finally, you have to change functional teams to agile teams. So, in the automation strategy here, most people will say we write a test to verify that things work as expected.

While that is true, it is not the whole truth. After all, we can be verified through manual tests as well. So there has to be something more to it. We want to move fast with confidence that things work. That's why a test should give us confidence, provide feedback, and make maintenance much easier. The confidence that the code does what it should.

Since we use realistic fixtures as inputs and expected outputs, we know that no matter what the code looks like inside, it does what it should. The feedback should be fast, accurate, reliable, and predictable. This is probably where we fall a bit short. The feedback we get fast enough, but when tests fail, we sometimes find ourselves following the stack traces from service rather than only the assertions—the maintenance, which is something that is commonly overlooked when writing tests.

Since we test from the edges, we can do maintenance with confidence that we don't break it. We go really fast. In the real world, achieving all three of these becomes an art. Let's first take a look at traditional testing strategy and where it fails us so that we can move on to be successful in our testing journey. The traditional approach for a long time, the traditional testing strategy was an extremely efficient way to organize our test.

In the microservices world, this is no longer the case, so it can be actively harmful. The biggest complexity in microservice is not within the service itself but in how it interacts with other services. And that deserves special attention. Having too many unit tests in microservices, which are small by definition, also restricts how we can change the code without having to change the test.

By having to change the test, we lose some confidence that the code still does what it should, and it has a negative impact on the speed as we iterate. One more fitting way of structuring our tests for microservices would be the testing honeycomb. That means we should focus on integration tests, have a few implementation detail tests, and even fewer integrated tests, ideally none.

Integrated tests: we define integrated tests as a test that will pass or fail based on the correctness of another system the signs that you have integrated tests are even if you spin up other services in your local testing environment or even if you test against other services in shared testing environment or changes to your system break test for other systems. These are some signs that you have integrated tests.

The integration tests we should aim for instead are the integration tests, which verify the correctness of our service in a more isolated fashion while focusing on the interaction points and making them very explicit. The implementation detail so when we use implementation detail test, we save implementation detail test for parts of the code that are naturally isolated and have an internal complexity of their own.

For example, we have parsing log files to give meaningful feedback to users so they don't need to go through the logs and search for issues themselves. Needless to say, log parsing is complex. Using implementation detail testing here saves us from having an integration test for every possible build issue. Instead, we have one integration test covering a failing build that makes sure the correct error message field is present.

The implementation test covered different failure scenarios. We'll look for them in the log file and ensure that we parse them as expected. Since we reuse the same fixtures for other services, we increase confidence that we won't accidentally break contracts between them. By the way, you may have noticed that we have been treating microservice as an isolated component, tested through its contracts.

In that sense, microservice has become our main unit, which is why we have avoided the use of term unit tests for microservices in favor of implementation detail tests. Test Automation as a Service. The need for a new test automation product has emerged in order to test services developed in microservice architecture and to easily provide the orchestration needed during testing.

In this project, which we call TaaS, there are many features from managing developer testing environments to preparing test data, from distributing the service subject to test to giving detailed test reports; engineering teams are easily able to make necessary changes, for example, cache eviction, database restore, and source deployment, etc.

And the data preparations, masking, and data obfuscation with developed tasks. In addition, functional and non-functional tests in the test library are run completely in isolation without affecting other working environments. The methodology taken into account Microsoft's architecture and isolated working environments, and we approach test subjects from different perspectives.

In the traditional approach, test automations are designed to weight unit tests. However, we paid attention to resource and integration tests mostly and primarily without writing unit tests. Our focus is only to create new test scenarios, not only to create new test scenarios, but also to ensure that the prepared test and automation infrastructure are easily understood and used by other teams.

The more this culture becomes widespread within the company, the more initial effort put into test automation will be utilized. So, the key to creating an effective test automation strategy is eliminating errors that can be expected in manual testing. The test automation framework enables teams to reduce the cost and force of the testing process.

Most importantly, it makes the process repeatable, which helps validate the app functions. Also, when test automation is updated with new test cases, this change adds a reusability factor to any other microservice. The idea and ultimate goal of all these and the way is to publish quality applications very quickly.

So what is the advantage of this? With TaaS, we adapted capabilities such as scalability, independent development, and distribution brought by Microsoft's architecture to test automation developed with a test automation framework developed in a separate project independent of source code so it can be run as a standalone microservice.

This project mainly contributes to the test library while running automation tests and developing a test module, which has become very easy for developers with our lean API approach, integration tests, and framework capabilities. A developer with an average Java programming language knowledge can start working with this system in less than 10 minutes which is the biggest advantage.

Here is the definition of our services and the product TaaS and our clients as mobile and web applications. We have 12 microservices in the TaaS product, and each of them has a function. For example, an issue tracking service centralizes all ticket information into one location by connecting popular project management software or ticket management software. The inspector service, sanity health check service for test bugs, test environments, and deployment status.

For example, the launcher service is responsible for executing test automation scenarios against target test environments. The reporting service also generates custom test results and coverage summaries for the test execution. We have a test execution matrix. How will the test services work? Each of these users in the system will have certain roles, and we will allow them to perform various actions with their authorization.

After we determine the target environment in which tests in our library will be run. We chose the test scope that we wanted to run. TaaS does the rest for us. The test automation code is like any other REST API. In TaaS, we have a test library. In the test library, we have functional and non-functional test scenarios. And we have backend services for testing here, which are microservices.

We also have some external services; when you trigger some set of tests on TaaS, TaaS hits the resource endpoints through the API gateway and for microservices. In the next slide, I want to mention these service interactions. The purpose of designing this system is actually to run test automation scripts. So basically, our test library is located in the middle.

As I mentioned before, we have a launcher service that is responsible for executing test automation scenarios against target test environments. The launcher service is always ready and waiting for commands from outside, and there is a test library within its dependency. The library contains functional and non-functional test scenarios.

For example, in UI test automation, we want to run UI automation scenarios against a target environment. Let's say test box one. Selenium tests create a browser instance to run the web interface layer of the relevant target environment. After tests are started, they are performed using a scalable structure in the Kubernetes environment, which we call Selenite Moon.

On the other hand, we have some resource automation for API testing. We have customized clients here, talk to microservices in the target environment at the API level, and perform some validations. The reporting service here is responsible for sending reports and emails to relevant users, then history service, archiving, and keeping the result in the database for further analytics.

As you know, as the software testing team, we have to test every developed feature. This is a process that takes place whether it is running manual or automated tests. Of course, we can produce quality software products when all the activities preparing environments while testing, performing, and finalizing the test are carried out on the same standard. As you can see, each service we have developed has undertaken a certain task within continuous testing phases. In this way, it is aimed to automate the quality process from end to end.

Let's look at our technology stack. Even if we mention our tech stack, our code base is written in Spring Boot. We are using Hibernate as ORM, Postgre for the database, and Redis for the cache. We mimic user interactions by sailing for zero. We customize API testing architecture by Apache's HTTP client for testing backend resources.

And we also use REST-Assured. There's also Selenite Moon, so we have capabilities to scale Selenium test execution on Kubernetes environment or in Redshift environment, OpenShift environment. Most of you know that Maven stands for project management. We also use Appium for mobile testing. Here is my presentation. Maybe, Kavya, I'll give you the word from here.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much, Murat, that was very interesting. I'm sure our audience must have found it insightful. But I wanted to ask a couple of questions if that is okay. So the first question that I had was, you know, how can TaaS tackle challenges when it comes to resources and integration tests, especially prioritizing them over traditional unit tests? Also, what are the benefits and drawbacks of this approach?

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Kavya, that's an insightful question about testing as a service. It has a role in addressing challenges in resource and integration tests, especially in comparison to traditional unit tests. TaaS can effectively tackle challenges in resource and integration tests by providing a comprehensive testing environment that simulates real-world scenarios. Unlike traditional unit tests focusing on individual components in isolation, TaaS allows for testing the entire system, ensuring that different modules intersect seamlessly and resources are utilized optimally.

The benefits are evident from real-world simulation to enhance efficiency through treating microservices as black boxes. However, it's essential to acknowledge potential drawbacks, such as the risk of overlooking unit-level issues. The decision to prioritize tasks over traditional unit tests depends on the specific needs and complexities of the project. While TaaS offers advantages in terms of realistic testing environments and end-to-end coverage, teams should carefully consider the associated complexities and costs to make informed decisions based on project requirements. Yeah, that is what we are doing in our projects.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much, Murat. That is very helpful. So the next question is, can we provide details on how TaaS seamlessly integrates with diverse DevOps tools like tracking software, source management systems, and CI/CD tools? Of course, the slide that you showed was very insightful, but yes, would love to double-click on that.

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Certainly, Kavya, the integration of TaaS with various DevOps tools is a crucial aspect of ensuring a seamless and efficient software development and testing process. A TaaS is designed to interact with a wide range of DevOps tools, including tracking software, source code management, and CI/CD tools. By ensuring such comprehensive integration, TaaS enables a collaborative and streamlined workflow within a DevOps environment.

It promotes faster development cycles, early bug detection, and improved overall software quality. For example, tracking software integration. This integration allows for easy communication between development and testing teams, ensuring that testing activities align with project timelines and requirements.

Also, the source code management system integration and the testing activities are closely tied to the code base, enabling efficient version control and continuous testing of code evolves. CI/CD integration automates the testing process, allowing for continuous testing at each stage of the development pipeline.

Test results are feedback into the CI/CD pipeline, providing rapid feedback for developers and notification and reporting integration; it provides detailed and customizable reporting, integrating with reporting tools and generating reports in formats compatible with other DevOps tools. This allows organizations to tailor the integration according to their specific needs and toolsets. I hope this should be answered to your question.

Kavya (Director of Product Marketing, LambdaTest) - Absolutely, very insightful. So the next question is, how does TaaS contribute to test automation scalability, independent development, and distribution, especially in large and complex microservice architectures?

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Yeah! TaaS plays a pivotal role in addressing challenges related to test automation scalability, independent development, and distribution in the context of large and complex microservice architectures. It provides the necessary tools and infrastructure to manage the challenges associated with testing in a microservice environment, promoting efficiency and reliability in the software development process.

For example, test automation scalability TaaS leverages cloud infrastructure and containerized technologies to provide on-demand resources for test execution. This ensures that testing scales seamlessly with the growing complexity of microservices. Automated tests can be executed concurrently across various microservices, enabling efficient and rapid testing without overwhelming local infrastructure.

The independent development TaaS also facilitates independent testing of microservices by offering isolated testing environments. Development teams can work on their respective microservices independently, and TaaS ensures that the testing environment for each microservice is isolated, preventing interference between different microservices.

This independence accelerates development cycles and allows the team to focus on specific functionalities without affecting others. In large architectures, TaaS also enables the distribution of test workloads across different testing environments. This is particularly crucial for us to execute in parallel across multiple microservices. So, TaaS can dynamically allocate resources based on the testing needs of each microservice, optimizing test execution and reducing overall testing time.

We also support parallel test execution, a key factor in handling microservices tests for different microservices can be run concurrently, allowing for faster feedback and reducing overall testing time. This parallelism is essential for maintaining agility in the development process, especially in large-scale architectures.

Kavya (Director of Product Marketing, LambdaTest) - Great, I mean, it is fascinating to see how TaaS streamlines testing, especially in large architectures, as you said, and makes the entire process more efficient. So the next question that I had was, how does updating the test automation framework, especially with new test cases, enhance functionality and reusability across microservices in the organization?

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Yeah, updating the framework with new test cases plays a crucial role in enhancing functionality and reusability across microservices in the organization. Let's talk about why it's essential to update our test automation framework with new test cases. It helps our testing process remain the same across all our microservices. Think of microservices like different puzzle pieces, each with its own unique features, and keeping our testing consistent is crucial for trustworthy and comparable results.

Secondly, when we add new test cases, we basically create reusable parts in our test automation framework. These parts act like building blocks that can be used for different microservices. This not only saves time but also makes us sure to follow the same testing approach every time. As the organization grows and adds new microservices or updated automation framework, adjust smoothly to these changes so we don't need to for a big makeover.

Now, let's talk about making regression testing more efficient by adding new test cases into our framework. We make a bunch of automated tests. This speeds up how we check existing functions across microservices, making sure everything works as it should. When we are bringing in new microservices or updated automation frameworks, make it fast and easy. The standard way we test things ensures that new services fit smoothly into our testing process. This helps them use existing test cases, and we can release very quickly.

And for the last thing that I want to say, is our microservices are always changing and updating the test automation framework. We are always updating our test automation framework, which is regularly filled with new test cases and has become a flexible and quick tool. It is ready to adjust to the changing nature of our microservices.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much. So it definitely underscores the importance of having a standardized approach across microservices, I suppose, very insightful. Thanks for that. So the next question that I had was, if you can explain how TaaS's lean API approach simplifies development and contributions for developers with diverse programming language expertise.

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Actually, this is the most crucial part of TaaS. Let's take a look at, look about how testing as a service keeps things simple and inclusive for developers, no matter their programming background. At first glance, TaaS has a user-friendly API. It is easy to understand and use, making integration a breeze for developers.

And guess what a task place is nice with any programming language. Developer can use their favorite language to interact with tasks. It's all about flexibility and inclusion. The best part, a task keeps it simple. The API is designed to be straightforward, and developers can do it quickly without getting the complex stuff. And now, here is the cool thing.

TaaS encourages teamwork. Developers from different language backgrounds can contribute. It is a collective effort that makes TaaS more powerful and adaptable. And if you are all about speed, if you are talking about speed, TaaS has your back. The simple API helps developers build and test projects quickly. It's like a shortcut to efficient development. In a nutshell, TaaS is all about keeping it easy and inclusive and speeding up a development game for everyone.

Kavya (Director of Product Marketing, LambdaTest) - Yeah, I mean, that's a really great insight. Good to hear how TaaS sort of helps foster collaboration among developers who have different skill sets. Great. So the last question that we have today is, how can TaaS manage versioning and backward compatibility, especially when integrating with DevOps tools, to ensure a smooth transition and minimize disruptions in testing processes?

Murat Mercan (Senior Expert Software Development, Turkish Airlines) - Thanks for the question, Kavya. TaaS keeps things smooth when integrating with DevOps tools by doing two key things. The first one, we use a clear versioning system. Each TaaS version and its API changes are well-documented, so users can easily understand what's going on. The second one, we are all about backward compatibility. When we roll out new versions, we make sure that they play nicely with old ones.

Your existing integrations with DevOps tools won't break, ensuring your testing process keeps running without a hitch. We also provide detailed release notes. So you are always in a loop about any changes, and if we ever decide to phase out a feature or version, we give you a heads-up with a clear timeline In a nutshell, we aim to make transitioning with tasks as smooth as possible, ensuring your testing process stays uninterrupted.

Kavya (Director of Product Marketing, LambdaTest) - Great, thank you so much, Murat. You know, once again, it's great to hear how TaaS adopts practices that ensure this transition is seamless and, of course, disruption-free. Great, Murat, I would say that it has been a very insightful session. Thank you so much for sharing the wealth of, you know, knowledge that you have.

To our audience, stay tuned for the episodes on the LambdaTest Experience (XP) Series. Until then, keep testing, keep innovating, and embrace the ever-evolving landscape of software quality. Once again, to Murat, thank you so much for giving us your time. Have a good day, everyone.