In-Depth with Playwright: A Modern Testing Framework

The full transcript

Kavya (Director of Product Marketing, LambdaTest) - Hi, everyone. Welcome to another exciting session of the LambdaTest Experience (XP) Series. Through XP Series, we dive into a world of insights and innovation featuring renowned industry experts and business leaders in the testing and QA ecosystem.

I'm your host, Kavya, Director of Product Marketing at LambdaTest, and it's a pleasure to have you with us today. Today's session is about deep diving into Playwright and exploring how this cutting-edge tool is transforming the automation testing landscape.

Before we kick start today's discussion, let me introduce you to our guest on the show, Parul Rana. Parul is the Lead QA Automation Engineer at Binmile and brings nearly 11 years of experience in quality assurance. Her expertise lies in ensuring the highest quality standards through rigorous testing and innovative solutions.

Beyond her professional accomplishments, Parul is an avid reader and passionate traveler. Parul will today guide us through the latest features of Playwright, demonstrating how it simplifies end-to-end testing and discussing real-world applications and best practices.

So, let's get straight into the heart of today's discussion. Parul, can you tell us a bit about your current role and what Binmile offers in the space of testing as a service?

Parul Rana (Lead QA Automation Engineer, Binmile) - Thank you, Kavya, for having me. I'm happy to be a part of the XP Series by LambdaTest. I have around 11 years of work experience in software development and testing while working on projects with clients from the US, Europe, and India. My industry focus varies from healthcare, banking, and payment solution domains.

Currently, at Binmile, I work as a Senior SDET and Test Automation Lead. In my current project, I cater to a European e-commerce chain, and I plan functional and regression tests and design automation test frameworks from scratch. I write utility libraries, and I have worked with tools like Selenium with Java and Playwright with TypeScript.

Binmile is a digital engineering company founded in 2014 by Mr. Avneesh Kamboj. As of today, we ranked 45th out of 500 high-growth companies in Asia-Pacific. We are the preferred software solutions provider in the industry in 15+ domains.

Kavya (Director of Product Marketing, LambdaTest) - That is great to hear that you're working across multiple industries parallel and have such vast experience. Also, congratulations to the Binmile team on the appellate, of course. So, moving on to the next question, right? How do you decide which test automation tool to choose from all the tools being offered in the market today?

Parul Rana (Lead QA Automation Engineer, Binmile) - There are many factors that I consider when I’m deciding on which automation tool to pick. The most important is to understand the project's requirements. What type of application will I be testing? Is it a web application, mobile application, desktop application, or API?

And what testing needs to be automated? Like if you're functionally testing an API or am I writing an end-to-end regression test for a web or mobile application? Or am I doing performance testing?

I also consider my team's skill set, like which programming language the testers are familiar with, they are happier with using a codeless tool, I have to see which language there, you know, if I pick up a new tool, will there be a learning curve for a new language or the tool.

And one other aspect is the project's budgetary constraints. Because there are more open-source and commercially licensed tools available on the market. So the cost of the tool, alongside the training my team might require, I need to consider those aspects.

Nowadays, there are also tools available with very intuitive interfaces and features that simplify test cases, test creation, and maintenance. So all those pointers also I keep in mind. And will I be able to do data-driven testing using the tool?

Because that would allow me to have more increased test coverage. And finally, is it the tool providing an easy way to integrate with the CI/CD pipeline, and I don't need to use any third-party tools? So these are the factors that I consider.

Kavya (Director of Product Marketing, LambdaTest) - Thank you, Parul. It is interesting to hear how you're navigating through all these options and even variables within the team, like budget, team skills, and, as you said, seamless integration of the tool with CI/CD pipelines.

And, of course, the UX part of it so as to find the perfect fit for your needs. Interestingly, based on the topic, of course, of the day, I know that you have chosen Playwright for your current project. So my question is, why did you choose Playwright to begin with?

Parul Rana (Lead QA Automation Engineer, Binmile) - The project I'm working on is a Node.js-based web application which is being developed using TypeScript. The clients wanted that the developers should be able to understand the test scripts that I write.

So TypeScript was a preferred language for the development of tests, and my aim was to write end-to-end tests quickly while also keeping in mind that the test should not be flaky and Playwright provides out-of-the-box features that streamline test writing and I prefer this for its capabilities like auto waiting that intelligently waits for you know elements to load and some processes to complete before it goes to the next test steps.

And this helps to remove test flakiness. And I also needed my tests to run in parallel for faster test execution. Playwright provides this out-of-the-box capability to do parallel testing. And if you see Selenium itself doesn't provide built-in support for parallel testing, you'll need to integrate it with other testing frameworks like TestNG or JUnit that offer parallel execution capabilities.

And you need to use annotations and XML configuration files to specify how the parallel test execution is to be done. Therefore, it's time-consuming to write the script using Selenium. So that is one thing I have to consider. And the other thing was that Playwright supports headless browser testing, which makes it ideal for CI/CD pipelines. So those are the reasons I chose Playwright.

Kavya (Director of Product Marketing, LambdaTest) - Thanks again, Parul. I mean, thanks for highlighting the out of the box capabilities of Playwright. That definitely seems to play a major role, especially given your role as Lead QA automation engineer. And what I am able to understand is when you have to look at two aspects of it.

One is from the client's perspective, how best to choose the right automation framework for them while also trying to understand your team's capabilities on the other hand. And yeah, thank you once again for mentioning how Playwright fits best over here. I know that you briefly highlighted this, but let's dive deep into it. How is the Playwright able to provide faster test execution?

Parul Rana (Lead QA Automation Engineer, Binmile) - So Playwright uses WebSocket connection for bi-directional client-server communication, and this method is faster and more efficient compared to traditional HTTP communication; for this reason, we see that WebSocket is being used by Selenium starting from version 4.5 also.

The other reason is that Playwright utilizes multiprocess architecture, where each browser instance runs in a separate process. So this allows parallel execution tests across different browsers or even within the same browser using multiple browser contexts to run in their own separate windows contexts.

Selenium, on the other hand, typically relies on a single browser instance per test. And this leads to sequential execution. In order to achieve parallel testing using Selenium, we need to integrate it with other testing frameworks, like I said, test ng, and which typically requires doing a lot of configurations. and for this reason, Playwright is a better tool, I feel, because it is providing parallelism out of the box for test development and this makes it faster in execution.

Additionally, Selenium WebDriver instances are not inherently thread-safe. So if you want to run multiple tests concurrently in a single WebDriver instance. This can lead to unexpected behavior. You'll need to address this. If you want to address this, you have to create separate WebDriver instances for each parallel test using techniques like thread local storage or isolate browser sessions.

On the other hand, Playwright uses a multi-process architecture where it uses different worker processes to isolate these tests. And each worker process has its own browser context, which is equivalent to like an incognito mode like profile okay, and this means that each test in its own it's running in its own isolated environment having its own local storage and session storage and cookies.

Now, another reason I think Playwright is faster is because it uses an auto-weighting mechanism. And that is that Playwright automatically weights the elements to load and actions to complete before proceeding to the next steps. And this eliminates using hardweights and explicit weight commands like WebDriver weights in Selenium.

And that can cause the test to be slow unnecessarily. And Playwright supports headless browsers, which are browsers that don't have a GUI, graphical user interface. And this further improves test execution speed as headless browsers are typically faster than traditional browsers.

And Playwright communicates with browsers using DevTools protocol, which is more efficient protocol than WebDriver protocol used by Selenium. And this also leads to faster test execution times.

Playwright includes a lot of built-in browser automation features, such as the ability to take screenshots, record videos, and interact with the browser's web elements. This eliminates the need to use third-party libraries and which further improves test execution speed.

And in addition to being faster, Playwright is also a more reliable framework because it is built on top of Chromium, which is the same engine that powers Google Chrome. And that means that Playwright is less likely to encounter compatibility issues with browsers.

Kavya (Director of Product Marketing, LambdaTest) - Thank you so much, Parul. So, two major factors are, of course, the faster test execution part of it. And then there is of course, the reliability factor that is sort of making Playwright more efficient and useful for automation engineers.

And of course, speed is a very crucial part when it comes to any project, any client, especially when we talk about enterprises, right? The speed and test execution can really make a huge difference in the development cycle per se.

So good to hear that. And you also mentioned that Playwright eliminates test flakiness. Can you explain how that is achieved?

Parul Rana (Lead QA Automation Engineer, Binmile) - Flaky tests often rely on fixed time delays, hard waits to account for page loading or element interaction. Playwright discourages this because the network speeds and rendering times, they can vary and the wait time can either be insufficient or excessive. Instead, Playwright provides a built-in waiting mechanism that is more robust than using the hard waits.

And these methods wait for a specific condition to be met before proceeding ensuring the test interacts with the page only when it's ready. For example, you can use waitForSelector that will wait for the element to be visible and present in the DOM before interacting with it, or waitForLoad, like it waits for different loading states of the browser, like DOM content loaded or network idle state.

So this eliminates the need to use any kind of hardweights. And PlayRed offers various locator strategies, like CSS selectors, text content, so that you can choose the most stable approach for your elements.

Playwright also configures test retries, and when a test fails, it can automatically get retried a certain number of times before being marked as definitely failed, and this can eliminate you know, occasional flakiness, which is caused by some external factors.

In special scenarios, you can also use the slow-mo option in Playwright configuration. This artificially slows down the test execution speed, giving the page more time to render and elements to become interactive. And it potentially helps to identify timing-related flakiness if there is.

Kavya (Director of Product Marketing, LambdaTest) - Thanks, Parul. Interesting, because at the end of the day, we are all trying to achieve that stability factor. And clearly, with Playwright, your team has been able to, in fact, not just reduce but kind of eliminate the test flakiness part of it. And that is a day-to-day struggle for all the testers out there.

So, I can only imagine how much of a relief that must have been for you. And, of course, all that you have mentioned about Playwright definitely seems interesting. But what exactly makes Playwright a modern test tool?

Parul Rana (Lead QA Automation Engineer, Binmile) - Well, Playwright leverages Chrome DevTools protocol as compared to other tools that use classic WebDriver. WebDriver is nothing but a remote control for your browser. It is an API that allows you to control the browser through your code. This helps you to automate tasks like navigating to a page, finding a web element, filling out the forms, and clicking a button.

So, it's basically stimulating the user's interaction with the browser's limited control over browser internals and debugging capabilities. Whereas Chrome DevTools protocol is a remote debugging API that allows developers to communicate with a running Chrome browser.

And it can be used to inspect the browser state and examine the underlying code of the web page, like the HTML structure, the CSS styling, and JavaScript. And you can control the behavior of the page by directly modifying the code and seeing the changes reflected on the page in real time. This allows for quick experimentation and debugging to stimulate different styling issues; that's one reason.

And DevTools also provide device emulation capabilities. And you can stimulate different screen sizes and different user agents and see how your web page looks and functions. So you can test how it will appear on desktop tablets or mobile phones.

And CDP also provides the ability to capture screenshots, monitor network requests and responses, and manage browser state like local storage cookies, Playwright leverage all these capabilities of CDP, helping it gain more control over the browser's behavior during test execution.

You can capture the screenshots, you can record browser sessions, you can see the browser state to the CDP, and you can access the detailed information of the DOM. And using this, you can assert on element properties, styles, and even the presence or absence of specific JavaScript functions in the page context. So that's why, you know, that's how the Playwright is, I feel, is a modern tool in all respects.

Kavya (Director of Product Marketing, LambdaTest) - Thank you, Parul, so much for shedding light on that. It is really interesting to see how Playwright has evolved to meet the current testing needs.

And especially now, I've recently read that Playwright has surpassed when it comes to weekly downloads, for instance. It has surpassed even Cypress. And that has happened for the very first time. So yeah, thank you so much for sharing those insights.

Moving on to the next question, can Playwright replace Selenium to become the most popular testing tool being used?

Parul Rana (Lead QA Automation Engineer, Binmile) - That's a trick question, I think. Currently, Playwright has the best features any modern testing tool can provide. You know, there is healthy competition in the industry today, and it is exciting to see how other automation tools like Selenium had to come up with its version 4 to make up for some of its shortcomings.

Currently, the Selenium team is working on their WebDriver Biodi project, which will revolutionize cross-browser testing. Playwright has its own set of drawbacks. It supports only Chromium-based browsers like Chrome Edge, but Playwright can't directly control the official Microsoft Edge browser. It can achieve some level of automation by launching the edge through Chromium emulation which might not perfectly replicate Edge behavior.

It has good support for WebKit, but Playwright uses custom-patched versions of Firefox for automation. So, it won't work with branded versions like the Firefox ESR Extended Support Release. And Playwright also prioritizes supporting the latest browser versions.

So, there might be limited functionality or no support for very old browsers like Internet Explorer version 11. So, considering all these drawbacks, it is difficult to say if Playwright is the winner.

Kavya (Director of Product Marketing, LambdaTest) - Thanks, Parul. As you said, it will be interesting to see how the testing community sort of responds to this. But yes, at the end of the day, each tool has its set of advantages and pros and cons, if I was to put it like that. And it could lead to very exciting developments.

And, of course, based on the project requirements, as you had initially said, right? Based on the project requirements. Each of these tools would be useful for the testing teams at the end of the day. So, moving on to the next question, you said that WebDriver BiDi aims to revolutionize cross-browser automation. Can you please elaborate on this a bit?

Parul Rana (Lead QA Automation Engineer, Binmile) - Selenium quickly realized that it had to move away from using classic WebDriver and, with its version 4, started using WebSockets. It also started using Chrome DevTools protocol. However, one of the biggest challenges was maintaining the compatibility between Selenium and CDP.

Because Chrome releases frequent updates, and CDP can change alongside those updates, causing a lot of breakage in the Selenium code. WebDriver Bidi protocol aims to overcome such issues. Like CDP, it uses web sockets for bi-directional communication between the browser and the test runner to make communication efficient and faster and have low network latency.

But again, like CDB, it aims to provide more fine-grained control over the browsers. However, CDP was made for Chrome, while BiDi aims to provide more flexibility for the testers to choose any browser as it is being designed to support all browsers. And it will be based on the W3C web standards, and all the browser vendors will be implementing it.

WebDriver BiDi also aims to provide more stability. Like I mentioned, Chrome DevTools is very tightly coupled to the Chrome versions, and they do not guarantee backward compatibility. WebDriver BiDi, on the other hand, to be more standardized and stable and will guarantee backward compatibility with different router versions. So that there is, you know, the least amount of test breakage.

And BiDi also helps test, it will help testers to subscribe to global events across multiple windows and tabs. With CDP, if you want to listen to any event like, you know, console logs or JavaScript exception, et cetera, you have to, you know, go to the specific window or tab, and then you have to send specific commands to attach to it.

And you have to create a separate CDP session with the tab or the window. And you can only listen to a specific window or a tab. However, BiDi is going to make it easy for you to listen to events globally across different windows and tabs, and this can be very helpful feature for the testers. BiDi also aims to reduce the code complexity as compared to CDP.

While CDP, we need to write many commands in order to create the CDP session and get some information from the browser. The same can be done with a lesser number of commands with BiDi. So it will reduce the complexity and also the network latency.

So we can say that BiDi protocol will definitely revolutionize the test industry and the only disadvantage as of today that I can think of is that it's still in development and we can't use it as of now. But I think they have begun to roll out the support for Java, but it is still in development for other language bindings.

Kavya (Director of Product Marketing, LambdaTest) - Good to hear how Selenium WebDriver BiDi is set to change the landscape when it comes to automation. And what is next? I mean, what do you think will be the next big thing apart from BiDi in test automation?

Parul Rana (Lead QA Automation Engineer, Binmile) - Well, artificial intelligence, AI is the buzzword today, and it is definitely going to bring a lot of changes in the way we test and design tests. AI will be able to automatically generate test cases based on usage patterns, user stories, and requirements, ensuring there is more coverage of all the test scenarios.

And AI algorithms will be able to detect changes in the UI and automatically update the affected test scripts, reducing the maintenance effort, and such scripts will be like able to self-heal from any core changes.

And currently, there are third-party tools in development as of today that will act like a wrapper around tools like Selenium and Playwright and help it enhance its capabilities. However, I haven't personally used any such tool yet.

However, at Binmile, I was involved in testing an AI-based chatbot that was trained on the vast rules and regulations of the banking domain. And I was mostly involved in workflow automation and it was quite a challenging project to work on.

Kavya (Director of Product Marketing, LambdaTest) - Thank you for also briefly sharing about an interesting project that we worked on. And coming to the current project, what challenges have you faced in it while using Playwright, and how have you overcome them?

Parul Rana (Lead QA Automation Engineer, Binmile) - Playwright, as you know, it supports parallel testing, but in some cases, it's not the ideal solution. For example, if your tests rely on data or state modifications from previous tests running in parallel, it can lead to unexpected results.

Imagine two tests trying to log into an application and the same username at the same time, and parallel execution can conflict and test failures. And suppose if you have test interactions with shared resources like a database connection or file system.

Parallel test execution might lead to race conditions or data corruption. So multiple tests access and modify, if they're modifying the same data simultaneously, there can be inconsistencies. So, in these cases, we need to use the serial mode of execution. So there were cases in my project also where I was facing this challenge.

However, as you know, I already said that I preferred to use Playwright because of its parallel testing out-of-the-box capabilities. So I wanted to find a workaround in case, how I can achieve parallel testing in cases where the same DBT tables or resources are being used.

So, I'll share the screen right now from my system and show you some code. So Playwright provides support to globally set up the authorization and authentication of the users. And you can handle different states of the users across your parallel tests. So I'll show you how it is done.

So, this is an auth setup file that I've created. That is basically having like multiple tests. They are called setups. I'm calling it set up here. And here, this you can see is login page, which is taking the page, the fixture. And the login page is nothing but your page object, which will contain all the web elements, and they will be mapped to your actual login page.

So here I'm just going on to the page and then I'm calling the method to log in. I'm sending the user ID and password. If you can see here, this is the environment configuration that I'm using. First, I will set all the emails and passwords in the process environment. And then I will wait. I will log in and then wait for a URL to come to the home page.

After that, I will store the state of that user using context .storage state. And here at this file path in the JSON format, the state of the user, authorized user, is saved. Similarly, in the next worker process, the same thing happens. And the only difference being that here, a different user ID and password are used.

So simultaneously in parallel execution, three different worker processes are happening they're running these tests in parallel, and all of them are using different users to login and after that there is another configuration that I have to do in the Playwright config .ts file here in the project section I'm setting up for the auth .setup file I'm setting it up as a project.

And I am doing the current project, which is using Chromium as a dependency for this setup. So this is always run before any tests in this project. So what happens is that before any test begins, the users will log in and that state of the user is saved to a file.

So next time, when the next test starts, the user doesn't have to log in or re-login. Different worker processes can maintain different user logged-in states, which will not interact or affect each other, and it will not if we can test scenarios in which in case we are using this like.

For example, replacing an order such cases in which the same resources might be used and it's not possible for different for the same user to be placing an order the same at the same time if a same user is placing or two different orders it's not possible in that case to test so for that I require like two different test users, and then I'm able to place two different orders at the same time and be possible.

So, we can use this for such cases. So here you can see that after I make the configurations and I run the test, these files will be created, which is just saving the state of the user like I said, cookie information, the access token values, and the state of the user like user ID, session IDs.

Okay, so this is the final file that is my test file which contains the actual tests that need to be done. So here I'm injecting in each of those tests the user's stored state, that authorized logged in state. So for the first test, I'm using the user one authorization; for the second test, I'm using the user two authorization.

So, these tests are running in parallel, and they're using different user states. And you can do many things from it like can go like add a product, place an order, you can write those steps in this part.

And the main thing is that you can use the stored state and inject it into your tests and run them in parallel, having different authenticated states of the users. And that can help you run those tests in parallel, which in general, is difficult to do.

And because in the serial mode, what happens is sometimes if you have like 200 tests and on the like 150th test, if it fails, then Playwright skips all the rest of the tests. So that becomes the difficult thing, you have to again reschedule the tests and, run all the tests, fix them again.

So, it becomes cumbersome. So, in many cases, we can avoid that and still go for the parallel test execution, in which case, even if there is a test failure, only that test is failed, and the execution of other tests keeps happening. So this is one scenario for how you can achieve it. Thank you.

Kavya (Director of Product Marketing, LambdaTest) - Thanks for sharing these insights, Parul. I'm sure these are some very useful tips and workarounds for testing teams that they can implement while they're working on a project, of course.

So, as we wrap up today's conversation, I would like to thank you, Parul, for your insightful presentation, as well as the tips that you had to share on insights on Playwright. I'm sure your practical demonstration, along with the insights, must have been very helpful for our audience when it comes to giving them a clear understanding of how to leverage Playwright, which is a powerful tool for modern testing needs.

To our audience, thank you so much for joining us today, and stay tuned for more episodes of the LambdaTest Experience Series, where we continue to bring cutting-edge insights from industry experts. Once again, thank you, Parul, so much for your time.

Parul Rana (Lead QA Automation Engineer, Binmile) - Thank you.