Playwright Visual Regression Testing Complete Guide

What is visual regression testing?

Visual regression testing focuses on image and pixel comparison - it allows us to define the view, area, or component of a website where we would like to perform visual analysis. We start with our baseline image, our ‘master/main’ branch equivalent, and what we expect the site to look like. We then compare it with our current screenshot, which may have been taken in a different environment or branch.

Here is an example. We have taken two screenshots from the Playwright homepage. We will label them A and B.

• A is what the website looks like right now, and
• B is what it looks like after we’ve made some changes on the local machine.

Can you tell the visual or styling issues in less than 10 seconds?

Did you find them? Three styling issues have been raised in our visual comparison below - the header navigation, hero buttons, and browser logos.

This comparison was executed with Playwright. It was run once to create a baseline and then edited for the purposes of this example. Finally, it was run the second time to compare our baseline to how the Playwright homepage looks.

Maybe some of you found 1 or 2, and maybe some of you found all 3 - but it should start to become clear that, as humans, we aren’t designed to pick up on these small visual changes in small spaces of time. The reason to find them in less than 10 seconds is how fast our visual tools can find them.

Benefits of visual regression testing

So why should we introduce visual testing? What does it bring to the team or product you are working on, and how can you show its value?

• Economical and scalable - in terms of time invested in setting up and getting to the point where you can execute tests, you’ll have something up and running in no time. Visual testing is built-in with the Playwright framework, so no additional installations or plugins are needed
• Super-fast execution - you’ll be able to implement it in your build and release pipelines with minimal impact on runtime, with tests executing in seconds, not minutes.
• Simplicity - as a testing type, you don’t need to debug it - if a test fails, you look at the visual output, and all issues are highlighted in red. It means the barrier to entry for all team members is low, and everyone can collaborate on it
• Code confidence - it’s another gate or safety net that you can add to your project and a way to raise the confidence in the quality of your code changes further.

What will visual regression testing not catch?

So far, visual regression sounds great, and you can’t wait to add it to your project, but keep in mind it won't replace your existing test suites. Here is a quick list of bugs or issues it won’t pick up on:

• Functional issues - clicking on links, buttons, and general interactions with your website
• API or Network issues - requests for content or data in the background
• Performance or security issues - response times or code vulnerabilities

Those are just a few, but you get the idea - visual testing focuses on the appearance and layout of the website and doesn’t dig deeper than that.

Implementing visual testing for a single web page

In this Playwright visual regression testing section, we will be using Playwright for our visual testing, so ensure you have the relevant frameworks and dependencies installed before we get started.

Once set up, delete unnecessary folders, so your structure looks like the one below. The example.spec.js contains a basic code snippet to get us started with Playwright visual regression testing:

// example.spec.js
const { test, expect } = require('@playwright/test');


test('example test', async ({ page }) => {
  await page.goto('https://ecommerce-playground.lambdatest.io/');
  await expect(page).toHaveScreenshot();
});

To run our basic visual test, we execute the following (these tests run headless, so you won’t be able to see the test running):

npx playwright test

You should notice a few things - first, all our tests failed, which is fine. And that we have some new folders created. The first execution has no baselines to run against, so the tests fail, and the baselines are created for us.

The folder tests/example.spec.js is where our baseline .png files are stored. And the test-results folder is where we can see our Playwright visual regression testing results. You will also notice we have run our visual tests across three different browsers - Chromium, Firefox, and Webkit.

Let’s run the tests a second time! This time we will notice all of our tests have passed, which is to be expected - the baseline images we took are being compared to the website in its current state, which is the same. You will also notice that our test-results sub-folders are empty this time (highlighted blue). This is because there were no failures to report on.

Next, let’s check that our Playwright visual regression testing is working correctly by making some small manual changes to one of our snapshot baselines - the changes we’ve made are meant to simulate a real-world situation where elements are ordered incorrectly, or missing entirely, due to a code change.

Although we have only made a breaking change to one image in this Playwright visual regression testing tutorial, two of them have failed - let’s look into both failures further and see what happened. There are two places where we can check the visual results - directly in the test results folder and by opening the files that end with diff.png or the generated HTML report.

This is the file view within Visual Studio Code; as you can see by the red highlights, we have locators out of place or missing entirely. This is good for a quick check, and we can switch between the actual, expected, and diff files to see what has changed.

There is also a nicer way of viewing it within the Playwright report. When your test execution finishes, another window should appear which contains the HTML report:

Serving HTML report at http://localhost:9323. Press Ctrl+C to quit.

If we open that page, we will see more useful information - the first is the error logs. It tells us that 744 pixels are different than what was expected, and the ratio comparison of all other pixels in total. This information is useful, as we can manage threshold ratios for our comparisons, which I will get into in more detail later.

 - 744 pixels (ratio 0.01 of all image pixels) are different.

If we scroll further down the page, we will see similar ‘actual’, ‘expected,’ and ‘diff’ images - and a nice additional feature, which lets us swipe between views:

That test was the one we intentionally changed. Let's look at the comparison for the file we didn’t change. As you can see, there is a small red highlighted area near the bottom of the web page. This could be due to the layout or elements on the page loading differently from the baseline, as it's a small banner image that has come into view.

This can happen occasionally and is another reason why it's good to tweak the thresholds at which the tests break and narrow the focus of our image comparison.

Implementing visual testing for a single element

Leading from our flaky visual test, let’s instead hone down our Playwright visual regression testing to a single element rather than the whole web view. We’ve updated our test to focus on the header logo:

// example.spec.js
const { test, expect } = require('@playwright/test');


test('example test', async ({ page }) => {
await page.goto('https://ecommerce-playground.lambdatest.io/');

const headerLogo = page.locator('#entry_217821 > figure > a > img')
await expect(headerLogo).toHaveScreenshot()
});

Before we perform Playwright visual regression testing, we will need to update our baseline images. We can do that by running this simple command:

npx playwright test --update-snapshots

If we check our snapshots folder, you will see the nice little icon for the LambdaTest Playground website:

Now, if we run our tests with the usual command, a visual comparison will be made just for the header logo locator snapshot:

npx playwright test

The test results should be three passing tests and an empty test results folder, as no discrepancies between the header logo visual comparison were found. We could break the whole website into smaller components for visual regression and minimize flakiness.

It would also bring the same benefits of breaking down UI tests into smaller chunks, we get more focused test results and outcomes, and finding the area for failure becomes easier.

Working with Thresholds

With Playwright visual regression testing, as with any type of automation testing, there will always be a margin for error. This is where thresholds come in useful. We can adjust and manage various threshold types depending on our application and use case.

Here are three examples below:

• maxDiffPixelRatio - this value can be between 0 and 1 and is defined as the acceptable amount of pixels that can differ from the total amount of pixels.
• maxDiffPixels - this can be any value and is just a count of how many pixels can be different - it's worth experimenting with your test execution and seeing what an acceptable difference is.
• Threshold - this value can be between 0 (strict) and 1 (lax) and is the acceptable perceived color difference between the same pixel in the compared images - again, this is worth experimenting with and seeing how strict you want it to be.

await expect(headerLogo).toHaveScreenshot({ maxDiffPixelRatio: 0.1 })
await expect(headerLogo).toHaveScreenshot({ maxDiffPixels: 100 })
await expect(headerLogo).toHaveScreenshot({ threshold: 0.1 })

Defining these values in every test can be repetitive if they are generally the same - we have the option to define them in the Playwright config file in this instance. This can be applied globally to all tests or just to a specific project.

module.exports = {
expect: {
toHaveScreenshot: { maxDiffPixels: 100 },
},
};

Ignoring sections of the webpage during comparison

Most websites have some form of dynamic content, like a hero carousel, gallery feed, or list of headings - in most cases this could change on a daily or weekly basis, and constantly updating your baseline images isn’t feasible.

The nice solution to this is to use an option called ‘mask,’ which allows us to ignore a locator or group of locators from our comparison. For this Playwright visual regression testing tutorial, let's continue to use the eCommerce website - the main area that stands out to me is the image carousel, which constantly rotates between different products.

If we update our test to the following and run:

npx playwright test --update-snapshots

// example.spec.js
const { test, expect } = require('@playwright/test');


test('example test', async ({ page }) => {
await page.goto('https://ecommerce-playground.lambdatest.io/');
await expect(page).toHaveScreenshot({mask: [page.locator('.carousel-inner')]})
});

We will have new baseline images generated which look like this:

The bright pink area is the locator we masked, which will be ignored from all test runs. So let’s perform Playwright visual regression testing with our new baseline.

npx playwright test

And we should have three passing tests - nice one! Now we can perform Playwright visual regression testing against the elements of our website, which we know are static, and any issues raised will be genuine and not the result of content changes.

Implementing full page visual comparisons

Another visual comparison we can perform is a full page - this will capture the full height of the webpage. This can be useful if your website has a lot of scrollable components or content you need to verify.

Remember that as we capture a larger area, the chance of failure may be higher. We can reduce this risk by using a few additional parameters.

• animations: “disabled” - this will stop any CSS animations or transitions on your webpage.
• maxDiffPixelRatio: 0.2 - which we covered earlier, will allow some room for minor differences.

This is what the test looks like with the additional parameters added.

// example.spec.js
const { test, expect } = require('@playwright/test');


test('example test', async ({ page }) => {
await page.goto('https://ecommerce-playground.lambdatest.io/');
await expect(page).toHaveScreenshot
({ fullPage: true, animations: "disabled", maxDiffPixelRatio: 0.2 });
});

And if we execute the update screenshots command, we can see how the baseline will look. As you can see, some content and animations have been disabled, but the core components of the website have been rendered. So our visual test will cover quite a lot while remaining stable.

The test execution will be the same as before, covering Chromium, Firefox, and Webkit for the full page view. If we perform Playwright visual regression testing, we should see a successful output for all 3.

Running your Playwright suite on cloud with SmartUI

You can power up your Playwright visual regression testing suite by executing your tests on LambdaTest SmartUI cloud, where you can access 3000+ browser versions and various OS combinations with a single test execution from your Continuous Integration (CI) seamlessly.

LambdaTest is a cloud-based testing platform that enables you to perform Playwright automated testing at scale, significantly reducing the time required to run your Playwright tests.

By using LambdaTest, you can execute your Playwright tests on a comprehensive online browser farm consisting of more than 50 different browser versions, including Chrome, Chromium, Microsoft Edge, Mozilla Firefox, and WebKit. This allows you to test your applications on various environments, ensuring maximum coverage and compatibility.

You can also subscribe to the LambdaTest YouTube Channel and stay updated with the latest tutorial around Selenium testing, Cypress E2E testing, Appium, and more.

Drive your scale for your application with continuous integration with LambdaTest SmartUI for your Playwright suite. The following are the feature differences between the local testing and LambdaTest SmartUI.

To summarize, Playwright visual regression testing can be helpful for your initial POCs or during the development period of an application, website, etc. If you are a team of QA engineers looking to collaborate to build a CI/CD pipeline with Playwright visual regression testing, it is highly recommended to host your tests with LamdbaTest SmartUI for your test suites.

Getting started with Smart UI testing using Playwright

This section of this Playwright visual regression testing tutorial will act as your step-by-step guide to perform Playwright tests with SmartUI.

• Basic understanding of the Playwright is required.
• Go to SmartUI and log in along with your credentials.
• Copy LT_USERNAME and LT_ACCESS_KEY credentials from the Access Key button on the top right of the dashboard.

export LT_USERNAME="YOUR_USERNAME"
export LT_ACCESS_KEY="YOUR ACCESS KEY"

The following steps will guide you in performing your first Playwright visual regression testing on the LambdaTest platform -

Step 1: Create a SmartUI Project.

The first step is to create a project with the application in which we will combine all your builds to run on the project. To create a SmartUI Project, follow these steps:

• Go to the Projects page.
• Click on the New Project button.
• Add the project's name, approvers for the changes found, and tags for any filter or easy navigation.
• Click on the Submit button.

Step 2: Configure your test with Playwright Desired Capabilities.

Once you have created a SmartUI Project, you can generate screenshots by running automation scripts. Follow the below steps to generate screenshots successfully.

• Install the node modules using the command
npm i
• Set up the LambdaTest credentials by following the instructions mentioned in the README.md file.
• Edit the required capabilities in your test file playwright-smartui.js.
• Add the following code snippet to run SmartUI with Playwright.

const { chromium } = require('playwright')
const { expect } = require('@playwright/test');

(async () => {
  const capabilities = {
    'browserName': 'Chrome', // Browsers allowed: 'Chrome', 'MicrosoftEdge', 'pw-chromium', 'pw-firefox' and 'pw-webkit'
    'browserVersion': 'latest',
    'LT:Options': {
      'platform': 'Windows 10',
      'build': 'Playwright SmartUI Build',
      'name': 'Playwright SmartUI Test',
      'user': process.env.LT_USERNAME,
      'accessKey': process.env.LT_ACCESS_KEY,
      'network': true,
      'video': true,
      'console': true,
      'smartUIProjectName': '<projectName>' //Add the required Smart UI Project name
    }
  }

  const browser = await chromium.connect({
    wsEndpoint: wss://cdp.lambdatest.com/playwright?capabilities=encodeURIComponent(JSON.stringify(capabilities))'
  })

  const page = await browser.newPage()

  await page.goto('https://www.bing.com')

  // Add the following command in order to take screenshot in SmartUI
  await page.evaluate((_) => {},
    'lambdatest_action: {"action":"smartui.takeScreenshot","arguments":{"fullPage":true,"screenshotName":"<Your Screenshot Name>"}}') // Add a relevant screenshot name here

  const element = await page.$('[id="sb_form_q"]')
  await element.click()
  await element.type('LambdaTest')
  await page.waitForTimeout(1000)
  await page.keyboard.press('Enter')
  await page.waitForSelector('[class=" b_active"]')
  const title = await page.title()

  try {
    expect(title).toEqual('LambdaTest - Search')
    // Mark the test as completed or failed
    await page.evaluate(_ => {}, 'lambdatest_action: {"action":"setTestStatus","arguments":{"status":"passed","remark":"Title matched"}}')
  } catch {
    await page.evaluate(_ => {}, 'lambdatest_action: {"action":"setTestStatus","arguments":{"status":"failed","remark":"Title not matched"}}')
  }

  await browser.close()
})()


• Execute the test using the following command.
node playwright-smartui.js

You can check the executed builds over at LambdaTest SmartUI.

If you are a developer seeking to showcase your expertise as a Playwright automation tester, LambdaTest's Playwright 101 certification program is an excellent opportunity. This program is designed for individuals who wish to demonstrate proficiency in performing Playwright end-to-end testing on contemporary web applications. It offers a chance to validate your skills and highlight your expertise in this field.

Conclusion

Playwright visual regression testing is a powerful layer to add to your quality gates, which can provide extensive test coverage with minimal effort. It can also reduce the number of UI test cases you currently have, which are just checking the element or component displayed. Add visual testing to your next project and see how quickly your confidence grows in the quality of changes being made.