How to Use Selenium and Cypress for Canvas Automation

The Canvas element in HTML (or <canvas>) is majorly used for rendering graphics, animations, and other interactive content on a web page. Though the Canvas element is a part of the DOM, content inside the Canvas is not a part of it! This essentially means that the traditional DOM-based validations do not hold well for the automated handling of Canvas elements.

Over & above, automating Canvas interactions with Cypress is a tad easier than Selenium. This is because Cypress runs within the browser’s context, providing you seamless access to JavaScript APIs like the Canvas API. In this blog, we will be covering using Selenium (Python) and Cypress for Canvas element automation with hands-on examples.

What Is Canvas Element in HTML?

The Canvas (<canvas>) element is used for drawing graphics or other visual content on-the-fly on a web page. Along with animations, the Canvas element can also be used for drawing charts, animations, and other custom visualizations on the page.

The Canvas element was introduced as a part of the HTML5 standard for reducing the dependencies on plugins like Adobe Flash and Microsoft SilverLight, both of which have already reached EOL (End Of Life).

The element, which is essentially a blank container, heavily relies on JavaScript for drawing any graphics and performing interactive actions. As stated in the Canvas API Documentation, the Canvas APIs primarily focus on 2D graphics. WebGL (Web Graphics Library), a JavaScript API for rendering high-performance interactive 2D & 3D graphics, also makes use of the <canvas> element.

Example 1:

As shown below, graphics in the LambdaTest Bug Smasher Game are housed in the <canvas> element. After locating the Canvas element using the ID – document.getElementById(“unity-canvas”), we get the details (i.e., dimensions/size) of the said element.

The above command, when run on the Browser console, provides the following output:

• The Canvas element is identified using the ID attribute (i.e., unity-canvas)
• The width=”1920″ and height=”1200″ attributes define the actual size of the Canvas element used for drawing/rendering
• style=”width: 960px; height: 600px;” defines the display size of the Canvas element on the page. In this case, a scaling factor of 0.5 (width = 960/1920 and height = 600/1200) is used when locating elements in the Canvas via its coordinates

Example 2:

Another example of the Canvas element is a rendering of a Bar Graph in CanvasJS. After locating the Canvas element using Selenium locators document.getElementById(“themes-chart”), we get the details (i.e., dimensions/size) of the said element.

The above command, when run on the browser console provides the following output:

• The Canvas element is identified using the ID attribute (i.e., themes-cart)
• The width=”1892″ and height=”882″ attributes define the actual size of the Canvas element used for drawing/rendering
• style=”width: 946px; height: 441px;” defines the display size of the Canvas element on the page. In this case, a scaling factor of 0.5 (width = 946/1892 and height = 441/882) is used when locating elements in the Canvas via its coordinates

As seen from the above examples, the internal resolution and display size difference are different (i.e., scaling—0.5) and can affect the quality of the rendering. When automating interactions with Canvas elements, it is important to take the scaling into account to ensure the accuracy of mouse movement actions.

In this blog, I will automate Canvas elements on the local Chrome browser and/or Chrome installed on the LambdaTest cloud grid.

LambdaTest is an AI-based test orchestration and execution platform that allows automated tests to be run at scale in more than 3,000 environments. It also supports automating Canvas elements on multiple browsers using popular frameworks like Selenium and Cypress.

The getBoundingClientRect() Method

Though the <canvas> element is a part of the DOM, the elements rendered inside it are not represented in the DOM. This makes automating Canvas challenging, as the elements within the DOM have to be located using the co-ordinates.

This is where the getBoundingClientRect() method of the Element class can be leveraged, as it provides the size of the element and its properties relative to the viewport. The getBoundingClientRect() method returns a DOMRect object that contains the following properties (in pixels):

Property	Description
X	X-coordinate of the element relative to the viewport
Y	Y-coordinate of the element relative to the viewport
width	Width of the element
height	Height of the element
top	Distance from the top of the viewport to the top of the element
right	Distance from the left of the viewport to the right of the element
bottom	Distance from the top of the viewport to the bottom of the element
left	Distance from the left of the viewport to the left of the element

Shown below is the pictorial view of the BoundingRect() method that deep dives in the approach used for calculating the properties:

BoundingRect()

When automating interactions with the Canvas element, set a consistent viewport size (e.g., 1024 * 768) and maintain a constant zoom-level (100% preferred) for reliable automation results.

Let’s fetch the DOMRect properties, by executing the document. getElementById(“unity-canvas”).getBoundingClientRect command on the browser console. Shown below is the command in action for the LambdaTest BugSmasher game:

Chrome (Maximized) + Zoom-level (100%)

Now, let’s resize the browser window and check the properties contained in the DOMRect object:

Chrome (Resized) + Zoom-level (100%)

As seen from the above screenshots, certain DOMRect properties, namely X, Y, top, right, bottom, and left, change with a resized browser window. Like other Selenium best practices, it is recommended to instantiate the respective browser in maximized mode so that the DOMRect properties (i.e., top, right, bottom, and left) are consistent across subsequent runs.

To summarize, the getBoundingClientRect() method plays a pivotal role in handling Canvas elements by providing accurate size and coordinates relative to the viewport.

Tracking Mouse Movements in Canvas Elements

As seen so far, interaction with Canvas content is based on precise coordinates rather than element-based tracking (which is normally used with DOM-based interactions). This is precisely why tracking mouse positions becomes essential to ensure that automation mimics real user behavior.

During the process of writing this blog, I tried automating different Canvas element scenarios with Selenium as well as Cypress. Since Cypress tests run directly in the browser, I found it relatively easy to use Cypress for Canvas automation.

As far as automation with Selenium is concerned, this insightful StackoverFlow thread helped a lot in calculating Canvas element coordinates more robustly. As stated in the W3C specification, the offset of action commands is calculated from the center of a WebElement.

moveToElement offset calculation on Chrome & Firefox browsers

Regardless of the automation framework used, such as Selenium or Cypress for Canvas automation, it is imperative to track mouse movements so that coordinates can be adjusted as needed. Since AI has made significant inroads into every aspect of software development, I decided to generate tests using AI.

I used the prompt “Show a dot wherever the mouse pointer is.” This code will be used for Selenium and Cypress for Canvas automation and fed into the AI tool that assisted me with code development.

Here is the AI-generated code that will show a Red (#FF0000) dot with every mouse movement.

Here is a short gist of the above AI-generated snippet, the primary purpose of which is to position a Red (#FF0000) colored dot relative to an element in the Canvas. Let’s dive deep into the essential aspects of the snippet:

Step 1. Cypress.Commands.add adds a Cypress custom command (named showClickCoordinates) that can be reused throughout the tests. It takes three input parameters:

• element: DOM element where the click needs to be highlighted
• x: X-coordinate calculated relative to the element’s bounding box
• y: Y-coordinate calculated relative to the element’s bounding box

Step 2: getBoundingClientRect() provides the element’s size and position relative to the viewport. For the current use case, we extract the top, left, width, and height properties of the said element.

Step 3: cy.window() helps provide access to the browser’s window object (i.e., win)

Step 4: This is the crux of the code where win.document.createElement helps in creating a new div element in the DOM (to act as a dot). Here is the styling of the dot:

• position: absolute: Places the dot at an exact mouse position
• width and height: Defines the size of the dot
• backgroundColor: ‘red’: Makes the dot visible as red (#FF0000)
• borderRadius: ‘50%’: Makes the dot a circle
• zIndex: ‘9999’: Ensures the dot appears above other elements

Step 5: The top (rect.top) and left (rect.left) styles are calculated to position the dot relative to the element in the viewport. Here, x and y are the coordinates offset inside the element.

The dot is finally appended to the body of the document, making it visible on the page.

Step 6: dot.remove() removes the displayed dot to clean up the DOM after a set timeout of 3 seconds.

Shown below is a sample use of the Mouse Movement tracking code when used for automating Canvas elements in the LambdaTest Bug Smasher game:

Mouse Movement Tracking Example

We will use the above AI-generated code while using Selenium and Cypress for Canvas automation examples, which will be demonstrated in later sections of the blog. Since Canvas interactions are heavily based on coordinates, you could also use Chrome/Brave extensions like Page Ruler, which lets you measure distances (in pixels) on a webpage.

Now, let’s get our hands dirty with some hands-on experiments with Selenium and Cypress or Canvas automation!

Using Cypress for Canvas Automation

For this demonstration, we will automate the play of LambdaTest Bug Smasher. If you are new to Cypress on the LambdaTest cloud, you can check out the blog on how to write your first Cypress test, which dives into the nuances of Cypress execution on LambdaTest.

Canvas automation demonstration should work seamlessly on Cypress LambdaTest cloud as well as on Cypress installed on your local machine. In both cases, Chrome is chosen as the browser for the test. The configuration for executing the Canvas tests on LambdaTest is available in lambdatest-config.json. You need to add your LambdaTest user name and access key, details of which can be fetched from the Account Settings Page.

Now that the setup is ready, let’s look at how we automated gameplay using Cypress; the entire implementation is available in canvas-element-handling.cy.js

Code Walkthrough:

Step 1: To get started, set the viewport to 1024 * 768 using the cy.viewport() method. Post that, and navigate to the LambdaTest Bug Smasher game page. It is important to note that the coordinate values used in the tests might need to be changed if the viewport size changes.

Step 2: Next, the cy.document() method yields the window.document object (i.e.,, doc). The doc.readyState property indicates the document’s current loading state.

The doc.readyState property value is checked if it is complete, which indicates that the document and its corresponding resources (e.g., images, stylesheets, etc.) are fully loaded.

Step 3 (Optional): This step simply retrieves the dimensions of the viewport (in pixels). In our case, we set the viewport size to 1024 (width) and 768 (height).

Once the window object is retrieved with the cy.window() method, the innerWidth and innerHeight properties are obtained to fetch the width & height of the viewport, respectively.

Step 4: The Canvas element is located using the ID locator (i.e., #unity-canvas), which is passed to the cy.get() method.

Once the <canvas> element is located, its width & height (when rendered on the page) properties are read using $canvas[0].width & $canvas[0].height respectively. The intrinsic resolution of the Canvas in pixels is set to width: 1920px; height: 1200px;

It is important to note that these properties indicate the CSS-rendered size of the Canvas on the document (i.e., width: 960px; height: 600px;).

Step 5: The remaining Canvas properties (X, Y, top, bottom, left, right) are also read using the getBoundingClientRect() method, which we discussed in earlier sections of the blog.

BoundingRect()

The CSS-rendered Canvas width & height are assigned to canvasDisplayedWidth & canvasDisplayedHeight variables, respectively.

Step 6: The scaleFactor is set to 0.5 since the CSS-rendered Canvas width is half of the Canvas displayed width. centerX & centerY of the Canvas are calculated for operations to be performed on the Canvas.

Step 7: When calculating the offset, I played around with a few values after getting the property values using the getBoundingClientRect() method.

The variables buttonXInCanvas and buttonYInCanvas are set to twice of Canvas center values.

Step 8: The X coordinate from the earlier step is multiplied by the scale factor (i.e., 0.5 in this case).

Step 9: The cy.wrap() wraps a DOM element (i.e., <canvas> element) into a Cypress chainable object.

Here, the Cypress custom command (e.g., showClickCoordinates), which shows a red-colored dot in the center of the Canvas (e.g., canvasCenterX & canvasCenterY), is wrapped in the cy.wrap() method.

Step 10: The next important step is to locate the Play Now button in the Canvas. As seen in the screenshot below, the position of the Play Now button can be computed by negating a couple of pixel values from buttonYInCanvasScaled (which is set to canvasCenterY * 2).

In our case, subtracting Top/Y (i.e., 89.5 pixels) from buttonYInCanvasScaled provided the exact coordinates of the Play Now button. Since the value of the Top/Y property might change with the zoom level and viewport size, it is always recommended to keep the zoom level to 100% and set the viewport size (e.g., 1024 * 768 in this case) before triggering the tests.

Step 11: Cypress invokes the click() method to click on the Play Now button. The force option in the click() method is set to true since it bypasses certain default checks (e.g., element visibility, interactability, etc.) that Cypress performs before interacting with elements.

The option might not be needed in our case since the Play Now button is also visible and interactable.

Step 12 (Final Game Play—1): This takes us inside the game arena! As mentioned so far, it would be time-consuming to locate the objects (e.g., bugs) and then act on them.

An easier approach would be to create random X and Y coordinate value pairs, taking into consideration the required properties returned by the getBoundingClientRect() method.

The getRandomCoordinates() function provides randomized X and Y coordinates using the following calculation: a float random number between 0 and 1 is generated using the Math.random() function in JavaScript.

BoundingRect()

The generated random number is multiplied by the Canvas width (for X) and height (for Y). The resultant is then added to the top property (for X) and left property (for Y). A buffer of 200 pixels is added to the final result to ensure that the generated X-Y coordinates are located within the game area (marked in red in the screenshot).

Game Area

If the generated values are greater than canvasWidth and/or canvasHeight, they are set to canvasCenterX and canvasCenterY, respectively.

Step 13 (Final Game Play – 2): Now that we have the coordinates handy, a click is generated every 500 ms using the cy.click() method. Since no object model detection is implemented in the code, there is a possibility of clicking on a bomb object, which results in the end of the game.

Step 14 (Clicking the LinkedIn Button—Outside Canvas): In the LambdaTest Bug Smasher game, the social media icons (e.g., LinkedIn and Twitter) are placed outside the Canvas.

First, the LinkedIn button is located with the CSS Selector locator. The window.open() method is stubbed with the cy.stub(win, ‘open’).as(‘windowOpen’), where win is the window handle.

Since the window is stubbed, it does not open an actual window or tab. This Cypress tutorial explains Cypress stubs and other important concepts related to the framework.

Finally, a click operation is performed on the window, and Cypress.sinon.match() is called to ensure that the string contains the substring linkedin.com.

Now, let’s execute the tests on Cypress on the LambdaTest cloud grid and Cypress installed on a local machine!

Execution

Before we start the tests, let’s set up the virtual environment (venv), which will help us better manage the environment and dependencies.

venv plays an instrumental role in providing isolation from the packages installed in the base environment. Run the commands virtualenv venv and source venv/bin/activate on the terminal to create the virtual environment.

Next, export the environment variables LT_USERNAME and LT_ACCESS_KEY on the terminal.

export LT_USERNAME=

export LT_ACCESS_KEY=

You can get these details from the Password & Security section. For execution on LambdaTest, first, install the dependencies by triggering the command make install from the root folder of the project.

With the dependencies installed & environment variables exported, you can trigger the command make cloud-canvas-automation-cypress from the root folder on the terminal. This makes the command internally execute the command lambdatest-cypress run that triggers the Canvas element test on the LambdaTest grid.

You can check out the support documentation on getting started with Cypress testing in case you encounter any issues with the execution. Once the tests are triggered, you can check the status on the LambdaTest Automation Dashboard.

LambdaTest Automation Dashboard

As seen above, we automated interactions with the Canvas element and also achieved a score of 85 🔥

The same test can also be executed on Canvas (on the local machine) by triggering the command make local-canvas-automation-cypress, which internally triggers npx cypress run-browser chrome-headed to the terminal.

With this, we have learned to use Cypress for Canvas automation! However, regardless of the Canvas element type (e.g., bar graphs, pie charts, etc.), you can still use the learnings from this section to automate Canvas interactions!

Using Selenium for Canvas Automation

There are scenarios where you might need to extract stats from graphs (e.g., pie charts, bar graphs, etc.) and report them to your organization’s higher stakeholders. For example, a bar graph can showcase an organization’s monthly sales data. Similar to this, the data can also be projected in the form of pie charts, line charts, and more!

Here is another example of a pie-chart deep diving into Employee Distribution by location. An automation script can be written to extract the tool-tip information from each quadrant in the <canvas> element.

Pie Chart Canvas Example – OrangeHRM

To demonstrate how Selenium handles Canvas elements, we will extract monthly sales data from a Canvas element in CanvasJS.

Sample Sales Data – CanvasJS

You can check out my Selenium Python tutorial if you want a quick refresher on automation with Selenium and PyUnit (& Pytest). In this Canvas bar graph example, we have used the PyUnit/unittest framework. However, the test can be easily ported for testing with the pytest framework.

Code Walkthrough:

Step 1: Since we are using the PyUnit (or unittest) framework for test automation, the browser & platform capabilities are set in the method implemented under the setUp() annotation.

The capabilities are generated using the LambdaTest Capabilities Generator, and the username and access key are added to helpers/utils.py.

Step 2: Since the test URL has dynamic content, scrolling vertically to the bottom of the page (and starting) ensures that all the required assets are available.

document.body.scrollHeight returns the total height of the page content. To allow the scrolling to complete and give the page content enough time to load, a blocking sleep of 2 seconds (i.e., time. sleep (2)) is added.

Lastly, window.scrollTo(0, 0) scrolls to the start of the page and a sleep of 2 seconds ensures that any UI changes triggered by scrolling are complete.

Step 3: Now that the page is loaded, the Canvas element is located using the ID locator in Selenium (i.e., themes-cart). The presence_of_element_located ExpectedCondition in Selenium is added to check if the Canvas element is present in the DOM.

The presence is checked with a WebDriverWait of 30 seconds. If the element is not located within this time, an exception is raised, and the test is considered failed.

Step 4: Akin to using Cypress for Canvas automation, the Canvas properties are extracted with the getBoundingClientRect() method in JavaScript.

BoundingRect()

Here, the JavaScriptExecutor in Selenium is leveraged with the execute_script method to execute JS code directly in the browser’s context. The implementation of this can be found in utils.py.

Step 5: Once the Canvas properties are derived, the center (X & Y) of the Canvas element is deduced by halving the display size of the <canvas> element on the web page (i.e., 946 * 441 pixels). The X & Y coordinates offset is set to 100, we will touch upon the reasons in the further section of this blog.

Step 6: To compute the month-on-month sales data, we first need to hover over the corresponding bar (for the respective month) and then read the tooltip. For this, the absolute X, Y value pair is read using the Mouse Coordinates Chrome extension.

X & Y coordinates of data marked as December

X & Y coordinates of data marked as November

As seen above, hovering over (1230, 389) should provide the sales data for December. On similar lines, hovering over (1130, 389) should provide the sales data for November. This means that the X offset can be set to 100 pixels. The coordinate value of the Y-axis can be set to Canvas center height (i.e. 221 +- 100 pixels). The logic mentioned for this test needs changes if the data is read from a pie chart or some other form of representation.

Since we need to find sales data for 12 months (January ~ December), a descending For loop from (12:0) is executed in the code. The move_to_element_with_offset() method which is a part of the Actions Class in Selenium, helps in moving to the respective X, Y pair in the Canvas.

The video below delves into the essential fundamentals of the move_to_element() method, including how to click on a specific area of an element.

Post the movement of the mouse pointer to a specific position relative to the center of the <canvas> element, a click operation is simulated at the current position.

Step 7: The sales data for the corresponding month are displayed as tooltip text on every bar. In Selenium, the tooltip element (inside the Canvas element) is located using the Class Name locator.

Next, the visible text content of the tooltip_elem is printed on the console using print(tooltip_elem.text).

After executing the descending For loop (from 1 to 12), the sales data for all 12 months should be printed on the terminal.

Execution

First up, export the environment variables LT_USERNAME and LT_ACCESS_KEY on the terminal.

export LT_USERNAME=LT_USERNAME

export LT_ACCESS_KEY=LT_ACCESS_KEY

You can get these details from the profile section on LambdaTest. In case you have not installed the dependencies, you can do the same by triggering the command make install from the root folder of the project.

Run the command make cloud-canvas-automation-python on the terminal, post which the tests would be executed on the LambdaTest grid and sales data is printed on the terminal.

Like before, you can also check the test execution status by visiting the LambdaTest Automation Dashboard.

LambdaTest Automation Dashboard Snapshot

Many more Canvas element scenarios can be automated using Selenium/Cypress, but covering all of them is beyond the scope of this blog.

Bonus: Automating Tasks Using PyAutoGUI

Until now, we have used Selenium and Cypress to automate interactions with the Canvas element on a webpage or document. Though the above-demonstrated tests can be executed seamlessly on local and LambdaTest cloud grids, the PyAutoGUI module can also be used to automate interactions on local machines.

PyAutoGUI is a cross-platform Python module for GUI automation. It is designed to programmatically control the mouse and keyboard. To install it, trigger pip3 install pyautogui (or pip install pyautogui) from the terminal.

GitHub

As stated in the official documentation, the x, y coordinates used by PyAutoGUI have the (0, 0) origin coordinates that start from the top left corner of the screen. On a screen that is 1920 * 1080 pixels in size, coordinates (0, 0) are for the top left while (1919, 1079) is for the bottom right.

It is important to note that PyAutoGUI interacts with the GUI directly by simulating mouse and keyboard events. Understandably, PyAutoGUI tests might fail or produce unexpected results if the window is switched while the execution is in progress.

For demonstration, we have automated LambdaTest Bug Smasher on Chrome (v131.0.6778.205) installed on macOS (Sequoia 15.2). Like before, the viewport size is set to (1024 * 768). The coordinates of the Play Now button and logic associated with the random generation of coordinates are similar to the Cypress Canvas Automation demo discussed in the earlier section of this blog.

The entire implementation is available in canvas_autogui.py.

The only difference is the use of the pyautogui.click() method, which moves to the X and Y coordinates and clicks the left mouse button.

Execution

Run the command make local-canvas-automation-python on the terminal. The Chrome browser will then be instantiated, and the PyAutoGUI module will automate gameplay interactions.

The Python tests showcased as a part of this blog can be easily ported to make it work with the pytest framework.

Challenges in Automating Canvas Interactions

The <canvas> element differs a lot from traditional HTML elements, primarily in how content within the Canvas is laid out and accessed. Unlike HTML elements that are a part of the DOM, Canvas content is outside the DOM structure.

Here are some of the major challenges when automating Canvas elements with frameworks like Selenium/Cypress:

Reliance on Pixel-Based Automation

While test automation frameworks like Selenium and Cypress rely on locator strategies and built-in APIs to realize element interactions, the automation of the Canvas element combines coordinate-based actions with the use of JavaScript APIs.

For example, we can automate actions on LambdaTest E-Commerce Playground by locating elements and using appropriate Selenium WebDriver APIs. The same can also be achieved with the help of the Cypress framework.

Example of DOM-based interaction

In case of a Canvas element, you first need to locate the Canvas element using the best-suited locators. Post that, move to the respective coordinates (or at an offset from the Canvas) and then perform respective actions (e.g., clicks, send keys, etc.) using built-in framework APIs or JavaScript injection.

As stated earlier, Canvas automation can be flaky because it relies on pixel-based rendering rather than a DOM-based structure.

Scaling, Zoom, and ViewPort Dependencies

Automation tests with Selenium can become flaky if the scaling, viewport size, and zoom levels are not set to constant values across tests. For instance, if the zoom level is changed from 100% to 200%, an ElementNotInteracable exception or an element not in the viewport may occur.

These pointers are valid, but you also need to ensure that the automation code can adapt to changes in viewport size, browser zoom level, and Device Pixel Ratio (DPR).

Canvas Example: LambdaTest Bug Smasher Game (Zoom – 100%)

Shown below is the browser snapshot that shows the change in the X & Y coordinates of the “Play Now” button once the zoom level is set to 200%

Canvas Example: LambdaTest Bug Smasher Game (Zoom – 200%)

Canvas automation code is more vulnerable to these variable factors than DOM-based Selenium tests, as element interactions are based on pixel positions within the Canvas.

Cross Browser Differences

The <canvas> element is compatible with widely-used browsers – Chrome, Firefox, Safari, and Edge. Since every browser uses its rendering engine, Canvas scaling and rendering will differ based on browser zoom levels, viewport settings, and devicePixelRatio.

Source

The underlying rendering engine of browsers can impact the layout and rendering of Canvas elements. This is because the implementation of the HTML5 Canvas API can differ from one engine to another.

In summary, subtle differences in the visual output can make tests that automate interactions with the Canvas elements flaky.

Complex Element Interactions

As elements in the Canvas are not a part of the DOM, interacting with frequently used elements like text boxes, buttons, sliders, etc., becomes a huge challenge. Operations like scrolling, moving by an offset, drag & drops, mouse actions etc., require a JavaScript-based approach or usage of Actions Class in Selenium.

Example – Entering personal information in text-boxes of the LambdaTest Bug Smasher involves locating the elements using move_to_element_with_offset() method (in Selenium) and respective its co-ordinates. This approach can be more cumbersome if there are more text-boxes in the Canvas 🙁

Over & above, you might need to put significant effort into finding coordinate offset if you have to test <canvas> interactions across different browsers, devices, and OS combinations!

Handling Dynamic Elements

Selenium Wait comes in handy when you are automating websites that have dynamic elements (or dynamic content). It helps in improving test reliability and addresses timing issues that arise due to factors like network throttling, WebElement visibility, and more.

However, dynamic content in Canvas that involves gestures, drawings, or animations cannot be automated in the way it is done in Selenium. Content in a <canvas> element is a single graphics block, owing to which it is difficult to predict when the said element will appear in the Canvas, even in non-throttling (or stable) conditions!

Over & above, Canvas content does not have a layered (or hierarchical structure) like the Document Object Model (DOM). Layers (or z-index-style ordering) must be simulated programmatically.

Example – Selenium/Cypress is not extensively used for game automation (that contains <canvas> element). However, we took up the LambdaTest BugSmasher example to showcase the potential challenges that come with automating dynamic content in Canvas.

LambdaTest Bug Smasher

As seen above, it is difficult to predict the presence of bugs placed at random locations in the Canvas. In such cases, the best possible solution is to randomize the X and Y coordinate pairs so that the click falls within the Canvas.

Other issues associated with automated testing of Canvas elements include accessibility issues, the need for image-based verification, and limited debugging capabilities.

Conclusion

Unlike DOM-based automation, automating Canvas elements can be challenging due to their reliance on pixel-based graphics. Test automation frameworks like Selenium, Cypress, Playwright, etc., can be leveraged to automate interactions with Canvas elements on a web page.

It is essential to adhere to Selenium best practices to minimize test flakiness and ensure consistency in test execution when automating Canvas elements. The learnings of this blog can help you get started with using Selenium and Cypress for Canvas automation – a choice that you should make based on the project & usecase requirements.

Happy Testing!

Citations

• HTML Living Standard: https://html.spec.whatwg.org/multipage/canvas.html
• Canvas API: https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API
• canvasJS: https://canvasjs.com/
• WebDriver W3C Recommendation: https://www.w3.org/TR/webdriver1/#pointer-actions
• Cypress document() method: https://docs.cypress.io/api/commands/document
• Cypress wrap() method: https://docs.cypress.io/api/commands/wrap
• Cypress.sinon: https://docs.cypress.io/api/utilities/sinon
• Mouse Coordinates Chrome extension: https://chromewebstore.google.com/detail/mouse-coordinates/gkkmpbaijflcgbbdfjgihbgmpkhgpgof?hl=en