Discover AI-native insights at Testμ Conference‘25

How to use test_wait_for_load_state_should_resolve_immediately_if_loaded method in Playwright Python

Best Python code snippet using playwright-python

test_navigation.py

Source:test_navigation.py

...509    await page.goto(server.PREFIX + "/one-style.html", wait_until="domcontentloaded")510    with pytest.raises(Error) as exc_info:511        await page.wait_for_load_state("load", timeout=1)512    assert "Timeout 1ms exceeded." in exc_info.value.message513async def test_wait_for_load_state_should_resolve_immediately_if_loaded(page, server):514    await page.goto(server.PREFIX + "/one-style.html")515    await page.wait_for_load_state()516async def test_wait_for_load_state_should_throw_for_bad_state(page, server):517    await page.goto(server.PREFIX + "/one-style.html")518    with pytest.raises(Error) as exc_info:519        await page.wait_for_load_state("bad")520    assert (521        "state: expected one of (load|domcontentloaded|networkidle)"522        in exc_info.value.message523    )524async def test_wait_for_load_state_should_resolve_immediately_if_load_state_matches(525    page, server526):527    await page.goto(server.EMPTY_PAGE)...

StackOverFlow community discussions

Questions

Discussion

cannot import name 'PageCoroutine' from 'scrapy_playwright.page'

Playwright auto-scroll to bottom of infinite-scroll page

Using Playwright for Python, how to I read the content of an input box

Is there a way to handle dynamically loaded selectors with clicks on pages in scrapy-playwright?

How to select an input according to a parent sibling label

Problem installing PlayWright Chrome on Heroku using Python

Web-scraping JavaScript page with Python

Playwright Python - Tab Link Not Visible

Playwright - how to find input that will contain a value?

What are the differences between Python Playwright sync vs. async APIs?

PageCoroutine is deprecated/obsolute. Use playwright_page_methods instead.

Working code as an example:

import scrapy
from scrapy_playwright.page import PageMethod

class TestSpider(scrapy.Spider):
    name = "test"
    def start_requests(self):
        yield scrapy.Request(

            url="https://shoppable-campaign-demo.netlify.app/#/",
            callback=self.parse,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", '.card-body'),
                ],
            },
        )

    def parse(self, response):
        
        products = response.xpath('//*[@class="card-body"]')
        for product in products:
            yield {
            'title':product.xpath('.//*[@class="card-title"]/text()').get()
          
            }

Output:

{'title': 'Oxford Loafers'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Ankle-length Slack'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'White Baseball Cap'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Triangle Bikini Top'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Short Blazer'}
2022-11-05 20:40:40 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-05 20:40:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 235,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 39851,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 41.370211,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 11, 5, 14, 40, 40, 261151),
 'item_scraped_count': 5,

https://stackoverflow.com/questions/74328303/cannot-import-name-pagecoroutine-from-scrapy-playwright-page

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

https://stackoverflow.com/questions/69183922/playwright-auto-scroll-to-bottom-of-infinite-scroll-page

Another way is to simply use the getAttribute() method of the elementHandle to retrieve the value:

handle = page.querySelector('//input[starts-with(@name, "Counter")]')
value = handle.getAttribute("value")

https://stackoverflow.com/questions/64790747/using-playwright-for-python-how-to-i-read-the-content-of-an-input-box

Maybe you can check the selector is visible and then you can click it?

if PageMethod("isVisible","#selector1"):
    PageMethod("click","#selector1")    
if PageMethod("isVisible","#selector2"):
    PageMethod("click","#selector2")    
if PageMethod("isVisible","#selector3"):
    PageMethod("click","#selector3")

I think you can usue the method isVisible, because scrapy_playwright support the same you have in playwright: https://github.com/scrapy-plugins/scrapy-playwright#supported-methods

And of course, isVisible() is a supported method in playwrgiht: https://playwright.dev/python/docs/api/class-page#page-is-visible

https://stackoverflow.com/questions/74380158/is-there-a-way-to-handle-dynamically-loaded-selectors-with-clicks-on-pages-in-sc

If you try to perform an action on a label which has associated input element Playwright will automatically retarget the action to the input.

If the DOM structure is always the same you can try something like this

page.locator('div:has( > :text-matches("Project")) + div input')

Otherwise layout selector is your best bet.

https://stackoverflow.com/questions/70986996/how-to-select-an-input-according-to-a-parent-sibling-label

Heroku's filesystem is both ephemeral and dyno-local. Any changes made to it are lost when the dyno restarts. More importantly at the moment, when you run something like heroku run bash you don't connect to a running dyno. Instead, you get a separate one-off dyno.

Installing Chromium (or making any other filesystem changes) in such an environment will never affect what is available in your other dynos.

It looks like Playwright really wants to manage its own browser binaries. Your best bet might be to run playwright install near the beginning of your script, e.g. something like this:

import subprocess
from playwright.sync_api import sync_playwright

subprocess.run(["playwright", "install"])

with sync_playwright() as playwright:
    # ...

This should make sure that the required binaries are always available at runtime.

https://stackoverflow.com/questions/73071242/problem-installing-playwright-chrome-on-heroku-using-python

EDIT Sept 2021: phantomjs isn't maintained any more, either

EDIT 30/Dec/2017: This answer appears in top results of Google searches, so I decided to update it. The old answer is still at the end.

dryscape isn't maintained anymore and the library dryscape developers recommend is Python 2 only. I have found using Selenium's python library with Phantom JS as a web driver fast enough and easy to get the work done.

Once you have installed Phantom JS, make sure the phantomjs binary is available in the current path:

phantomjs --version
# result:
2.1.1

#Example To give an example, I created a sample page with following HTML code. (link):

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Javascript scraping test</title>
</head>
<body>
  <p id='intro-text'>No javascript support</p>
  <script>
     document.getElementById('intro-text').innerHTML = 'Yay! Supports javascript';
  </script> 
</body>
</html>

without javascript it says: No javascript support and with javascript: Yay! Supports javascript

#Scraping without JS support:

import requests
from bs4 import BeautifulSoup
response = requests.get(my_url)
soup = BeautifulSoup(response.text)
soup.find(id="intro-text")
# Result:
<p id="intro-text">No javascript support</p>

#Scraping with JS support:

from selenium import webdriver
driver = webdriver.PhantomJS()
driver.get(my_url)
p_element = driver.find_element_by_id(id_='intro-text')
print(p_element.text)
# result:
'Yay! Supports javascript'

You can also use Python library dryscrape to scrape javascript driven websites.

#Scraping with JS support:

import dryscrape
from bs4 import BeautifulSoup
session = dryscrape.Session()
session.visit(my_url)
response = session.body()
soup = BeautifulSoup(response)
soup.find(id="intro-text")
# Result:
<p id="intro-text">Yay! Supports javascript</p>

https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python

From the docs, you should use a string as selector.

page.waitForSelector(selector[, options])

selector <string> A selector to to query for

page.click(selector[, options])

selector A selector to search for an element

selector = '//*[@id="sections-scroll-target"]/ul/ul/li[8]/a'
    
page.wait_for_selector(selector, timeout=10000)

page.hover(selector)

Looks like you want https://www.testdummysite.com/news/markets?mod=nav_left_section

If so this will work

//*[@href= "https://www.testdummysite.com/news/markets?mod=nav_left_section" ]

but these get a different link https://www.testdummysite.com/?mod=nav_left_section.

//*[@id="sections-scroll-target"]/ul/ul/li[8]/a

#sections-scroll-target > ul > ul > li:nth-child(8) > a

Because these URLs are similar, I wonder if you've posted the HTML accurately?

https://stackoverflow.com/questions/71631162/playwright-python-tab-link-not-visible

to_have_value supports regular expressions. You can do something like this:

expect(self.page.locator(input_locator).first).
to_have_value(re.compile(r"123.+"), timeout=20 * 1000)

https://stackoverflow.com/questions/74983650/playwright-how-to-find-input-that-will-contain-a-value

Blogs

Check out the latest blogs from LambdaTest on this topic:

Selenium 4.0 and The Future: Manoj Kumar [Testμ 2022]

We were eager to listen to Manoj Kumar, VP Developer Relations, LambdaTest, speak on the importance of Selenium 4.0 and how bright the future is. This was the agenda of the speech:

Webinar: Test Orchestration using HyperExecute

The speed at which tests are executed and the “dearth of smartness” in testing are the two major problems developers and testers encounter.

How To Use Playwright For Web Scraping with Python

In today’s data-driven world, the ability to access and analyze large amounts of data can give researchers, businesses & organizations a competitive edge. One of the most important & free sources of this data is the Internet, which can be accessed and mined through web scraping.

How To Handle Captcha In Selenium

With the rapidly evolving technology due to its ever-increasing demand in today’s world, Digital Security has become a major concern for the Software Industry. There are various ways through which Digital Security can be achieved, Captcha being one of them.Captcha is easy for humans to solve but hard for “bots” and other malicious software to figure out. However, Captcha has always been tricky for the testers to automate, as many of them don’t know how to handle captcha in Selenium or using any other test automation framework.

Getting Started With DevOps – A Beginner’s Guide

Are you looking to get started with DevOps or willing to shift gears in your professional career by adding DevOps as a skill? If your answer is yes, you have arrived at the right place!

Playwright tutorial

LambdaTest’s Playwright tutorial will give you a broader idea about the Playwright automation framework, its unique features, and use cases with examples to exceed your understanding of Playwright testing. This tutorial will give A to Z guidance, from installing the Playwright framework to some best practices and advanced concepts.

Chapters:

What is Playwright : Playwright is comparatively new but has gained good popularity. Get to know some history of the Playwright with some interesting facts connected with it.
How To Install Playwright : Learn in detail about what basic configuration and dependencies are required for installing Playwright and run a test. Get a step-by-step direction for installing the Playwright automation framework.
Playwright Futuristic Features: Launched in 2020, Playwright gained huge popularity quickly because of some obliging features such as Playwright Test Generator and Inspector, Playwright Reporter, Playwright auto-waiting mechanism and etc. Read up on those features to master Playwright testing.
What is Component Testing: Component testing in Playwright is a unique feature that allows a tester to test a single component of a web application without integrating them with other elements. Learn how to perform Component testing on the Playwright automation framework.
Inputs And Buttons In Playwright: Every website has Input boxes and buttons; learn about testing inputs and buttons with different scenarios and examples.
Functions and Selectors in Playwright: Learn how to launch the Chromium browser with Playwright. Also, gain a better understanding of some important functions like “BrowserContext,” which allows you to run multiple browser sessions, and “newPage” which interacts with a page.
Handling Alerts and Dropdowns in Playwright : Playwright interact with different types of alerts and pop-ups, such as simple, confirmation, and prompt, and different types of dropdowns, such as single selector and multi-selector get your hands-on with handling alerts and dropdown in Playright testing.
Playwright vs Puppeteer: Get to know about the difference between two testing frameworks and how they are different than one another, which browsers they support, and what features they provide.
Run Playwright Tests on LambdaTest: Playwright testing with LambdaTest leverages test performance to the utmost. You can run multiple Playwright tests in Parallel with the LammbdaTest test cloud. Get a step-by-step guide to run your Playwright test on the LambdaTest platform.
Playwright Python Tutorial: Playwright automation framework support all major languages such as Python, JavaScript, TypeScript, .NET and etc. However, there are various advantages to Python end-to-end testing with Playwright because of its versatile utility. Get the hang of Playwright python testing with this chapter.
Playwright End To End Testing Tutorial: Get your hands on with Playwright end-to-end testing and learn to use some exciting features such as TraceViewer, Debugging, Networking, Component testing, Visual testing, and many more.
Playwright Video Tutorial: Watch the video tutorials on Playwright testing from experts and get a consecutive in-depth explanation of Playwright automation testing.