How to use test_should_fire method in Playwright Python

https://www.tessco.com/product/207882 loads quit slow. Try to extend the default timeout of 30000ms adding a timeout to page.goto(link):

page.goto(link, timeout = 0)

With setting timeout to 0 you disable the timeout. Documentation

Alternatively, you can disable timeout with the following:

page.set_default_timeout(0)
page.goto(link)

I would do this:

1. Register a listener with expose_function.
2. Use evaluate to register a listener to newPosts. A Listener on the browser side.
3. Inside that listener call the function you exposed on 1.

You can see from the documentation here that you can use pseudo-class of has-text

... the :has-text() pseudo-class

and maybe some some more advance matching selector with pseudo-class text-matches

the :text-matches() pseudo-class

but you need to remember that two ways is better if you combine the usage with some css selector.

Maybe you can add more details on where you want to search the string from is it literally in the entire document or not. (and I can update the answer with more detail code example according to your target)

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

You could use a regular expression for your locator:

page.locator(text=/\d{1,2}\sseconds?\slater/i)

If you don't want to use context managers then you can start the browser without one using the .start() method:

from playwright.sync_api import sync_playwright

p = sync_playwright().start()
browser = p.chromium.launch(headless=False)
page = browser.new_page()
p.stop()

Remember to use the .stop() method once you are done.

I had a similar problem: I had PM2 run by an Apache Airflow Celery worker running a playwright task, hourly. The job hung.

I suspect it is the environment variables littered by Byobu and PM2. What worked for me was wrapping the Python script in a bash script and sanitizing the environment variable like the following:

#!/bin/bash
[ "$HOME" != "" ] && exec -c $0
# Your python script should come below:
/home/ubuntu/usr/venv-3.10-airflow/bin/airflow celery worker

The first line clears the env vars (as suggested in Sanitize environment with command or bash script?). Now you can run this script with pm2 start command. My worker ran fine for about 24 hours as of now. I hope this works for you too.

There is several good practices for creating locators/selectors. Using playwright there is official documentation for each common and unique selector on how-to and what-is doing.

More information in https://playwright.dev/docs/selectors#text-selector

About your case, i would suggest always to use an parent selector for locating an element. When there is a button, try to find its unique parent.

• By id
• By unique class
• Something else unique.

Example:

<dv id=test>
 <button id=continue-test>Continue</button>
</div>

In this case you can use the unique id of the button and not the text. Selector css: #continue-test But if you, don't have an unique identifier for the button you can use the parent and go down to the button. Selector css: #test > button

Matching text using css is not possible, but with XPATH can look like this:

//button[text()="Continue"] 

This selector MATCHES the text using "equals".

Using playwright:

button:has-text("Continue")

Using has-text and quotes - matches the text using equals. If you are using another selector for example text=Continue, this will match all elements that CONTAINS the text "Continue"

All this is explained with example in the official documentation for playwright selectors. That does not mean to not use XPATH to achieve the goals.

CSS selectors are fast but kind of restricted to work with text.

Xpath is quite slower but much more powerful to work in text/parent/child elements etc.

I would suggest always to use an parent element with unique identifier and go down to reach your actual element, which will receive the interaction.

The content function relies on document.documentElement.outerHTML, so you might get a formatted value. If the request returns a JSON you could grab the text from the response that goto returns:

response = page.goto("https://raw.githubusercontent.com/corysimmons/colors.json/master/colors.json")
jsonContent = response.text()
jsonResult = json.loads(jsonContent)
print(jsonResult)

If there is some processing in the middle, you could ask for the inner_text of the :root element:

jsonContent = page.inner_text(':root')
jsonResult = json.loads(jsonContent)
print(jsonResult)