How to use reset method in Playwright Python

As mentioned in the comments, your tests should be independent and indeed this can be achieved with database fixtures. One way to achieve this and worked really well for me is by using testcontainers: https://github.com/testcontainers/testcontainers-python.

A fixture might look like this in pytest with, in my case, a Postgress database:

@pytest.fixture(scope="module", name="session_for_db_in_testcontainer")
def db_engine():
    """
    Creates testcontainer with Postgres db
    """
    pg_container = PostgresContainer('postgres:latest')
    pg_container.start()

    # Fireup the SQLModel engine with the uri of the container
    db_engine = create_engine(pg_container.get_connection_url())
    sqlmodel_metadata.create_all(db_engine)

    with Session(db_engine) as session_for_db_in_testcontainer:
        # add some rows to start, for test get requests and posting existing data
        add_data_to_test_db(database_input_path, session_for_db_in_testcontainer)
        yield session_for_db_in_testcontainer

    # Will be executed after the last test
    session_for_db_in_testcontainer.close()
    pg_container.stop()

Like this during the test run a (Postgres) DB is created it only runs during a session, module or function depending on the scope of the fixture. If you want, you can add test data to the db as well like in the example.

The locator will help you find an element. But from there, if you don't have an element to pierce the Shadow DOM, you might need to do that manually.

text = page.locator('some-element').first.evaluate("node => node.shadowRoot.innerHTML")

Type F12 in Google Chrome and go to the console.

Write $x() and place the xpath selector inside the parenthesis. Hit the enter key and make sure you actually find the right selector.

No need to use e.value, you can do something like this:

p1 = page.locator("//*[@id='rc-tabs-1-panel-apikey']/div/span/input")
p1text = p1.inner_text

Also - use * as less as possible. If the element with the id is a div write div for example and not *. It makes the searching faster.

After trying many different variants, I guessed a working syntax

handle.selectOption({"label": "Banana"})

It's been suggested by the developers of scrapy_playwright to instantiate the DOWNLOAD_HANDLERS and TWISTER_REACTOR into your script.

A similar comment is provided here

Here's a working script implementing just this:

import scrapy
from scrapy_playwright.page import PageCoroutine
from scrapy.crawler import CrawlerProcess

class ProductSpider(scrapy.Spider):
    name = 'product'

    def start_requests(self):
        yield scrapy.Request(
            'https://shoppable-campaign-demo.netlify.app/#/',
            callback = self.parse,
            meta={
                'playwright': True,
                'playwright_include_page': True,
                'playwright_page_coroutines': [
                    PageCoroutine("wait_for_selector", "div#productListing"),
                ]
            }
        )

    async def parse(self, response):
        container = response.xpath("(//div[@class='col-md-6'])[1]")
        for items in container:
            yield {
                'products':items.xpath("(//h3[@class='card-title'])[1]//text()").get()
            }
        # parses content

if __name__ == "__main__":
    process = CrawlerProcess(
        settings={
            "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
            "DOWNLOAD_HANDLERS": {
                "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
                "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            },
            "CONCURRENT_REQUESTS": 32,
            "FEED_URI":'Products.jl',
            "FEED_FORMAT":'jsonlines',
        }
    )
    process.crawl(ProductSpider)
    process.start()

And we get the following output:

{'products': 'Oxford Loafers'}

You need playwright added to your PATH.

However, a better way to do this (if python is added to your path) without adding it to your PATH is by running:

python -m playwright install

This runs the playwright module as a script. Use python -h for more information on these flags, and python -m playwright for more information on the flags supported specifically by playwright

The locator will help you find an element. But from there, if you don't have an element to pierce the Shadow DOM, you might need to do that manually.

text = page.locator('some-element').first.evaluate("node => node.shadowRoot.innerHTML")

you cant get the contents of a dynamic website like youtube using requests. sorry to be so direct, but this is the truth.

you need first to get to the url, then render the response using something like chromium in the background, then pass the results to beautiful soup.

the rendering will take 1-2 seconds. this is how its done.

there is a snippet for extracting the dynamic website contents which then are passed to BeautifulSoup

# pip install playwright
from playwright.sync_api import sync_playwright
# after installing you will get prompted
# to install `chromium`, the `thing` i was talking about
from bs4 import BeautifulSoup


def get_dynamic_soup(url: str) -> BeautifulSoup:
    with sync_playwright() as p:
        # Launch the browser
        browser = p.chromium.launch()

        # Open a new browser page
        page = browser.new_page()

        # Open our test file in the opened page
        page.goto(url)

        # Process extracted content with BeautifulSoup
        soup = BeautifulSoup(page.content(), "html.parser")

        browser.close()

        return soup

# quote is defined in your code
_url = "https://www.youtube.com/results?search_query=" + quote(" ".join(searchTerm[:-2]))
soup = get_dynamic_soup(_url)
# now you can do whatever you want with the soup

then you can do your stuff:

videos = soup.findAll(attrs={"class":"yt-uix-tile-link"})[1:4]

to install playwright

python -m pip install playwright # this installs the python package
python -m playwright install # this install the chromium executable

docs for installation

EDIT i found a bug in your code this line

videos = soup.findAll(attrs={"class":"yt-uix-tile-link"})[1:4]

is wrong because you need to specify the HTML element you want to search for

a good example is:

videos = soup.findAll("div", attrs={
    "class": "yt-uix-tile-link"
})[1:4]
# or 
videos = soup.findAll("span", attrs={
    "class": "yt-uix-tile-link"
})[1:4]
# or whatever element it is

There is several good practices for creating locators/selectors. Using playwright there is official documentation for each common and unique selector on how-to and what-is doing.

More information in https://playwright.dev/docs/selectors#text-selector

About your case, i would suggest always to use an parent selector for locating an element. When there is a button, try to find its unique parent.

• By id
• By unique class
• Something else unique.

Example:

<dv id=test>
 <button id=continue-test>Continue</button>
</div>

In this case you can use the unique id of the button and not the text. Selector css: #continue-test But if you, don't have an unique identifier for the button you can use the parent and go down to the button. Selector css: #test > button

Matching text using css is not possible, but with XPATH can look like this:

//button[text()="Continue"] 

This selector MATCHES the text using "equals".

Using playwright:

button:has-text("Continue")

Using has-text and quotes - matches the text using equals. If you are using another selector for example text=Continue, this will match all elements that CONTAINS the text "Continue"

All this is explained with example in the official documentation for playwright selectors. That does not mean to not use XPATH to achieve the goals.

CSS selectors are fast but kind of restricted to work with text.

Xpath is quite slower but much more powerful to work in text/parent/child elements etc.

I would suggest always to use an parent element with unique identifier and go down to reach your actual element, which will receive the interaction.