How to use inner_html method in Playwright Python

For Typescript/Javascript there is playwright-dompath which you can install with:

npm install playwright-dompath

Basically you just call the xPath function with the element you selected, and you get back the full xPath. Under the hood, it uses the same functions from Chrome DevTools.

you cant get the contents of a dynamic website like youtube using requests. sorry to be so direct, but this is the truth.

you need first to get to the url, then render the response using something like chromium in the background, then pass the results to beautiful soup.

the rendering will take 1-2 seconds. this is how its done.

there is a snippet for extracting the dynamic website contents which then are passed to BeautifulSoup

# pip install playwright
from playwright.sync_api import sync_playwright
# after installing you will get prompted
# to install `chromium`, the `thing` i was talking about
from bs4 import BeautifulSoup


def get_dynamic_soup(url: str) -> BeautifulSoup:
    with sync_playwright() as p:
        # Launch the browser
        browser = p.chromium.launch()

        # Open a new browser page
        page = browser.new_page()

        # Open our test file in the opened page
        page.goto(url)

        # Process extracted content with BeautifulSoup
        soup = BeautifulSoup(page.content(), "html.parser")

        browser.close()

        return soup

# quote is defined in your code
_url = "https://www.youtube.com/results?search_query=" + quote(" ".join(searchTerm[:-2]))
soup = get_dynamic_soup(_url)
# now you can do whatever you want with the soup

then you can do your stuff:

videos = soup.findAll(attrs={"class":"yt-uix-tile-link"})[1:4]

to install playwright

python -m pip install playwright # this installs the python package
python -m playwright install # this install the chromium executable

docs for installation

EDIT i found a bug in your code this line

videos = soup.findAll(attrs={"class":"yt-uix-tile-link"})[1:4]

is wrong because you need to specify the HTML element you want to search for

a good example is:

videos = soup.findAll("div", attrs={
    "class": "yt-uix-tile-link"
})[1:4]
# or 
videos = soup.findAll("span", attrs={
    "class": "yt-uix-tile-link"
})[1:4]
# or whatever element it is

You can see from the documentation here that you can use pseudo-class of has-text

... the :has-text() pseudo-class

and maybe some some more advance matching selector with pseudo-class text-matches

the :text-matches() pseudo-class

but you need to remember that two ways is better if you combine the usage with some css selector.

Maybe you can add more details on where you want to search the string from is it literally in the entire document or not. (and I can update the answer with more detail code example according to your target)

There are many things you can improve there.

• First, you can read the text area value using page.locator('#startText').value()
• Then, you click on the check button. Don't wait using wait_for_load_state, because the page is not navigating.
• After the check action there is an SPAN element inside the "#inside" div. You can do page.locator('#inside SPAN').inner_html(). The inner_html function will wait for the element for you.

I managed to make a code that closes only a specific tab!

all_pages = page.context.pages
await all_pages[1].close()

You can use the addInitScript function.

await page. add_init_script("() => Object.defineProperty(navigator,'webdriver'...");

Closing actual popups

I think browser.once("targetcreated", ...) as described in this issue https://stackoverflow.com/a/57246544/2202112 could be something that you are looking for.

It allows you to setup a callback on newly created targets.

For reference, see puppeteer docs on "targetcreated"

Closing modals

1. Find the selector for the button that dismisses the modal, for example .button-close or .button-dismiss.
2. Use the page.click(selector, [options]) method to click the button before continuing further scraping

to_have_value supports regular expressions. You can do something like this:

expect(self.page.locator(input_locator).first).
to_have_value(re.compile(r"123.+"), timeout=20 * 1000)

Closing actual popups

I think browser.once("targetcreated", ...) as described in this issue https://stackoverflow.com/a/57246544/2202112 could be something that you are looking for.

It allows you to setup a callback on newly created targets.

For reference, see puppeteer docs on "targetcreated"

Closing modals

1. Find the selector for the button that dismisses the modal, for example .button-close or .button-dismiss.
2. Use the page.click(selector, [options]) method to click the button before continuing further scraping