How to use goto method in Playwright Python

You can use the addInitScript function.

await page. add_init_script("() => Object.defineProperty(navigator,'webdriver'...");

After investigating and trying several things, looks like the problem is the user_agent of the browser when is in headless mode, for some reason the default user agent does not like to that page, try with:

def extract_html(self):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36')
        page.goto('http://book.flygofirst.com/Flight/Select?inl=0&CHD=0&s=True&o1=BOM&d1=BLR&ADT=1&dd1=2022-12-10&gl=0&glo=0&cc=INR&mon=true')
        html = page.inner_html('#sectionBody')
        return html

You can define your own function like this one:

def my_own_wait_for_selector(page, selector, time_out):
    try:
        page.wait_for_selector(selector, timeout=time_out)
        return True
    except:
        return False

That function will wait for a selector and if given some time (miliseconds) the element is not there it will return False instead of trowing an error.

Then in your code you could try something like:

while not my_own_wait_for_selector(page, "my_button_selector", 5000):
    page.reload()
page.locator("my_button_selector").click()

As you can see we are checking if our button is present, and we will wait only 5 seconds, if after 5 seconds the button is not there, we will reload, and again and again until our button is there, once our button is there code will continue and then we will click the button.

Be careful, because if your button is never there, your script will reload and your script will never finish, so you should add some other condition to your while

Maybe this simple example will help you:

for CURRENT_XPATH in ['FIRST_XPATH', 'SECOND_XPATH']:

    with page.expect_download() as download_info:
        page.click(CURRENT_XPATH)

    Download = await download_info.value
    await Download.save_as(Download.suggested_filename)

The locator will help you find an element. But from there, if you don't have an element to pierce the Shadow DOM, you might need to do that manually.

text = page.locator('some-element').first.evaluate("node => node.shadowRoot.innerHTML")

I believe you need to call your go_to_url function using the same recipe:

asyncio.get_event_loop().run_until_complete(go_to_url())

I would do this:

1. Register a listener with expose_function.
2. Use evaluate to register a listener to newPosts. A Listener on the browser side.
3. Inside that listener call the function you exposed on 1.

There are two things to notice:

1. Python executable
2. Alternatives priority

Executable

Running in the base image:

# which python
# which python3
/usr/bin/python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8

With the following Dockerfile (from the question);

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 2

python is now added and controlled by alternatives, but python3 is untouched:

# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:10 /usr/bin/python -> /etc/alternatives/python
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 18 Dec 16 04:10 /etc/alternatives/python -> /usr/bin/python3.8

Priority

From alternative's manpage:

If the group is in automatic mode, and the newly added alternatives' priority is higher than any other installed alternatives for this group, the symlinks will be updated to point to the newly added alternatives.

This is numeric, so it's not that 1 is the winner and 2 is second-best, but rather 2 is higher than 1.

Resolution

By changing both the executable form python to python3, and the priority so that 3.10 will be 2 and 3.8 will be 1 (as such):

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.10 2
RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.8 1

the default python version should be 3.10:

# which python
# which python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:16 /usr/bin/python3 -> /etc/alternatives/python
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 19 Dec 16 04:16 /etc/alternatives/python -> /usr/bin/python3.10
# python3 --version
Python 3.10.9

If you need to handle new page without event listener (i.e. new tab open by link click), you can try this code:

    from playwright import sync_playwright

    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.newContext()
        page = context.newPage()
        page.goto('<site url>')
        with context.expect_page() as tab:
            page.click('.newTabByLink')
        # do some steps
        ...
        tab.close()
        
        browser.close()