How to use test_should_specify_repeat_property method in Playwright Python

https://www.piwheels.org is where wheels for Raspberry Pi are published, not PyPI. This is the list of wheels for "playwright": https://www.piwheels.org/project/playwright/ . Version 0.162.2 is the latest available release (click on "Show more releases").

To install from PyPI try to install from both PyPI and PiWheels:

pip install --index-url=https://pypi.org/simple/ --extra-index-url=https://www.piwheels.org/simple/ playwright

Install these first

playwright

scrapy-playwright

Scrapy

And then run in the terminal

playwright install (This will install the driver required for playwright)

Original answer:

Using querySelector() / querySelectorAll with XPath (XML Path Language) lets you retrieve the elementHandle (respectively a collection of handles). Generally speaking, XPath can be used to navigate through elements and attributes in an XML document.

from playwright import sync_playwright

with sync_playwright() as p:
    for browser_type in [p.chromium, p.firefox, p.webkit]:
        browser = browser_type.launch(headless=False)
        page = browser.newPage()
        page.goto('https://duckduckgo.com/')
        element = page.querySelector('input[id=\"search_form_input_homepage\"]')
        
        parent = element.querySelector('xpath=..')
        grandparent = element.querySelector('xpath=../..')
        siblings = element.querySelectorAll('xpath=following-sibling::*')
        children = element.querySelectorAll('xpath=child::*')

        browser.close()

Update (2022-07-22):

It seems that browser.newPage() is deprecated, so in newer versions of playwright, the function is called browser.new_page() (note the different function name).

Optionally create a browser context first (and close it afterwards) and call new_page() on that context.

The way the children/parent/grandparent/siblings are accessed stays the same.

from playwright import sync_playwright

with sync_playwright() as p:
    for browser_type in [p.chromium, p.firefox, p.webkit]:
        browser = browser_type.launch(headless=False)
        context = browser.new_context()
        page = context.new_page()
        page.goto('https://duckduckgo.com/')
        element = page.querySelector('input[id=\"search_form_input_homepage\"]')
        
        parent = element.querySelector('xpath=..')
        grandparent = element.querySelector('xpath=../..')
        siblings = element.querySelectorAll('xpath=following-sibling::*')
        children = element.querySelectorAll('xpath=child::*')

        context.close()
        browser.close()

PLAYWRITH_BROWSERS_PATH=0

Please make sure your above configuration was correct. There are some spelling mistakes in the settings. Its like PLAYWRIGHT_BROWSERS_PATH = 0

In your local.settings.json it should be

{

"IsEncrypted": false,

    "Values": {
    
        "AzureWebJobsStorage": "",
        
        "FUNCTIONS_WORKER_RUNTIME": "python",
        
        "PLAYWRIGHT_BROWSERS_PATH": "0"
    
    }

}

And In your requirement.txt file it should be

azure-functions
playwright==1.18.0

here we can see the playwright running in local as well as azure after deployment Local:

Azure:

Chromium will not allow you to override the Host header. I presume this is because Host is something the browser manages internally. Source.

Why do you want to modify this header?

Closing actual popups

I think browser.once("targetcreated", ...) as described in this issue https://stackoverflow.com/a/57246544/2202112 could be something that you are looking for.

It allows you to setup a callback on newly created targets.

For reference, see puppeteer docs on "targetcreated"

Closing modals

1. Find the selector for the button that dismisses the modal, for example .button-close or .button-dismiss.
2. Use the page.click(selector, [options]) method to click the button before continuing further scraping

Playwright has a built in code generator which will make things easy for you.

Run this command and try it out

npx playwright codegen

You should be able to get that button by text:

page.locator('"Log in"').click()

The main problem:

context.new_page() calls before variable declaration(context = ...)

Should works fine:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=50)
    context = browser.new_context(storage_state="website1.json")
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill("input#input-username", "demo")
    page.fill("input#input-password", "demo")
    page.click("button[type=submit]")

Also you can use Persistent authentication:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    user_data_dir = 'FULL_PATH_TO_BROWSER_PROFILE'
    browser = p.chromium.launch_persistent_context(user_data_dir, headless=False)
    # login only 1 time...

JFYI: opencart uses user_token in GET parameters and validates it. playwright(storage_state, user_data_dir etc) should works fine. Example:

# run docker container with playwright
docker rm -fv example && docker run --name example -it mcr.microsoft.com/playwright/python:v1.27.1-focal bash
# create empty state
echo {} >> /tmp/state.json
# run python in interactive mode
python
# Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
# [GCC 9.4.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.

Paste the next script:

from time import sleep
from urllib.parse import parse_qs

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # login
    browser = p.chromium.launch()
    context = browser.new_context(storage_state='/tmp/state.json')
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill('input#input-username', 'demo')
    page.fill('input#input-password', 'demo')
    page.click('button[type=submit]')
    sleep(5)  # just wait for redirect - you can wait for a element...
    page.context.storage_state(path='/tmp/state.json')
    page.screenshot(path='/tmp/after_state_init.png')
    # parse user_token for GET requests
    parsed_url = parse_qs(page.url)
    user_token = parsed_url['user_token'][0]
    print(f'user token {user_token}')
    page.close()
    browser.close()


with sync_playwright() as p:
    # open admin dashboard without login... 
    browser = p.chromium.launch()
    context = browser.new_context()
    page = browser.new_page(storage_state='/tmp/state.json')
    page.goto(f'https://demo.opencart.com/admin/index.php?route=common/dashboard&user_token={user_token}')
    sleep(5)
    page.screenshot(path='/tmp/open_using_saved_state.png')
    page.close()
    browser.close()

Now open a new terminal and copy screens from docker container:

docker cp example:/tmp/after_state_init.png ./
docker cp example:/tmp/open_using_saved_state.png ./

You'll see that they are the same - admin dashboard works without login.