How to use test_click_throw_for_hidden_nodes_with_force method in Playwright Python

Doing something similar for a project of mine, this is how I would do it.

from playwright.sync_api import sync_playwright

posts = ['https://playwright.dev/','https://playwright.dev/python/',]

def scrap_post_info(context, post):
    page = context.new_page()
    page.goto(post)
    print(page.title())
    # do whatever scraping you need to
    page.close()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    for post in posts:
        scrap_post_info(context, post)
        # some time delay

browser.close()

Thing is the code snippet from the playwright docs is more about opening a new page after clicking a link on an existing page. Since you already have the urls ready, you can just visit each page one by one, and do your scraping.

You can use one of those options :

const element = await page.$$("text='element'");
if (element) {
    // ...
}

or

const element = await page.$$("text='element'");
if (element.length) {
    // ...
}

or

await expect(page.locator(".MyClass")).toHaveCount(0)

You can pass both selectors separated by a comma. Ref.

new_selector = "text=Your are a winner, text=Better luck next time"
handle = query_selector(new_selector)

Instead of copying the curl, try copying the fetch and put it in like:

driver.execute_script(```
  fetch(...)
```)

You can access the href in an almost similar way to that of selenium.

element_handle = page.locator("//a")
link = element_handle.get_attribute('href')

Here is my solution:

page.eval_on_selector(
  selector="textarea", # Modify the selector to fit yours.
  expression="(el) => el.style.display = 'inline-block'",
)

Here is the document of eval_on_selector().

The main problem:

context.new_page() calls before variable declaration(context = ...)

Should works fine:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=50)
    context = browser.new_context(storage_state="website1.json")
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill("input#input-username", "demo")
    page.fill("input#input-password", "demo")
    page.click("button[type=submit]")

Also you can use Persistent authentication:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    user_data_dir = 'FULL_PATH_TO_BROWSER_PROFILE'
    browser = p.chromium.launch_persistent_context(user_data_dir, headless=False)
    # login only 1 time...

JFYI: opencart uses user_token in GET parameters and validates it. playwright(storage_state, user_data_dir etc) should works fine. Example:

# run docker container with playwright
docker rm -fv example && docker run --name example -it mcr.microsoft.com/playwright/python:v1.27.1-focal bash
# create empty state
echo {} >> /tmp/state.json
# run python in interactive mode
python
# Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
# [GCC 9.4.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.

Paste the next script:

from time import sleep
from urllib.parse import parse_qs

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # login
    browser = p.chromium.launch()
    context = browser.new_context(storage_state='/tmp/state.json')
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill('input#input-username', 'demo')
    page.fill('input#input-password', 'demo')
    page.click('button[type=submit]')
    sleep(5)  # just wait for redirect - you can wait for a element...
    page.context.storage_state(path='/tmp/state.json')
    page.screenshot(path='/tmp/after_state_init.png')
    # parse user_token for GET requests
    parsed_url = parse_qs(page.url)
    user_token = parsed_url['user_token'][0]
    print(f'user token {user_token}')
    page.close()
    browser.close()


with sync_playwright() as p:
    # open admin dashboard without login... 
    browser = p.chromium.launch()
    context = browser.new_context()
    page = browser.new_page(storage_state='/tmp/state.json')
    page.goto(f'https://demo.opencart.com/admin/index.php?route=common/dashboard&user_token={user_token}')
    sleep(5)
    page.screenshot(path='/tmp/open_using_saved_state.png')
    page.close()
    browser.close()

Now open a new terminal and copy screens from docker container:

docker cp example:/tmp/after_state_init.png ./
docker cp example:/tmp/open_using_saved_state.png ./

You'll see that they are the same - admin dashboard works without login.

From the docs, you should use a string as selector.

page.waitForSelector(selector[, options])

selector <string> A selector to to query for

page.click(selector[, options])

selector A selector to search for an element

selector = '//*[@id="sections-scroll-target"]/ul/ul/li[8]/a'
    
page.wait_for_selector(selector, timeout=10000)

page.hover(selector)

Looks like you want https://www.testdummysite.com/news/markets?mod=nav_left_section

If so this will work

//*[@href= "https://www.testdummysite.com/news/markets?mod=nav_left_section" ]

but these get a different link https://www.testdummysite.com/?mod=nav_left_section.

//*[@id="sections-scroll-target"]/ul/ul/li[8]/a

#sections-scroll-target > ul > ul > li:nth-child(8) > a

Because these URLs are similar, I wonder if you've posted the HTML accurately?

The sync_api is simply a wrapper around the asyncio_api that abstracts asyncio usage away from you. As such, the capabilities are largely the same, but the async_api may afford some more flexibility in complex scenarios (for example, in previous playwright-python releases the only way to run instances in a multi-threaded fashion on Unix + Python 3.7 was to use the async_api for reasons I won't get into here).

Chances are you won't need that flexibility though, so I'd just suggest using whatever you're comfortable with.