How to use test_locators_all_inner_texts method in Playwright Python

The main problem:

context.new_page() calls before variable declaration(context = ...)

Should works fine:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=50)
    context = browser.new_context(storage_state="website1.json")
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill("input#input-username", "demo")
    page.fill("input#input-password", "demo")
    page.click("button[type=submit]")

Also you can use Persistent authentication:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    user_data_dir = 'FULL_PATH_TO_BROWSER_PROFILE'
    browser = p.chromium.launch_persistent_context(user_data_dir, headless=False)
    # login only 1 time...

JFYI: opencart uses user_token in GET parameters and validates it. playwright(storage_state, user_data_dir etc) should works fine. Example:

# run docker container with playwright
docker rm -fv example && docker run --name example -it mcr.microsoft.com/playwright/python:v1.27.1-focal bash
# create empty state
echo {} >> /tmp/state.json
# run python in interactive mode
python
# Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
# [GCC 9.4.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.

Paste the next script:

from time import sleep
from urllib.parse import parse_qs

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # login
    browser = p.chromium.launch()
    context = browser.new_context(storage_state='/tmp/state.json')
    page = context.new_page()
    page.goto('https://demo.opencart.com/admin')
    page.fill('input#input-username', 'demo')
    page.fill('input#input-password', 'demo')
    page.click('button[type=submit]')
    sleep(5)  # just wait for redirect - you can wait for a element...
    page.context.storage_state(path='/tmp/state.json')
    page.screenshot(path='/tmp/after_state_init.png')
    # parse user_token for GET requests
    parsed_url = parse_qs(page.url)
    user_token = parsed_url['user_token'][0]
    print(f'user token {user_token}')
    page.close()
    browser.close()


with sync_playwright() as p:
    # open admin dashboard without login... 
    browser = p.chromium.launch()
    context = browser.new_context()
    page = browser.new_page(storage_state='/tmp/state.json')
    page.goto(f'https://demo.opencart.com/admin/index.php?route=common/dashboard&user_token={user_token}')
    sleep(5)
    page.screenshot(path='/tmp/open_using_saved_state.png')
    page.close()
    browser.close()

Now open a new terminal and copy screens from docker container:

docker cp example:/tmp/after_state_init.png ./
docker cp example:/tmp/open_using_saved_state.png ./

You'll see that they are the same - admin dashboard works without login.

There are two things to notice:

1. Python executable
2. Alternatives priority

Executable

Running in the base image:

# which python
# which python3
/usr/bin/python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8

With the following Dockerfile (from the question);

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 2

python is now added and controlled by alternatives, but python3 is untouched:

# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:10 /usr/bin/python -> /etc/alternatives/python
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 18 Dec 16 04:10 /etc/alternatives/python -> /usr/bin/python3.8

Priority

From alternative's manpage:

If the group is in automatic mode, and the newly added alternatives' priority is higher than any other installed alternatives for this group, the symlinks will be updated to point to the newly added alternatives.

This is numeric, so it's not that 1 is the winner and 2 is second-best, but rather 2 is higher than 1.

Resolution

By changing both the executable form python to python3, and the priority so that 3.10 will be 2 and 3.8 will be 1 (as such):

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.10 2
RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.8 1

the default python version should be 3.10:

# which python
# which python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:16 /usr/bin/python3 -> /etc/alternatives/python
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 19 Dec 16 04:16 /etc/alternatives/python -> /usr/bin/python3.10
# python3 --version
Python 3.10.9

After reading the entire doc :

page.evaluate("() => dataLayer")

It's been suggested by the developers of scrapy_playwright to instantiate the DOWNLOAD_HANDLERS and TWISTER_REACTOR into your script.

A similar comment is provided here

Here's a working script implementing just this:

import scrapy
from scrapy_playwright.page import PageCoroutine
from scrapy.crawler import CrawlerProcess

class ProductSpider(scrapy.Spider):
    name = 'product'

    def start_requests(self):
        yield scrapy.Request(
            'https://shoppable-campaign-demo.netlify.app/#/',
            callback = self.parse,
            meta={
                'playwright': True,
                'playwright_include_page': True,
                'playwright_page_coroutines': [
                    PageCoroutine("wait_for_selector", "div#productListing"),
                ]
            }
        )

    async def parse(self, response):
        container = response.xpath("(//div[@class='col-md-6'])[1]")
        for items in container:
            yield {
                'products':items.xpath("(//h3[@class='card-title'])[1]//text()").get()
            }
        # parses content

if __name__ == "__main__":
    process = CrawlerProcess(
        settings={
            "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
            "DOWNLOAD_HANDLERS": {
                "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
                "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            },
            "CONCURRENT_REQUESTS": 32,
            "FEED_URI":'Products.jl',
            "FEED_FORMAT":'jsonlines',
        }
    )
    process.crawl(ProductSpider)
    process.start()

And we get the following output:

{'products': 'Oxford Loafers'}

as the graph is being injected dynamically into the website via JavaScript, you have two ways of going about this task:

1. Emulate the script (unnecessary hard but fast)

2. WebScrape using a browser client (slower, but much easier) You can use Python libraries such as Selenium or Playwright (the second option being more modern) to handle this task. The documentation for playwright is pretty intuitive, and you can use css selectors in order to traverse the dom, which, I suppose, you already have knowledge of.

Let me know if that helps!

The sync_api is simply a wrapper around the asyncio_api that abstracts asyncio usage away from you. As such, the capabilities are largely the same, but the async_api may afford some more flexibility in complex scenarios (for example, in previous playwright-python releases the only way to run instances in a multi-threaded fashion on Unix + Python 3.7 was to use the async_api for reasons I won't get into here).

Chances are you won't need that flexibility though, so I'd just suggest using whatever you're comfortable with.

You can use the not_be_visible expect assertion for this.

expect('element CSS here').not_to_be_visible()

By default expect assertions have a 5 second timeout, in case you want to increase the timeout, you can do like this:

expect('element CSS here').not_to_be_visible(timeout=7000)

Doing something similar for a project of mine, this is how I would do it.

from playwright.sync_api import sync_playwright

posts = ['https://playwright.dev/','https://playwright.dev/python/',]

def scrap_post_info(context, post):
    page = context.new_page()
    page.goto(post)
    print(page.title())
    # do whatever scraping you need to
    page.close()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    for post in posts:
        scrap_post_info(context, post)
        # some time delay

browser.close()

Thing is the code snippet from the playwright docs is more about opening a new page after clicking a link on an existing page. Since you already have the urls ready, you can just visit each page one by one, and do your scraping.

After trying many different variants, I guessed a working syntax

handle.selectOption({"label": "Banana"})