How to use _on_worker method in Playwright Python

page.hover(selector[, options])

After that you can do

page.waitForSlector(selector,{timeout:6000})

And then do what you need for your elem. You can read this:

https://playwright.dev/docs/api/class-page#page-hover

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

You can use the API request feature:

browser = playwright.chromium.launch()
context = browser.new_context(base_url="https://api.github.com")
api_request_context = context.request
page = context.new_page()

response = api_request_context.post(
  "/user/repos",
  headers={
      "Accept": "application/vnd.github.v3+json",
      "Authorization": f"token {API_TOKEN}",
  },
  data={"name": REPO},
)

Playwright has no method get_by_role.

Try this:

page.locator('link[name="Log in"]').click()

You can give the locator method any CSS or Xpath expression.

This error means that python-playwright or selenium is not launching correctly. What I did is

1.Check the version of your chrome Right upper corner -> help -> About Google Chrome

2.Download the chromedriver from chromedriver

3.let your API find the driver

api = TikTokApi.get_instance(use_selenium=True, executablePath="/path/to/chromedriver")

Functions assigned to events (in any module) can't return value and you may need to use global variable for this. But lambda can't use global and can't assign value to variable so it needs normal function

url = ""

def on_response(response):
    global url  # inform functio to use external variable (instead of local variable) when you use `=` to assign value
    
    if ".m3u8" in response.url:
        url = response.url

page.on("response", on_response) 

I found out that the problem was i was trying to close the context before closing the page first, and that was the thing which raises unhandleable exceptions. The solution is replacing the close_playwright() function with this:

def close_playwright():

    page.close()
    context.close()
    browser.close()
    playwright.dispose()

PageCoroutine is deprecated/obsolute. Use playwright_page_methods instead.

Working code as an example:

import scrapy
from scrapy_playwright.page import PageMethod

class TestSpider(scrapy.Spider):
    name = "test"
    def start_requests(self):
        yield scrapy.Request(

            url="https://shoppable-campaign-demo.netlify.app/#/",
            callback=self.parse,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", '.card-body'),
                ],
            },
        )

    def parse(self, response):
        
        products = response.xpath('//*[@class="card-body"]')
        for product in products:
            yield {
            'title':product.xpath('.//*[@class="card-title"]/text()').get()
          
            }

Output:

{'title': 'Oxford Loafers'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Ankle-length Slack'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'White Baseball Cap'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Triangle Bikini Top'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Short Blazer'}
2022-11-05 20:40:40 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-05 20:40:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 235,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 39851,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 41.370211,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 11, 5, 14, 40, 40, 261151),
 'item_scraped_count': 5,

Microsoft released a python docker image for Playwright

Dockerfile

# Build Environment: Playwright
FROM mcr.microsoft.com/playwright/python:v1.21.0-focal

# Add python script to Docker
COPY index.py /

# Run Python script
CMD [ "python", "index.py" ]

Check Playwright - Docker docs for the latest playwright version.