How to use _patch_case method in Playwright Python

Please refer this stack overflow link for more details. hope this will help to solve your issue

You have to write the page.on("dialog", lambda dialog: dialog.accept()) code before you press on button which is generating the popup. Playwright works with the listner so you should first start listner then click on element which is generating the popup.

page.on("dialog", lambda dialog: dialog.accept())
page.locator(#xyz).click() // Element opening alert window

To get the full HTML content of the page you can use page.content().

Playwright does not support key filtering at the moment. But you can filter for the job.id which is part of the props:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.glassdoor.co.uk/Job/qa-engineer-jobs-SRCH_KO0,11.htm")
    page.locator("_react=q[job.id=1007630619432]").click()
    browser.close()

If you try to perform an action on a label which has associated input element Playwright will automatically retarget the action to the input.

If the DOM structure is always the same you can try something like this

page.locator('div:has( > :text-matches("Project")) + div input')

Otherwise layout selector is your best bet.

Heroku doesn't support webkit at the time, with other browsers, you can use this buildpack: https://github.com/mxschmitt/heroku-playwright-buildpack/

After trying many different variants, I guessed a working syntax

handle.selectOption({"label": "Banana"})

You can define your own function like this one:

def my_own_wait_for_selector(page, selector, time_out):
    try:
        page.wait_for_selector(selector, timeout=time_out)
        return True
    except:
        return False

That function will wait for a selector and if given some time (miliseconds) the element is not there it will return False instead of trowing an error.

Then in your code you could try something like:

while not my_own_wait_for_selector(page, "my_button_selector", 5000):
    page.reload()
page.locator("my_button_selector").click()

As you can see we are checking if our button is present, and we will wait only 5 seconds, if after 5 seconds the button is not there, we will reload, and again and again until our button is there, once our button is there code will continue and then we will click the button.

Be careful, because if your button is never there, your script will reload and your script will never finish, so you should add some other condition to your while

Here is a fully working solution by using scrapy-playwright, I got the idea the following issue 61 and the users profile @lime-n.

We download the xhr requests sent and store these into a dict with both the playwright tools and scrapy-playwright. I include the playwright_page_event_handlers to integrate playwright tools for this.

Depending on the video size, it will take a few minutes to download.

main.py

import scrapy
from playwright.async_api import Response as PlaywrightResponse
import jsonlines
import pandas as pd
import json
from video.items import videoItem

headers = {
    'authority': 'api.equidia.fr',
    'accept': 'application/json, text/plain, */*',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'content-type': 'application/json',
    'origin': 'https://www.equidia.fr',
    'referer': 'https://www.equidia.fr/',
    'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-site',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

class videoDownloadSpider(scrapy.Spider):
    name = 'video'
    start_urls = ['https://www.equidia.fr/courses/2022-07-31/R1/C1']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                #callback = self.parse, 
                meta = {
                    'playwright':True,
                    'playwright_page_event_handlers':{
                    "response": "handle_response",
                }
                
                }               
            )

    async def handle_response(self, response: PlaywrightResponse) -> None:
        """
        We can grab the post data with response.request.post - there are three different types for different needs.
        The method below helps grab those resource types of 'xhr' and 'fetch' until I can work out how to only send these to the download request.
        """
        self.logger.info(f'test the log of data: {response.request.resource_type, response.request.url, response.request.method}')
        jl_file = "videos.jl"
        data = {}
        if response.request.resource_type == "xhr":
            if response.request.method == "GET":
                if 'videos' in response.request.url:

                    data['resource_type']=response.request.resource_type,
                    data['request_url']=response.request.url,
                    data['method']=response.request.method
                    with jsonlines.open(jl_file, mode='a') as writer:
                        writer.write(data)

    def parse(self, response):
        video = pd.read_json('videos.jl', lines=True)
        print('KEST: %s' % video['request_url'][0][0])

        yield scrapy.FormRequest(
            url = video['request_url'][0][0],
            headers=headers,
            callback = self.parse_video_json
        )
    def parse_video_json(self, response):
        another_url = response.json().get('video_url')
        yield scrapy.FormRequest(
            another_url,
            headers=headers,
            callback=self.extract_videos
        )

    def extract_videos(self, response):
        videos = response.json().get('mp4')
        for keys, vids in videos.items():
            loader = videoItem()
            loader['video'] = [vids]
            yield loader

items.py

class videoItem(scrapy.Item):
    video = scrapy.Field()

pipelines.py

class downloadFilesPipeline(FilesPipeline):
    def file_path(self, request, response=None, info=None, item=None):
        file = request.url.split('/')[-1]
        video_file = f"{file}.mp4"
        return video_file

settings.py

from pathlib import Path
import os

BASE_DIR = Path(__file__).resolve().parent.parent

FILES_STORE = os.path.join(BASE_DIR, 'videos')

FILES_URLS_FIELD = 'video'
FILES_RESULT_FIELD = 'results'


DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

ITEM_PIPELINES = {
    'video.pipelines.downloadFilesPipeline': 150,
}