How to use test_locators_dispatch_event_should_work method in Playwright Python

If you know that the click will take a while, you can set a timeout:

page.locator("text=Download").click(timeout=60000)

you can use the channel msedge

robotframework example

New Browser chromium headless=false channel=msedge

New Context viewport={'width': 1920, 'height': 1080}

New Page https://marketsquare.github.io/robotframework-browser/Browser.html

Here is a fully working solution by using scrapy-playwright, I got the idea the following issue 61 and the users profile @lime-n.

We download the xhr requests sent and store these into a dict with both the playwright tools and scrapy-playwright. I include the playwright_page_event_handlers to integrate playwright tools for this.

Depending on the video size, it will take a few minutes to download.

main.py

import scrapy
from playwright.async_api import Response as PlaywrightResponse
import jsonlines
import pandas as pd
import json
from video.items import videoItem

headers = {
    'authority': 'api.equidia.fr',
    'accept': 'application/json, text/plain, */*',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'content-type': 'application/json',
    'origin': 'https://www.equidia.fr',
    'referer': 'https://www.equidia.fr/',
    'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-site',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

class videoDownloadSpider(scrapy.Spider):
    name = 'video'
    start_urls = ['https://www.equidia.fr/courses/2022-07-31/R1/C1']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                #callback = self.parse, 
                meta = {
                    'playwright':True,
                    'playwright_page_event_handlers':{
                    "response": "handle_response",
                }
                
                }               
            )

    async def handle_response(self, response: PlaywrightResponse) -> None:
        """
        We can grab the post data with response.request.post - there are three different types for different needs.
        The method below helps grab those resource types of 'xhr' and 'fetch' until I can work out how to only send these to the download request.
        """
        self.logger.info(f'test the log of data: {response.request.resource_type, response.request.url, response.request.method}')
        jl_file = "videos.jl"
        data = {}
        if response.request.resource_type == "xhr":
            if response.request.method == "GET":
                if 'videos' in response.request.url:

                    data['resource_type']=response.request.resource_type,
                    data['request_url']=response.request.url,
                    data['method']=response.request.method
                    with jsonlines.open(jl_file, mode='a') as writer:
                        writer.write(data)

    def parse(self, response):
        video = pd.read_json('videos.jl', lines=True)
        print('KEST: %s' % video['request_url'][0][0])

        yield scrapy.FormRequest(
            url = video['request_url'][0][0],
            headers=headers,
            callback = self.parse_video_json
        )
    def parse_video_json(self, response):
        another_url = response.json().get('video_url')
        yield scrapy.FormRequest(
            another_url,
            headers=headers,
            callback=self.extract_videos
        )

    def extract_videos(self, response):
        videos = response.json().get('mp4')
        for keys, vids in videos.items():
            loader = videoItem()
            loader['video'] = [vids]
            yield loader

items.py

class videoItem(scrapy.Item):
    video = scrapy.Field()

pipelines.py

class downloadFilesPipeline(FilesPipeline):
    def file_path(self, request, response=None, info=None, item=None):
        file = request.url.split('/')[-1]
        video_file = f"{file}.mp4"
        return video_file

settings.py

from pathlib import Path
import os

BASE_DIR = Path(__file__).resolve().parent.parent

FILES_STORE = os.path.join(BASE_DIR, 'videos')

FILES_URLS_FIELD = 'video'
FILES_RESULT_FIELD = 'results'


DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

ITEM_PIPELINES = {
    'video.pipelines.downloadFilesPipeline': 150,
}

Absent seeing the actual site, a traditional selection and get_attribute should be sufficient:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.set_content("""
    <iframe title="IFRAME_NAME" src="https://www.data_I_want_TO_get.com"></iframe>
    """)
    src = page.get_attribute('iframe[title="IFRAME_NAME"]', "src")
    print(src)  # => https://www.data_I_want_TO_get.com
    browser.close()

If the frame isn't immediately visible you can wait for it by replacing the src = line with

src = (
    page.wait_for_selector('iframe[title="IFRAME_NAME"]')
        .get_attribute("src")
)

Playwright has a built in code generator which will make things easy for you.

Run this command and try it out

npx playwright codegen

It is not possible to do it with the URL as I can see

Anyways, I prepared an script for making what you wanted. I used an auxiliar function that you can use. I hope it works for you.

import time
from playwright.sync_api import Playwright, sync_playwright, expect

#This function wait for an element, if the element is not present it return false instead of throwing an exception and stopping the script. If element is not present after the timeout, it return false and we can use that false for our desired flow
def wait_for_selector_with_no_exception(page, selector, timeout):
    try:
        page.wait_for_selector(selector, timeout=timeout)
        return True
    except:
        return False

with sync_playwright() as p:
    #We define the browser, the context and the page
    browser = p.chromium.launch(headless=False)
    context = browser.new_context(
        user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36')
    page = browser.new_page()

    #We navigate to URL of the car we want (In this case the Tesla from your example)
    url_for_car = "https://turo.com/us/en/car-rental/united-states/austin-tx/tesla/model-s/1733237"
    page.goto(url_for_car)

    #We do click into start date dropdown element
    page.locator("#reservationDates-dateRangePicker").click()

    #Here we define the selector for the start date we want (You can replace it with your desired date, of course)
    selector_for_start_date="//td[contains(@aria-label,'December 29, 2022')]"

    #We use our auxiliar function to see if the start date is within the DOM, if not we click on arrow for moving to next month
    while not wait_for_selector_with_no_exception(page=page, selector=selector_for_start_date, timeout=2000):
        page.locator(".DayPickerNavigationNext_icon").click()

    #Once desired start date is within the DOM we click on it
    page.locator(selector_for_start_date).click()

    #This selector is the second datepicker element once is open
    selector_when_date_end_is_open = '//div[@class="DateInput_displayText DateInput_displayText__focused DateInput_displayText__has-input"]/..//input[@id="reservationDates-dateRangePicker-endDate"]'

    #We wait till the second datepicker is already open
    page.wait_for_selector(selector_when_date_end_is_open)

    #Here we define the selector for the end date we want (Also you can replace it with your desired end date)
    selector_for_end_date="//td[contains(@aria-label,'March 23, 2023')]"

    #Same we did with start date but with end date, we wait for our end date element, if is not present we click on next arrow
    while not wait_for_selector_with_no_exception(page=page, selector=selector_for_end_date, timeout=2000):
        page.locator(".DayPickerNavigationNext_icon").click()

    #Once our end date is present we click on it
    page.locator(selector_for_end_date).click()

    #This sleep you can remove it, is only for you to see that it worked as expected
    time.sleep(10)

    #We close the context and the browser
    context.close()
    browser.close()

As you can see, I used xpath for my selectors because it is what I have under control, if you find a way of doing it with CSS go ahead.

Step 1: make sure you're in Linux Container in Docker.

Step 2: Go to C:\Users\[User Name]\.docker\config.json Replace credsStore to credStore

Step 3: create Dockerfile, .dockerignore and docker-compose.yml file

Step 4: docker-compose format.

version: "3.9"
services:
frontend:
build: .
ports:
  - "8000:80"
backend:
image: "mcr.microsoft.com/mssql/server"
environment:
  SA_PASSWORD: "Docker123!"
  ACCEPT_EULA: "Y"
ports:
  - "1440:1433"

try to use sam ports as I mentioned here. User = sa, Database = master It will run perfectly fine. IA.

I am not used to use async and I am not sure of your question, but I think this is what you want:

import asyncio
from playwright.async_api import async_playwright


async def main():
    async with async_playwright() as p:
        for browser_type in [p.chromium, p.firefox, p.webkit]:
            browser = await browser_type.launch(headless=False)
            page = await browser.new_page()


            # Now, we gonna expect a request, but that request we have to expect it after doing something, in my example is the navigation to google, in your example could be clicking a button or something else.
            async with page.expect_request("https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png") as first:
                await page.goto("https://google.com")
            first_request = await first.value
            print(f"The method of the request was: {first_request.method}")
            print(f"The url of the request was: {first_request.url}")
            print()
            print("And the headers:")
            # You can use this first_request.all_headers() as a Json
            print(await first_request.all_headers())

            await asyncio.sleep(5)

asyncio.run(main())

I did it with google, you should do it with your own page and knowing what should be the request url.

Here some doc: https://playwright.dev/python/docs/api/class-page#page-wait-for-request

As a possible workaround, if you are redirected(to the broken page) after the token/cookie is granted, you can as well navigate to a normal site url, and you should be logged in