How to use test_expose_function method in Playwright Python

Just in case you had the same query or were struggling with dynamic web scraping I found a way to scrape the data without using selenium and bs4.

I used playwright which is far more straightforward and has a very much appreciated inner_html() function which reads straight into the dynamic flex HTML code. Here is the code for reference.

#part of the help to write the script I got from https://stackoverflow.com/questions/64303326/using-playwright-for-python-how-do-i-select-or-find-an-element

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(slow_mo=1000)

    page = browser.new_page()
    page.goto('https://vemcount.app/embed/widget/uOCRuLPangWo5fT')
    central = page.query_selector("p.w-full span");
    print({'central': central.inner_html()})
        
    browser.close()
 

If there is a better way I am more than happy to hear your suggestions.

Yours,

A Beginner in Python. :)

You have to use inner_text for this:

email = page.locator.inner_text("#cphMain_hlOrgInboundEmail")
print(email)

It's been suggested by the developers of scrapy_playwright to instantiate the DOWNLOAD_HANDLERS and TWISTER_REACTOR into your script.

A similar comment is provided here

Here's a working script implementing just this:

import scrapy
from scrapy_playwright.page import PageCoroutine
from scrapy.crawler import CrawlerProcess

class ProductSpider(scrapy.Spider):
    name = 'product'

    def start_requests(self):
        yield scrapy.Request(
            'https://shoppable-campaign-demo.netlify.app/#/',
            callback = self.parse,
            meta={
                'playwright': True,
                'playwright_include_page': True,
                'playwright_page_coroutines': [
                    PageCoroutine("wait_for_selector", "div#productListing"),
                ]
            }
        )

    async def parse(self, response):
        container = response.xpath("(//div[@class='col-md-6'])[1]")
        for items in container:
            yield {
                'products':items.xpath("(//h3[@class='card-title'])[1]//text()").get()
            }
        # parses content

if __name__ == "__main__":
    process = CrawlerProcess(
        settings={
            "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
            "DOWNLOAD_HANDLERS": {
                "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
                "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            },
            "CONCURRENT_REQUESTS": 32,
            "FEED_URI":'Products.jl',
            "FEED_FORMAT":'jsonlines',
        }
    )
    process.crawl(ProductSpider)
    process.start()

And we get the following output:

{'products': 'Oxford Loafers'}

They are saved in the same way, but in the second case you are overwriting the variable with return value of .screenshot method, which seems to be of the type bytes

Please refer this stack overflow link for more details. hope this will help to solve your issue

If you want a Playwright image with Python instead of the Playwright image with Node you have currently set,

do

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal

instead. You won't then need to install python3-pip (or playwright!) at all.

Additionally, you'll probably want to explicitly spell out python3 instead of python:

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal
RUN pip3 install TikTokApi
RUN playwright install
COPY . .

In playwright, there are three alternatives to select dropdown element

1.  select by label

2. select by index

3.  select by value

You can select dropdown element by visible text(label) or by value instead and select dropdow element by attribute is not supported in playwright even not in Selenium too.

Example:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.firefox.launch()
    page = browser.new_page() 
    page.goto(url)  
    page.locator('//select[@name="dropdown_selected_size_name"]').select_option(label="4,0 x 30 mm 500 Stück")
    #page.locator('//select[@name="dropdown_selected_size_name"]').select_option(value="B08XWN4HM6")
    page.wait_for_timeout(4000)
    browser.close()

you can use the channel msedge

robotframework example

New Browser chromium headless=false channel=msedge

New Context viewport={'width': 1920, 'height': 1080}

New Page https://marketsquare.github.io/robotframework-browser/Browser.html

import requests
import threading

totalRequests = 0
numberOfThreads = 10
threads = [0] * numberOfThreads


def worker(thread):
    r = requests.get("url")
    threads[thread] = 0  # free thread


while totalRequests < 100:
    for thread in range(numberOfThreads):
        if threads[thread] == 0:
            threads[thread] = 1  # occupy thread
            t = threading.Thread(target=worker, args=(thread,))
            t.start()
            totalRequests += 1