How to use serialize_call_stack method in Playwright Python

Looks like soup is unable to find anything and its returning None and Nonetype has no text property

The error looks here for soup in clothes: as you have overridden the soup parser

PS:

You can use get_text() to extract the text

price = soup.find('span', {'class' : 'price'}).get_text()

Sometimes you may have to use contents depending on the return type

price = soup.find('span', {'class' : 'price'}).contents[0]

At the end of each iteration of "y" you issue a mouse up event. However, you dont issue a mouse move event to the start of the new line before putting the mouse down again.

Try the below code:

    for y in range(100, 1000, 100):
        # Generate mouse points
        points = []
        wm(start_point, y, x, y, M_0=15, D_0=12, move_mouse=lambda x, y: points.append([x, y]))

        page.mouse.up()
        # Move mouse to start of new line
        page.mouse.move(points[0][0], points[0][1])
        # Mouse back down
        page.mouse.down()
        #Iterate the remaining points
        for point in points[1:]:
            page.mouse.move(point[0], point[1])
        page.mouse.up()

Your xpath selector is not correct. Try this

'Year of Est.': response.xpath("//td[contains(text(), 'Year Established')]/following-sibling::td/div/div/div/text()").extract()

I also note some errors in your code such as the line below which will raise an error. You may want to recheck how you extract links from the search page.

data = self.link_extractor.extract(response.text, base_url=response.url)

Edit: The year of establishment is loaded once the company tab is clicked. You have to simulate the click using selenium or scrapy-playwright. My simple implementation using scrapy-playwright is as below.

import scrapy
from scrapy.crawler import CrawlerProcess
import os
from selectorlib import Extractor
from scrapy_playwright.page import PageCoroutine


class Spider(scrapy.Spider):
    name = 'alibaba_crawler'
    allowed_domains = ['alibaba.com']
    start_urls = ['http://alibaba.com/']
    link_extractor = Extractor.from_yaml_file(os.path.join(os.path.dirname(__file__), "../resources/search_results_searchpage.yml"))

    def start_requests(self):
        search_text = "Headphones"
        url = "https://www.alibaba.com/trade/search?fsb=y&IndexArea=product_en&CatId=&SearchText={0}&viewtype=G".format(
            search_text)
        yield scrapy.Request(url, callback=self.parse, meta={"search_text": search_text})

    def parse(self, response):
        data = self.link_extractor.extract(
            response.text, base_url=response.url)
        for product in data['products']:
            parsed_url = product["link"]

            yield scrapy.Request(parsed_url, callback=self.crawl_mainpage, meta={"playwright": True, 'playwright_page_coroutines': {
                    "click": PageCoroutine("click", selector="//span[@title='Company Profile']"),
                },})

    def crawl_mainpage(self, response):
        yield {
            'name': response.xpath("//h1[@class='module-pdp-title']/text()").extract(),
            'Year of Establishment': response.xpath("//td[contains(text(), 'Year Established')]/following-sibling::td/div/div/div/text()").extract()
        }


if __name__ == "__main__":
    process = CrawlerProcess(settings={
        'DOWNLOAD_HANDLERS': {
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        'TWISTED_REACTOR' :"twisted.internet.asyncioreactor.AsyncioSelectorReactor"
    })
    process.crawl(Spider)
    process.start()

Below is a sample log of running the scraper using python crawler.py. The year 2010 is shown in the output

I had a similar problem: I had PM2 run by an Apache Airflow Celery worker running a playwright task, hourly. The job hung.

I suspect it is the environment variables littered by Byobu and PM2. What worked for me was wrapping the Python script in a bash script and sanitizing the environment variable like the following:

#!/bin/bash
[ "$HOME" != "" ] && exec -c $0
# Your python script should come below:
/home/ubuntu/usr/venv-3.10-airflow/bin/airflow celery worker

The first line clears the env vars (as suggested in Sanitize environment with command or bash script?). Now you can run this script with pm2 start command. My worker ran fine for about 24 hours as of now. I hope this works for you too.

I guess you are using codegen probably.

That page is a little bit strange for urls as I can see.

Anyways, I fixed your code:

from playwright.sync_api import Playwright, sync_playwright, expect


def run(playwright: Playwright) -> None:
    browser = playwright.chromium.launch(headless=False)
    context = browser.new_context()

    # Open new page
    page = context.new_page()

    # Go to https://www.elections.il.gov/CampaignDisclosure/ContributionSearchByCommittees.aspx?T=637994490317517425
    page.goto(
        "https://www.elections.il.gov/CampaignDisclosure/ContributionSearchByCommittees.aspx")

    # Fill input[name="ctl00\$ContentPlaceHolder1\$txtCmteID"]
    page.locator("input[name=\"ctl00\\$ContentPlaceHolder1\\$txtCmteID\"]").fill("34589")

    page.locator("input:has-text(\"Search\")").click()

    page.locator("text=Download This List").click()

    # Click text=CSV File
    with page.expect_download() as download_info:
        page.locator("text=CSV File").click()
    download = download_info.value
    download.save_as(download.suggested_filename)


    # ---------------------
    context.close()
    browser.close()


with sync_playwright() as playwright:
    run(playwright)

If you need to handle new page without event listener (i.e. new tab open by link click), you can try this code:

    from playwright import sync_playwright

    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.newContext()
        page = context.newPage()
        page.goto('<site url>')
        with context.expect_page() as tab:
            page.click('.newTabByLink')
        # do some steps
        ...
        tab.close()
        
        browser.close()

In playwright, there are three alternatives to select dropdown element

1.  select by label

2. select by index

3.  select by value

You can select dropdown element by visible text(label) or by value instead and select dropdow element by attribute is not supported in playwright even not in Selenium too.

Example:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.firefox.launch()
    page = browser.new_page() 
    page.goto(url)  
    page.locator('//select[@name="dropdown_selected_size_name"]').select_option(label="4,0 x 30 mm 500 Stück")
    #page.locator('//select[@name="dropdown_selected_size_name"]').select_option(value="B08XWN4HM6")
    page.wait_for_timeout(4000)
    browser.close()

I have the same problem. As David said, this error means that python-playwright or selenium is not launching correctly. What I did is

1. Check the version of your chrome Right upper corner -> help -> About Google Chrome
2. Download the chromedriver from chromedriver!
3. let your API find the driver

api = TikTokApi.get_instance(use_selenium=True, executablePath="/path/to/chromedriver")

Just use some condition instead of print method, for example you could check if response contains some key in its json:

def run(playwright):
    chromium = playwright.chromium
    browser = chromium.launch()
    page = browser.new_page()
    # Subscribe to "request" and "response" events.
    page.on("request", lambda request: print(">>", request.method, request.url))
    page.on("response", lambda response: print("<<", response.status, response.url))
    page.goto("https://example.com")
    browser.close()

For Example:

page.on("response", lambda response: response if key in response.body())

There should be waitForResponse for python too, and you could use that.