How to use sha256 method in Playwright Python

Maybe you can check the selector is visible and then you can click it?

if PageMethod("isVisible","#selector1"):
    PageMethod("click","#selector1")    
if PageMethod("isVisible","#selector2"):
    PageMethod("click","#selector2")    
if PageMethod("isVisible","#selector3"):
    PageMethod("click","#selector3")

I think you can usue the method isVisible, because scrapy_playwright support the same you have in playwright: https://github.com/scrapy-plugins/scrapy-playwright#supported-methods

And of course, isVisible() is a supported method in playwrgiht: https://playwright.dev/python/docs/api/class-page#page-is-visible

From the docs, you should use a string as selector.

page.waitForSelector(selector[, options])

selector <string> A selector to to query for

page.click(selector[, options])

selector A selector to search for an element

selector = '//*[@id="sections-scroll-target"]/ul/ul/li[8]/a'
    
page.wait_for_selector(selector, timeout=10000)

page.hover(selector)

Looks like you want https://www.testdummysite.com/news/markets?mod=nav_left_section

If so this will work

//*[@href= "https://www.testdummysite.com/news/markets?mod=nav_left_section" ]

but these get a different link https://www.testdummysite.com/?mod=nav_left_section.

//*[@id="sections-scroll-target"]/ul/ul/li[8]/a

#sections-scroll-target > ul > ul > li:nth-child(8) > a

Because these URLs are similar, I wonder if you've posted the HTML accurately?

If you don't want to use context managers then you can start the browser without one using the .start() method:

from playwright.sync_api import sync_playwright

p = sync_playwright().start()
browser = p.chromium.launch(headless=False)
page = browser.new_page()
p.stop()

Remember to use the .stop() method once you are done.

Thanks for the question. I'll share what's possible today, and we would be happy to make changes if you could share your scenario as an issue on playwright-python.

playwright-python can work with environment variables like outlined in the linked doc. For example, you can set environment variable for PLAYWRIGHT_BROWSERS_PATH to specify a custom location for your browser binaries.

It is recommended to use Playwright against bundled browser binaries, since Playwright can often depend on browser-side changes that might not be available in older versions. These browser-side changes are done with the intent to improve execution reliability and breadth of Playwright capabilities.

In case you do want to use Playwright against a non-bundled browser binary, you can set the executablePath while launching a browser. For example, the script below launches Edge instead of the bundled Chromium binary.

Python 3.8.0 (default, Sep 21 2020, 12:25:40)
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from playwright import sync_playwright
>>> pw = sync_playwright().start()
>>> browser = pw.chromium.launch(headless=False, executablePath='/Applications/Microsoft Edge Dev.app/Contents/MacOS/Microsoft Edge Dev')

We do not provide browser specific packages like playwright-chromium for Python today, but we are happy to collect feedback on it and make changes.

Have you tried to download it as Binary package?

Maybe you can go over this libasound.so.2 package page and see if you can download it.

wget https://vault.centos.org/centos/8/AppStream/aarch64/os/Packages/alsa-lib-1.2.5-4.el8.aarch64.rpm

When I searched within apt, ubuntu package tool I don't see anything there

root@21be95fca6e6:/# apt search libasound.so.2
Sorting... Done
Full Text Search... Done

For instance, Vim package would appears like below

root@21be95fca6e6:/# apt search vim           
Sorting... Done
Full Text Search... Done
apvlv/jammy 0.4.0-2 arm64
  PDF viewer with Vim-like behaviour

If you want a Playwright image with Python instead of the Playwright image with Node you have currently set,

do

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal

instead. You won't then need to install python3-pip (or playwright!) at all.

Additionally, you'll probably want to explicitly spell out python3 instead of python:

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal
RUN pip3 install TikTokApi
RUN playwright install
COPY . .

I had the same issue as well. I ran this command, then restarted my command line and it started working.

playwright install-deps

I hope this helps.

I have the same problem. As David said, this error means that python-playwright or selenium is not launching correctly. What I did is

1. Check the version of your chrome Right upper corner -> help -> About Google Chrome
2. Download the chromedriver from chromedriver!
3. let your API find the driver

api = TikTokApi.get_instance(use_selenium=True, executablePath="/path/to/chromedriver")

Here is a fully working solution by using scrapy-playwright, I got the idea the following issue 61 and the users profile @lime-n.

We download the xhr requests sent and store these into a dict with both the playwright tools and scrapy-playwright. I include the playwright_page_event_handlers to integrate playwright tools for this.

Depending on the video size, it will take a few minutes to download.

main.py

import scrapy
from playwright.async_api import Response as PlaywrightResponse
import jsonlines
import pandas as pd
import json
from video.items import videoItem

headers = {
    'authority': 'api.equidia.fr',
    'accept': 'application/json, text/plain, */*',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'content-type': 'application/json',
    'origin': 'https://www.equidia.fr',
    'referer': 'https://www.equidia.fr/',
    'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-site',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

class videoDownloadSpider(scrapy.Spider):
    name = 'video'
    start_urls = ['https://www.equidia.fr/courses/2022-07-31/R1/C1']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                #callback = self.parse, 
                meta = {
                    'playwright':True,
                    'playwright_page_event_handlers':{
                    "response": "handle_response",
                }
                
                }               
            )

    async def handle_response(self, response: PlaywrightResponse) -> None:
        """
        We can grab the post data with response.request.post - there are three different types for different needs.
        The method below helps grab those resource types of 'xhr' and 'fetch' until I can work out how to only send these to the download request.
        """
        self.logger.info(f'test the log of data: {response.request.resource_type, response.request.url, response.request.method}')
        jl_file = "videos.jl"
        data = {}
        if response.request.resource_type == "xhr":
            if response.request.method == "GET":
                if 'videos' in response.request.url:

                    data['resource_type']=response.request.resource_type,
                    data['request_url']=response.request.url,
                    data['method']=response.request.method
                    with jsonlines.open(jl_file, mode='a') as writer:
                        writer.write(data)

    def parse(self, response):
        video = pd.read_json('videos.jl', lines=True)
        print('KEST: %s' % video['request_url'][0][0])

        yield scrapy.FormRequest(
            url = video['request_url'][0][0],
            headers=headers,
            callback = self.parse_video_json
        )
    def parse_video_json(self, response):
        another_url = response.json().get('video_url')
        yield scrapy.FormRequest(
            another_url,
            headers=headers,
            callback=self.extract_videos
        )

    def extract_videos(self, response):
        videos = response.json().get('mp4')
        for keys, vids in videos.items():
            loader = videoItem()
            loader['video'] = [vids]
            yield loader

items.py

class videoItem(scrapy.Item):
    video = scrapy.Field()

pipelines.py

class downloadFilesPipeline(FilesPipeline):
    def file_path(self, request, response=None, info=None, item=None):
        file = request.url.split('/')[-1]
        video_file = f"{file}.mp4"
        return video_file

settings.py

from pathlib import Path
import os

BASE_DIR = Path(__file__).resolve().parent.parent

FILES_STORE = os.path.join(BASE_DIR, 'videos')

FILES_URLS_FIELD = 'video'
FILES_RESULT_FIELD = 'results'


DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

ITEM_PIPELINES = {
    'video.pipelines.downloadFilesPipeline': 150,
}