How to use test_should_expose_keyIdentifier_in_webkit method in Playwright Python

This error means that python-playwright or selenium is not launching correctly. What I did is

1.Check the version of your chrome Right upper corner -> help -> About Google Chrome

2.Download the chromedriver from chromedriver

3.let your API find the driver

api = TikTokApi.get_instance(use_selenium=True, executablePath="/path/to/chromedriver")

This should get you started, because you don't really need to do fancy image trickery.

All reddit posts have data-testid=post-container selectors which let you get the top level post. All comments have a .Comment class you can use, so you can enumerate all the comments on any given reddit post and capture them separately if that's what you want.

Some major caveats of this code:

• It does not respect comment nesting - I only took a brief glance at Reddit's page source but they don't seem to have easy ways of getting comment chains.

• Reddit could change their selector names/CSS classes any time, and they do have some inconsistencies in how they seem to name things anyway.

• You specify you want to capture AskReddit threads, which can be quite large, so you could easily end up with, say, 1000+ comment screenshots on popular threads, or even 10K+ screenshots if you just let it run with abandon. I set a limit of 10, modify code as needed.

• I wrote this as a demonstration, I have no idea how it will perform on giant posts or such, but hopefully it shows you how to actually pull the information you want. If you need to do more work, you will likely have to inspect Reddit page layouts with your browser's developer tools.

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright, Browser

MAX_COMMENT_LIMIT = 10


async def capture(browser: Browser,  url: str) -> None:
    page = await browser.new_page()

    await page.goto(url)
    posts = page.locator("data-testid=post-container")

    # reddit loads differently sometimes, so you need to
    # find the first post on the page
    await posts.first.screenshot(path="post.png")

    # this does not respect comment-nesting
    comments = page.locator(".Comment")
    count = await comments.count()
    print(f"Found {count} comments")
      
    for i in range(count):
        await comments.nth(i).screenshot(path=f"comment-{i}.png")

        if i+1 == MAX_COMMENT_LIMIT:
            print(f"Screened {MAX_COMMENT_LIMIT} images, stopping.")
            break


async def main() -> None:
    reddit_url = "https://www.reddit.com/r/cpp/comments/w9xkmf/c_for_scientific_programming/"
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await capture(browser, reddit_url)
        await browser.close()

asyncio.run(main())

The thread I picked generated some images like this...

Top-level post text

A comment

The download.path() in playwright is just a random GUID (globally unique identifier). It's designed to validate the download works - not to keep the file.

Playwright is a testing tool and imagine running tests across every major browser on every code change - any downloads would quickly take up a lot of space and it would hack people off if you need to manually clear them out.

Good news is you are very close - If you want to keep the file you just need to give the file a name in the save_as.

instead of this:

   download.save_as(path)

use this:

   download.save_as(download.suggested_filename)

That saves the file in the same location as the script.

In playwright, there are three alternatives to select dropdown element

1.  select by label

2. select by index

3.  select by value

You can select dropdown element by visible text(label) or by value instead and select dropdow element by attribute is not supported in playwright even not in Selenium too.

Example:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.firefox.launch()
    page = browser.new_page() 
    page.goto(url)  
    page.locator('//select[@name="dropdown_selected_size_name"]').select_option(label="4,0 x 30 mm 500 Stück")
    #page.locator('//select[@name="dropdown_selected_size_name"]').select_option(value="B08XWN4HM6")
    page.wait_for_timeout(4000)
    browser.close()

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

Not sure exactly if by "browser to show up" you refer at the fact that you need the browser to not run in headless mode.

There are two possible ways:

1. Install Playwright Test for VSCode as suggest by @MingJie-MSFT. You'll have a Show browser checkbox there in case you want to run your tests headless or not
2. In the config file, for use, add:

headless: false

Example for both options

You could use a regular expression for your locator:

page.locator(text=/\d{1,2}\sseconds?\slater/i)

You can install your playwright by adding it in requirement.txt. Once you re-deploy, it will be installed automatically together with your app.

example in requirement.txt:

playwright==1.9.2

But if it needs another command for installment, you should use App Engine Flexible. In App Engine Flexible, you can use Dockerfile to run another command and other configuration to install your additional dependencies and applications.

I guess you are using codegen probably.

That page is a little bit strange for urls as I can see.

Anyways, I fixed your code:

from playwright.sync_api import Playwright, sync_playwright, expect


def run(playwright: Playwright) -> None:
    browser = playwright.chromium.launch(headless=False)
    context = browser.new_context()

    # Open new page
    page = context.new_page()

    # Go to https://www.elections.il.gov/CampaignDisclosure/ContributionSearchByCommittees.aspx?T=637994490317517425
    page.goto(
        "https://www.elections.il.gov/CampaignDisclosure/ContributionSearchByCommittees.aspx")

    # Fill input[name="ctl00\$ContentPlaceHolder1\$txtCmteID"]
    page.locator("input[name=\"ctl00\\$ContentPlaceHolder1\\$txtCmteID\"]").fill("34589")

    page.locator("input:has-text(\"Search\")").click()

    page.locator("text=Download This List").click()

    # Click text=CSV File
    with page.expect_download() as download_info:
        page.locator("text=CSV File").click()
    download = download_info.value
    download.save_as(download.suggested_filename)


    # ---------------------
    context.close()
    browser.close()


with sync_playwright() as playwright:
    run(playwright)