How to use test_jshandle_as_element_work method in Playwright Python

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

This should get you started, because you don't really need to do fancy image trickery.

All reddit posts have data-testid=post-container selectors which let you get the top level post. All comments have a .Comment class you can use, so you can enumerate all the comments on any given reddit post and capture them separately if that's what you want.

Some major caveats of this code:

• It does not respect comment nesting - I only took a brief glance at Reddit's page source but they don't seem to have easy ways of getting comment chains.

• Reddit could change their selector names/CSS classes any time, and they do have some inconsistencies in how they seem to name things anyway.

• You specify you want to capture AskReddit threads, which can be quite large, so you could easily end up with, say, 1000+ comment screenshots on popular threads, or even 10K+ screenshots if you just let it run with abandon. I set a limit of 10, modify code as needed.

• I wrote this as a demonstration, I have no idea how it will perform on giant posts or such, but hopefully it shows you how to actually pull the information you want. If you need to do more work, you will likely have to inspect Reddit page layouts with your browser's developer tools.

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright, Browser

MAX_COMMENT_LIMIT = 10


async def capture(browser: Browser,  url: str) -> None:
    page = await browser.new_page()

    await page.goto(url)
    posts = page.locator("data-testid=post-container")

    # reddit loads differently sometimes, so you need to
    # find the first post on the page
    await posts.first.screenshot(path="post.png")

    # this does not respect comment-nesting
    comments = page.locator(".Comment")
    count = await comments.count()
    print(f"Found {count} comments")
      
    for i in range(count):
        await comments.nth(i).screenshot(path=f"comment-{i}.png")

        if i+1 == MAX_COMMENT_LIMIT:
            print(f"Screened {MAX_COMMENT_LIMIT} images, stopping.")
            break


async def main() -> None:
    reddit_url = "https://www.reddit.com/r/cpp/comments/w9xkmf/c_for_scientific_programming/"
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await capture(browser, reddit_url)
        await browser.close()

asyncio.run(main())

The thread I picked generated some images like this...

Top-level post text

A comment

This here worked for me:

with page.expect_download() as download_info:    
    page.goto("Link")
    page.click('xpath=button')
download = download_info.value
path = download.path()
download.save_as("filename.csv")
print(path)

#---------------------
context.close()
browser.close()

I hope it helps you!

Playwright has no method get_by_role.

Try this:

page.locator('link[name="Log in"]').click()

You can give the locator method any CSS or Xpath expression.

There are two things to notice:

1. Python executable
2. Alternatives priority

Executable

Running in the base image:

# which python
# which python3
/usr/bin/python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8

With the following Dockerfile (from the question);

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 2

python is now added and controlled by alternatives, but python3 is untouched:

# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:10 /usr/bin/python -> /etc/alternatives/python
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 18 Dec 16 04:10 /etc/alternatives/python -> /usr/bin/python3.8

Priority

From alternative's manpage:

If the group is in automatic mode, and the newly added alternatives' priority is higher than any other installed alternatives for this group, the symlinks will be updated to point to the newly added alternatives.

This is numeric, so it's not that 1 is the winner and 2 is second-best, but rather 2 is higher than 1.

Resolution

By changing both the executable form python to python3, and the priority so that 3.10 will be 2 and 3.8 will be 1 (as such):

FROM mcr.microsoft.com/playwright
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y python3.10

RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.10 2
RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.8 1

the default python version should be 3.10:

# which python
# which python3
# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Dec 16 04:16 /usr/bin/python3 -> /etc/alternatives/python
-rwxr-xr-x 1 root root 5565184 Dec  7 01:12 /usr/bin/python3.10
-rwxr-xr-x 1 root root 5230592 Jun 22 20:18 /usr/bin/python3.8
# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 19 Dec 16 04:16 /etc/alternatives/python -> /usr/bin/python3.10
# python3 --version
Python 3.10.9

There is another alternative package which works for scraping google play store data, below is the link

https://pypi.org/project/google-play-scraper/

In case, your looking straight for sample code follow the below steps

from google_play_scraper import app

result = app(
'com.nianticlabs.pokemongo',
lang='en', # defaults to 'en'
country='us' # defaults to 'us'
)
print(result)

There are many things you can improve there.

• First, you can read the text area value using page.locator('#startText').value()
• Then, you click on the check button. Don't wait using wait_for_load_state, because the page is not navigating.
• After the check action there is an SPAN element inside the "#inside" div. You can do page.locator('#inside SPAN').inner_html(). The inner_html function will wait for the element for you.

You have to use inner_text for this:

email = page.locator.inner_text("#cphMain_hlOrgInboundEmail")
print(email)

Try this:

# Needed libs
from playwright.sync_api import Playwright, sync_playwright, expect

with sync_playwright() as p:
    #We define the browser, the context and the page
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = context.new_page()
    # Go to URL for this example
    page.goto("https://demoqa.com/browser-windows")

    # With the current context we wait for a new page, and we do whatever we have to do inside for opening that new page (tab)
    with context.expect_page() as new_page_info:
        # Needed actions for opening the new tab, in my case clicling a button
        page.locator('#tabButton').click()  # Opens a new tab
    # Now we have the new page
    new_page = new_page_info.value
    # We do whatever we want with our new page, in my case it is only taking a text from an element and printing it out
    text_of_h1_from_new_page = new_page.locator('#sampleHeading').inner_text()
    print(text_of_h1_from_new_page)
    # Close the context and browser
    context.close()
    browser.close()

More info here: https://playwright.dev/python/docs/pages#handling-new-pages