How to use wait_for_function method in Playwright Python

Playwright will work with proprietary Chromium-based browsers (including Opera, Edge, Chrome etc.), you can install the browser independently from Playwright and then pass path to its executable via execuable_path launch option:

browser = playwrights.chromium.launch(executable_path='/opt/path_to_opera_bin')

For automating WebKit and Firefox Playwright currently requires remote debugging protocol which is only available in the custom builds.

This here worked for me:

with page.expect_download() as download_info:    
    page.goto("Link")
    page.click('xpath=button')
download = download_info.value
path = download.path()
download.save_as("filename.csv")
print(path)

#---------------------
context.close()
browser.close()

I hope it helps you!

You can do like this:

expect("SELECTOR").to_have_attribute("aria-disabled", "value", timeout=5000)

Turns out I was close, but to get the iframe correctly, I needed to call the contentFrame() method.

Returns the content frame for element handles referencing iframe nodes, or null otherwise

Then, querySelector() will return the respective elementHandle just fine:

with sync_playwright() as p:
    for browser_type in [p.chromium, p.firefox, p.webkit]:
        browser = browser_type.launch(headless=False)
        page = browser.newPage()
        page.goto('https://www.w3schools.com/tags/tryit.asp?filename=tryhtml_option')
        iframe = page.querySelector('iframe').contentFrame()
        element = iframe.querySelector('select')
        print(type(element))
        print(element.innerHTML())
        browser.close()

successfully prints

<class 'playwright.sync_api.ElementHandle'>

  <option value="volvo">Volvo</option>
  <option value="saab">Saab</option>
  <option value="opel">Opel</option>
  <option value="audi">Audi</option>

Note: if there are multiple iframes, you can just use an attribute when retrieving the handle. To get the iframe by its id in the above example, e.g. use

iframe = page.querySelector('iframe[id=\"iframeResult\"]').contentFrame()

First of all, Playwright already has a full suite of selectors that work on the live page, so to eliminate a dependency, speed up your scrape, use less code and avoid weird errors when the static HTML snapshot gets out of sync with the live page, I suggest skipping BS.

On to the main problem, you've done good by printing the HTML to see what sort of response you're dealing with. The 404 page indicates you've been detected as a bot when running headlessly, but this can often manifest as a captcha, Cloudflare browser check page, or other "are you a robot?" notice.

As with everything in scraping, there's no one-size-fits-all solution, but one typical approach is to set a custom user agent string:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    ua = (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/69.0.3497.100 Safari/537.36"
    )
    url = (
        "https://www.apple.com/br/shop/product/MV7N2BE/A/airpods-com-estojo-de-recarga"
    )
    page = browser.new_page(user_agent=ua)
    page.goto(url, wait_until="domcontentloaded")
    sel = "span.as-price-installments:last-child"
    text = (
        page.wait_for_selector(sel)
        .text_content()
        .replace("à vista (10% de desconto)", "")
        .strip()
    )
    print(text)  # => R$ 1.399,50
    browser.close()

You should be able to get that button by text:

page.locator('"Log in"').click()

This should get you started, because you don't really need to do fancy image trickery.

All reddit posts have data-testid=post-container selectors which let you get the top level post. All comments have a .Comment class you can use, so you can enumerate all the comments on any given reddit post and capture them separately if that's what you want.

Some major caveats of this code:

• It does not respect comment nesting - I only took a brief glance at Reddit's page source but they don't seem to have easy ways of getting comment chains.

• Reddit could change their selector names/CSS classes any time, and they do have some inconsistencies in how they seem to name things anyway.

• You specify you want to capture AskReddit threads, which can be quite large, so you could easily end up with, say, 1000+ comment screenshots on popular threads, or even 10K+ screenshots if you just let it run with abandon. I set a limit of 10, modify code as needed.

• I wrote this as a demonstration, I have no idea how it will perform on giant posts or such, but hopefully it shows you how to actually pull the information you want. If you need to do more work, you will likely have to inspect Reddit page layouts with your browser's developer tools.

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright, Browser

MAX_COMMENT_LIMIT = 10


async def capture(browser: Browser,  url: str) -> None:
    page = await browser.new_page()

    await page.goto(url)
    posts = page.locator("data-testid=post-container")

    # reddit loads differently sometimes, so you need to
    # find the first post on the page
    await posts.first.screenshot(path="post.png")

    # this does not respect comment-nesting
    comments = page.locator(".Comment")
    count = await comments.count()
    print(f"Found {count} comments")
      
    for i in range(count):
        await comments.nth(i).screenshot(path=f"comment-{i}.png")

        if i+1 == MAX_COMMENT_LIMIT:
            print(f"Screened {MAX_COMMENT_LIMIT} images, stopping.")
            break


async def main() -> None:
    reddit_url = "https://www.reddit.com/r/cpp/comments/w9xkmf/c_for_scientific_programming/"
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await capture(browser, reddit_url)
        await browser.close()

asyncio.run(main())

The thread I picked generated some images like this...

Top-level post text

A comment

I'd use the technique from can i run playwright outside of 'with'? and How to start playwright outside 'with' without context managers on the interactive repl:

PS C:\Users\foo\Desktop> py
Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from playwright.sync_api import sync_playwright
>>> p = sync_playwright().start()
>>> browser = p.chromium.launch(headless=False)
>>> page = browser.new_page()
>>> page.goto("https://www.example.com")
<Response url='https://www.example.com/' request=<Request url='https://www.example.com/' method='GET'>>
>>> page.title()
'Example Domain'
>>> page.close()
>>> browser.close()
>>> p.stop()

If you use page.pause(), try running playwright.resume() in the browser dev tools console to resume the Python repl.

If you really need to do this from a script rather than the Python repl, you could use the code interpreter or roll your own, but I'd try to avoid this if possible.

I'd suggest to use the Playwright codegen to let it generate selectors for you: https://playwright.dev/docs/cli#generate-code

And use Locators instead of ElementHandles, they provide easy utility methods like .nth(42), .first, .last and automatically wait for an element to appear with the given selector. See here: https://playwright.dev/python/docs/api/class-locator

For more information about selectors see here: https://playwright.dev/docs/selectors