How to use _on_request method in Playwright Python

Type F12 in Google Chrome and go to the console.

Write $x() and place the xpath selector inside the parenthesis. Hit the enter key and make sure you actually find the right selector.

No need to use e.value, you can do something like this:

p1 = page.locator("//*[@id='rc-tabs-1-panel-apikey']/div/span/input")
p1text = p1.inner_text

Also - use * as less as possible. If the element with the id is a div write div for example and not *. It makes the searching faster.

I was able to get the titles of the news articles with the following code:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
  browser = p.chromium.launch(headless=False)
  page = browser.new_page()
  page.goto('https://blog.coinbase.com/', wait_until='domcontentloaded')
  elements = page.query_selector_all('*[data-post-id]')
  titles = []
  for element in elements:
    try:
      title = element.query_selector('h3 div')
      title = title.text_content()
      if not title in titles:
        titles.append(title)
    except Exception as e:
      continue
  print(titles)

It might not be exactly what you're looking for but hopefully it gets you in the right direction!

The issue is the docker image you have selected (alpine) architecture tags dont match any that playwright supports.

python:3.9:alpine

/code/src # pip debug --verbose | grep "py3-"
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
  py3-none-musllinux_1_2_x86_64
  py3-none-musllinux_1_1_x86_64
  py3-none-musllinux_1_0_x86_64
  py3-none-linux_x86_64
  py3-none-any

playwright tags

/code/src # pip install playwright -vv | grep "Skipping link" | cut -f2 -d'(' | cut -f1 -d')' | grep "^py3" | sort -u
ERROR: Could not find a version that satisfies the requirement playwright (from versions: none)
ERROR: No matching distribution found for playwright
py3-none-macosx_10_13_x86_64
py3-none-macosx_11_0_arm64
py3-none-macosx_11_0_universal2
py3-none-manylinux1_x86_64
py3-none-manylinux2014_aarch64, py3-none-manylinux_2_17_aarch64
py3-none-manylinux_2_27_x86_64
py3-none-win32
py3-none-win_amd64

This is why pip is telling you there is no matching distribution because its checking all the builds for playwright against the builds alpine linux wants and there isnt a common one, so it cannot install. You will need to use a docker image with an architecture that playwright is built against and supports. You can see this in more detail if you run pip install playwright -vv in the alpine linux image

  Skipping link: none of the wheel's tags (py3-none-manylinux2014_aarch64, py3-none-manylinux_2_17_aarch64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/a6/c5/142449e50253fef
ff667584793ef303c1bb1ecf1cb849cd8b83cd9a653c1/playwright-1.27.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (from https://pypi.org/simple/playwright/) (requires-python:>=3.7)
  Skipping link: none of the wheel's tags (py3-none-win32) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/79/54/cc0b824dd92507f4ed48e2a71424af2d0b4f91d72f3932ad4fefa7c68981/pla
ywright-1.27.1-py3-none-win32.whl (from https://pypi.org/simple/playwright/) (requires-python:>=3.7)
  Skipping link: none of the wheel's tags (py3-none-win_amd64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/5d/a0/e7009f47b7fef143617a2907fad14170a5ae765fe7bad8050402addd3cab
/playwright-1.27.1-py3-none-win_amd64.whl (from https://pypi.org/simple/playwright/) (requires-python:>=3.7)

Skipping link: not a file: https://pypi.org/simple/playwright/
Given no hashes to check 0 links for project 'playwright': discarding no candidates
ERROR: Could not find a version that satisfies the requirement playwright (from versions: none)
ERROR: No matching distribution found for playwright

You can see the architectures it builds against on there github https://github.com/microsoft/playwright-python/blob/release-1.27/setup.py#L79

Playwright handle differently with elements than the older python browsers...

in dropdown you should page.select_option(selector)

that's also why in Playwright you can't click a check box, you should page.check(selector) https://playwright.dev/python/docs/api/class-page#page-select-option

So, it seems that this is simply not directly possible with python-playwright, nor is it going to be. From this discussion, I gathered that the dialog...

"[...] is not a part of the web, it is a part of the browser, so we don't automate it."

There are workarounds, e.g. to trigger a download in a different way, but this would then not involve handling the browser dialog, as stated in the question.

Not guaranteed for success. Change the parameters in function async_cf_retry if it tends to fail. I wrote this code by modifying my heavy modules, so it may not work elegantly.

import re
import asyncio
from playwright.async_api import async_playwright, Error, Page
from cf_clearance import stealth_async
import httpx

# local_client = httpx.AsyncClient(verify=False)


async def async_cf_retry(page: Page, tries=10) -> bool:
    # use tries=-1 for infinite retries
    # excerpted from `from cf_clearance import async_retry`
    success = False
    while tries != 0:
        try:
            title = await page.title()
        except Error:
            tries -= 1
            await asyncio.sleep(1)
        else:
            # print(title)
            if title == 'Please Wait... | Cloudflare':
                await page.close()
                raise NotImplementedError('Encountered recaptcha. Check whether your proxy is an elite proxy.')
            elif title == 'Just a moment...':
                tries -= 1
                await asyncio.sleep(5)
            elif "www." in title:
                await page.reload()
                tries -= 1
                await asyncio.sleep(5)
            else:
                success = True
                break
    return success


async def get_client_with_clearance(proxy: str = None):
    async def get_one_clearance(proxy=proxy, logs=False):
        # proxy = {"server": "socks5://localhost:7890"}
        if type(proxy) is str:
            proxy = {'server': proxy}
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False, proxy=proxy,
                                              args=["--window-position=1000,800", "--disable-web-security",
                                                    "--disable-webgl"])
            page = await browser.new_page()#viewport={"width": 0, "height": 0})
            await stealth_async(page)
            
            # Download
            async def on_download(download):
                print('download', download)
                await download.save_as(download.suggested_filename)
            page.on("download", on_download)

            if logs:
                def log_response(intercepted_response):
                    print("a response was received:", intercepted_response.url)
                
                page.on("response", log_response)
            
            await page.goto("https://www.historique-meteo.net/site/export.php?ville_id=1067")
            res = await async_cf_retry(page)
            if res:
                cookies = await page.context.cookies()
                cookies_for_httpx = {cookie['name']: cookie['value'] for cookie in cookies}
                ua = await page.evaluate('() => {return navigator.userAgent}')
                # print(ua)
            else:
                await page.close()
                raise InterruptedError("cf challenge fail")
            await asyncio.sleep(10000)
            return ua, cookies_for_httpx
    
    ua, cookies_for_httpx = await get_one_clearance(logs=True)
    print(cookies_for_httpx)


print(asyncio.get_event_loop().run_until_complete(get_client_with_clearance(
    # proxy='http://localhost:8888'
    # use proxifier on windows as an elite proxy
)))

If download <Download url='https://www.historique-meteo.net/site/export.php?ville_id=1067' suggested_filename='export-aix-les-bains.csv'> is shown in the console, the browser must be downloading your file.
This code may keep trying to get a cloudflare clearance even if the file has been downloaded. Edit your strategies in async_cf_retry if this happens.
Use persisent context offered by playwright to save and reuse your cloudflare clearance. Consider downloading with httpx.

You can simply do as below: (JavaScript)

 const list = await page.locator('locator').allTextContents()
 console.log("The Paragraphs List: => "+ list)

I know you said you're sure the element exists but I would still suggest using page.wait_for_selector() instead because I've run into a few situations where the element simply wasn't ready yet even though it seemed like it should be:

table = page.wait_for_selector('#grvPedidos')
print(table.inner_html())

Note that I'm using synchronous Playwright.

I have come up with a solution in Node js puppeteer and I think it work the same way in python. As far as why the button isn't being pressed, I have no idea. But, what does work is entering an address and then clicking the first address suggestion.

Here is my code:

const puppeteer = require('puppeteer');
async function start(){
    const browser = await puppeteer.launch()
    const page = await browser.newPage()
    await page.goto("https://www.ubereats.com/ca")
    await page.type("#location-typeahead-home-input", "Address")
    await page.waitForSelector('#location-typeahead-home-item-0');
    await page.click("#location-typeahead-home-item-0")
    await page.waitForNavigation()

    console.log(page.url())
    await browser.close()
}

start()