How to use test_wait_for_request_should_work_with_url_match method in Playwright Python

Based on this answer, one trick would be loading the browser with a persistent_context, closing the browser, and then edit the preference file setting the Preserve log value.

user_data_dir = './prefs'
pref_file_path = user_data_dir + '/Default/Preferences'
browser = playwright.chromium.launch_persistent_context(user_data_dir, headless=False, args= ['--auto-open-devtools-for-tabs'])
browser.close()

with open(pref_file_path, 'r') as pref_file:
    data = json.load(pref_file)

data['devtools'] = {
    'preferences': {
        'network_log.preserve-log': '"true"'
    }
}

with open(pref_file_path, 'w') as pref_file:
    json.dump(data, pref_file)

browser = playwright.chromium.launch_persistent_context(user_data_dir, headless=False, args= ['--auto-open-devtools-for-tabs'])
page = browser.new_page()
page.goto('https://stackoverflow.com/questions/63661366/puppeteer-launch-chromium-with-preserve-log-enabled')

I tried this and it worked (I used an element handler instead of arguments[0]):

selector = page.query_selector('.my-image')
img_base64 = page.evaluate("""(selector_handle) => {
    var cnv = document.createElement('canvas');
    cnv.width = 300; cnv.height = 300;
    cnv.getContext('2d').drawImage(selector_handle, 0, 0);
    return cnv.toDataURL('image/jpeg').substring(22);
}""", selector)
with open(r"image.jpg", 'wb') as f:
    f.write(base64.b64decode(img_base64))

So, it seems that this is simply not directly possible with python-playwright, nor is it going to be. From this discussion, I gathered that the dialog...

"[...] is not a part of the web, it is a part of the browser, so we don't automate it."

There are workarounds, e.g. to trigger a download in a different way, but this would then not involve handling the browser dialog, as stated in the question.

Your locator should look like this. Replace ":" with ">>"

div[class="some-class"]>>nth=3

page.goto returns the Response of the page where its navigating to. So you can do something like that:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    response = page.goto("http://whatsmyuseragent.org/")
    assert response.ok

    browser.close()

There are some options to render javascript pages. If you would like to use Python you can use Scrapy-Splash / Selenium / Playwright.

If you are ok with NodeJs you can use Puppeteer / Playwright

I prefer Puppeteer if I need to render a page.

Here is the solution for you.

npm i puppeteer

File name: price.js

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.target.com/p/red-blend-wine-750ml-bottle-california-roots-8482/-/A-52525405#lnk=sametab');
  let price = await page.evaluate(()=>{
      return document.querySelector('div[data-test="product-price"]').innerText;
  });

  await browser.close();
})();

Unfortunately, I do not think this is possible. For web scraping, you might use requests or urllib to download the page then use an HTML parser, such as BeautifulSoup4 or lxml, to scrape the webpage. While browsers may have a URL for the new tab (Chrome, for example, has chrome://newtab), requests and urllib aren't able to establish connections to the URLs.

For requests:

>>> import requests
>>> res = requests.get("chrome://newtab")
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    res = requests.get("chrome://newtab")
  ...
requests.exceptions.InvalidSchema: No connection adapters were found for 'chrome://newtab'

For urllib:

>>> from urllib.request import urlopen
>>> res = urlopen("chrome://newtab")
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    res = urlopen("chrome://newtab")
  File "C:\Users\JLpyt\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  ...
urllib.error.URLError: <urlopen error unknown url type: chrome>

With browser automation packages (selenium, playwright, pyppeteer), you are able to access the new tab page, by sending the Ctrl+T/Cmd+T virtual keystrokes. From there, you would be able to pass the page content to an HTML parser and scrape the page. While this is possible for 'classic' browsers (Chrome, Firefox, Internet Explorer, etc.), I don't know of any package for Brave Browser.

Perhaps there is something I am missing, but, as far as I know, you cannot scrape the new tab page of Brave Browser.

From the docs, you should use a string as selector.

page.waitForSelector(selector[, options])

selector <string> A selector to to query for

page.click(selector[, options])

selector A selector to search for an element

selector = '//*[@id="sections-scroll-target"]/ul/ul/li[8]/a'
    
page.wait_for_selector(selector, timeout=10000)

page.hover(selector)

Looks like you want https://www.testdummysite.com/news/markets?mod=nav_left_section

If so this will work

//*[@href= "https://www.testdummysite.com/news/markets?mod=nav_left_section" ]

but these get a different link https://www.testdummysite.com/?mod=nav_left_section.

//*[@id="sections-scroll-target"]/ul/ul/li[8]/a

#sections-scroll-target > ul > ul > li:nth-child(8) > a

Because these URLs are similar, I wonder if you've posted the HTML accurately?

You can use the not_be_visible expect assertion for this.

expect('element CSS here').not_to_be_visible()

By default expect assertions have a 5 second timeout, in case you want to increase the timeout, you can do like this:

expect('element CSS here').not_to_be_visible(timeout=7000)