How to use get_driver_env method in Playwright Python

Yes, according to your link, all proxies are "dead"

Before using proxies try checking them here is one possible solution:

import json
import requests
from pythonping import ping
from concurrent.futures import ThreadPoolExecutor


check_proxies_url = "https://httpbin.org/ip"
good_proxy = set()

# proxy_lst = requests.get("https://www.proxy-list.download/api/v1/get", params={"type": "https"})
# proxies = [proxy for proxy in proxy_lst.text.split('\r\n') if proxy]
proxy_lst = requests.get("http://proxylist.fatezero.org/proxy.list")
proxies = (f"{json.loads(data)['host']}:{json.loads(data)['port']}" for data in proxy_lst.text.split('\n') if data)

def get_proxies(proxy):
    proxies = {
        "https": proxy,
        "http": proxy
    }
    try:
        response = requests.get(url=check_proxies_url, proxies=proxies, timeout=2)
        response.raise_for_status()
        if ping(target=proxies["https"].split(':')[0], count=1, timeout=2).rtt_avg_ms < 150:
            good_proxy.add(proxies["https"])
            print(f"Good proxies: {proxies['https']}")
    except Exception:
        print(f"Bad proxy: {proxies['https']}")

with ThreadPoolExecutor() as executor:
    executor.map(get_proxies, proxies)

print(good_proxy)

Get a list of active proxies with ping up to 150ms.

Output:

{'209.166.175.201:8080', '170.39.194.156:3128', '20.111.54.16:80', '20.111.54.16:8123'}

But in any case, this is a shared proxy and their performance is not guaranteed. If you want to be sure that your parser will work, then it is better to buy a proxy.

I ran your code with received proxy '170.39.194.156:3128' and for now it works

Doing something similar for a project of mine, this is how I would do it.

from playwright.sync_api import sync_playwright

posts = ['https://playwright.dev/','https://playwright.dev/python/',]

def scrap_post_info(context, post):
    page = context.new_page()
    page.goto(post)
    print(page.title())
    # do whatever scraping you need to
    page.close()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    for post in posts:
        scrap_post_info(context, post)
        # some time delay

browser.close()

Thing is the code snippet from the playwright docs is more about opening a new page after clicking a link on an existing page. Since you already have the urls ready, you can just visit each page one by one, and do your scraping.

The Download historical data button is inside an iframe that points to this URL.

https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=1476&url=copper-prices-historical-chart-data

But it needs Javascript to work, meaning, you have to load the page in a browser (that executes javascript) to click that button [^1].

Luckily, there are tools to automate browser interactions. Playwright is a library available in many languages, but we'll use it with Python.

1. Install the latest version of Python.
2. Update the pip (Python package manager) to the latest version:

   python -m pip install --user --upgrade pip
   

3. Install playright

   python -m pip install --user playwright
   

4. Use playwright to install a browser. It will be used by playwright only.

   python -m playwright install chromium
   

5. Open the URL with playwright inspector:

   python -m playwright open 'https://www.macrotrends.net/1476/copper-prices-historical-chart-data'
   

A browser window should pop up. Change the Target to python and click Record button in Playwright inspector window.

Then do what you need to do to click & download the file from the page.

Playwright will generate some code for us.

Stop recording and copy the script.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    with p.chromium.launch(headless=False) as browser:
        page = browser.new_page(accept_downloads=True)
        page.goto('https://www.macrotrends.net/1476/copper-prices-historical-chart-data')
        
        # paste the script here

In my runs, playwright generated this script for me:

# Click button:has-text("Download Historical Data")
with page.expect_download() as download_info:
    page.frame(name="chart_iframe").click("button:has-text(\"Download Historical Data\")")

Combining both and adding the code to save the file, we have:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    print('launching the browser')
    with p.chromium.launch(headless=False) as browser:
        page = browser.new_page(accept_downloads=True)
        print('visiting the page')
        page.goto('https://www.macrotrends.net/1476/copper-prices-historical-chart-data')
        
        print('clicking download button')
        # Click button:has-text("Download Historical Data")
        with page.expect_download() as download_info:
            page.frame(name="chart_iframe").click("button:has-text(\"Download Historical Data\")")
        download = download_info.value
        
        print('saving the file')
        download.save_as('prices.csv')

One important point you should look out for is that Python is a whitespace-sensitive language, meaning you have to align the code indentation properly to make it run.

Save this file as download_copper_prices.py, and when we run the script,

python download_copper_prices.py

A browser should pop up, visit the page, download and save the prices as prices.csv in the working directory.

[^1]: Actually, we can scrape the prices right off the page, as they're serialized as JSON. With a bit of regex, we can extract the JSON from the HTML. But that solution seemed to me a bit too job-specific, so I opted for a more hands-on solution that he can apply for other occasions. Scripting browser interactions is an important tool every power user should learn.

That said, I'm not knowledgeable in AutoIt, but assuming it has capable Regex tools, it's entirely possible to achieve the same result in AutoIt by downloading the HTML of the URL that the iframe points to (https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=1476&url=copper-prices-historical-chart-data) then using this regex to extract the JSON:

var originalData = (.*);

Running this code on the browser console returns an array of 15k+ entries:

JSON.parse(document.body.innerHTML.match('var originalData = (.*);')[1])

consisting of date and closing prices:

[{
    "date": "1959-07-02",
    "close": "0.2930"
}, {...}, ...]

Then it should be easy to convert this JSON to CSV.

References

• https://playwright.dev/python/docs/intro

Each test must be independent from the rest. As you said, your test_delete_user can not depend on your test_create_user.

What I do? For my test test_create_user:

• Load website
• Create user
• Use expect statement to verify it worked
• In case you don't want to leave trash in the DB, then you can have another DB query for deleting that new user, but if this query does not work, the test should not fail.

For my test test_edit_user:

• Load website
• I always edit same user
• Use expect statement to verify it worked

For my test test_delete_user:

• Precondition: I have a query to DB to insert an user, then my test starts.
• Load website
• Delete user
• Use expect statement to verify it worked

I do it like this because each test is independent, lets imagine the function for deleting an user is working but your function for create user is not working, if your test were like this:

• Load website
• Create user
• Delete user
• Use expect statement to verify it worked

You will get a falsy error saying that your test_delete_user is not working, but actually it is.

I'd use the technique from can i run playwright outside of 'with'? and How to start playwright outside 'with' without context managers on the interactive repl:

PS C:\Users\foo\Desktop> py
Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from playwright.sync_api import sync_playwright
>>> p = sync_playwright().start()
>>> browser = p.chromium.launch(headless=False)
>>> page = browser.new_page()
>>> page.goto("https://www.example.com")
<Response url='https://www.example.com/' request=<Request url='https://www.example.com/' method='GET'>>
>>> page.title()
'Example Domain'
>>> page.close()
>>> browser.close()
>>> p.stop()

If you use page.pause(), try running playwright.resume() in the browser dev tools console to resume the Python repl.

If you really need to do this from a script rather than the Python repl, you could use the code interpreter or roll your own, but I'd try to avoid this if possible.

The only way I found is wrap the goto inside a try-except block

Content of main.py

from playwright.sync_api import Playwright, sync_playwright, expect

with sync_playwright() as p:
    #We define the browser, the context and the page
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = browser.new_page()
    with page.expect_download() as download_info:
        #If we navigate without try, it will throw an exception and it will stop our script, so, we wrap it inside a try except block
        try:
            page.goto("https://manueltgomes.com/wp-content/uploads/2021/09/ParseHTMLLInks_1_0_0_1.zip")
        except:
            pass
    download = download_info.value
    download.save_as(f"./{download.suggested_filename}")
    context.close()
    browser.close()

Then just run python3 main.py

The file will be saved in the same path of the main.py file

Quite a few questions there!

"Why does Selenium with Python have such a delay to issue commands"

I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.

This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?

Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.

Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.

"Why does Puppeteer seem to have a lower delay when interacting with the same browser?" Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.

However - how much faster is it really?

In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).

You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.

When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)

Some other options to think about if you really need speed:

• Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
• Try other browsers
• If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
• Another option that isn't selenium or puppeteer is javascript directly in the browser
• Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.

If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.

Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)

https://www.tessco.com/product/207882 loads quit slow. Try to extend the default timeout of 30000ms adding a timeout to page.goto(link):

page.goto(link, timeout = 0)

With setting timeout to 0 you disable the timeout. Documentation

Alternatively, you can disable timeout with the following:

page.set_default_timeout(0)
page.goto(link)

First of all, Playwright already has a full suite of selectors that work on the live page, so to eliminate a dependency, speed up your scrape, use less code and avoid weird errors when the static HTML snapshot gets out of sync with the live page, I suggest skipping BS.

On to the main problem, you've done good by printing the HTML to see what sort of response you're dealing with. The 404 page indicates you've been detected as a bot when running headlessly, but this can often manifest as a captcha, Cloudflare browser check page, or other "are you a robot?" notice.

As with everything in scraping, there's no one-size-fits-all solution, but one typical approach is to set a custom user agent string:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    ua = (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/69.0.3497.100 Safari/537.36"
    )
    url = (
        "https://www.apple.com/br/shop/product/MV7N2BE/A/airpods-com-estojo-de-recarga"
    )
    page = browser.new_page(user_agent=ua)
    page.goto(url, wait_until="domcontentloaded")
    sel = "span.as-price-installments:last-child"
    text = (
        page.wait_for_selector(sel)
        .text_content()
        .replace("à vista (10% de desconto)", "")
        .strip()
    )
    print(text)  # => R$ 1.399,50
    browser.close()