How to use test_request_fulfill_should_work_with_request_interception method in Playwright Python

You can define your own function like this one:

def my_own_wait_for_selector(page, selector, time_out):
    try:
        page.wait_for_selector(selector, timeout=time_out)
        return True
    except:
        return False

That function will wait for a selector and if given some time (miliseconds) the element is not there it will return False instead of trowing an error.

Then in your code you could try something like:

while not my_own_wait_for_selector(page, "my_button_selector", 5000):
    page.reload()
page.locator("my_button_selector").click()

As you can see we are checking if our button is present, and we will wait only 5 seconds, if after 5 seconds the button is not there, we will reload, and again and again until our button is there, once our button is there code will continue and then we will click the button.

Be careful, because if your button is never there, your script will reload and your script will never finish, so you should add some other condition to your while

I see what the problem here is. It does have to do with pyautogui and playwright not functioning together correctly. I tried searching around to find how to actually get the current mouse position with playwright, but I did not find anything.

So, here is my conclusion:

How did I figure out the problem, and what actually is the problem?

The problem here is that pyautogui actually controls the computer's mouse in this case. It seems like playwright only controls that of the browser. If you add a time.sleep(10) (In order to actually see if the cursor has changed position and to have time to move the cursor at a different position) at the end of the function like so:

def test_simple_move():
    mouse_start_position = pyautogui.position()
    print(mouse_start_position)
    
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=False, slow_mo=10)
        page = browser.new_page()
        page.goto(r"http://www.uitestingplayground.com/")
        page.mouse.move(100,400)
        mouse_final_position = pyautogui.position()
        print(mouse_final_position)

        time.sleep(10)

You will then see that the actual cursor does not move. It seems like what playwright is doing here is just moving a "virtual" cursor in its browser. In fact, what I did to prove this was:

1. Launch the code
2. While the delay of 10 seconds is going, move the cursor to another point on the screen.
3. Wait for the code to finish.
4. You will now see how the points are different. Here are my points

Quite a few questions there!

"Why does Selenium with Python have such a delay to issue commands"

I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.

This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?

Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.

Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.

"Why does Puppeteer seem to have a lower delay when interacting with the same browser?" Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.

However - how much faster is it really?

In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).

You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.

When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)

Some other options to think about if you really need speed:

• Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
• Try other browsers
• If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
• Another option that isn't selenium or puppeteer is javascript directly in the browser
• Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.

If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.

Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)

Reading your question I understand that your problem is not the download itself, your problem is the way of clicking the link that you need.

From that moment, you have to options:

1 - You know the date (exact text):

date = "2022-09-26"
with page.expect_download() as download_info:
    page.locator(f"//a[text()='{date}_TEST.zip']").click()
    download = download_info.value
    page.wait_for_url("http://192.xx")
    download_filepath = os.path.join(download_path, suggested_filename)
    download.save_as(download_filepath)
    context.close()
    browser.close()

As you can see you can use f"..{}.." for your locator string in order to use some vars for the locator, basically a locator is a string.

Of course you should know the date before, in my example is hardcoded.

2 - You don't know the date (Contains text):

with page.expect_download() as download_info:
    page.locator(f"//a[contains(text(),'_TEST.zip')]").click()
    download = download_info.value
    page.wait_for_url("http://192.xx")
    download_filepath = os.path.join(download_path, suggested_filename)
    download.save_as(download_filepath)
    context.close()
    browser.close()

And being honest, I think the line page.wait_for_url("http://192.xx") is not needed. Playwright will wait for your download, actually you don't need to wait for the url, I think.

For my examples I used selector with Xpath.

You can do:

hrefs_of_page = page.eval_on_selector_all("a[href^='/wiki/']", "elements => elements.map(element => element.href)")

which should work for your use-case. This will lookup for all the link tags which have a href attribute which starts with /wiki. Then on the browser side JavaScript gets evaluated which maps from an array of elements to the href attribute so a string array gets returned on the Python side.

Just use some condition instead of print method, for example you could check if response contains some key in its json:

def run(playwright):
    chromium = playwright.chromium
    browser = chromium.launch()
    page = browser.new_page()
    # Subscribe to "request" and "response" events.
    page.on("request", lambda request: print(">>", request.method, request.url))
    page.on("response", lambda response: print("<<", response.status, response.url))
    page.goto("https://example.com")
    browser.close()

For Example:

page.on("response", lambda response: response if key in response.body())

There should be waitForResponse for python too, and you could use that.

this worked for me in java:

Page page = browser.newPage();
Map<String, String> headers = new HashMap<>();
headers.put("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) `enter code here`AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
page.setExtraHTTPHeaders(headers );

First of all, Playwright already has a full suite of selectors that work on the live page, so to eliminate a dependency, speed up your scrape, use less code and avoid weird errors when the static HTML snapshot gets out of sync with the live page, I suggest skipping BS.

On to the main problem, you've done good by printing the HTML to see what sort of response you're dealing with. The 404 page indicates you've been detected as a bot when running headlessly, but this can often manifest as a captcha, Cloudflare browser check page, or other "are you a robot?" notice.

As with everything in scraping, there's no one-size-fits-all solution, but one typical approach is to set a custom user agent string:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    ua = (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/69.0.3497.100 Safari/537.36"
    )
    url = (
        "https://www.apple.com/br/shop/product/MV7N2BE/A/airpods-com-estojo-de-recarga"
    )
    page = browser.new_page(user_agent=ua)
    page.goto(url, wait_until="domcontentloaded")
    sel = "span.as-price-installments:last-child"
    text = (
        page.wait_for_selector(sel)
        .text_content()
        .replace("à vista (10% de desconto)", "")
        .strip()
    )
    print(text)  # => R$ 1.399,50
    browser.close()

The best way to do it is using playwright docker image as base image, then you won't need to install its dependencies. Take a look to the documentation here: https://playwright.dev/docs/docker