How to use expect_request_finished method in Playwright Python

You can use the select_options for selecting 30.

page.select_option('select#MainContent_ddlRecords', '30')

You can use the text selector and then click the checkboxes like this:

page.locator("text=Tabloul avocaţilor stagiari").click()

It looks like there is some kind of lazy load, at first, first 8 elements are loaded, if you scroll down, more content will load

as mentioned in https://github.com/microsoft/playwright-python/issues/480

Jupyter notebook uses asyncio event loop, so you should use async api.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

# await page.screenshot(path="example.png")
# await browser.close()
# await playwright.stop()

If you use sync API, it will throw an Error like this:

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()

'''
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
'''

The accepted answer does not work with the newer versions of Playwright. (Thanks @576i for pointing this out)

Here is the Python code that works with the newer versions (tested with version 1.5):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto('http://whatsmyuseragent.org/')
    ua = page.query_selector(".user-agent");
    print(ua.inner_html())
    browser.close()

To get only the text, use the inner_text() function.

print(ua.inner_text())

I solved my problem. In fact my docker container (frontend) is called "app" which is also domain name of fronend application. My application is running locally on http. Chromium and geko drivers force httpS connection for some domain names one of which is "app". So i have to change name for my docker container wich contains frontend application.

Referencing this:

1. pip install --upgrade pip
   

2. pip uninstall playwright
   

3. pip install playwright
   

4. Change to your script directory

5. playwright install
   

The issue had nothing to do with really any details in my original post. The issue was I was using Chromium, and it is a known bug that post_data does not contain file/blob data.

Quite a few questions there!

"Why does Selenium with Python have such a delay to issue commands"

I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.

This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?

Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.

Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.

"Why does Puppeteer seem to have a lower delay when interacting with the same browser?" Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.

However - how much faster is it really?

In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).

You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.

When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)

Some other options to think about if you really need speed:

• Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
• Try other browsers
• If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
• Another option that isn't selenium or puppeteer is javascript directly in the browser
• Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.

If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.

Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)

For the 1st form:

def get_playwright():
    with sync_playwright() as playwright:
        yield playwright

print(get_playwright())  # <generator object get_playwright at 0x108aac580>

That is expected because get_playwright is a generator, which returns a generator iterator, which you have to call next(...) on to get each yielded value from the iterator.

Consider a simpler, non-playwright example:

In [14]: def generate_nums():
    ...:     for num in range(10):
    ...:         yield num
    ...: 

In [15]: nums = generate_nums()

In [16]: nums
Out[16]: <generator object generate_nums at 0x11115e6d0>

In [17]: next(nums)
Out[17]: 0

In [18]: next(nums)
Out[18]: 1

In [19]: next(nums)
Out[19]: 2

For more examples, see Understanding generators in Python.

Since your get_playwright returns an iterator, you need to call next() once to get the actual object:

from playwright.sync_api import sync_playwright


def get_playwright():
    with sync_playwright() as playwright:
        yield playwright

playwright_generator = get_playwright()
print(playwright_generator)  # <generator object get_playwright at 0x104031580>

playwright = next(playwright_generator)
print(playwright)  # <playwright._impl._playwright.Playwright object at 0x1041aabb0>

For the 2nd form:

@pytest.fixture()
def get_playwright():
    with sync_playwright() as playwright:
        yield playwright

def test(get_playwright):
    print(get_playwright)

It should be the same case, but it's just that pytest automatically calls next() on the fixture value if it's a generator. I could not find documentation for this behavior from the pytest docs, but it was mentioned by one of the pytest author's/maintainer's in a different answer:

Here's roughly the execution here

• pytest notices your fixture is used for the test function
• pytest calls the fixture function
  • since it is a generator, it returns immediately without executing code
• pytest notices it is a generator, calls next(...) on it
  • this causes the code to execute until the yield and then "pausing". you can think of it kind of as a co-routine ...
• pytest then executes your test function

...which is probably why the value passed to your test function is already the next-ed value, the playwright object.