How to use test_check_the_box_outside_label method in Playwright Python

Install these first

playwright

scrapy-playwright

Scrapy

And then run in the terminal

playwright install (This will install the driver required for playwright)

At the end of each iteration of "y" you issue a mouse up event. However, you dont issue a mouse move event to the start of the new line before putting the mouse down again.

Try the below code:

    for y in range(100, 1000, 100):
        # Generate mouse points
        points = []
        wm(start_point, y, x, y, M_0=15, D_0=12, move_mouse=lambda x, y: points.append([x, y]))

        page.mouse.up()
        # Move mouse to start of new line
        page.mouse.move(points[0][0], points[0][1])
        # Mouse back down
        page.mouse.down()
        #Iterate the remaining points
        for point in points[1:]:
            page.mouse.move(point[0], point[1])
        page.mouse.up()

Heroku doesn't support webkit at the time, with other browsers, you can use this buildpack: https://github.com/mxschmitt/heroku-playwright-buildpack/

as mentioned in https://github.com/microsoft/playwright-python/issues/480

Jupyter notebook uses asyncio event loop, so you should use async api.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

# await page.screenshot(path="example.png")
# await browser.close()
# await playwright.stop()

If you use sync API, it will throw an Error like this:

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()

'''
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
'''

The only way I found is wrap the goto inside a try-except block

Content of main.py

from playwright.sync_api import Playwright, sync_playwright, expect

with sync_playwright() as p:
    #We define the browser, the context and the page
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = browser.new_page()
    with page.expect_download() as download_info:
        #If we navigate without try, it will throw an exception and it will stop our script, so, we wrap it inside a try except block
        try:
            page.goto("https://manueltgomes.com/wp-content/uploads/2021/09/ParseHTMLLInks_1_0_0_1.zip")
        except:
            pass
    download = download_info.value
    download.save_as(f"./{download.suggested_filename}")
    context.close()
    browser.close()

Then just run python3 main.py

The file will be saved in the same path of the main.py file

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

Unfortunately, I do not think this is possible. For web scraping, you might use requests or urllib to download the page then use an HTML parser, such as BeautifulSoup4 or lxml, to scrape the webpage. While browsers may have a URL for the new tab (Chrome, for example, has chrome://newtab), requests and urllib aren't able to establish connections to the URLs.

For requests:

>>> import requests
>>> res = requests.get("chrome://newtab")
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    res = requests.get("chrome://newtab")
  ...
requests.exceptions.InvalidSchema: No connection adapters were found for 'chrome://newtab'

For urllib:

>>> from urllib.request import urlopen
>>> res = urlopen("chrome://newtab")
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    res = urlopen("chrome://newtab")
  File "C:\Users\JLpyt\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  ...
urllib.error.URLError: <urlopen error unknown url type: chrome>

With browser automation packages (selenium, playwright, pyppeteer), you are able to access the new tab page, by sending the Ctrl+T/Cmd+T virtual keystrokes. From there, you would be able to pass the page content to an HTML parser and scrape the page. While this is possible for 'classic' browsers (Chrome, Firefox, Internet Explorer, etc.), I don't know of any package for Brave Browser.

Perhaps there is something I am missing, but, as far as I know, you cannot scrape the new tab page of Brave Browser.

I had a similar problem: I had PM2 run by an Apache Airflow Celery worker running a playwright task, hourly. The job hung.

I suspect it is the environment variables littered by Byobu and PM2. What worked for me was wrapping the Python script in a bash script and sanitizing the environment variable like the following:

#!/bin/bash
[ "$HOME" != "" ] && exec -c $0
# Your python script should come below:
/home/ubuntu/usr/venv-3.10-airflow/bin/airflow celery worker

The first line clears the env vars (as suggested in Sanitize environment with command or bash script?). Now you can run this script with pm2 start command. My worker ran fine for about 24 hours as of now. I hope this works for you too.

in the docker, you don't need to go inside python. Just run from bash:

$ npx playwright open google.com

For more options, check out the doc: https://playwright.dev/docs/cli/