How to use text_content method in Playwright Python

Closing actual popups

I think browser.once("targetcreated", ...) as described in this issue https://stackoverflow.com/a/57246544/2202112 could be something that you are looking for.

It allows you to setup a callback on newly created targets.

For reference, see puppeteer docs on "targetcreated"

Closing modals

1. Find the selector for the button that dismisses the modal, for example .button-close or .button-dismiss.
2. Use the page.click(selector, [options]) method to click the button before continuing further scraping

So I found a working solution.

What I did was to combine Javascript with python Playwright code.

I start the setInterval with a timer of 200ms to scroll down on the page with page.evaluate() and then I follow it up with a python loop that checks every second whether the total height of the page (scroll included) has changed. If it changes it continues to scroll and if it hasn't changed than the scroll is over.
This is what it looks like:

page.evaluate(
    """
    var intervalID = setInterval(function () {
        var scrollingElement = (document.scrollingElement || document.body);
        scrollingElement.scrollTop = scrollingElement.scrollHeight;
    }, 200);

    """
)
prev_height = None
while True:
    curr_height = page.evaluate('(window.innerHeight + window.scrollY)')
    if not prev_height:
        prev_height = curr_height
        time.sleep(1)
    elif prev_height == curr_height:
        page.evaluate('clearInterval(intervalID)')
        break
    else:
        prev_height = curr_height
        time.sleep(1)

EDIT

See the below answer using the new mouse.wheel(x, y) feature for an up to date way to scroll using playwright. Combine my answer with his to lessen the need to use JS.

Looks like soup is unable to find anything and its returning None and Nonetype has no text property

The error looks here for soup in clothes: as you have overridden the soup parser

PS:

You can use get_text() to extract the text

price = soup.find('span', {'class' : 'price'}).get_text()

Sometimes you may have to use contents depending on the return type

price = soup.find('span', {'class' : 'price'}).contents[0]

The accepted answer does not work with the newer versions of Playwright. (Thanks @576i for pointing this out)

Here is the Python code that works with the newer versions (tested with version 1.5):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto('http://whatsmyuseragent.org/')
    ua = page.query_selector(".user-agent");
    print(ua.inner_html())
    browser.close()

To get only the text, use the inner_text() function.

print(ua.inner_text())

I had a similar problem: I had PM2 run by an Apache Airflow Celery worker running a playwright task, hourly. The job hung.

I suspect it is the environment variables littered by Byobu and PM2. What worked for me was wrapping the Python script in a bash script and sanitizing the environment variable like the following:

#!/bin/bash
[ "$HOME" != "" ] && exec -c $0
# Your python script should come below:
/home/ubuntu/usr/venv-3.10-airflow/bin/airflow celery worker

The first line clears the env vars (as suggested in Sanitize environment with command or bash script?). Now you can run this script with pm2 start command. My worker ran fine for about 24 hours as of now. I hope this works for you too.

If you want a Playwright image with Python instead of the Playwright image with Node you have currently set,

do

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal

instead. You won't then need to install python3-pip (or playwright!) at all.

Additionally, you'll probably want to explicitly spell out python3 instead of python:

FROM mcr.microsoft.com/playwright/python:v1.23.0-focal
RUN pip3 install TikTokApi
RUN playwright install
COPY . .

You can use below (Reference).

await page.locator('td > small:below(:text("{pan_ids[number]}")').text_content()

Or, You can also use the Adjacent sibling combinator

await page.locator(':text("{pan_ids[number]}") + td > small').text_content()

You didn't set the cookie properly. You have

  # 1. Set the fields of the cookie.
  #    Give the cookie a value from the 'yourname' variable,
  #    a domain (localhost), and a max-age.
  c['value'] = yourname
  c['domain'] = 'localhost'
  c['max_age'] = 60

and the solution is

  c['yourname'] = yourname
  c['yourname']['domain'] = 'localhost'
  c['yourname']['max-age'] = 60

You can use the Page.innerText(".score") method to accomplish that, see here for an example:

// @ts-check
const playwright = require("playwright");

(async () => {
  const browser = await playwright.chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.setContent(`
    <span _ngcontent-fnp-c148="" class="score"> &nbsp;<!--bindings={
"ng-reflect-ng-if": "false"
}--> 96%, </span>
`);
  const content = await page.innerText(".score")
  console.log(content)
  await page.screenshot({ path: `example.png` });
  await browser.close();
})();

Or interactively here: https://try.playwright.tech/?s=bl1fed4

For Python usage see here: https://microsoft.github.io/playwright-python/sync_api.html#playwright.sync_api.Page.innerText