How to use test_locators_evaluate_handle_should_work method in Playwright Python

Step 1: make sure you're in Linux Container in Docker.

Step 2: Go to C:\Users\[User Name]\.docker\config.json Replace credsStore to credStore

Step 3: create Dockerfile, .dockerignore and docker-compose.yml file

Step 4: docker-compose format.

version: "3.9"
services:
frontend:
build: .
ports:
  - "8000:80"
backend:
image: "mcr.microsoft.com/mssql/server"
environment:
  SA_PASSWORD: "Docker123!"
  ACCEPT_EULA: "Y"
ports:
  - "1440:1433"

try to use sam ports as I mentioned here. User = sa, Database = master It will run perfectly fine. IA.

It looks like there is some kind of lazy load, at first, first 8 elements are loaded, if you scroll down, more content will load

The download.path() in playwright is just a random GUID (globally unique identifier). It's designed to validate the download works - not to keep the file.

Playwright is a testing tool and imagine running tests across every major browser on every code change - any downloads would quickly take up a lot of space and it would hack people off if you need to manually clear them out.

Good news is you are very close - If you want to keep the file you just need to give the file a name in the save_as.

instead of this:

   download.save_as(path)

use this:

   download.save_as(download.suggested_filename)

That saves the file in the same location as the script.

No, the webpage isn't loaded content dynamically by JavaScript rather it's entirely static HTML DOM

from bs4 import BeautifulSoup
import requests

page = requests.get('https://order.mandarake.co.jp/order/listPage/list?categoryCode=07&keyword=naruto&lang=en')
soup = BeautifulSoup(page.content,'lxml')

data = []
for e in soup.select('div.title'):

    d = {
        'title':e.a.get_text(strip=True),
        
        }
    
    data.append(d)

print(data)

Output:

[{'title': 'NARUTO THE ANIMATION CHRONICLE\u3000genga made for sale'}, {'title': 'Plex DPCF Haruno Sakura Reboru ring of the eyes'}, {'title': 'Naruto: Shippuden\u3000(replica)  ナルト'}, {'title': 'Naruto: Shippuden\u3000(replica)  ナルト'}, {'title': 'Naruto: Shippuden\u3000(replica)  NARUTO -ナルト-'}, {'title': 'Naruto: Shippuden ナルト\u3000(replica)'}, {'title': 'Naruto Shippuuden\u3000(replica) NARUTO -ナルト-'}, {'title': 'NARUTO -ナルト- 疾風伝\u3000(複製セル)'}, {'title': 'MegaHouse    ちみ メガ Petit Chara Land NARUTO SHIPPUDEN ナルト blast-of-wind intermediary   Even [swirl ナルト special is a volume on ばよ.  
  All 6 types set] inner bag not opened/box damaged'}, {'title': 'NARUTO -ナルト- 疾風伝\u3000(複製セル)'}, {'title': 'NARUTO -ナルト- 疾風伝\u3000(複製セル)'}, {'title': 'NARUTO -ナルト- 疾風伝'}, {'title': 'NARUTO -ナルト- 疾風伝'}, {'title': 'NARUTO -ナルト-'}]

After trying many different variants, I guessed a working syntax

handle.selectOption({"label": "Banana"})

PageCoroutine is deprecated/obsolute. Use playwright_page_methods instead.

Working code as an example:

import scrapy
from scrapy_playwright.page import PageMethod

class TestSpider(scrapy.Spider):
    name = "test"
    def start_requests(self):
        yield scrapy.Request(

            url="https://shoppable-campaign-demo.netlify.app/#/",
            callback=self.parse,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", '.card-body'),
                ],
            },
        )

    def parse(self, response):
        
        products = response.xpath('//*[@class="card-body"]')
        for product in products:
            yield {
            'title':product.xpath('.//*[@class="card-title"]/text()').get()
          
            }

Output:

{'title': 'Oxford Loafers'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Ankle-length Slack'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'White Baseball Cap'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Triangle Bikini Top'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Short Blazer'}
2022-11-05 20:40:40 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-05 20:40:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 235,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 39851,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 41.370211,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 11, 5, 14, 40, 40, 261151),
 'item_scraped_count': 5,

Original answer:

Using querySelector() / querySelectorAll with XPath (XML Path Language) lets you retrieve the elementHandle (respectively a collection of handles). Generally speaking, XPath can be used to navigate through elements and attributes in an XML document.

from playwright import sync_playwright

with sync_playwright() as p:
    for browser_type in [p.chromium, p.firefox, p.webkit]:
        browser = browser_type.launch(headless=False)
        page = browser.newPage()
        page.goto('https://duckduckgo.com/')
        element = page.querySelector('input[id=\"search_form_input_homepage\"]')
        
        parent = element.querySelector('xpath=..')
        grandparent = element.querySelector('xpath=../..')
        siblings = element.querySelectorAll('xpath=following-sibling::*')
        children = element.querySelectorAll('xpath=child::*')

        browser.close()

Update (2022-07-22):

It seems that browser.newPage() is deprecated, so in newer versions of playwright, the function is called browser.new_page() (note the different function name).

Optionally create a browser context first (and close it afterwards) and call new_page() on that context.

The way the children/parent/grandparent/siblings are accessed stays the same.

from playwright import sync_playwright

with sync_playwright() as p:
    for browser_type in [p.chromium, p.firefox, p.webkit]:
        browser = browser_type.launch(headless=False)
        context = browser.new_context()
        page = context.new_page()
        page.goto('https://duckduckgo.com/')
        element = page.querySelector('input[id=\"search_form_input_homepage\"]')
        
        parent = element.querySelector('xpath=..')
        grandparent = element.querySelector('xpath=../..')
        siblings = element.querySelectorAll('xpath=following-sibling::*')
        children = element.querySelectorAll('xpath=child::*')

        context.close()
        browser.close()

Heroku doesn't support webkit at the time, with other browsers, you can use this buildpack: https://github.com/mxschmitt/heroku-playwright-buildpack/

as mentioned in https://github.com/microsoft/playwright-python/issues/480

Jupyter notebook uses asyncio event loop, so you should use async api.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

# await page.screenshot(path="example.png")
# await browser.close()
# await playwright.stop()

If you use sync API, it will throw an Error like this:

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()

'''
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
'''