How to use _page_closed method in Playwright Python

Functions assigned to events (in any module) can't return value and you may need to use global variable for this. But lambda can't use global and can't assign value to variable so it needs normal function

url = ""

def on_response(response):
    global url  # inform functio to use external variable (instead of local variable) when you use `=` to assign value
    
    if ".m3u8" in response.url:
        url = response.url

page.on("response", on_response) 

From the docs, you should use a string as selector.

page.waitForSelector(selector[, options])

selector <string> A selector to to query for

page.click(selector[, options])

selector A selector to search for an element

selector = '//*[@id="sections-scroll-target"]/ul/ul/li[8]/a'
    
page.wait_for_selector(selector, timeout=10000)

page.hover(selector)

Looks like you want https://www.testdummysite.com/news/markets?mod=nav_left_section

If so this will work

//*[@href= "https://www.testdummysite.com/news/markets?mod=nav_left_section" ]

but these get a different link https://www.testdummysite.com/?mod=nav_left_section.

//*[@id="sections-scroll-target"]/ul/ul/li[8]/a

#sections-scroll-target > ul > ul > li:nth-child(8) > a

Because these URLs are similar, I wonder if you've posted the HTML accurately?

import requests
import threading

totalRequests = 0
numberOfThreads = 10
threads = [0] * numberOfThreads


def worker(thread):
    r = requests.get("url")
    threads[thread] = 0  # free thread


while totalRequests < 100:
    for thread in range(numberOfThreads):
        if threads[thread] == 0:
            threads[thread] = 1  # occupy thread
            t = threading.Thread(target=worker, args=(thread,))
            t.start()
            totalRequests += 1

To get the full HTML content of the page you can use page.content().

I was doing similar thing using playwright Python API. All you need to do is store your authentication as a json file. Then you have to use the file using your account. You need some knowledge about HTML and CSS. You can easily get the needed information using beautifulsoup module.

The best way to do it is using playwright docker image as base image, then you won't need to install its dependencies. Take a look to the documentation here: https://playwright.dev/docs/docker

You can access the href in an almost similar way to that of selenium.

element_handle = page.locator("//a")
link = element_handle.get_attribute('href')

So with the help of Joran I was able to make this code work, tx!

I used his suggestion for using a future twice to get the data outside the main() function.

mainRespPromise = asyncio.Future()

async def main():
    async with async_playwright() as p:
        myRespPromise = asyncio.Future()
        async def handle_response(response): 
        # the endpoint we are insterested in 
            if ('eindpoint/name' in response.url): 
                json_data = await response.json()
                # "resolve the promise"
                myRespPromise.set_result(json_data)

        browser = await p.chromium.launch()
        page = await browser.new_page()
                
        # go to directly to searchpage
        await page.goto("website_url", wait_until='networkidle')
        
        page.on('response', handle_response)
        print("Made call, now await response...")

        await page.fill('input[id=zoeklocatie]', 'search_query')
        
        # Use two enters to first make button visible
        await page.keyboard.press("Enter")
        await page.keyboard.press("Enter")
        
        result_json = await myRespPromise
        print("GOT RESULT:",result_json)

        await page.wait_for_timeout(3000)
        
        await browser.close()
        mainRespPromise.set_result(result_json)

await main()

json_data = await mainRespPromise

After trying many different variants, I guessed a working syntax

handle.selectOption({"label": "Banana"})