How to use test_should_run_with_custom_categories_if_provided method in Playwright Python

Chrome can't launch itself with a User Profile folder, it must launch from a User Data folder. If you want a specific profile, you can set a profile as your default Chrome profile and then copy the new User Data folder elsewhere for your application (or use original User Data folder).

The misconception here seems to be that visible == clickable. That is not the case. In fact the docs explain what exact actionability checks are performed that determines whether the element is clickable

For example, for page.click(selector, **kwargs), Playwright will ensure that:

element is Attached to the DOM

element is Visible

element is Stable, as in not animating or completed animation

element Receives Events, as in not obscured by other elements

element is Enabled

Functionality to check whether all this is True for any element is currently not exposed in playwright (specifically, for checking whether the element is stable and reactive). The best you can do is change if locator.is_visible() to if locator.is_visible() and locator.is_enabled() to additionally check whether it is enabled as well (which would be done as part of the actionability checks later).

You can track the relevant issue here

In your special case, each page has their own link. It is your base link and the page number with an hyphen (-) in between.

You can see this behaviour when clicking on the second page, compare your base-link with the link you have now: https://forum.donanimhaber.com/turkcell-25-000-dk-12-000-sms-70-gb-internet-12-ay-320-tl-ana-konu--151777938-2

(notice the -2 at the end)

One way to do it, would be to change your url in a for-loop, iterating up to 24 and scrape all of those pages individually.

You can see from the documentation here that you can use pseudo-class of has-text

... the :has-text() pseudo-class

and maybe some some more advance matching selector with pseudo-class text-matches

the :text-matches() pseudo-class

but you need to remember that two ways is better if you combine the usage with some css selector.

Maybe you can add more details on where you want to search the string from is it literally in the entire document or not. (and I can update the answer with more detail code example according to your target)

The page you're navigating to shows this on a GET request:

HTTP ERROR 401 You must provide a http header 'JWT'

This means that this page requires a level of authorization to be accessed.

See JWTs.

"Authorization: This is the most common scenario for using JWT. Once the user is logged in, each subsequent request will include the JWT, allowing the user to access routes, services, and resources that are permitted with that token."

You can access the root page just fine, but once you navigate to more user specific pages or "routes", you will need to provide a JWT to access that page's content.

There is a way to get past this using scraping. You will have to log in to the site as a user using your scraper and collect the JWT which is created by the server and sent back to your client. Then use that JWT in your request's headers:

token = "randomjwtgibberishshahdahdahwdwa"
HEADERS = {
    "Authorization": "Bearer " + token
}

So with the help of Joran I was able to make this code work, tx!

I used his suggestion for using a future twice to get the data outside the main() function.

mainRespPromise = asyncio.Future()

async def main():
    async with async_playwright() as p:
        myRespPromise = asyncio.Future()
        async def handle_response(response): 
        # the endpoint we are insterested in 
            if ('eindpoint/name' in response.url): 
                json_data = await response.json()
                # "resolve the promise"
                myRespPromise.set_result(json_data)

        browser = await p.chromium.launch()
        page = await browser.new_page()
                
        # go to directly to searchpage
        await page.goto("website_url", wait_until='networkidle')
        
        page.on('response', handle_response)
        print("Made call, now await response...")

        await page.fill('input[id=zoeklocatie]', 'search_query')
        
        # Use two enters to first make button visible
        await page.keyboard.press("Enter")
        await page.keyboard.press("Enter")
        
        result_json = await myRespPromise
        print("GOT RESULT:",result_json)

        await page.wait_for_timeout(3000)
        
        await browser.close()
        mainRespPromise.set_result(result_json)

await main()

json_data = await mainRespPromise

I have come up with a solution in Node js puppeteer and I think it work the same way in python. As far as why the button isn't being pressed, I have no idea. But, what does work is entering an address and then clicking the first address suggestion.

Here is my code:

const puppeteer = require('puppeteer');
async function start(){
    const browser = await puppeteer.launch()
    const page = await browser.newPage()
    await page.goto("https://www.ubereats.com/ca")
    await page.type("#location-typeahead-home-input", "Address")
    await page.waitForSelector('#location-typeahead-home-item-0');
    await page.click("#location-typeahead-home-item-0")
    await page.waitForNavigation()

    console.log(page.url())
    await browser.close()
}

start()

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

.NET syntax - this worked for me

await page.Locator(":nth-match(:text('your text'), 2)").ClickAsync();