How to use tap method in Playwright Python

After reading the entire doc :

page.evaluate("() => dataLayer")

Have you tried to download it as Binary package?

Maybe you can go over this libasound.so.2 package page and see if you can download it.

wget https://vault.centos.org/centos/8/AppStream/aarch64/os/Packages/alsa-lib-1.2.5-4.el8.aarch64.rpm

When I searched within apt, ubuntu package tool I don't see anything there

root@21be95fca6e6:/# apt search libasound.so.2
Sorting... Done
Full Text Search... Done

For instance, Vim package would appears like below

root@21be95fca6e6:/# apt search vim           
Sorting... Done
Full Text Search... Done
apvlv/jammy 0.4.0-2 arm64
  PDF viewer with Vim-like behaviour

In your special case, each page has their own link. It is your base link and the page number with an hyphen (-) in between.

You can see this behaviour when clicking on the second page, compare your base-link with the link you have now: https://forum.donanimhaber.com/turkcell-25-000-dk-12-000-sms-70-gb-internet-12-ay-320-tl-ana-konu--151777938-2

(notice the -2 at the end)

One way to do it, would be to change your url in a for-loop, iterating up to 24 and scrape all of those pages individually.

Just in case you had the same query or were struggling with dynamic web scraping I found a way to scrape the data without using selenium and bs4.

I used playwright which is far more straightforward and has a very much appreciated inner_html() function which reads straight into the dynamic flex HTML code. Here is the code for reference.

#part of the help to write the script I got from https://stackoverflow.com/questions/64303326/using-playwright-for-python-how-do-i-select-or-find-an-element

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(slow_mo=1000)

    page = browser.new_page()
    page.goto('https://vemcount.app/embed/widget/uOCRuLPangWo5fT')
    central = page.query_selector("p.w-full span");
    print({'central': central.inner_html()})
        
    browser.close()
 

If there is a better way I am more than happy to hear your suggestions.

Yours,

A Beginner in Python. :)

So how do I scrape a website which has dynamic content?

there are a few options:

1. Use Selenium, which allows you to simulate opening a browser, letting the page render, then pull the html source code
2. Sometimes you can look at the XHR and see if you can fetch the data directly (like from an API)
3. Sometimes the data is within the <script> tags of the html source. You could search through those and use json.loads() once you manipulate the text into a json format

what exactly is the difference between dynamic and static content?

Dynamic means the data is generated from a request after the initial page request. Static means all the data is there at the original call to the site

How do I extract other information like price and image from the website? and how to get particular classes for example like a price?

Refer to your first question

how would I know that data is dynamically created?

You'll know it's dynamically created if you see it in the dev tools page source, but not in the html page source you first request. You can also see if the data is generated by additional requests in the dev tool and looking at Network -> XHR

Lastly

Amazon does offer an API to access the data. Try looking into that as well

Use the .start() method:

>>> from playwright.sync_api import Playwright, sync_playwright, expect
>>> playwright = sync_playwright().start()
>>> browser = playwright.chromium.launch(headless=False)
>>> page = browser.new_page()

Alternatively, if you just want an interactive browser, and don't care about an interactive shell, you can also use the wait_for_timeout function instead (only applicable on Page objects) and set the timeout to a high value:

from playwright.sync_api import sync_playwright

with sync_playwright() as playwright:
    browser = playwright.chromium.launch(headless=False)
    page = browser.new_page()
    page.wait_for_timeout(10000)

Quite a few questions there!

"Why does Selenium with Python have such a delay to issue commands"

I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.

This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?

Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.

Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.

"Why does Puppeteer seem to have a lower delay when interacting with the same browser?" Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.

However - how much faster is it really?

In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).

You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.

When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)

Some other options to think about if you really need speed:

• Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
• Try other browsers
• If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
• Another option that isn't selenium or puppeteer is javascript directly in the browser
• Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.

If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.

Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)

You didn't set the cookie properly. You have

  # 1. Set the fields of the cookie.
  #    Give the cookie a value from the 'yourname' variable,
  #    a domain (localhost), and a max-age.
  c['value'] = yourname
  c['domain'] = 'localhost'
  c['max_age'] = 60

and the solution is

  c['yourname'] = yourname
  c['yourname']['domain'] = 'localhost'
  c['yourname']['max-age'] = 60

I would do this:

1. Register a listener with expose_function.
2. Use evaluate to register a listener to newPosts. A Listener on the browser side.
3. Inside that listener call the function you exposed on 1.