How to use timeoutOrTerminationPromise method in Puppeteer

Puppeteer uses Chrome (or FireFox) browser under the hood, so in case:

• This is not the first visit (cache filled)
• Resources has proper cache headers and not expired (cache-control, etc.)
• You didn't disable cache manually using

await page.setCacheEnabled(false);
await pageSession.send('Network.setCacheDisabled', { cacheDisabled: true });

Resources will be already cached and you don't need to do anything manually.

However, if you want to do testing on cached page, you will need to warm it up simply pre visiting it before tests, like in the example:

async function warmingBrowser(url: URL, pageInstance: Page) {
    await pageInstance.goto(url.href, { waitUntil: 'networkidle0' });
    await pageInstance.close();
}

The code is taken from the perfrunner

In case you want to make it work completely offline - Puppeteer will not help with that, you need to implement your own caching strategy using the ServiceWorker.

But there are some pitfalls on this step (exactly with caching and invalidating the cache) so be aware.

You could wait for some reaction to your click (some page change, or HTTP request made), for example:

• wait for certain element state change https://playwright.dev/docs/api/class-locator#locator-wait-for
• wait for a new element to appear on the page (like a dialog open) https://playwright.dev/docs/api/class-page#page-wait-for-selector
• wait until a request fired https://playwright.dev/docs/api/class-page#page-wait-for-request
• wait until a response received https://playwright.dev/docs/api/class-page#page-wait-for-response
• or wait for an event https://playwright.dev/docs/api/class-page#page-wait-for-event

Maybe the element's not loaded yet. You could wait for that xpath.

const linkHandlers = await page.waitForXPath("//div[contains(text(), 'My saved leads')]");

waitForXPath will not only wait but return that element.

The error caused by the click and hover methods called on an ElementHandle can be mitigated by ensuring the element is "in view" separately from the method (despite the docs stating that the method "scrolls it into view if needed", seems like sometimes the element does not become visible).

Assuming the element is indeed attached to DOM - you can check that via isConnected property, try using something like this (the sample is in TypeScript) - scrollIntoView should make sure the element is centered on:

const safeHover = async <E extends ElementHandle<Element>>(
  elem: E
) => {

  await elem.evaluate((e) =>
    e.scrollIntoView({ block: "center", inline: "center" })
  );

  await elem.click();
};

Note that the sample uses elementHandle's click method, not Page's. Since you call waitForSelector before clicking, use the returned elementHandle:

//...
const pres = await ig.waitForSelector('[role=presentation]'); 
pres && await safeHover(pres);
//...

Also, these Q&As may prove useful:

1. Puppeteer in NodeJS reports 'Error: Node is either not visible or not an HTMLElement'
2. Puppeteer throws error with "Node not visible..."

I think the easiest way is to use puppeteer $eval. https://pptr.dev/#?product=Puppeteer&version=v5.2.1&show=api-pageevalselector-pagefunction-args-1

Select element by id and just ask for it's value will give you the selected value in the dropdown.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({headless: false})
  const page = await browser.newPage();
  await page.goto('https://afrekenen.bol.com/nl/winkelwagentje/direct-toevoegen?id=1001024332369558:2');

  const option = await page.$eval("#tst_quantity_dropdown", node => 
     node.value);
  console.log('Selected option is:', option)
})();

Ouputs: Selected option is: 2

$x() is not a standard JavaScript method to select element by XPath. $x() it's only a helper in chrome devtools. They claim this in the documentation:

Note: This API is only available from within the console itself. You cannot access the Command Line API from scripts on the page.

And page.evaluate() is treated here as a "scripts on the page".

You have two options:

1. Use document.evaluate

Here is a example of selecting element (featured article) inside page.evaluate():

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://en.wikipedia.org', { waitUntil: 'networkidle2' });

    const text = await page.evaluate(() => {
        // $x() is not a JS standard -
        // this is only sugar syntax in chrome devtools
        // use document.evaluate()
        const featureArticle = document
            .evaluate(
                '//*[@id="mp-tfa"]',
                document,
                null,
                XPathResult.FIRST_ORDERED_NODE_TYPE,
                null
            )
            .singleNodeValue;

        return featureArticle.textContent;
    });

    console.log(text);
    await browser.close();
})();

2. Select element by Puppeteer page.$x() and pass it to page.evaluate()

This example achieves the same results as in the 1. example:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://en.wikipedia.org', { waitUntil: 'networkidle2' });

    // await page.$x() returns array of ElementHandle
    // we are only interested in the first element
    const featureArticle = (await page.$x('//*[@id="mp-tfa"]'))[0];
    // the same as:
    // const featureArticle = await page.$('#mp-tfa');

    const text = await page.evaluate(el => {
        // do what you want with featureArticle in page.evaluate
        return el.textContent;
    }, featureArticle);

    console.log(text);
    await browser.close();
})();

Here is a related question how to inject $x() helper function to your scripts.

If you are attempting to fill out and submit a login form, you can use the following:

await page.goto('https://www.example.com/login');

await page.type('#username', 'username');
await page.type('#password', 'password');

await page.click('#submit');

await page.waitForNavigation();

console.log('New Page URL:', page.url());

In my current project I am using the following library from Google, puppeteer. I personally found it to be very easy to use, and it even provides access to the dev protocol that Google Chrome has.

I've done some reading around and found a lot about headless browsers but not sure if that's the right path to go down as they seem to rely on other applications like chromium and they're focused on testing sites.

Yes, they are often used for testing, to assure that the correct things are rendered on screen etc. However, in many scenarios, like yours, the use of a headless browser to interact with a website is totally legit in a non-testing scenario.

It happens because Chrome specifically blocks local file access this way for security reasons.

Anyway, you don't need to open the version page to get a user agent. Because there is a method for it:

await browser.userAgent(); 

It returns: <Promise<string>> Promise which resolves to the browser's original user agent.