How to use setCacheEnabled method in Puppeteer

Your XPath expression is valid, but you are returning text() instead of the node itself. page.$x expects the XPath to return an element, therefore your code does not work. To return the node you need to query for the span element.

const element = await page.$x("//span[contains(translate(text(), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'two')]");

Please note, that text() only works for text-only nodes. If you have mixed content (containing elements and text), you should use the string value (. instead of text()):

const element = await page.$x("//span[contains(translate(., 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'two')]");

To compare the expressions I put them below each other:

//span//text()[contains(translate(., 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'two')]
//span[contains(translate(text(), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'two')
//span[contains(translate(., 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'two')]

The first one is the expression (given by you) for the text of the span node. The second one queries the node itself by using text(). The last one uses the string value to query the node.

You may find that Puppeteer does not behave as expected when controlling pages that incorporate audio and video. (For example, video playback/screenshots is likely to fail.) There are two reasons for this:

• Puppeteer is bundled with Chromium — not Chrome — and so by default, it inherits all of Chromium's media-related limitations. This means that Puppeteer does not support licensed formats such as AAC or H.264. (However, it is possible to force Puppeteer to use a separately-installed version Chrome instead of Chromium via the executablePath option to puppeteer.launch. You should only use this configuration if you need an official release of Chrome that supports these media formats.)
• Since Puppeteer (in all configurations) controls a desktop version of Chromium/Chrome, features that are only supported by the mobile version of Chrome are not supported. This means that Puppeteer does not support HTTP Live Streaming (HLS).

Source: FAQ: What features does puppeteer not support?

Late answer from someone looking for similar options due to the convenience of some browser SVG APIs:

My first recommendation, as someone who has written a fair amount of my own audio visualization software, is to use a graphics library and language that don't require a browser or GPU, like Gd or Anti-grain Geometry or Cairo with any server-side language. You might also check out Processing.org (which I haven't used), not sure if there's a headless version.

If that's not possible, I've found these so far but haven't tried them:

• https://github.com/tungs/timecut

• https://github.com/myplanet/headless-render

• https://wave.video/blog/how-we-render-animated-content-from-html5-canvas/

I encountered the same issue but solved it by replacing

{waitUntil: 'networkidle2'}

with:

{waitUntil: 'domcontentloaded'}

More information here:

[https://github.com/puppeteer/puppeteer/issues/2482][1]

Just to add more information, some sites have a lot of content and so the page might not have loaded within the default timeout of 30 seconds. So do away with this, you can totally remove timeout by doing so:

await page.setDefaultNavigationTimeout(0)

So this can look like this:

const puppeteer = require('puppeteer');

(async () => {

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.setDefaultNavigationTimeout(0);
await page.goto('https://www.example.com/', {
waitUntil: 'networkidle2'
});

There may be a library that wraps promises and awaits then more gracefully, but normally I mix and match Promises and async/await to make code more readable.
For instance, in your example, it would go somewhat like this:

let title = await page.$("#introduction")
  .then(intro => intro.$x(".."))
  .then(results => results[0])
  .then(first => first.$eval("span:first-child", el => el.innerText));

That way, await is only used to unbox the value once.

A simple solution is to use the fetch api to execute the GET request. This way you can read the response, pass it to your backend and save it to disk.

Use this sample code as reference:

import fs from 'fs';

async function downloadImage(page: any, url: string, fullpath: string) {
  const data = await page.evaluate(
    // tslint:disable-next-line no-shadowed-variable
    async ({ url }) => {
      function readAsBinaryStringAsync(blob) {
        return new Promise((resolve, reject) => {
          const fr = new FileReader();
          fr.readAsBinaryString(blob);
          fr.onload = () => {
            resolve(fr.result);
          };
        });
      }

      const r = await fetch(url, {
        credentials: 'include',
        headers: {
          accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp, */*;q=0.8',
          'cache-control': 'no-cache',
          pragma: 'no-cache',
          'sec-fetch-mode': 'navigate',
          'sec-fetch-site': 'same-site',
          'upgrade-insecure-requests': '1'
        },
        referrerPolicy: 'no-referrer-when-downgrade',
        body: null,
        method: 'GET',
        mode: 'cors'
      });

      return await readAsBinaryStringAsync(await r.blob());
    },
    { url }
  );

  fs.writeFileSync(fullpath, data, { encoding: 'binary' });
}

First of all, return the promise instead of assigning it to a variable.

const findInputFieldByName = (page, inputName) => {
  return new Promise(((resolve, reject) => { // <-- Return it

After that, you can throw error if need.

await findInputFieldByName(page,'identifier').catch(error => {
    throw ('The following error occurred: ' + error) // <-- pass the error to main program
});

As all the possible click solutions fails to click on the element, (even with page.waitForSelector): as an ultimate hack you can try to wait until the link becomes clickable by forcing some milliseconds of wait. It can be done with page.waitFor, which was deprecated with pptr 5.3.0, since that page.waitForTimeout can be used.

< Puppeteer 5.3.0

await page.waitFor(4000);
await page.$eval('#reboot > .block-row > .showForNodes', elem => elem.click());

Puppeteer 5.3.0+

await page.waitForTimeout(4000);
await page.$eval('#reboot > .block-row > .showForNodes', elem => elem.click());

The error

You need to mark the function you're passing inside as async, an async function is only lexically scoped to the function itself.

Any functions you call inside it (like .forEach or evaluate in this case) need to be marked as async themselves (assuming they support it):

//            Here
page.evaluate(async () => {

Fixing the larger issue

In your specific case, you should extract the loop outside of the evaluate to avoid timeouts:

while(await page.$$('.Ns6lhs9 _gfh3').length < counter) {
  await page.evaluate(() => window.scrollBy(0, window.innerHeight);
}

If your evaluate takes too long puppeteer will time out and it is generally preferable to not do expensive stuff inside evaluate.