How to use addScriptContent method in Puppeteer

I would recommend this rather simple approach:

async function retry(promiseFactory, retryCount) {
  try {
    return await promiseFactory();
  } catch (error) {
    if (retryCount <= 0) {
      throw error;
    }
    return await retry(promiseFactory, retryCount - 1);
  }
}

This function calls the promiseFactory, and waits for the returned Promise to finish. In case an error happens the process is (recursively) repeated until retryCount reaches 0.

Code Sample

You can use the function like this:

await retry(
  () => page.waitForXPath('//*[contains(@class, ".customer_name")]/ancestor::li'),
  5 // retry this 5 times
);

You can also pass any other function returning a Promise like Promise.all:

await retry(
  () => Promise.all([
    page.goto(url),
    page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
  ]),
  1 // retry only once
);

Don't combine await and catch

Another advice: You should not combine await with .then or .catch as this will result in unexpected problems. Either use await and surround your code with a try..catch block or use .then and .catch. Otherwise your code might be waiting for the results of a catch function to finish, etc.

Instead, you use try..catch like this:

try {
  // ...
} catch (error) {
  logMyErrors(error);
}

The problem is actually not with puppeteer, try running your code in a chrome repl (it will give the same result). Object.defineProperty doesn't accept a value as third argument, but a "descriptor". I am guessing you put the function right there in the third argument like

Object.defineProperty(HTMLCanvasElement.prototype, "toBlob", () => {
  console.log('toBlob!');
});

but it wants it like this

Object.defineProperty(HTMLCanvasElement.prototype, "toBlob", {
  value: () => {
    console.log('toBlob!');
  }
});

Example code that worked for me: https://gist.github.com/dralletje/37d0fc8a564ad377d60881d7c7429f6b

You can use page.waitForNavigation() to wait for the new page to load completely before generating a PDF:

await page.goto(fullUrl, {
  waitUntil: 'networkidle0',
});

await page.type('#username', 'scott');
await page.type('#password', 'tiger');

await page.click('#Login_Button');

await page.waitForNavigation({
  waitUntil: 'networkidle0',
});

await page.pdf({
  path: outputFileName,
  displayHeaderFooter: true,
  headerTemplate: '',
  footerTemplate: '',
  printBackground: true,
  format: 'A4',
});

If there is a certain element that is generated dynamically that you would like included in your PDF, consider using page.waitForSelector() to ensure that the content is visible:

await page.waitForSelector('#example', {
  visible: true,
});

This script should work

'use strict';

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless : false});
  const page = await browser.newPage();

  console.log("Opening page");
  await page.goto('https://www.google.com/recaptcha/api2/demo');
  console.log("Opened page");

  const frame = await page.frames().find(f => f.name().startsWith("a-"));
  await frame.waitForSelector('div.recaptcha-checkbox-border');
  console.log("Captcha exists!");

  await browser.close();
})();

It appears that the captcha is inside an iframe that always starts with name=a- (however I can't confirm that with my limited testing)

You first need to get the iframe then await for the selector after the iframe has loaded. This is the output, try changing the iframe name or the selector name to see it fail.

It works fine with a user agent string:

const puppeteer = require("puppeteer"); // ^19.0.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: true});
  const [page] = await browser.pages();
  const ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36";
  await page.setExtraHTTPHeaders({"Accept-Language": "en-US,en;q=0.9"});
  await page.setUserAgent(ua);
  const url = "https://etherscan.io/tx/0xb06c7d09611cb234bfcd8ccf5bcd7f54c062bee9ca5d262cc5d8f3c4c923bd32";
  await page.goto(url);
  const btn = await page.waitForSelector("#txfeebutton");
  console.log(await btn.evaluate(el => el.textContent.trim())); // => ($0.56)
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

One debugging strategy for this trying the same script with headless: false and seeing if that works, then checking page.content() when running headlessly. You can see Cloudflare is detecting your scraper and presenting a captcha.

Related:

• Puppeteer can't find elements when Headless TRUE
• Why does headless need to be false for Puppeteer to work?

As an aside, configureBrowser leaks a reference to the browser object, so you'll never be able to call browser.close() and gracefully terminate the process. I recommend the above boilerplate and avoiding writing premature abstractions.

Great question, had the same issue myself.????

The main problem is videos URLs don't have to end with a specific suffix and indeed they don't on many websites (YouTube for example).

But there this function:

request.resourceType()

Contains the request's resource type as it was perceived by the rendering engine. ResourceType will be one of the following: document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other.

A bit of Googling and found this MDN:

media

Resources loaded by a <video> or <audio> element.

And as I don't mind on blocking audio requests as well (thanks God, they are not in use in normal web pages...)

Eventually this is what I ended up doing:

page.on('request', request => {
  const url = request.url().toLowerCase()
  const resourceType = request.resourceType()
  
  if (resourceType == 'media' ||
    url.endsWith('.mp4') ||
    url.endsWith('.avi') ||
    url.endsWith('.flv') ||
    url.endsWith('.mov') ||
    url.endsWith('.wmv')) {
    
    console.log(`ABORTING: video`)
    request.abort();
  }
  else
    request.continue();
})

I use "waitForSelector" method and after that try to get the text

await page.waitForSelector('your selector')
let element = await page.$('your selector')
let value = await page.evaluate(el => el.textContent, element)

You can select any element by property by using:

'[property="value"]'

in your case this should do the trick:

'[placeholder="Email"]'

Your main issue is probably that you're only searching for lastImage before all the clicking has been done, which means it doesn't exist yet. A fix would be to move that line to within the while loop, but in general the code you've given all looks a bit verbose. I'd probably use some kind of recursive function instead, similar to the below:

function updateContent() {
  setTimeout(function() {
      var button = document.getElementById('button');
      button.innerHTML = parseInt(button.innerHTML) + 1
      if (button.innerHTML === '5') {
          button.className = "unclickable"
          button.disabled = true
      } else {
          button.click()
      }
  }, 500)
}

<button id="button" class="clickable" onclick="updateContent()">1</button>