How to use _ensureIsolatedWorld method in Puppeteer

You should configure Network.setRequestInterception before Network.requestIntercepted. Here is working example:

const url = 'http://yahoo.com';
const puppeteer = require('puppeteer');

puppeteer.launch({ userDataDir: './data/' }).then(async browser => {
  const page = await browser.newPage();
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');

  // added configuration
  await client.send('Network.setRequestInterception', {
    patterns: [{ urlPattern: '*' }],
  });

  await client.on('Network.requestIntercepted', async e => {
    console.log('EVENT INFO: ');
    console.log(e.interceptionId);
    console.log(e.resourceType);
    console.log(e.isNavigationRequest);

    // pass all network requests (not part of a question)
    await client.send('Network.continueInterceptedRequest', {
      interceptionId: e.interceptionId,
    });
  });

  await page.goto(url);
  await browser.close();
});

I may be very late on this. Nevertheless for others, try:

const browser = await puppeteer.launch({defaultViewport: null});

Set the defaultViewport option to null as above to disable the 800x600 resolution. It takes the max resolution then.

I think using $eval might even be a little bit cleaner:

await page.$eval('input[name=email]', e => e.blur());

Try generating random useragent using this npm package. This usually solves the user agent-based protection.

In puppeteer pages can override browser user agent with page.setUserAgent

var userAgent = require('user-agents');
...
await page.setUserAgent(userAgent.random().toString())

Additionally, you can add these two extra plugins,

puppeteer-extra-plugin-recaptcha - Solves reCAPTCHAs automatically, using a single line of code: page.solveRecaptchas()

NOTE: puppeteer-extra-plugin-recaptcha uses a paid service 2captcha

puppeteer-extra-plugin-stealth - Applies various evasion techniques to make detection of headless puppeteer harder.

I had a similar issue and found a working answer here. Hopefully this works for you too. The interval was a little slow so I changed it from 100 to 30.

Just setting the viewports height and weight is not enough to fully simulate a mobile browser. Right now the page assumes that you just have a very small browser window.

The easiest way to simulate a mobile device is by using the the function page.emulate and the default DeviceDesriptors, which contain information about a large number of mobile devices.

Quote from the docs for page.emulate:

Emulates given device metrics and user agent. This method is a shortcut for calling two methods:

• page.setUserAgent(userAgent)
• page.setViewport(viewport)

To aid emulation, puppeteer provides a list of device descriptors which can be obtained via the require('puppeteer/DeviceDescriptors') command. [...]

Example

Here is an example on how to simulate an iPhone when visiting the page.

const puppeteer = require('puppeteer');
const devices = require('puppeteer/DeviceDescriptors');
const iPhone = devices['iPhone 6'];

const url = '...';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.emulate(iPhone);
  await page.goto(url);

  // Simlified page.evaluate
  let producData = await page.evaluate(() => ({
    name: document.querySelector('#a-page h1#title').innerText,
    price: document.querySelector('#a-page #newBuyBoxPrice').innerText
  }));

  console.dir(producData);
})();

I also simplified your page.evaluate a little, but you can of course also use your original code after the page.goto. This returned the name and the price of the product for me.

Kind of a late response, but might be helpful as a reference. You could debug your client script like that:

await page.evaluate(() => {
  debugger;
  const btn = document.querySelector(...);
  btn.click();
});

Now just launch puppeteer using:

puppeteer.launch({devtools: true})

Chromium will open and stop on your breakpoint.

I would suggest using a loop that you break out of when successful because then there is no build-up of resources in any way (like promises) and you can call your function an infinite number of times if required without stressing resource usage.

async function waitForThisToFinish() {
    while (true) {
        try {
            let val = await findItemOnPage()
            // use break or return here to break out of the loop
            return val;
        } catch (ex) {
            console.log("still waiting and trying again")
        }
    }
}

In addition, you also should make several additional changes:

1. Check the actual error to make sure it's the type of error that can be fixed by retry (such as a timeout). If it's a permanent error, you will retry forever.
2. Implement a delay before retrying to avoid causing avalanche failures on the server-side and to avoid server blocking from rate limiting.
3. Implement a progressive back-off in the delay.

Since you generally don't want to write code that hammers a server (by repeatedly making the same request over and over again in rapid succession) whenever there's an error because this can cause avalanche server failures where a small problem turns into a big problem very quickly, you should probably implement a delay before trying again that backs off into a longer delay the longer the issue persists.

function delay(t) {
    return new Promise(resolve => {
        setTimeout(resolve, t);
    });
}


async function waitForThisToFinish() {
    let waitTime = 100;
    while (true) {
        try {
            let val = await findItemOnPage()
            // use break or return here to break out of the loop
            return val;
        } catch (ex) {
            // Should also check if the actual error is one that is likely
            // temporary.  Otherwise, you may loop forever on a permanent error
            console.log("still waiting and trying again")
            // implement super simple backoff (there are much more elegant algorithms available)
            await delay(waitTime);
            waitTime += 300;
        }
    }
}

First of all, [name="link"] should be [name="arivalink"] to match your DOM. I assume that's a typo.

As another aside, I recommend using the Promise.all navigation pattern instead of waitForTimeout which can cause race conditions (although this doesn't appear to be related to the problem in this case).

As for the main issue, the external file is working just fine, so that's a red herring. You can prove that by calling page.evaluate(() => fileFunction()) right after navigating to sample.html.

The real problem is that when you navigate with window.location.replace('https://www.geeksforgeeks.org');, Chromium isn't pushing that action onto the history stack. It's replacing the current URL, so page.goBack() goes back to the original about:blank rather than sample.html as you expect. about:blank doesn't have fileFunction in it, so Puppeteer throws.

Now, when you click [name="link"] with Puppeteer, that does push the history stack, so goBack works just fine.

You can reproduce this behavior by loading sample.html in a browser and navigating it by hand without Puppeteer.

Long story short, if you're calling a function in browser context using evaluate that runs window.location.replace, you can't rely on page.goBack. You'll need to use page.goto to get back to sample.html.

There's an interesting nuance: if you use page.click to invoke JS that runs location.replace("..."), Puppeteer will push the history stack and page.goBack will behave as expected. If you invoke the same JS logic with page.evaluate(() => location.replace("..."));, Puppeteer won't push the current URL to the history stack and page.goBack won't work as you expect. The evaluate behavior better aligns with "manual" browsing (i.e. as a human with a mouse and keyboard on a GUI).

Here's code to demonstrate all of this. Everything goes in the same directory and node index.js runs Puppeteer (I used Puppeteer 9.0.0).

script.js

const replaceLocation = () => location.replace("https://www.example.com");
const setLocation = () => location = "https://www.example.com";

sample.html

<!DOCTYPE html>
<html lang="en">
<head>
  <title>sample</title>
</head>
<body>
  <div>
    <a href="https://www.example.com">normal link</a> | 
    <a href="#" onclick="replaceLocation()">location.replace()</a> | 
    <a href="#" onclick="setLocation()">location = ...</a>
  </div>
  <script src="script.js"></script>
</body>
</html>

index.js

const puppeteer = require("puppeteer");

const url = "file:///Users/sam.gajjar/SG/Projects/headless-chrome/sample.html";
const log = (() => {
  let logId = 0;
  return (...args) => console.log(logId++, ...args);
})();
let browser;

(async () => {
  browser = await puppeteer.launch({
    headless: false, 
    slowMo: 500,
  });
  const [page] = await browser.pages();
  await page.goto(url);

  // display the starting location
  log(page.url()); // 0 sample.html
  
  // click the normal link and pop the browser stack with goBack
  await Promise.all([
    page.waitForNavigation(),
    page.click("a:nth-child(1)"),
  ]);
  log(page.url()); // 1 example.com
  await page.goBack();
  log(page.url()); // 2 sample.html
  
  // fire location.replace with click
  await Promise.all([
    page.waitForNavigation(),
    page.click("a:nth-child(2)"), // pushes history (!)
  ]);
  log(page.url()); // 3 example.com
  await page.goBack();
  log(page.url()); // 4 sample.html

  // fire location.replace with evaluate
  await Promise.all([
    page.waitForNavigation(),
    page.evaluate(() => replaceLocation()), // doesn't push history
  ]);
  log(page.url()); // 5 example.com
  await page.goBack();
  log(page.url()); // 6 about:blank <--- here's your bug!
  
  await page.goto(url); // go to sample.html from about:blank <-- here's the fix
  log(page.url()); // 7 sample.html
  
  // use location = and see that goBack takes us to sample.html
  await Promise.all([
    page.waitForNavigation(),
    page.evaluate(() => setLocation()), // same behavior as page.click
  ]);
  log(page.url()); // 8 example.com
  await page.goBack();
  log(page.url()); // 9 sample.html
})()
  .catch(err => console.error(err))
  .finally(async () => await browser.close())
;