Mareks solution is fine for a few pages, but in case you want to crawl a large number of pages concurrently, I recommend looking at my library puppeteer-cluster.

It runs tasks in parallel (like Mareks solution), but also takes care of error handling, retrying and some other things. You can see a minimal example below. It's also possible to use the library in more complex settings.

const { Cluster } = require('puppeteer-cluster');

(async () => {
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT, // use one browser per worker
    maxConcurrency: 4, // cluster with four workers
  });

  // Define a task to be executed for your data
  await cluster.task(async ({ page, data: url }) => {
    await page.goto(url);
    const screen = await page.screenshot();
    // ...
  });

  // Queue URLs
  cluster.queue('http://www.google.com/');
  cluster.queue('http://www.wikipedia.org/');
  // ...

  // Wait for cluster to idle and close it
  await cluster.idle();
  await cluster.close();
})();

You can get the iframe using contentFrame as you are doing now, and then call $.

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

await page.goto('http://www.espn.com/login')

const elementHandle = await page.waitForSelector('div#disneyid-wrapper iframe');
const frame = await elementHandle.contentFrame();
await frame.waitForSelector('[ng-model="vm.username"]');
const username = await frame.$('[ng-model="vm.username"]');
await username.type('foo');
await browser.close()

These lines are inside your for loop:

                browser.close();
                return resolve(products);

So as part of the first iteration, you close the browser and return the function. You should move this out of your for loop and store products inside an array like this:

              const urls = /* ... */;
              const productsList = [];

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(/* ... */);
                productsList.push(products);
              }
              browser.close();
              return resolve(productsList); // resolve with an array containing the aggregated products

In case you are looking for a more elegant solution (for crawling the pages in parallel), you might want to have a look at the library puppeteer-cluster (disclaimer: I'm the author).

Debugging: - Puppeteer has various page events used for debugging purpose here. - Puppeteer recently added ASYNC stack trace so you can track errors more precisely.

Event emitting, You can use the default events module and exposeFunction to build your own events system.

Refer to the following code snippet that has all of the mentioned methods,

const EventEmitter = require("events");
const myEventTracker = new EventEmitter();
myEventTracker.on("some-event", function(...data) {
  // do something on event
  console.log(data);
});

// usage in puppeteer
const puppeteer = require("puppeteer");
puppeteer.launch({ headless: false }).then(async browser => {
  const page = await browser.newPage();

  // normal console events forwarded to node context
  page.on("console", msg => console.log(msg.text()));

  // we can use this to make our own reverse events
  myEventTracker.on("change-viewport", async function(data) {
    // do something on event
    await page.setViewport({ width: data.width, height: data.height });
  });

  // or we can expose the emit function
  await page.exposeFunction("emitter", (...data) =>
  myEventTracker.emit(...data)
  );

  // emit however we want
  await page.evaluate(async => {
    emitter("change-viewport", { width: 1200, height: 800 });
    emitter("some-event", "inside browser");
  });

  await page.goto("http://viewportsizes.com/mine/");
  // await browser.close();
});

It will become a blog post if we want to explain all, I will update my answer otherwise.

1. await can only be used in async functions.
2. The function in evaluate will be evaluated in the page context so either:

Use setTimeout():

page.evaluate(() => {
     //Code to execute before
     setTimeout(function() {
       //Code to execute after the 1000ms time delay
     }, 1000);    
 });

Or write your async delay function similar to page.waitFor in puppeteer.

function waitFor(delay) {
    return new Promise(resolve => setTimeout(resolve, delay));
}

Then you will have:

page.evaluate(async() => {
     //Define your waitFor function
     //Code to execute before
     await waitFor(1000);
     //Code to execute after the 1000ms time delay
 });

You can use mouse.move() for this, but it seems this would not fix the issue as the document body is not scrollable. You need to find the scrollable element to scroll just it:

'use strict';

const puppeteer = require('puppeteer');

(async function main() {
  try {
    const browser = await puppeteer.launch({ headless: false, defaultViewport: null });
    const [page] = await browser.pages();

    await page.goto('https://www.vudu.com/content/movies/movieslist');
    await page.waitForSelector('.nr-pt-40');

    const data = await page.evaluate(() => {
      const container = document.querySelector('.nr-pt-40');
      container.scrollBy(0, container.clientHeight);
    });

    // await browser.close();
  } catch (err) {
    console.error(err);
  }
})();

Another programmer suggested that maybe the page.evaluate context is the headless browser environment and not Node.js.

Right, this is one of the keys to Puppeteer, code within the evaluate callback is evaluated in the browser, not in your Node process (even though that's not at all obvious looking at the source of code using Puppeteer).

You can respond to the console event of the Page object, which is raised when any console method (log, error, etc.) is called in the client-side code. From that link:

page.on('console', msg => {
  for (let i = 0; i < msg.args().length; ++i)
    console.log(`${i}: ${msg.args()[i]}`);
});

The console event receives a ConsoleMessage object, which tells you what type of call it was (log, error, etc.), what the arguments were (args()), etc.

For me worked doing this

While this is annoying behaviour, there is a workaround: close all pages before closing the browser

let pages = await browser.pages(); await Promise.all(pages.map(page =>page.close())); await browser.close();

https://www.gitmemory.com/issue/puppeteer/puppeteer/6563/739149056

page.authentication

The correct format for proxy-server argument is,

--proxy-server=HOSTNAME:PORT

If it's HTTPS proxy, you can pass the username and password using page.authenticate before even doing a navigation,

page.authenticate({username:'user', password:'password'});

Complete code would look like this,

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
        headless:false,
        ignoreHTTPSErrors:true,
        args: ['--no-sandbox','--proxy-server=HOSTNAME:PORT']
  });
  const page = await browser.newPage();

  // Authenticate Here 
  await page.authenticate({username:user, password:password});
  await page.goto('https://www.example.com/');
})();

Proxy Chain

If somehow the authentication does not work using above method, you might want to handle the authentication somewhere else.

There are multiple packages to do that, one is proxy-chain, with this, you can take one proxy, and use it as new proxy server.

The proxyChain.anonymizeProxy(proxyUrl) will take one proxy with username and password, create one new proxy which you can use on your script.

const puppeteer = require('puppeteer');
const proxyChain = require('proxy-chain');

(async() => {
    const oldProxyUrl = 'http://username:password@hostname:8000';
    const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);

    // Prints something like "http://127.0.0.1:12345"
    console.log(newProxyUrl);

    const browser = await puppeteer.launch({
        args: [`--proxy-server=${newProxyUrl}`],
    });

    // Do your magic here...
    const page = await browser.newPage();
    await page.goto('https://www.example.com');
})();