I have the same error if I run evaluated function in a browser. It seems the issue is that canseled events do not have sign-up links.

Actually it's not a popup but a popover. It will visible and accessible in the DOM after you actually click on the selector

await page.evaluate(() =>{
   document.querySelector('.nav-a.nav-a-2.a-popover-trigger.a-declarative').click();
});

To unsure you can work with the popover you need to wait to be load.

await page.waitForSelector(".a-popover.a-popover-modal.a-declarative");

After that you can click and type the text you want in the popover.

If you get Node.js and Puppeteer installed, you can use this simple script to get the HTML with JavaScript executed. Use it as:

node script.js url pagename

For test purposes, the default url is 'http://example.com/' and the default pagename is 'page-timestamp.html' in the current directory.

const fs = require('fs');
const puppeteer = require('puppeteer');

const url = process.argv[2] || 'http://example.com/';
const path = process.argv[3] || `page-${Date.now()}.html`;

(async function main() {
  const browser = await puppeteer.launch();
  const [page] = await browser.pages();

  await page.goto(url, { waitUntil: 'networkidle0' });
  fs.writeFileSync(path, await page.content());

  await browser.close();
})().catch(console.error);

Iterating puppeteer async methods in for loop vs. Array.map()/Array.forEach()

As all puppeteer methods are asynchronous it doesn't matter how we iterate over them. I've made a comparison and a rating of the most commonly recommended and used options.

For this purpose, I have created a React.Js example page with a lot of React buttons here (I just call it Lot Of React Buttons). Here (1) we are able set how many buttons to be rendered on the page; (2) we can activate the black buttons to turn green by clicking on them. I consider it an identical use case as the OP's, and it is also a general case of browser automation (we expect something to happen if we do something on the page). Let's say our use case is:

Scenario outline: click all the buttons with the same selector
  Given I have <no.> black buttons on the page
  When I click on all of them
  Then I should have <no.> green buttons on the page

There is a conservative and a rather extreme scenario. To click no. = 132 buttons is not a huge CPU task, no. = 1320 can take a bit of time.

---

I. Array.map

In general, if we only want to perform async methods like elementHandle.click in iteration, but we don't want to return a new array: it is a bad practice to use Array.map. Map method execution is going to finish before all the iteratees are executed completely because Array iteration methods execute the iteratees synchronously, but the puppeteer methods, the iteratees are: asynchronous.

Code example

const elHandleArray = await page.$$('button')

elHandleArray.map(async el => {
  await el.click()
})

await page.screenshot({ path: 'clicks_map.png' })
await browser.close()

Specialties

• returns another array
• parallel execution inside the .map method
• fast

132 buttons scenario result: ❌

Duration: 891 ms

By watching the browser in headful mode it looks like it works, but if we check when the page.screenshot happened: we can see the clicks were still in progress. It is due to the fact the Array.map cannot be awaited by default. It is only luck that the script had enough time to resolve all clicks on all elements until the browser was not closed.

1320 buttons scenario result: ❌

Duration: 6868 ms

If we increase the number of elements of the same selector we will run into the following error: UnhandledPromiseRejectionWarning: Error: Node is either not visible or not an HTMLElement, because we already reached await page.screenshot() and await browser.close(): the async clicks are still in progress while the browser is already closed.

---

II. Array.forEach

All the iteratees will be executed, but forEach is going to return before all of them finish execution, which is not the desirable behavior in many cases with async functions. In terms of puppeteer it is a very similar case to Array.map, except: for Array.forEach does not return a new array.

Code example

const elHandleArray = await page.$$('button')

elHandleArray.forEach(async el => {
  await el.click()
})

await page.screenshot({ path: 'clicks_foreach.png' })
await browser.close()

Specialties

• parallel execution inside the .forEach method
• fast

132 buttons scenario result: ❌

Duration: 1058 ms

By watching the browser in headful mode it looks like it works, but if we check when the page.screenshot happened: we can see the clicks were still in progress.

1320 buttons scenario result: ❌

Duration: 5111 ms

If we increase the number of elements with the same selector we will run into the following error: UnhandledPromiseRejectionWarning: Error: Node is either not visible or not an HTMLElement, because we already reached await page.screenshot() and await browser.close(): the async clicks are still in progress while the browser is already closed.

---

III. page.$$eval + forEach

The best performing solution is a slightly modified version of bside's answer. The page.$$eval (page.$$eval(selector, pageFunction[, ...args])) runs Array.from(document.querySelectorAll(selector)) within the page and passes it as the first argument to pageFunction. It functions as a wrapper over forEach hence it can be awaited perfectly.

Code example

await page.$$eval('button', elHandles => elHandles.forEach(el => el.click()))

await page.screenshot({ path: 'clicks_eval_foreach.png' })
await browser.close()

Specialties

• no side-effects of using async puppeteer method inside a .forEach method
• parallel execution inside the .forEach method
• extremely fast

132 buttons scenario result: ✅

Duration: 711 ms

By watching the browser in headful mode we see the effect is immediate, also the screenshot is taken only after every element has been clicked, every promise has been resolved.

1320 buttons scenario result: ✅

Duration: 3445 ms

Works just like in case of 132 buttons, extremely fast.

---

IV. for...of loop

The simplest option, not that fast and executed in sequence. The script won't go to page.screenshot until the loop is not finished.

Code example

const elHandleArray = await page.$$('button')

for (const el of elHandleArray) {
  await el.click()
}

await page.screenshot({ path: 'clicks_for_of.png' })
await browser.close()

Specialties

• async behavior works as expected by the first sight
• execution in sequence inside the loop
• slow

132 buttons scenario result: ✅

Duration: 2957 ms

By watching the browser in headful mode we can see the page clicks are happening in strict order, also the screenshot is taken only after every element has been clicked.

1320 buttons scenario result: ✅

Duration: 25 396 ms

Works just like in case of 132 buttons (but it takes more time).

---

Summary

• Avoid using Array.map if you only want to perform async events and you aren't using the returned array, use forEach or for-of instead. ❌
• Array.forEach is an option, but you need to wrap it so the next async method only starts after all promises are resolved inside the forEach. ❌
• Combine Array.forEach with $$eval for best performance if the order of async events doesn't matter inside the iteration. ✅
• Use a for/for...of loop if speed is not vital and if the order of the async events does matter inside the iteration. ✅

---

Sources / Recommended materials

• Sebastien Chopin: JavaScript: async/await with forEach() (codeburst.io)
• Antonio Val: Making array iteration easy when using async/await (Medium)
• Using async/await with a forEach loop (Stackoverflow)
• Await with array foreach containing async await (Stackoverflow)

I ran into a similar issue and found the workaround function on the Puppeteer Control networkidle wait time issue to suit my needs: https://github.com/GoogleChrome/puppeteer/issues/1353#issuecomment-356561654

Essentially, you can create a custom function, and call that prior to any additional steps:

function waitForNetworkIdle(page, timeout, maxInflightRequests = 0) {
  page.on('request', onRequestStarted);
  page.on('requestfinished', onRequestFinished);
  page.on('requestfailed', onRequestFinished);

  let inflight = 0;
  let fulfill;
  let promise = new Promise(x => fulfill = x);
  let timeoutId = setTimeout(onTimeoutDone, timeout);
  return promise;

  function onTimeoutDone() {
    page.removeListener('request', onRequestStarted);
    page.removeListener('requestfinished', onRequestFinished);
    page.removeListener('requestfailed', onRequestFinished);
    fulfill();
  }

  function onRequestStarted() {
    ++inflight;
    if (inflight > maxInflightRequests)
      clearTimeout(timeoutId);
  }

  function onRequestFinished() {
    if (inflight === 0)
      return;
    --inflight;
    if (inflight === maxInflightRequests)
      timeoutId = setTimeout(onTimeoutDone, timeout);
  }
}

// Example
await Promise.all([
  page.goto('https://google.com'),
  waitForNetworkIdle(page, 500, 0), // equivalent to 'networkidle0'
]);

The answer to this lies in using page.setRequestInterception(true); and monitoring subsequent requests, waiting for them to resvolve before moving on to the next task (thanks @Guarev for the point in the right direction).

This module (https://github.com/jtassin/pending-xhr-puppeteer) does exactly that, but for XHR requests. I modified it to look for 'image' and 'font' types.

Final code looks something like this:

// launch puppeteer
const browser = await puppeteer.launch({});

// load new page
const page = await browser.newPage();

// go to URL and wait for initial requests to resolve
await page.goto(pageUrl, {
  waitUntil: "networkidle0"
});

// enable this here because we don't want to watch the initial page asset requests (which page.goto above triggers) 
await page.setRequestInterception(true);

// custom version of pending-xhr-puppeteer module
let monitorRequests = new PuppeteerNetworkMonitor(page);

// START LOOP
for (let value of lotsOfValues) {

  // interact with select menu
  await page.select('select', value);

  // wait for network requests to complete (images, fonts)
  await monitorRequests.waitForAllRequests();

  // screenshot page with new content
  await pageElement.screenshot({
    type: "jpeg",
    quality: 100
  });

} // END LOOP

// close
await browser.close();

NPM Module

class PuppeteerNetworkMonitor {

    constructor(page) {
        this.promisees = [];
        this.page = page;
        this.resourceType = ['image'];
        this.pendingRequests = new Set();
        this.finishedRequestsWithSuccess = new Set();
        this.finishedRequestsWithErrors = new Set();
        page.on('request', (request) => {
            request.continue();
            if (this.resourceType.includes(request.resourceType())) {
                this.pendingRequests.add(request);
                this.promisees.push(
                    new Promise(resolve => {
                        request.resolver = resolve;
                    }),
                );
            }
        });
        page.on('requestfailed', (request) => {
            if (this.resourceType.includes(request.resourceType())) {
                this.pendingRequests.delete(request);
                this.finishedRequestsWithErrors.add(request);
                if (request.resolver) {
                    request.resolver();
                    delete request.resolver;
                }
            }
        });
        page.on('requestfinished', (request) => {
            if (this.resourceType.includes(request.resourceType())) {
                this.pendingRequests.delete(request);
                this.finishedRequestsWithSuccess.add(request);
                if (request.resolver) {
                    request.resolver();
                    delete request.resolver;
                }
            }
        });
    }

    async waitForAllRequests() {
        if (this.pendingRequestCount() === 0) {
            return;
        }
        await Promise.all(this.promisees);
    }

    pendingRequestCount() {
        return this.pendingRequests.size;
    }
}

module.exports = PuppeteerNetworkMonitor;

It is highly recommended to watch for errors when debugging headless scripts — making screenshots, waiting for a necessary element to appear and also catching js errors from the page context, like this:

page.on('pageerror', err => console.log(err));

With this callback in place it becomes clear why the resutls is undefined:

[Error: ReferenceError: bar is not defined

It seems that the bar variable is not defined on the target page, so assignment to testCase cannot be made, therefore it stays undefined.

Or if you're sure that bar is indeed defined, maybe it's created later in page's lifetime, after you're doing page.evaluate. In this case you'd want to use page.waitFor to wait for it:

await page.goto(url)
await page.waitFor(() => typeof(testCase) !== "undefined") // <== WAITING
console.log(await page.evaluate(() => testCase));

You can wait for navigation asynchronously to avoid getting null on redirection,

await Promise.all([
    page.click('button[type=submit]'),
    page.waitForNavigation({waitUntil: 'networkidle2'})
]);

This will help you if the page.click already triggers a navigation.

Your selector should be like const nodes = element.querySelectorAll('.post-item-info');. Then to access individual items in the returned collection, use a traditional for loop like

for(let i = 0; i < nodes.length; i++){
      const currentNode = nodes[i];
      // doStuffWith(currentNode);
    }