How to use downloadBrowser method in Puppeteer

Actually it's not a popup but a popover. It will visible and accessible in the DOM after you actually click on the selector

await page.evaluate(() =>{
   document.querySelector('.nav-a.nav-a-2.a-popover-trigger.a-declarative').click();
});

To unsure you can work with the popover you need to wait to be load.

await page.waitForSelector(".a-popover.a-popover-modal.a-declarative");

After that you can click and type the text you want in the popover.

There are two scenarios in you case First,If you are using version less than v0.13.0 you won't be able to do it Second if you are using node version 6 you have to build node components manually

cd node_modules/puppeteer
npm run build

You can use the clip option of elementHandle.screenshot() in conjunction with elementHandle.boundingBox() to set the width and height of an element screenshot.

The example below will take a screenshot of an element, and clip the element if it exceeds the current viewport:

await page.setViewport({
  width: 800,
  height: 1000,
});

const example = await page.$('#example');
const bounding_box = await example.boundingBox();

await example.screenshot({
  path: 'example.png',
  clip: {
    x: bounding_box.x,
    y: bounding_box.y,
    width: Math.min(bounding_box.width, page.viewport().width),
    height: Math.min(bounding_box.height, page.viewport().height),
  },
});

You need parens () around your braces {}:

This is valid:

let func = (div, index) => ({
        title: "abc",
        category: "abcd",
        price: 23
         })

This is invalid:

let func = (div, index) => {
        title: "abc",
        category: "abcd",
        price: 23
         }

The reason the parens are required is because otherwise, the curly braces would be interpreted as a code block instead of an object literal.

The page is using frames. You are only seeing the main content of the page (without the content of the frames). To also get the content of the frame, you need to first find the frame (e.g. via page.$) and then get its frame handle via elementHandle.contentFrame. You can then call frame.content() to get the content of the frame.

Simple Example

const frameElementHandle = await page.$('#selector iframe');
const frame = await frameElementHandle.contentFrame();
const frameContent = await frame.content();

Depending on the structure of the page, you need to do this for multiple frames to get all contents or you even need to do it for a frame inside the frame (what seems to be the case for the given page).

Example to read all frame contents

Below is an example that recursively read the contents of all frames on the page.

const contents = [];
async function extractFrameContents(pageOrFrame) {
  const frames = await pageOrFrame.$$('iframe');
  for (let frameElement of frames) {
    const frame = await frameElement.contentFrame();
    const frameContent = await frame.content();

    // do something with the content, example:
    contents.push(frameContent);

    // recursively repeat
    await extractFrameContents(frame); 
  }
}
await extractFrameContents(page);

I don't think that document.querySelector normally exists inside a node.js program. Puppeteer does provide page.$, which is a close analog. It returns a Promise.

Update: With the new information in the question, it sounds like the part of the DOM tree that contains #buttonToClick is constructed after the DOMContentLoaded event. I've had good results using page.waitForNavigation({ waitUntil: 'networkidle0' }), though 'networkidle2' or some other option might be better if you have lingering network connections. See the various options here: https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagewaitfornavigationoptions

I suspect this should work:

await page.waitForNavigation({ waitUntil: 'networkidle0' });
if (await page.$('#buttonToClick') !== null) {
    await page.click('#buttonToClick');
  } else {
    await page.waitForSelector('#otherButton');
    await page.click('#otherButton');
  }

A script should not lose to a human who has just manual controls like F5 button.

It happens because sometimes the rules that puppeteer follows are much stricter than what we consider as a "fully loaded webpage". Even if you as a human can decide whether your desired element is in the DOM already (because you see the element is there) or it is not there (because you don't see it). E.g.: you will see that your button is not there even if the background image is still loading in the background, or the webfonts are still not loaded and you have the fallback fonts, but puppeteer waits for specific events in the background to get the permission either to go to the catch block (timeout) or to grab the desired element (waitForSelector succeeds). It can really depends on the site you are visiting, but you are able to speed up the process of recognition of your desired element.

I give some examples and ideas how you can achieve this.

Ways to speed up recognition of the desired element

1.) If you don't need every network connections for your task you could speed up page loading by replacing waitUntil: 'networkidle2' to waitUntil: 'domcontentloaded' as this event happens usually earlier and will be fired when #ourButton will be already present in the DOM.

The possible options of page.goto/page.reload:

• load - consider navigation to be finished when the load event is fired.
• domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
• networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
• networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.

You are winning over the script because of networkidle2 is too strict. You may need this option (e.g. you are visiting a single-page application or later you will need data from the 3rd party network connection e.g. cookies) but in case it is not mandatory you will experience better performance with domcontentloaded.

2.) Instead of constantly navigating to the same url you could use page.reload method in a loop, e.g.:

await page.goto(url, { waitUntil: 'domcontentloaded' })
let selectorExists = await page.$('#ourButton')

while (selectorExists === null) {
  await page.reload({ waitUntil: 'domcontentloaded' })
  console.log('reload')
  selectorExists = await page.$('#ourButton')
}
await page.click('#ourButton')
// code goes on...

Its main benefit is that you are able to shorten and simplify your pageRefresher function. But I experienced also better performance (however I did no benchmarking but I felt it much faster than re-opening a page).

3.) If you don't need every resource type for your task you could also speed up page loading by disabling images or css with the following script:

await page.setRequestInterception(true)
page.on('request', (request) => {
  if (request.resourceType() === 'image') request.abort()
  else request.continue()
})

[source]

List of resourceType-s.

The main difference between those lines of code is the interaction between the Node.js and the browser environment.

The first code snippet will do the following:

• Run document.querySelector in the browser and return the element handle (to the Node.js environment)
• Run getProperty on the handle and return the result (to the Node.js environment)
• Click an element inside the browser

The second code snippet simply does this:

• Run the given function in the browser context (and return results to the Node.js environment)

Performance

Regarding the performance of these statements, one has to remember that puppeteer communicates via WebSockets with the browser. Therefore the second statement will run faster as there is just one command send to the browser (in contrast to three).

This might make a big difference if the browser you are connecting to is running on a different machine (connected to using puppeteer.connect). It will likely only result in a few milliseconds difference if the script and the browser are located on the same machine. In the latter case it might therefore not make a big difference.

Advantage of using element handles

Using element handles has some advantages. First, functions like elementHandle.click will behave more "human-like" in contrast to using document.querySelector('...').click(). puppeteer will for example move the mouse to the location and click in the center of the element instead of just executing the click function.

When to use what

In general, I recommend to use page.evaluate whenever possible as this API is also a lot easier to debug. When an error happens, you can simply reproduce the error by opening the DevTools in your Chrome browser and rerunning the same lines in your browser. If you are mixing a lot of page.$ statements together it might be much harder to understand what the problem is and whether it happened inside the Node.js or the browser runtime.

Use the element handles if you need the element for longer (because you maybe have make some complex calculations or wait for an external event before you can extract information from them).

I really doubt puppeteer can achieve this since it is just a nodejs driver for headless chrome.

If you need to use custom module with cmd require(...) inside, you would probably need to use module bundler like webpack to generate a bundle file with all the dependencies, then inject into puppeteer page.

Just my 2 cents.