If await page.click(NEXT_BUTTON_SELECTOR); is causing a pageload to happen then whatever happens next in the script is lost. To get around that you'd have to do:

page.click(NEXT_BUTTON_SELECTOR); // Notice no `await`
await page.waitForNavigation();

Check out more about that API from the docs page here!

They just released this feature :)

page.xpath(expression)

expression <string> Expression to evaluate

returns:<Promise<?ElementHandle>> Promise which resolves to ElementHandle pointing to the page element.

check it out here

The reason it might work in UI mode but not headless is that sites who aggressively fight scraping will detect that you are running in a headless browser.

Some possible workarounds:

Use puppeteer-extra

Found here: https://github.com/berstend/puppeteer-extra Check out their docs for how to use it. It has a couple plugins that might help in getting past headless-mode detection:

1. puppeteer-extra-plugin-anonymize-ua -- anonymizes your User Agent. Note that this might help with getting past headless mode detection, but as you'll see if you visit https://amiunique.org/ it is unlikely to be enough to keep you from being identified as a repeat visitor.
2. puppeteer-extra-plugin-stealth -- this might help win the cat-and-mouse game of not being detected as headless. There are many tricks that are employed to detect headless mode, and as many tricks to evade them.

Run a "real" Chromium instance/UI

It's possible to run a single browser UI in a manner that let's you attach puppeteer to that running instance. Here's an article that explains it: https://medium.com/@jaredpotter1/connecting-puppeteer-to-existing-chrome-window-8a10828149e0

Essentially you're starting Chrome or Chromium (or Edge?) from the command line with --remote-debugging-port=9222 (or any old port?) plus other command line switches depending on what environment you're running it in. Then you use puppeteer to connect to that running instance instead of having it do the default behavior of launching a headless Chromium instance: const browser = await puppeteer.connect({ browserURL: ENDPOINT_URL });. Read the puppeteer docs here for more info: https://pptr.dev/#?product=Puppeteer&version=v5.2.1&show=api-puppeteerlaunchoptions

The ENDPOINT_URL is displayed in the terminal when you launch the browser from the command line with the --remote-debugging-port=9222 option.

This option is going to require some server/ops mojo, so be prepared to do a lot more Stack Overflow searches. :-)

There are other strategies I'm sure but those are the two I'm most familiar with. Good luck!

No, you cannot pass functions like that. The passed data needs to be serializable via JSON.stringify, which is not possible for functions.

Alternative: Expose the function

To use a function from your Node.js environment inside the page, you need to expose it via page.exposeFunction. When called the function is executed inside your Node.js environment

await page.exposeFunction('myfunction', text => `great ${text}`);
await page.evaluate(async (object) => {
  return await window.myfunction(object); // 'great example'
}, 'example');

Alternative: Define the function inside page.evaluate

To use a function inside the page context, you can either define it inside of the context. This way, the function does not have access to your Node.js variables.

await page.evaluate((obj) => {
  const myfunction = (stuff) => `great ${stuff}`;
  return myfunction(obj); // 'great example'
}, 'example');

If it is not a must to launch chrome from bash, but it is OK to launch by the Node script itself then you could try the following. (In my example I used websocket endpoint to connect to instead of browserUrl, feel free to choose which suits better your needs.)

const browser = await puppeteer.launch({ 
  headless: true, 
  args: [
    `--user-data-dir=${task.profile}`
  ]
})
const browserWSEndpoint = await browser.wsEndpoint()
const browser2 = await puppeteer.connect({ browserWSEndpoint, defaultViewport: null })

Note: Make sure your --user-data-dir is correct. If you are on Windows you will need something like: '--user-data-dir=%userprofile%\\AppData\\Local\\Google\\Chrome\\User Data\\ProfileName'

To determine the user data directory for a running Chrome instance:

1. Navigate to chrome://version
2. Look for the Profile Path field. This gives the path to the profile directory.
3. The user data directory is the parent of the profile directory.

Example (Windows):

• [Profile Path] C:\Users\Alice\AppData\Local\Google\Chrome\User Data\Default
• [User Data Dir] C:\Users\Alice\AppData\Local\Google\Chrome\User Data

[source]

You can try to wrap your promises inside a Promise.all.

So instead of returning result, you return Promise.all(result) which will resolve once it get's all of the data.

Also, there are many different ways to go thru the list or selectors.

$$eval

This runs Array.from(document.querySelectorAll(selector)) within the page and passes it as the first argument to pageFunction.

Example code:

await page.$$eval('a', a => a.href)

evaluate

This is golden and you can execute any code that you can execute on your browsers console.

const result = await page.evaluate(() =>
  [...document.querySelectorAll('input')] // or anything that you want to use normally
)

Useful APIs

There are several useful APIs that can be used wisely:

• .type: types a text on selector
• .click: Clicks an elementhandle/selector etc
• .select: Selects something from a element

Know what doesn't work

elementHandle.attribute() has been removed on #638. Which means getting attribute should be handled by $eval from now.

I can think of two dubious and inconvenient solutions.

1. Using eval() or Function() constructor inside the evaluated function and transfer the function code as a string (difficult for code writing, linting etc. — this can be mitigated a bit by using Function.prototype.toString()).

2. Placing all the utility functions in a file and using page.addScriptTag() to inject this file so that the evaluated function can call any of the functions declared in this file (this can pollute the global scope).

Edit:

function evaluateXPath(page: puppeteer.Page, xpath: string, evalFunc: (value: any, index: number, array: any[]) => any): Promise<any[]> {
    return page.waitForXPath(xpath)
        .then(() => page.$x(xpath))
        .then(handles => page.evaluate((evalFuncStr, ...handles) =>
            handles.map(eval(`(${evalFuncStr})`)),
            evalFunc.toString(),
            ...handles))
}

I prefer to work with first-page instead of opening a new one, It's faster than opening a new page, but I don't think that really can boost performance and speed.

const [ page ] = await browser.pages();

Your selector should be like const nodes = element.querySelectorAll('.post-item-info');. Then to access individual items in the returned collection, use a traditional for loop like

for(let i = 0; i < nodes.length; i++){
      const currentNode = nodes[i];
      // doStuffWith(currentNode);
    }