How to use setOfflineMode method in Puppeteer

Unfortunately, Array.prototype.forEach's iterator function is not executed in an async manner as you would expect when defining it as async. Using a for loop should work for what you're trying to do.

for (let i = 0; i < uniqueLinks.length; i ++) {
  await page.goto(uniqueLinks[i], { waitUntil: "load" });
}

I found the answer here https://github.com/heroku/heroku-buildpack-google-chrome/issues/37. Heroku doesn't have any swap space thus resulting in crash. Adding the --disable-dev-shm-usage and --single-process flags fixed it.

const browser = await puppeteer.launch({
      headless: true,
      args: [
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-dev-shm-usage',
        '--single-process'
      ],
    });

You should restructure your Promise Chaining example to use the then() method before the catch() method.

Consider the following example using page.waitForSelector():

// Correct Method
await page.waitForSelector('#example').then(() => {
  console.log('SUCCESS');
}).catch(e => {
  console.log('FAIL');
});

If the element does not exist, then FAIL will be logged to the console. Otherwise, if the element does exist, the output will be SUCCESS.

On the other hand, take a look at the example below in which then() and catch() are reversed:

// Incorrect Method
await page.waitForSelector('#example').catch(e => {
  console.log('FAIL');
}).then(() => {
  console.log('SUCCESS - not necessarily');
});

If the element does not exist, then FAIL will be logged to the console, but regardless of whether the element exists or not, SUCCESS will also be written to the console. This is because logging SUCCESS is the next immediate step in the chain after attempting to catch an error.

Using then() before catch() will allow you to print one of two messages and achieve your desired result.

The async and await keywords are tools to manage promises.

client.decode takes a callback (you aren't calling then() on it) so it, presumably, doesn't return a promise.

The await keyword thus has no effect.

You can create a promise around your callback function.

Further reading:

• How do I convert an existing callback API to promises?
• How do I return the response from an asynchronous call?

1. Is There Any Test That Process POST Request?..

const [response] = await Promise.all([
  page.click('input[type="submit"]'), // After clicking the submit
  page.waitForNavigation() // This will set the promise to wait for navigation events
// Then the page will be send POST and navigate to target page
]);
// The promise resolved

2. How Do I Know Test Has Been Completely Finished?

const [response] = await Promise.all([
  page.click('a.my-link'), // Clicking the link will indirectly cause a navigation
  page.waitForNavigation('networkidle2') // The promise resolves after navigation has finished after no more than 2 request left
]);
// The promise resolved

3. How to Get The Redirect Page URL While Test Running?

For example, if the website http://example.com has a single redirect to https://example.com, then the chain will contain one request:

const response = await page.goto('http://example.com');
const chain = response.request().redirectChain();
console.log(chain.length); // Return 1
console.log(chain[0].url()); // Return string 'http://example.com'

If the website https://google.com has no redirects, then the chain will be empty:

const response = await page.goto('https://google.com');
const chain = response.request().redirectChain();
console.log(chain.length); // Return 0

You're properly catching the waitForSelector and its chained promises, but you're not doing the same for the launch and newPage calls - they're not connected to the catch later.

Because async functions automatically return Promises already, you might consider avoiding the Promise constructor entirely:

var readHtml = async (url) => {
  try {
    var browser = await puppeteer.launch()
    var page    = await browser.newPage()
  } catch(e) {
    // handle initialization error
  }

  await page.waitForSelector('.allDataLoaded')
    .then(() => {
    console.log ("Finished reading: " + url)
    return resolve("COOL");
  })
    .catch((err) => {
    console.log ("Timeout or other error: ", err)
    return resolve("TRYAGAIN");
  });
}

Or, you might consider putting the catch in the consumer of readHtml:

var readHtml = async (url) => {
  var browser = await puppeteer.launch()
  var page    = await browser.newPage()
  await page.waitForSelector('.allDataLoaded')
  console.log ("Finished reading: " + url)
};
readHtml(someurl)
  .catch((e) => console.log('err: ' + e));

There is a special convenience method page.$$eval for this task in puppeteer:

let result = await page.$$eval('.nameToRetrieve', names => names.map(name => name.textContent));
console.log(result);

This method runs Array.from(document.querySelectorAll(selector)) within the page and passes it as the first argument to pageFunction.

The result will be:

[ ' Name 1 ', ' Name 2 ', ' Name 3 ' ]

I tried straight forward approaches to solve this problem and they didn't work. Even obscure apis like CSS expressions and counters don't work to solve this. Fortunately there seems to be a simple enough work around.

We print each page separately using the pageRange parameter and then combine all the pages to generate the required pdf. This enables us to print each header/footer if it were a function of the pageNumber. For example:

const footerTemplate = function (pageNumber) {
    return `<div>Page number: ${pageNumber + 24}</div>`;
};

We need to iterate over each page and print it.

const printPage = function (pageNumber) {
    return {
        ...
        path: `html-page-${pageNumber}.pdf`,
        footerTemplate: footerTemplate(pageNumber),
        pageRanges: String(pageNumber)
    };
};


(async function () {
    ...
    const page = await browser.newPage();
    var pageNumber = 1;
    try {
        while (pageNumber > 0) {
            await page.pdf(printPage(pageNumber));
            pageNumber += 1;
        }
    } catch (e) {
    } finally {
       // Merge and clean up
    }
})();

There is no trivial way to determine the total number of pages to print. So we don't know when to stop. Fortunately, Chrome sends an error when we try to print a page that's out of range. So we can use that to stop our printing.

Attached below is a working sample with page numbers offset by 24. Run with dependencies: fs, pdf-merger-js and puppeteer.

const puppeteer = require("puppeteer");
const PDFMerger = require('pdf-merger-js');
const fs = require("fs");

const footerTemplate = function (pageNumber) {
    return `<div style="font-size: 10px; display: flex; flex-direction: row; justify-content: space-between; width: 100%" id='template'>
        <div>Page number: ${pageNumber + 24}</div>
    </div>`;
};

const mergePdfs = async function (totalPages, fileName) {
    var merger = new PDFMerger();
    for (var pageNumber = 1; pageNumber < totalPages; pageNumber++) {
        await merger.add(`html-page-${pageNumber}.pdf`);
    }
    await merger.save(fileName);
};

const cleanup = function (totalPages) {
    for (var pageNumber = 1; pageNumber < totalPages; pageNumber++) {
        var path = `html-page-${pageNumber}.pdf`
        fs.rmSync(path);
    }
};

const pageSetup = function (pageNumber) {
    return {
        path: `html-page-${pageNumber}.pdf`,
        format: 'Letter',
        printBackground: true,
        displayHeaderFooter: true,
        footerTemplate: footerTemplate(pageNumber),
        pageRanges: String(pageNumber),
        margin: {
            top: '1in',
            right: '0in',
            bottom: '1in',
            left: '0in'
        }
    };
};


(async function () {
    const browser = await puppeteer.launch({
        ignoreHTTPSErrors: true,
        dumpio: true,
        headless: true
    });
    const page = await browser.newPage();
    await page.goto('http://worrydream.com/KillMath/');
    var pageNumber = 1;
    try {
        while (pageNumber > 0) {
            await page.pdf(pageSetup(pageNumber));
            pageNumber += 1;
        }
    } catch (e) {
        await mergePdfs(pageNumber, 'html-page.pdf');
        cleanup(pageNumber);
    }
    await browser.close();
})();

According to the Puppeteer Documentation:

page.title()

• returns: <Promise<string>> Returns page's title.

Shortcut for page.mainFrame().title().

Therefore, you should use page.title() for getting the title of the newly opened page.

Alternatively, you can gain a slight performance boost by using the following:

page._frameManager._mainFrame.evaluate(() => document.title)

Note: Make sure to use the await operator when calling page.title(), as the title tag must be downloaded before Puppeteer can access its content.