How to use setOfflineMode method in Puppeteer

Short answer

This XPath expression will query a button which contains the text "Button text":

const [button] = await page.$x("//button[contains(., 'Button text')]");
if (button) {
    await button.click();
}

To also respect the <div class="elements"> surrounding the buttons, use the following code:

const [button] = await page.$x("//div[@class='elements']/button[contains(., 'Button text')]");

Explanation

To explain why using the text node (text()) is wrong in some cases, let's look at an example:

<div>
    <button>Start End</button>
    <button>Start <em>Middle</em> End</button>
</div>

First, let's check the results when using contains(text(), 'Text'):

• //button[contains(text(), 'Start')] will return both two nodes (as expected)
• //button[contains(text(), 'End')] will only return one nodes (the first) as text() returns a list with two texts (Start and End), but contains will only check the first one
• //button[contains(text(), 'Middle')] will return no results as text() does not include the text of child nodes

Here are the XPath expressions for contains(., 'Text'), which works on the element itself including its child nodes:

• //button[contains(., 'Start')] will return both two buttons
• //button[contains(., 'End')] will again return both two buttons
• //button[contains(., 'Middle')] will return one (the last button)

So in most cases, it makes more sense to use the . instead of text() in an XPath expression.

The architecture I ended up with is as follows:

C++ using FFmpeg (libavdevice/libavcodec/libavformat etc.) feeds into the device, which I created using v4l2loopback. Then Chrome can detect this pseudo-device. (as long as you use exclusive_caps=1 option as shown below)

So the first thing I do is set up the v4l2loopback device. This is a faux device that will output like a normal camera, but it will also take input like a capture device or similar.

git clone https://github.com/umlaeute/v4l2loopback
cd v4l2loopback
git reset --hard b2b33ee31d521da5069cd0683b3c0982251517b6 # ensure v4l2loopback functionality changes don't affect this script
make
sudo insmod v4l2loopback.ko exclusive_caps=1 video_nr=$video_nr card_label="My_Fake_Camera"

The browser will see the device in navigator.mediaDevices.enumerateDevices() when and only when you're publishing to it. To test that it's working before you feed to it through C++, you can use ffmpeg -re -i test.avi -f v4l2 /dev/video$video_nr. For my needs, I'm using Puppeteer, so it was relatively easy to test, but keep in mind a long-lasting browser session will cache the devices and refresh them somewhat infrequently, so make sure test.avi (or any video file) is quite long (1 min+) so you can try to reset your environment fully. I've never figured out what the caching strategy is exactly, so Puppeteer turned out to be very helpful here, but I had already been using it, so I didn't have to set it up. YMMV.

Now (for me) the hard part was getting FFmpeg (libav-* version 2.8) to output to this device. I can't/won't share all my code, but here are the parts and some guiding wisdom:

Set up:

• Create an AVFormatContext using avformat_alloc_output_context2(&formatContext->pb, NULL, "v4l2", "/dev/video5")
• Set up the AVCodec using avcodec_find_encoder and create an AVStream using avformat_new_stream
• There are a bunch of little flags you should be setting, but I won't walk through all of them in this answer. This snippet as well as some others include a lot of this work, but they're all geared towards writing to disk rather than to a device. The biggest thing you need to change is creating the AVFormatContext using the device rather than the file (see first step).

For each frame:

• Convert your image to the proper colorspace (mine is BGR, which is OpenCV's default) using OpenCV's cvtColor
• Convert the OpenCV matrix to a libav AVFrame (using sws_scale)
• Encode the AVFrame into an AVPacket using avcodec_encode_video2
• Write the packet to the AVFormatContext using av_write_frame

As long as you do all this right, it should feed it into the device and you should be able to consume your video feed in the browser (or anywhere that detects cameras).

The one thing I'll add that's necessary specifically for Docker is that you have to make sure to share the v4l2 device between the host and the container, assuming you're consuming the device outside of the container (which I am). This means you'll run the docker run command with --device=/dev/video$video_nr.

You can use page.evaluate to send a post request using fetch API

await page.evaluate(() => {
    return fetch('url', {method: 'POST', body: 'test' }).then(res => res.json())
})

So then you can make a request after the requestfinished event fired.

page.setRequestInterception(true)
page.on('requestfinished', async (request: Request) => {
    let response = request.response()                   // Getting the response request
    let responseHeaders = response.headers()            // Check response headers
    let responseBuffer = await response.buffer()        // Get the buffer required
    let responseJSON = await response.json()            // Get parsed JSON body
    await page.evaluate(([headers, buffer, json]) => {  // Replay request with buffer received
        let someData1 = buffer.toString()               // Change buffer to string type
        let someData2 = headers['Content-Type']         // or maybe use some headers data
        let someData3 = json.properties.value           // or use response data object properties
                                                        // This fetch API below will do the rest
        return fetch('url', { method: 'POST', body: 'test' }).then(res => res.json())
    }, [responseHeaders, responseBuffer, responseJSON])
})

You can use page.waitForNavigation() as an alternative to jQuery's document.ready() function:

await page.waitForNavigation({waitUntil: 'load'});             // consider navigation to be finished when the load event is fired.
await page.waitForNavigation({waitUntil: 'domcontentloaded'}); // consider navigation to be finished when the DOMContentLoaded event is fired.
await page.waitForNavigation({waitUntil: 'networkidle0'});     // consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
await page.waitForNavigation({waitUntil: 'networkidle2'});     // consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.

Alternatively, you can use the waitUntil option in page.goto() to wait for the document to load:

await page.goto('https://example.com/', {waitUntil: 'load'});
await page.goto('https://example.com/', {waitUntil: 'domcontentloaded'});
await page.goto('https://example.com/', {waitUntil: 'networkidle0'});
await page.goto('https://example.com/', {waitUntil: 'networkidle2'});

resolve or reject should be used inside a promise to have functionality (e.g. Promise.resolve). Additionally,

new Promise((resolve, reject) => setTimeout(resolve, 1000))
//and
new Promise((x, y) => setTimeout(x, 1000))

are the same, so you need to have a promise for reject to have a meaning.

After catch, you're throwing the error again. Unless you're catching it somewhere else outside of getMarkup you'll see the same warning.

try {
    await page.goto(url, {timeout: 60000});
    if (this.waitForElement !== null)
        await page.waitForSelector(this.waitForElement, {timeout: 60000});
    return await page.content();
} catch (err) {
    //HERE
    console.error(err.message);
}

Finally, if the error is happening in the webpage's Javascript, I'll suggest using window.onerror event in the page. You can find more info here.

You can just hand over your puppeteer Instance like following:

const puppeteer = require('puppeteer-extra');
const _StealthPlugin = require('puppeteer-extra-plugin-stealth');
const _AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');

puppeteer.use(_StealthPlugin());
puppeteer.use(_AdblockerPlugin());

const browser = await puppeteer.launch({
    puppeteer,
});

Src: https://github.com/thomasdondorf/puppeteer-cluster#clusterlaunchoptions

You can set chrome window size during puppeteer.launch with flag --window-size

Here is usage in your example:

const browser = await puppeteer.launch({
    headless: false, // The browser is visible
    ignoreHTTPSErrors: true,
    args: [`--window-size=${options.width},${options.height}`] // new option
});

Mareks solution is fine for a few pages, but in case you want to crawl a large number of pages concurrently, I recommend looking at my library puppeteer-cluster.

It runs tasks in parallel (like Mareks solution), but also takes care of error handling, retrying and some other things. You can see a minimal example below. It's also possible to use the library in more complex settings.

const { Cluster } = require('puppeteer-cluster');

(async () => {
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT, // use one browser per worker
    maxConcurrency: 4, // cluster with four workers
  });

  // Define a task to be executed for your data
  await cluster.task(async ({ page, data: url }) => {
    await page.goto(url);
    const screen = await page.screenshot();
    // ...
  });

  // Queue URLs
  cluster.queue('http://www.google.com/');
  cluster.queue('http://www.wikipedia.org/');
  // ...

  // Wait for cluster to idle and close it
  await cluster.idle();
  await cluster.close();
})();

You can use evaluateHandle to get the element handle, and then call the type function on that element.

const el = await page.evaluateHandle(() => document.activeElement);
await el.type('message');