How to use processClip method in Puppeteer

You have to get the frame element, from there you can get the frame itself so you can call evaluate inside that frame:

const elementHandle = await page.$('frame[name=SWContent]');
const frame = await elementHandle.contentFrame();
let urls = await frame.evaluate(() => {
  let results = [];
  let items = document.querySelectorAll("input").length;
  return items;
});

Just setting the viewports height and weight is not enough to fully simulate a mobile browser. Right now the page assumes that you just have a very small browser window.

The easiest way to simulate a mobile device is by using the the function page.emulate and the default DeviceDesriptors, which contain information about a large number of mobile devices.

Quote from the docs for page.emulate:

Emulates given device metrics and user agent. This method is a shortcut for calling two methods:

• page.setUserAgent(userAgent)
• page.setViewport(viewport)

To aid emulation, puppeteer provides a list of device descriptors which can be obtained via the require('puppeteer/DeviceDescriptors') command. [...]

Example

Here is an example on how to simulate an iPhone when visiting the page.

const puppeteer = require('puppeteer');
const devices = require('puppeteer/DeviceDescriptors');
const iPhone = devices['iPhone 6'];

const url = '...';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.emulate(iPhone);
  await page.goto(url);

  // Simlified page.evaluate
  let producData = await page.evaluate(() => ({
    name: document.querySelector('#a-page h1#title').innerText,
    price: document.querySelector('#a-page #newBuyBoxPrice').innerText
  }));

  console.dir(producData);
})();

I also simplified your page.evaluate a little, but you can of course also use your original code after the page.goto. This returned the name and the price of the product for me.

This is the problem the library puppeteer-cluster (I'm the author) is addressing. It allows you to build a pool of pages (or browsers) to use and run tasks inside.

You find several general code samples in the repository (and also on stackoverflow). Let me address your specific use case of running different tasks with an example.

Code Sample

The following code creates two tasks:

• crawl: Opens the page and extracts an URL to then start the second task
• screenshot: Takes a screenshot of the extracted URL

The process is started by queuing the crawl task with the URLs.

const { Cluster } = require('puppeteer-cluster');

(async () => {
    const cluster = await Cluster.launch({ // use four pages in parallel
        concurrency: Cluster.CONCURRENCY_PAGE,
        maxConcurrency: 4,
    });

    // We define two tasks
    const crawl = async ({ page, data: url }) => {
        await page.goto(url);
        const extractedURL = /* ... */; // extract an URL (or multiple) from the document somehow
        cluster.queue(extractedURL, screenshot);
    };

    const screenshot = async ({ page, data: url }) => {
        await page.goto(url);
        await page.screenshot();
    };

    // Crawl some pages
    cluster.queue('https://www.google.com/', crawl);
    cluster.queue('https://github.com/', crawl);

    // Wait until everything is done and close the cluster
    await cluster.idle();
    await cluster.close();
})();

This is a minimal example. I left out error handling, monitoring and the setup options.

I managed to do it. I've found older Chrome version (65), and got it running with puppeteer-extra.

The versions of the libs that I used and are working:

PepperFlashPlugin version: 32.0.0.223

Google Chrome: 65.0.3325.181

Puppeteer-core: 1.7.0 (check the tags for the corresponding version if you need different than 65)

puppeteer-extra: 2.1.3 puppeteer: 1.0.0

puppeteer-extra-plugin-flash: 2.1.3

Launching the browser looks like this:

const browser = await PuppeteerExtra.launch({
        headless: false,
        executablePath: process.env.CHROME_EXECUTABLE,
        args: [
          '--window-size=800,600',
          '--enable-webgl',
          '--enable-accelerated-2d-canvas',
        ],
      });
const page = await this.fBrowser.newPage();

      await page.setViewport({ width: 800, height: 600});
      await page.goto('http://ultrasounds.com', { waitUntil: 'networkidle2' });

And it works! ????

I found out: node LTS (AKA node 6) is not supporting async / await mechanism right now. See :

See here for details : https://www.infoq.com/news/2017/02/node-76-async-await

puppeteer is completely async, so you have to wait it's initialization before using this.browser.

But setWorldConstructor is sync function, so you can't wait there. In my example I used Before hook

My example: https://gist.github.com/dmitrika/7dee618842c00fbc35418b901735656b

You need to do at least two things to get puppeteer working with Brave.

First, you need to enable remote debugging on brave. You need to go to chrome://settings/privacy and then enable Remote debugging.

Second, Brave doesn't like many default command-line arguments that puppeteer sends. So you might want to ignore default arguments.

(async()=>{
    const browser = await puppeteer.launch({
        executablePath:"/Applications/Brave Browser.app/Contents/MacOS/Brave Browser",
        headless:false,
        ignoreDefaultArgs: true
    })
    const page = await browser.newPage()
    page.goto("https://www.google.com")
})()

You get the front page because you request page's content immediately after clicking on the "next" button, but you need to wait for Ajax request to finish. It can be done by observing a "tree palm" ajax loader: when it's not visible, the results are in.

// Utility function to pass time: await timeout(ms)
const timeout = ms => new Promise(resolve => setTimeout(resolve, ms));

// emulate a realistic client's screen size
await page.property('viewportSize', { width: 1280, height: 720 });

const status = await page.open('https://www.dubailand.gov.ae/English/services/Eservices/Pages/Brokers.aspx');

await page.evaluate(function() {
    document.querySelector('#ctl00_ctl42_g_26779dcd_6f3a_42ae_903c_59dea61690e9_dpPager > a.NextPageLink').click();
});

// Give it time to start request
await timeout(1000);

// Wait until the loader is gone
while(1 == await page.evaluate(function(){ 
    return jQuery(".Loader_large:visible").length 
}))
{
    await timeout(1000);
    console.log(".");
}

// Now for scraping
let contacts = await page.evaluate(function(){

    var contacts = []; 
    jQuery("#tbBrokers tr").each(function(i, row){
        contacts.push({"title" : jQuery(row).find("td:nth-child(2)").text().trim(), "phone" : jQuery(row).find("td:nth-child(4)").text().trim() })
    })

    return contacts;
});

console.log(contacts);

Summary:

The Puppeteer function page.exposeFunction() essentially allows you to access Node.js functionality within the Page DOM Environment.

On the other hand, page.evaluateOnNewDocument() evaluates a predefined function when a new document is created and before any of its scripts are executed.

The Puppeteer Documentation for page.exposeFunction() states:

page.exposeFunction(name, puppeteerFunction)

• name <string> Name of the function on the window object
• puppeteerFunction <function> Callback function which will be called in Puppeteer's context.
• returns: <Promise>

The method adds a function called name on the page's window object. When called, the function executes puppeteerFunction in node.js and returns a Promise which resolves to the return value of puppeteerFunction.

If the puppeteerFunction returns a Promise, it will be awaited.

NOTE Functions installed via page.exposeFunction survive navigations.

An example of adding an md5 function into the page:

const puppeteer = require('puppeteer');
const crypto = require('crypto');

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text()));
  await page.exposeFunction('md5', text =>
    crypto.createHash('md5').update(text).digest('hex')
  );
  await page.evaluate(async () => {
    // use window.md5 to compute hashes
    const myString = 'PUPPETEER';
    const myHash = await window.md5(myString);
    console.log(`md5 of ${myString} is ${myHash}`);
  });
  await browser.close();
});

An example of adding a window.readfile function into the page:

const puppeteer = require('puppeteer');
const fs = require('fs');

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text()));
  await page.exposeFunction('readfile', async filePath => {
    return new Promise((resolve, reject) => {
      fs.readFile(filePath, 'utf8', (err, text) => {
        if (err)
          reject(err);
        else
          resolve(text);
      });
    });
  });
  await page.evaluate(async () => {
    // use window.readfile to read contents of a file
    const content = await window.readfile('/etc/hosts');
    console.log(content);
  });
  await browser.close();
});

Furthermore, the Puppeteer Documentation for page.evaluateOnNewDocument explains:

page.evaluateOnNewDocument(pageFunction, ...args)

• pageFunction <function|string> Function to be evaluated in browser context
• ...args <...Serializable> Arguments to pass to pageFunction
• returns: <Promise>

Adds a function which would be invoked in one of the following scenarios:

• whenever the page is navigated
• whenever the child frame is attached or navigated. In this case, the function is invoked in the context of the newly attached frame

The function is invoked after the document was created but before any of its scripts were run. This is useful to amend the JavaScript environment, e.g. to seed Math.random.

An example of overriding the navigator.languages property before the page loads:

// preload.js

// overwrite the `languages` property to use a custom getter
Object.defineProperty(navigator, "languages", {
  get: function() {
    return ["en-US", "en", "bn"];
  }
});

// In your puppeteer script, assuming the preload.js file is in same folder of our script
const preloadFile = fs.readFileSync('./preload.js', 'utf8');
await page.evaluateOnNewDocument(preloadFile);