How to use getExceptionMessage method in Puppeteer

The answer is in the error msg:

Missing X server or $DISPLAY

You can only run chrome in headless mode by default on the server as it doesn't have a display screen. That's why you get the error.

headless: false,

line should be either removed or modified to:

headless: true,

There are alternative ways to fake headful mode on the server, but it heavily depends on your environment what works for you and these are non-stable workaround solutions. e.g.

• https://www.mattzeunert.com/2018/07/21/running-headful-chrome-on-ubuntu-server.html
• https://github.com/beemi/puppeteer-headful

You can use timeout: 0 to disable timeout errors if you're loading a heavy page.

Use it in your page.goto like:

await page.goto('url'+tableCell04Val, {waitUntil: 'load', timeout: 0});

You can see the PR made to Pupeteer here which added the change, along with documentation and the unit tests that implement it.

page.evaluate():

You can use page.evaluate() to get the innerHTML of an element:

const inner_html = await page.evaluate(() => document.querySelector('#myElement').innerHTML);

console.log(inner_html);

elementHandle.getProperty() / .jsonValue():

Alternatively, if you must use page.$(), you can access the innerHTML using a combination of elementHandle.getProperty() and elementHandle.jsonValue():

const inner_html = await (await (await page.$('#myElement')).getProperty('innerHTML')).jsonValue();

console.log(inner_html);

Update: page.emulateMedia() is dropped in favor of page.emulateMediaType()

This isn't actually related to puppeteer at all. On line 4, you have a zero width space character, which doesn't look like anything, but makes your Node program invalid. This sometimes shows up if you're copying a line from a site or some other source.

If you delete that line and insert a blank new line, you should be fine. I tested by copying out your code, installing the dependencies, and running it. Depending on your editor, you should be able to show unexpected unicode characters like this.

Middle click using page.click:

let options = {button : 'middle'};
await page.click('a[href="'+accountsClickElements[i]+'"]', options)

It might take some time for the new tab to open, which you can wait for with await page.waitForTimeout(1000).

Full code:

let accountsClickElements = await page.$$eval('.result-lockup__name a', el => el.map(x => x.getAttribute('href')));

browser.on('targetcreated', function(){                                            
    console.log(accountsClickElements[i]) 
})   

let options = {
    button : 'middle'
};
for (let i = 0; i<25; i++) {
    await page.waitForTimeout(2000);
    await page.focus('a[href="'+accountsClickElements[i]+'"]')                                                    
    await page.click('a[href="'+accountsClickElements[i]+'"]', options)
    const [tab1, tab2, tab3] = await browser.pages();
    await page.waitForTimeout(2000);
    await tab3.bringToFront();                                                                               
    await ListenCompanyPageNewTab(tab3); 
    await tab2.bringToFront();                                                            
} 

• You've posted the definition of getContent, not searchTarget.
• You're passing the result of searchTarget to setInterval rather than passing a reference to the function searchTarget.
• You say you're getting errors but haven't said what those errors are.
• You can use an async function with setInterval.
• You don't need to use await with console.log (you shouldn't use await because as console.log doesn't return anything, let alone a Promise<T>).

I assume what you actually want is this:

async function getContent() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(url);
    const content =  await page.content();
    console.log(content);
    await browser.close();
}

const timerId = window.setInterval( getContent, 13 * 1000 );

// You can call `window.clearInterval( timerId )` to stop the interval loop.

I recommend adding error-handling that stops the interval loop on the first error:

/** @type {number} */
let intervalTimerId = null;

async function getContent() {
    try {
        const browser = await puppeteer.launch();
        const page = await browser.newPage();
        await page.goto(url);
        const content =  await page.content();
        await console.log(content);
        await browser.close();
    }
    catch( err ) {
        console.error( "Error in getContent: %o", err );
        if( intervalTimerId ) {
            window.clearInterval( intervalTimerId );
        }
    }
}

intervalTimerId = window.setInterval( getContent, 13 * 1000 );

Alternative approach:

As other users like @mplungjan and @RohanAsokan have pointed out, there are potential design issues with your code because setInterval will call getContent every 13 seconds even if the previous call to getContent still hasn't finished yet - this could happen if (for example) await page.content() took longer than 13 seconds to run.

In which case the solution is to use window.setTimeout instead of window.setInterval and to call it from within getContent, like so:

/** @type {number} */
let lastSetTimeoutId = null;

async function getContentAndStartLoop() {
    try {
        const browser = await puppeteer.launch();
        const page = await browser.newPage();
        await page.goto(url);
        const content =  await page.content();
        console.log(content);
        await browser.close();

        lastSetTimeoutId = window.setTimeout( getContentAndStartLoop, 13 * 1000 );
    }
    catch( err ) {
        console.error( "Error in getContent: %o", err );
    }
}

Note that getContentAndStartLoop will resolve after the first loop but keep on running itself until an error is thrown.

Better approach:

I think it'd be better to structure it like so (using a Promise-adapter for setTimeout named delay):

async function myProgram() {
    
    const url     = "https://thispersondoesnotexist.com/":
    const browser = await puppeteer.launch();
    const page    = await browser.newPage();
    
    try {
        while( true ) {
        
            await page.goto( url );
        
            const content = await page.content();
            console.log( content );

            await delay( 13 * 1000 );
        }
    }
    catch( err ) {
        console.error( "Error in myProgram: %o", err );
    }
    finally {
        await browser.close();
    }
    
}

async function delay( ms, state = null ) {
    
    return new Promise( ( resolve, reject ) => {
        window.setTimeout( () => resolve( state ), ms );
    } );
}

Let's go back to the first, you can use fullPage to take the screenshot.

await page.screenshot({path: 'example.png', fullPage: true});

If you really want to download all resources to offline, yes you can:

const fse = require('fs-extra');

page.on('response', (res) => {
    // save all the data to SOMEWHERE_TO_STORE
    await fse.outputFile(SOMEWHERE_TO_STORE, await res.buffer());
});

Then, you can browser the website offline through puppeteer with everything all right.

await page.setRequestInterception(true);
page.on('request', (req) => {
    // handle the request by responding data that you stored in SOMEWHERE_TO_STORE
    // and of course, don't forget THE_FILE_TYPE
    req.respond({
        status: 200,
        contentType: THE_FILE_TYPE,
        body: await fse.readFile(SOMEWHERE_TO_STORE),
    });
});

You are attempting to load your local file from the Page DOM Environment using $.get(), which will not work because it is a violation of the same-origin policy.

The Chromium flag --allow-file-access-from-files is described below:

By default, file:// URIs cannot read other file:// URIs. This is an override for developers who need the old behavior for testing.

This flag does not apply to your scenario.

You can use the Node.js function fs.readFileSync() instead to obtain the content of your file and pass it to page.evaluate():

const fs = require('fs');

const file_path = '/..../cors.js';
const file_content = fs.existsSync(file_path) ? fs.readFileSync(file_path, 'utf8') : '';

await page.evaluate(file_content => {
  console.log(file_content);
}, file_content);