How to use EnableGzip method of PuppeteerSharp.TestServer.SimpleServer class

You can use different browser instances for each logical instances. I mean instead of trying to set different proxy for each page/tab with different proxy just create new browser instance and set proxy via launch args.

If this solution doesn't fit your needs, check this question. There is library for NodeJS which give ability to use different proxy per each page/tab. You can check that library source code and implement same things inside your C# application.

That library is using very simple method. Instead of sending requests via puppeter's browser/page library send request via nodejs http tools. It can be done by using method page.setRequestInterception. So library intercept each request from page, after that gather data and send request via http tools. I used C# a long time ago. So maybe I am wrong, but you can try to use HttpWebRequest or something similar. After you get result you should use method request.respond and pass response results there. In this way you can put any kind of proxy inside your application. Check here code of library.

To use XPath based and not CSS selection use page.XPathAsync(xpath-expression) e.g.

var tds = page.XPathAsync(".//td[contains(., 'ABC-DEF-GHI')]");
await tds[0].ClickAsync();

You could try some wait for navigation options. Something like this:

await page.SetContentAsync(
    htmlDoc.SaveToString(), 
    new NavigationOptions { WaitUntil = new[] { WaitUntilNavigation. Networkidle0 }});

You can close the page using CloseAsync:

var page = browser.NewPageAsync();
////
await page.CloseAsync();

An using block will also close the page:

using (var page = await new browser.PageAsync())
{
///
}

Puppeteer-Sharp v2.0.3+ also supports await using blocks

await using (var page = await new browser.PageAsync())
{
 ///
}

You can use the LaunchOptions.ExecutablePath property. If you use ExecutablePath you don't even need to call BrowserFetcher. It would look something like this:

var options = new LaunchOptions
{
    Headless = true,
    ExecutablePath = "<CHROME.EXE FULL PATH>"
};

using (var browser = await Puppeteer.LaunchAsync(options))
{
}

Although there are still many improvements going on, Puppeteer-Sharp is thread-safe. To improve loading performance, there are a few approaches you can take.

Launch one browser and then connect to it

You can launch one (real) browser and then use the ConnectAsync method to connect to it.

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = false,
});

var theBrowser1 = await Puppeteer.ConnectAsync(new ConnectOptions { BrowserWSEndpoint = browser.WebSocketEndpoint });
var theBrowser2 = await Puppeteer.ConnectAsync(new ConnectOptions { BrowserWSEndpoint = browser.WebSocketEndpoint });
var page1 = await theBrowser1.NewPageAsync();
var page2 = await theBrowser2.NewPageAsync();

await Task.WhenAll(
    page1.GoToAsync("https://www.stackoverflow.com"),
    page2.GoToAsync("https://serverfault.com/")
);

I know that code is not running in parallel, but you'll get the idea about reusing the same browser.

Create new pages on the same browser

If you are using TPL, you shouldn't have any issues creating new pages from different threads using the same browser.

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = false,
});

var urls = new string[]
{
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com"
};

await Task.WhenAll(
    urls.Select(url => Task.Factory.StartNew(async () =>
    {
        var page = await browser.NewPageAsync();
        return page.GoToAsync(url);
    })));

Again, this example is just to give you an idea of how this could be accomplished.

Pages queue

There is one user who created a queue of X pages (for x from 0 to X => NewPage) and then he would grab pages from that queue. You can see the example here.

This is how you can use System.Net.WebSockets.Managed on Puppeteer-Sharp

var browserOptions = new LaunchOptions
{
    Headless = true,
    WebSocketFactory = (uri, socketOptions, cancellationToken)
        => System.Net.WebSockets.SystemClientWebSocket.ConnectAsync(uri, cancellationToken)
};
using (var browser = await Puppeteer.LaunchAsync(browserOptions))
using (var page = await browser.NewPageAsync())
{

}

So, after many tries and a lot of searched, I ended up switching to Selenium, which seems to have better documentation and a lot of examples here. This is the code, that worked for me:

string locator1 = "td:Contains(\'41818397111\')";
string locator2 = "td[id = \'InvoiceDetails\']";
IWebElement webElement = driver.FindElement(Selenium.WebDriver.Extensions.By.JQuerySelector(locator1).Parent().Find(locator2));
webElement.Click();

There are few ways to implement what you need (considering a C# console application).

Maybe the easiest one is to use tools that interact with an instance of a browser, i.e. Selenium (used for unit tests). So:

1. Install Selenium.WebDriver nuget package
2. Install a browser where your application will run (let's suppose chrome)
3. Download the browser driver (chromedriver)

4. Write something like:

   IWebDriver driver = null;
   try
   {
       ChromeOptions options = new ChromeOptions();
       options.AddArguments("--incognito");
   
       driver = new ChromeDriver(options);
       driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(5);
       driver.Url = "https://www.wedj.com/dj-photo-video.nsf/firstdance.html";
   
       var musicTable = driver.FindElement(By.Id("musicTable"));
       // interact with driver to get data from the page.
    }
    finally
    {
       if (driver != null)
          driver.Dispose();
    }
   

Otherwise, you need to investigate a little bit more how the webpage works. As far as I can see, the page loads a javascript, https://www.wedj.com/dj-photo-video.nsf/musiclist.js, that it is responsible to load the list of music from server. This js script basically load data from following url: https://www.wedj.com/gbmusic.nsf/musicList?open&wedj=1&list=category_firstdance&count=100 (you can open it also in a browser). Excluding "(" and ")", the result is a json you can parse (maybe using newtonsoft.json package):

{
  "more": "yes",
  "title": "<h1>Most Requested Wedding First Dance Songs<\/h...",
  "event": "<table class='musicTable g6-table-all g6-small' id='musicTable' borde..."
}

The event property contains the data you need (you can use HtmlAgilityPack nuget package to parse it).

Pro Selenium:

1. easy to interact with
2. the behavior is the same of what you see by the browser

Cons Selenium:

1. you need chrome or another browser installed
2. the browser is running when you interact with it
3. the browser download the full page (images, html, js, css...)

Pro manual:

1. you load only what you need
2. no dependencies to external programs (i.e. browsers)

Cons manual:

1. you need to understand how html/js works
2. you need to manually parse the json/html

In this specific case, I prefer the second options.