How to use SetAuth method of PuppeteerSharp.TestServer.SimpleServer class

EDIT: New Suggestion HTML Renderer for PDF using PdfSharp

(After trying wkhtmltopdf and suggesting to avoid it)

HtmlRenderer.PdfSharp is a 100% fully C# managed code, easy to use, thread safe and most importantly FREE (New BSD License) solution.

Usage

1. Download HtmlRenderer.PdfSharp nuget package.

2. Use Example Method.

   public static Byte[] PdfSharpConvert(String html)
   {
       Byte[] res = null;
       using (MemoryStream ms = new MemoryStream())
       {
           var pdf = TheArtOfDev.HtmlRenderer.PdfSharp.PdfGenerator.GeneratePdf(html, PdfSharp.PageSize.A4);
           pdf.Save(ms);
           res = ms.ToArray();
       }
       return res;
   }
   

A very Good Alternate Is a Free Version of iTextSharp

Until version 4.1.6 iTextSharp was licensed under the LGPL licence and versions until 4.16 (or there may be also forks) are available as packages and can be freely used. Of course someone can use the continued 5+ paid version.

I tried to integrate wkhtmltopdf solutions on my project and had a bunch of hurdles.

I personally would avoid using wkhtmltopdf - based solutions on Hosted Enterprise applications for the following reasons.

1. First of all wkhtmltopdf is C++ implemented not C#, and you will experience various problems embedding it within your C# code, especially while switching between 32bit and 64bit builds of your project. Had to try several workarounds including conditional project building etc. etc. just to avoid "invalid format exceptions" on different machines.
2. If you manage your own virtual machine its ok. But if your project is running within a constrained environment like (Azure (Actually is impossible withing azure as mentioned by the TuesPenchin author) , Elastic Beanstalk etc) it's a nightmare to configure that environment only for wkhtmltopdf to work.
3. wkhtmltopdf is creating files within your server so you have to manage user permissions and grant "write" access to where wkhtmltopdf is running.
4. Wkhtmltopdf is running as a standalone application, so its not managed by your IIS application pool. So you have to either host it as a service on another machine or you will experience processing spikes and memory consumption within your production server.
5. It uses temp files to generate the pdf, and in cases Like AWS EC2 which has really slow disk i/o it is a big performance problem.
6. The most hated "Unable to load DLL 'wkhtmltox.dll'" error reported by many users.

--- PRE Edit Section ---

For anyone who want to generate pdf from html in simpler applications / environments I leave my old post as suggestion.

TuesPechkin

https://www.nuget.org/packages/TuesPechkin/

or Especially For MVC Web Applications (But I think you may use it in any .net application)

Rotativa

https://www.nuget.org/packages/Rotativa/

They both utilize the wkhtmtopdf binary for converting html to pdf. Which uses the webkit engine for rendering the pages so it can also parse css style sheets.

They provide easy to use seamless integration with C#.

Rotativa can also generate directly PDFs from any Razor View.

Additionally for real world web applications they also manage thread safety etc...

I think you should not do a .ConfigureAwait on the controller level, look at this article for more information: https://blog.stephencleary.com/2017/03/aspnetcore-synchronization-context.html.

ASP.NET team dropped the use of SynchronizationContext, so using it in your controller is pointless.

As the article states, you should still use it on your service level, as you don't know whether or not a UI could plug itself to the service and use it, but on your WEB API, you can drop it.

I think you will find easy to translate the code from puppeteer recorder to Puppeteer-Sharp. At least you can get the selectors from there.

Once you have those selectors you could do 3 things:

One option is setting a directory where the browser can store the cookies using the UserDataDir property. You could open the browser BrowserFetcher downloads and click that button.

A better solution is executing a script using EvaluateFunctionAsync to check if you have the cookies button and click it (using the selector you got from the recorder). Your code will be more flexible.

Last option would be downloading Chromie 77 or higher and launch it using ExecutablePath.

You can call EvaluateFunction passing that ElementHandle as an argument

var content = await Page.EvaluateFunctionAsync<string>("e => e.textContent", elementHandle);

If you have many scenarios like that, you can build an extension method to solve that for you ;)

Just replacing navigation with WaitUntilNavigation.Networkidle2 worked to wait until Javascript is finished to excute.

using PuppeteerSharp;

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
Browser browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = true // false if you need to see the browser
});
var page = await browser.NewPageAsync();
page.DefaultTimeout = 5000; // or you can set this as 0
await page.GoToAsync("https://www.google.com", WaitUntilNavigation.Networkidle2);
var content = await page.GetContentAsync();

Console.WriteLine(content);

System.ValueTuple was added in .NET 4.7. Upgrading your project to 4.7 would be easiest.

If you can't, but your project is targetting a version 4.5+ but less than 4.7, then you can install the System.ValueTuple NuGet package. If you run into any trouble there, see this answer.

Although there are still many improvements going on, Puppeteer-Sharp is thread-safe. To improve loading performance, there are a few approaches you can take.

Launch one browser and then connect to it

You can launch one (real) browser and then use the ConnectAsync method to connect to it.

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = false,
});

var theBrowser1 = await Puppeteer.ConnectAsync(new ConnectOptions { BrowserWSEndpoint = browser.WebSocketEndpoint });
var theBrowser2 = await Puppeteer.ConnectAsync(new ConnectOptions { BrowserWSEndpoint = browser.WebSocketEndpoint });
var page1 = await theBrowser1.NewPageAsync();
var page2 = await theBrowser2.NewPageAsync();

await Task.WhenAll(
    page1.GoToAsync("https://www.stackoverflow.com"),
    page2.GoToAsync("https://serverfault.com/")
);

I know that code is not running in parallel, but you'll get the idea about reusing the same browser.

Create new pages on the same browser

If you are using TPL, you shouldn't have any issues creating new pages from different threads using the same browser.

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = false,
});

var urls = new string[]
{
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com",
    "https://www.stackoverflow.com"
};

await Task.WhenAll(
    urls.Select(url => Task.Factory.StartNew(async () =>
    {
        var page = await browser.NewPageAsync();
        return page.GoToAsync(url);
    })));

Again, this example is just to give you an idea of how this could be accomplished.

Pages queue

There is one user who created a queue of X pages (for x from 0 to X => NewPage) and then he would grab pages from that queue. You can see the example here.

Try to disable some of the security features that can be disabled when launching puppeteer.

Check in puppeteer chrome://flags/ in case there's something blocking iframe access, maybe is insecure content or maybe you have to be explicit about isolation trial.

My 2 cents on this, it should allow it to access it from non secure

Args = new[]
    {
        "--disable-web-security",
        "--disable-features=IsolateOrigins,site-per-process,BlockInsecurePrivateNetworkRequests",
        "--disable-site-isolation-trials"
    }

EvaluateExpressionAsync won't return DOM elements. You should build a serialized output. For instance, a string[] with the review text.

var reviews = "Array.from(document.querySelectorAll('.comments-content')).map(r => r.innerText);";
var reviews = await page.EvaluateExpressionAsync<string[]>(reviews);
Console.WriteLine(reviews);