How to use DetachAsync method of PuppeteerSharp.CDPSession class

For anybody still encountering this issue, the answer by jamie-tillman on this github issue solved it for me:

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
#####################
#PUPPETEER RECIPE based on https://github.com/hardkoded/puppeteer-sharp/issues/1180#issuecomment-1015532968
#####################
RUN apt-get update && apt-get -f install && apt-get -y install wget gnupg2 apt-utils
RUN wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' >> /etc/apt/sources.list
RUN apt-get update \
&& apt-get install -y google-chrome-stable --no-install-recommends --allow-downgrades fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf
######################
#END PUPPETEER RECIPE
######################
ENV PUPPETEER_EXECUTABLE_PATH "/usr/bin/google-chrome-stable"
WORKDIR /app
EXPOSE 80
EXPOSE 443

The path to the executable is automatically detected by Puppeteer. However, I still had to disable sandboxmode as seen in the snippet below:

var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = true,
    Args = new [] {
      "--disable-gpu",
      "--disable-dev-shm-usage",
      "--disable-setuid-sandbox",
      "--no-sandbox"}
});

More useful information is found in the issue thread so I recommend skimming through it if you still have trouble.

After reading this issue, it seems that we should use absolute path with UserDataDir, so this works fine:

browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = true,
    UserDataDir = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, ".local-chromium", "Win64-884014", "chrome-win", "UserData").Replace(@"\", @"\\")
});

But another problem appeared because browser doesn't save any cookie. Again, some issues on github said that the browser should be closed in order to save the cookies. I tried this:

await page.CloseAsync();
await browser.CloseAsync();

But it doesn't works as expected because the browser is still running in the background. The final solution for my was this:

var chromes = System.Diagnostics.Process.GetProcesses().Where(x => x.ProcessName == "chrome").ToList();

foreach (var ch in chromes)
{
    try
    {
        ch.Kill();
    }
    catch
    {
    }
}

The above code will close all chrome browsers, so it may need some changes in order to close the current browser only.

If you want to get an element from the browser you should get an ElementHandle, which is a pointer to an element in the browser, using EvaluateExpressionHandleAsync:

var element = await page.EvaluateExpressionHandleAsync($"document.querySelectorAll('[{attributeName}=\"{attributeValue}\"]')[0];");

Add WaitForNavigationAsync() after your SetContentAsync() call to wait for the page content to load. By default it waits for the onLoad event, but you can use parameters to change it to wait for a different event.

await page.WaitForNavigationAsync();

EvaluateExpressionAsync won't return DOM elements. You should build a serialized output. For instance, a string[] with the review text.

var reviews = "Array.from(document.querySelectorAll('.comments-content')).map(r => r.innerText);";
var reviews = await page.EvaluateExpressionAsync<string[]>(reviews);
Console.WriteLine(reviews);

I would use WaitForSelectorAsync instead:

var frameElement = await page.WaitForSelectorAsync("#google_ads_frame5");
var frame = await frameElement.ContentFrameAsync();
var frameContent = await frame.GetContentAsync();

I think you should not do a .ConfigureAwait on the controller level, look at this article for more information: https://blog.stephencleary.com/2017/03/aspnetcore-synchronization-context.html.

ASP.NET team dropped the use of SynchronizationContext, so using it in your controller is pointless.

As the article states, you should still use it on your service level, as you don't know whether or not a UI could plug itself to the service and use it, but on your WEB API, you can drop it.

Should be due to PuppeteerSharp requires to run on Linux system in Azure (which can download and install Chromium by coding). If want to test on Windows, need to pre-install Chrome/Chromium and add ExecutablePath in LaunchOptions

// DEBUG: for running on local
ExecutablePath = "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe"

Please share if you have other solutions.

From the examples in the docs, it looks like you have to do this:

// select the tag first
var element = await _page.QuerySelectorAsync("h1 > strong > a");

// then get the attribute
var url = await element.EvaluateFunctionAsync<string>("e => e.getAttribute('href')");
// OR
var url = await _page.EvaluateFunctionAsync<string>("e => e.getAttribute('href')", element);
// OR
var url = await (await element.GetPropertyAsync("href")).JsonValueAsync<string>();

// OR all in one
var url = await _page.EvaluateExpressionAsync<string>($"document.querySelector({"'h1 > strong > a'"}).getAttribute('href')");

You gotta apply this to the value of a select option, of course. Probably 'value' instead of 'href' and the correct selector.