How to use SessionMapServer class of org.openqa.selenium.grid.sessionmap.httpd package

With the IE driver, the order in which the windows appear in the collection is not guaranteed. That is, the 0th window in the collection is not necessarily the first window opened by the session. Given that is the case, you'll need to do something like the following:

private string FindNewWindowHandle(IWebDriver driver, IList<string> existingHandles, int timeout)
{
    string foundHandle = string.Empty;
    DateTime endTime = DateTime.Now.Add(TimeSpan.FromSeconds(timeout));
    while (string.IsNullOrEmpty(foundHandle) && DateTime.Now < endTime)
    {
        IList<string> currentHandles = driver.WindowHandles;
        if (currentHandles.Count != existingHandles.Count)
        {
            foreach (string currentHandle in currentHandles)
            {
                if (!existingHandles.Contains(currentHandle))
                {
                    foundHandle = currentHandle;
                    break;
                }
            }
        }

        if (string.IsNullOrEmpty(foundHandle))
        {
            System.Threading.Thread.Sleep(250);
        }
     }

     // Note: could optionally check for handle found here and throw
     // an exception if no window was found.
     return foundHandle;
}

The usage of the above function would be something like the following:

IList<string> handles = driver.WindowHandles;
// do whatever you have to do to invoke the popup
element.Click();
string popupHandle = FindNewWindowHandle(driver, handles, 10);
if (!string.IsNullOrEmpty(popupHandle))
{
    driver.SwitchTo().Window(popupHandle);
}

Just use //element[@someAttribute] for your XPath expression.

In order to use ChromeDriver undetectable to Distil checkpoints (which are described nicely in this stackoverflow post), you will need to ensure that your browser does not contain any variable in its window or document prototypes that reveal that you are using a webdriver, as the one you mention.

You can use software as Selenium along with ChromeDriver and Chrome as long as you take some precautions and make some fixes to the binaries. This method will apply only to headed version, if you wish to use headless, you would need to take additional measurements to pass window/rendering tests, described here.

1. Fix Chrome binary, or use an old version

First, lets deal with that navigator.webdriver set to True. This is defined by W3C protocol here as part of 'NavigatorAutomationInformation' of browsers, which extends the Navigator interface. How to remove it? That project has a lot of files, third party stuff, blink web runtimes, etc. So, instead of becoming crazy trying to figure out how this works, as Chromium is open-source, just be clever and search google for the commit which incorporated that. Here is the link. Pay attention to these files:

• third_party/WebKit/Source/core/frame/Navigator.h, which holds the line of code:

   `bool webdriver() const { return true; }` 
  
  This method is supossed to always return true, as you can see.
  

• third_party/WebKit/Source/core/frame/Navigator.idl, which contains the extensions of Navigators, included our

  `Navigator implements NavigatorAutomationInformation;` 
  
  which is being commited. Interesting, isn't it?
  

• third_party/WebKit/Source/core/frame/NavigatorAutomationInformation.idl contains the extension itself, with a read-only variable, which is webdriver:

       `[
           NoInterfaceObject, // Always used on target of 'implements'
           Exposed=(Window),
           RuntimeEnabled=AutomationControlled
       ] interface NavigatorAutomationInformation {
           readonly attribute boolean webdriver;
       };`
  

To get rid of this functionality, it should be enough commenting the line in Navigator.idl which extends Navigator with this functionality, and compile the source (compiling in linux here). However, this is a laborious task for almost any computer and can take several hours. If you look the date of the commit, it was on Oct 2017, so an option is to download any version of Chrome released before that date. To search for mirrors, you can google for inurl:/deb/pool/main/g/google-chrome-stable/.

2. Fix ChromeDriver

Distil checks the regex rule '/\$[a-z]dc_/' against window variables, and ChromeDriver adds one as mentioned here which satisfies that condition. As they mention, you have to edit call_function.js amongst the source code, and redefine the variable var key = '$cdc_asdjflasutopfhvcZLmcfl_';. with something else. Also, probably easier, you can use an hex editor to update the existing binary.

If you decided to use an older version of Chrome -I guess you did-, you will need to use an appropiate version of ChromeDriver. You can know which one is fine for your Chrome version in the ChromeDriver downloads webpage. For example, for Chrome v61 (which fits your needs), you could use ChromeDriver 2.34. Once done, just put the ChromeDriver binary on '/usr/bin/local'.

3. Take other precautions

• Pay attention to your user-agent.
• Don't perform too many repeated requests.
• Use a (random) delay between requests.
• Use the Chrome arguments used here to mimic a normal user profile.

There is a page documenting Advanced User Interactions; which has a lot of great examples on how to generate a sequence of actions, you can find it here

// Configure the action
Actions builder = new Actions(driver);

builder.keyDown(Keys.CONTROL)
   .click(someElement)
   .click(someOtherElement)
   .keyUp(Keys.CONTROL);

// Then get the action:
Action selectMultiple = builder.build();

// And execute it:
selectMultiple.perform();   

or

Actions builder = new Actions(driver);

Action dragAndDrop = builder.clickAndHold(someElement)
   .moveToElement(otherElement)
   .release(otherElement)
   .build();

dragAndDrop.perform();

You can use

WebElement activeElement = driver.switchTo().activeElement(); 
String className =  activeElement.getAttribute("class"); 
String id = activeElement.getAttribute("id");

The window obviously has no name, so you can't select it by name.

1. If the window is opened via JavaScript and you can change the script, try changing window.open("someUrl"); to window.open("someUrl", "someName");, you'll be then able to select the window by the set name. More information on the MDN doc for window.open().

2. Selenium RC doesn't support <a href="someUrl" target="_blank"> links (which open the link in a new window). Therefore, if the window is opened by a link of this type, you have to find this <a> element, get the href attribute and call

   selenium.openWindow(theFoundUrl, "theNewWindow");
   selenium.selectWindow("id=theNewWindow");
   

3. If it is opened via JavaScript before or during the onload event, you'll need to call

   selenium.openWindow("", "theNewWindow");
   selenium.selectWindow("id=theNewWindow");
   

   More information on this in the bug SEL-339 or in the openWindow() and selectWindow() JavaDocs.

4. If you have only two windows / want to open the newest one, you can try

   selenium.selectPopup()

   That is, obviously, the easiest way, because it selects the first non-top window. Therefore, it's only useful when you want to select the newest popup.

5. If the new window has a unique title, you can do

   selenium.selectPopup("Title of the window");
   

   or selenium.selectWindow("title=Title of the window");

6. Otherwise, you must iterate over selenium.getAllWindowNames() to get the right name (Selenium creates names for windows without one). However, you can't hardcode that name into your testcase, because it will change every time, so you'll need to work out some dynamic logic for this.

7. You won't like this: Go for WebDriver. It should be far more resistant to such problems.

I had an extra ":" in my feature file after Given, When and Then.

It's working now.

    String base = driver.getWindowHandle();

    Set <String> set = driver.getWindowHandles();

    set.remove(base);
    assert set.size()==1;

    driver.switchTo().window(set.toArray(new String[0]));

    driver.close();
    driver.switchTo().window(base);

This works for me...

First the update ¹

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.

Preventing Detection ²

To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:

• Adding the argument --disable-blink-features=AutomationControlled

  from selenium import webdriver
  
  options = webdriver.ChromeOptions() 
  options.add_argument('--disable-blink-features=AutomationControlled')
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get("https://www.website.com")
  

You can find a relevant detailed discussion in Selenium can't open a second page

• Rotating the user-agent through execute_cdp_cmd() command as follows:

  #Setting up Chrome/83.0.4103.53 as useragent
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
  

• Change the property value of the navigator for webdriver to undefined

  driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
  

• Exclude the collection of enable-automation switches

  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  

• Turn-off useAutomationExtension

  options.add_experimental_option('useAutomationExtension', False)
  

Sample Code ³

Clubbing up all the steps mentioned above and effective code block will be:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

History

As per the W3C Editor's Draft the current implementation strictly mentions:

The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.

Further,

Navigator includes NavigatorAutomationInformation;

It is to be noted that:

The NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.

The NavigatorAutomationInformation interface is defined as:

interface mixin NavigatorAutomationInformation {
    readonly attribute boolean webdriver;
};

which returns true if webdriver-active flag is set, false otherwise.

Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.

Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.

Update (6-Nov-2019)

As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions() class to add a couple of arguments to:

• Exclude the collection of enable-automation switches
• Turn-off useAutomationExtension

through an instance of ChromeOptions as follows:

• Java Example:

  System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
  ChromeOptions options = new ChromeOptions();
  options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
  options.setExperimentalOption("useAutomationExtension", false);
  WebDriver driver =  new ChromeDriver(options);
  driver.get("https://www.google.com/");
  

• Python Example

  from selenium import webdriver
  
  options = webdriver.ChromeOptions()
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
  driver.get("https://www.google.com/")
  

• Ruby Example

    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("--disable-blink-features=AutomationControlled")
    driver = Selenium::WebDriver.for :chrome, options: options
  

Legends

¹: Applies to Selenium's Python clients only.

²: Applies to Selenium's Python clients only.

³: Applies to Selenium's Python clients only.