How to use newSession method of org.openqa.selenium.grid.distributor.local.LocalDistributor class

First the update ¹

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.

---

Preventing Detection ²

To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:

• Adding the argument --disable-blink-features=AutomationControlled

  from selenium import webdriver
  
  options = webdriver.ChromeOptions() 
  options.add_argument('--disable-blink-features=AutomationControlled')
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get("https://www.website.com")
  

You can find a relevant detailed discussion in Selenium can't open a second page

• Rotating the user-agent through execute_cdp_cmd() command as follows:

  #Setting up Chrome/83.0.4103.53 as useragent
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
  

• Change the property value of the navigator for webdriver to undefined

  driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
  

• Exclude the collection of enable-automation switches

  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  

• Turn-off useAutomationExtension

  options.add_experimental_option('useAutomationExtension', False)
  

---

Sample Code ³

Clubbing up all the steps mentioned above and effective code block will be:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

---

History

As per the W3C Editor's Draft the current implementation strictly mentions:

The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.

Further,

Navigator includes NavigatorAutomationInformation;

It is to be noted that:

The NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.

The NavigatorAutomationInformation interface is defined as:

interface mixin NavigatorAutomationInformation {
    readonly attribute boolean webdriver;
};

which returns true if webdriver-active flag is set, false otherwise.

Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.

Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.

---

Update (6-Nov-2019)

As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions() class to add a couple of arguments to:

• Exclude the collection of enable-automation switches
• Turn-off useAutomationExtension

through an instance of ChromeOptions as follows:

• Java Example:

  System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
  ChromeOptions options = new ChromeOptions();
  options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
  options.setExperimentalOption("useAutomationExtension", false);
  WebDriver driver =  new ChromeDriver(options);
  driver.get("https://www.google.com/");
  

• Python Example

  from selenium import webdriver
  
  options = webdriver.ChromeOptions()
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
  driver.get("https://www.google.com/")
  

• Ruby Example

    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("--disable-blink-features=AutomationControlled")
    driver = Selenium::WebDriver.for :chrome, options: options
  

---

Legends

¹: Applies to Selenium's Python clients only.

²: Applies to Selenium's Python clients only.

³: Applies to Selenium's Python clients only.

I had a bad experience with HtmlUnitDriver. Some time ago I wrote testing framework which were supposed to be fired at Hudson, and finally I decided to use Firefox driver which was more predictable and easier to debug. The point is that in my case it was a page full of javascripts - dynamically loaded fields, etc., and working with HtmlUnitDriver was really a can of worms.

If you really need to use HtmlUnitDriver try do debug 'pagesource' which is accessible for Selenium in 'current' (hanging) moment.

As, when you open a feature file in Eclipse and right-click, then you do not find an option 'Run As->Cucumber Feature', it appears that your Eclipse do not have the required Cucumber Eclipse Plug-in. You can install this plug-in using following steps:

1. Launch Eclipse and navigate to 'Help->Install New Software'.
2. Click on 'Add' button
3. Enter any name e.g. 'Cucumber' and type '' as location.
4. Click on OK.
5. 'Cucumber Eclipse Plugin' option will appear in the available software list.
6. Select the check box against it and click on 'Next' button.
7. Read the terms of license agreement and click on 'I accept the terms of the license agreement” if you agree with the license terms.
8. Click on 'Finish' button.
9. Let it install, it will take few seconds to complete.

After successful installation of this plug-in, if you open any feature file (i.e. file with .feature extension) and right-click, you should see 'Run As->Cucumber Feature' option.

Update 1

In Eclipse, go to 'Windows->Preferences->Run/Debug/Launching/Launch Configuration' and uncheck 'Cucumber Feature' checkbox if it is checked. Refer to below screen shot for more details:

You can get immediate parent using ..

By.xpath("//input[contains(@id,'custProglabel')]/..")

As a side note, xpath should start with lower case 'x'.

My recommendation is Testng because, click here the Advantages of Testng & Advantages of TestNG over Junit. also check this

There is a known issue with the new Version of the FirefoxDriver namely the GeckoDriver, which doesn´t support the Action class see:

Selenium web driver moveToElement (Actions) throwing error with marionette driver?

https://github.com/SeleniumHQ/selenium/issues/3348

Without more info I´d assume this is also your problem. If you need to test with FF then use an older version or Chrome with the ChromeDriver

The Selenium client bindings will try to locate the geckodriver executable from the system PATH. You will need to add the directory containing the executable to the system path.

• On Unix systems you can do the following to append it to your system’s search path, if you’re using a bash-compatible shell:

  export PATH=$PATH:/path/to/geckodriver
  

• On Windows you need to update the Path system variable to add the full directory path to the executable. The principle is the same as on Unix.

All below configuration for launching latest firefox using any programming language binding is applicable for Selenium2 to enable Marionette explicitly. With Selenium 3.0 and later, you shouldn't need to do anything to use Marionette, as it's enabled by default.

To use Marionette in your tests you will need to update your desired capabilities to use it.

Java :

As exception is clearly saying you need to download latest geckodriver.exe from here and set downloaded geckodriver.exe path where it's exists in your computer as system property with with variable webdriver.gecko.driver before initiating marionette driver and launching firefox as below :-

//if you didn't update the Path system variable to add the full directory path to the executable as above mentioned then doing this directly through code
System.setProperty("webdriver.gecko.driver", "path/to/geckodriver.exe");

//Now you can Initialize marionette driver to launch firefox
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability("marionette", true);
WebDriver driver = new MarionetteDriver(capabilities); 

And for Selenium3 use as :-

WebDriver driver = new FirefoxDriver();

If you're still in trouble follow this link as well which would help you to solving your problem

.NET :

var driver = new FirefoxDriver(new FirefoxOptions());

Python :

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

caps = DesiredCapabilities.FIREFOX

# Tell the Python bindings to use Marionette.
# This will not be necessary in the future,
# when Selenium will auto-detect what remote end
# it is talking to.
caps["marionette"] = True

# Path to Firefox DevEdition or Nightly.
# Firefox 47 (stable) is currently not supported,
# and may give you a suboptimal experience.
#
# On Mac OS you must point to the binary executable
# inside the application package, such as
# /Applications/FirefoxNightly.app/Contents/MacOS/firefox-bin
caps["binary"] = "/usr/bin/firefox"

driver = webdriver.Firefox(capabilities=caps)

Ruby :

# Selenium 3 uses Marionette by default when firefox is specified
# Set Marionette in Selenium 2 by directly passing marionette: true
# You might need to specify an alternate path for the desired version of Firefox

Selenium::WebDriver::Firefox::Binary.path = "/path/to/firefox"
driver = Selenium::WebDriver.for :firefox, marionette: true

JavaScript (Node.js) :

const webdriver = require('selenium-webdriver');
const Capabilities = require('selenium-webdriver/lib/capabilities').Capabilities;

var capabilities = Capabilities.firefox();

// Tell the Node.js bindings to use Marionette.
// This will not be necessary in the future,
// when Selenium will auto-detect what remote end
// it is talking to.
capabilities.set('marionette', true);

var driver = new webdriver.Builder().withCapabilities(capabilities).build();

Using RemoteWebDriver

If you want to use RemoteWebDriver in any language, this will allow you to use Marionette in Selenium Grid.

Python:

caps = DesiredCapabilities.FIREFOX

# Tell the Python bindings to use Marionette.
# This will not be necessary in the future,
# when Selenium will auto-detect what remote end
# it is talking to.
caps["marionette"] = True

driver = webdriver.Firefox(capabilities=caps)

Ruby :

# Selenium 3 uses Marionette by default when firefox is specified
# Set Marionette in Selenium 2 by using the Capabilities class
# You might need to specify an alternate path for the desired version of Firefox

caps = Selenium::WebDriver::Remote::Capabilities.firefox marionette: true, firefox_binary: "/path/to/firefox"
driver = Selenium::WebDriver.for :remote, desired_capabilities: caps

Java :

DesiredCapabilities capabilities = DesiredCapabilities.firefox();

// Tell the Java bindings to use Marionette.
// This will not be necessary in the future,
// when Selenium will auto-detect what remote end
// it is talking to.
capabilities.setCapability("marionette", true);

WebDriver driver = new RemoteWebDriver(capabilities); 

.NET

DesiredCapabilities capabilities = DesiredCapabilities.Firefox();

// Tell the .NET bindings to use Marionette.
// This will not be necessary in the future,
// when Selenium will auto-detect what remote end
// it is talking to.
capabilities.SetCapability("marionette", true);

var driver = new RemoteWebDriver(capabilities); 

Note : Just like the other drivers available to Selenium from other browser vendors, Mozilla has released now an executable that will run alongside the browser. Follow this for more details.

You can download latest geckodriver executable to support latest firefox from here

Use .getOptions() method and store them in a list .Then find its size.

Select se = new Select(driver.findElement(By.id("select drop down locator")));

List<WebElement> l = se.getOptions();
l.size();

-Ajay

If you don't have the option to regenerate the same image e.g. by appending a request parameter in your test, you can take a screenshot

File sreenshotFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

and extract the image from the screenshot, which should be easy if you know what you are looking for.

Alternatively as per this answer you could configure Selenium driver to use a known directory to store the browser data with:

ChromeOptions chromeProfile = new ChromeOptions();
chromeProfile.addArguments("user-data-dir=" + chromeProfilePath);
driver = new ChromeDriver(options);

and perhaps the random image will be saved on the disk somewhere under chromeProfilePath. It's possible but the process is complex.