How to use setHasResidentKey method of org.openqa.selenium.virtualauthenticator.VirtualAuthenticatorOptions class

If your tests are dependent, you can put them in the same class with dependsOnMethod={method1} defined on method2 so that the order is gauranteed. If it is between different classes, you might consider extending your AddNormalEE class from the LoginScript class.

To run tests in the same browser, your driver instance needs to be shared between your classes or it has to be the same in all your @Tests. Either make it static or consider having a threadlocal webdriver variable in case you plan to run parallely some day. In the above case, you can also have a method getDriver() in your loginScript which returns the driver object to the AddNormalEE class if static needs to be avoided.

As a general practice, it is good to have independent tests. You can make use of parallel runs to overcome the time issue with independent tests. Making login as a method and not a test since we are not asserting any behavior as per your code above. If I am testing login, I would have separate tests that test login functionality only.

The difference of findElement between WebElement and WebDriver is the context.

Whereas the context of WebDriveris the current page, the context of WebElement is that element. WebDriver will search across the whole document, while WebElement will try to find the first child element from that node.

Note that when searching with WebElement via XPath, starting with // will still search across the entire document, not just the children of the current node. You can use .// to limit the search results to the children of that WebElement.

First the update ¹

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.

Preventing Detection ²

To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:

• Adding the argument --disable-blink-features=AutomationControlled

  from selenium import webdriver
  
  options = webdriver.ChromeOptions() 
  options.add_argument('--disable-blink-features=AutomationControlled')
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get("https://www.website.com")
  

You can find a relevant detailed discussion in Selenium can't open a second page

• Rotating the user-agent through execute_cdp_cmd() command as follows:

  #Setting up Chrome/83.0.4103.53 as useragent
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
  

• Change the property value of the navigator for webdriver to undefined

  driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
  

• Exclude the collection of enable-automation switches

  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  

• Turn-off useAutomationExtension

  options.add_experimental_option('useAutomationExtension', False)
  

Sample Code ³

Clubbing up all the steps mentioned above and effective code block will be:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

History

As per the W3C Editor's Draft the current implementation strictly mentions:

The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.

Further,

Navigator includes NavigatorAutomationInformation;

It is to be noted that:

The NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.

The NavigatorAutomationInformation interface is defined as:

interface mixin NavigatorAutomationInformation {
    readonly attribute boolean webdriver;
};

which returns true if webdriver-active flag is set, false otherwise.

Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.

Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.

Update (6-Nov-2019)

As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions() class to add a couple of arguments to:

• Exclude the collection of enable-automation switches
• Turn-off useAutomationExtension

through an instance of ChromeOptions as follows:

• Java Example:

  System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
  ChromeOptions options = new ChromeOptions();
  options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
  options.setExperimentalOption("useAutomationExtension", false);
  WebDriver driver =  new ChromeDriver(options);
  driver.get("https://www.google.com/");
  

• Python Example

  from selenium import webdriver
  
  options = webdriver.ChromeOptions()
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
  driver.get("https://www.google.com/")
  

• Ruby Example

    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("--disable-blink-features=AutomationControlled")
    driver = Selenium::WebDriver.for :chrome, options: options
  

Legends

¹: Applies to Selenium's Python clients only.

²: Applies to Selenium's Python clients only.

³: Applies to Selenium's Python clients only.

The problem

The problem you are probably facing is that the method returns the right (and valid!) element, but when you're trying to access it a second later, it is stale and throws.

This usually arises when:

1. You click something that loads a new page asynchronously or at least changes it.
2. You immediatelly (before the page load could finish) search for an element ... and you find it!
3. The page finally unloads and the new one loads up.
4. You try to access your previously found element, but now it's stale, even though the new page contains it, too.

The solutions

There are four ways to solve it I know about:

1. Use proper waits

   Use proper waits after every anticipated page-load when facing asynchronous pages. Insert an explicit wait after the initial click and wait for the new page / new content to load. Only after that you can try to search for the element you want. This should be the first thing you'll do. It will increase the robustness of your tests greatly.

2. The way you did it

   I have been using a variant of your method for two years now (together with the technique above in solution 1) and it absolutely works most of the time and fails only on strange WebDriver bugs. Try to access the found element right after it is found (before returning from the method) via a .isDisplayed() method or something. If it throws, you already know how to search again. If it passes, you have one more (false) assurance.

3. Use a WebElement that re-finds itself when stale

   Write a WebElement decorator that remembers how it was found and re-find it when it's accessed and throws. This obviously forces you to use custom findElement() methods that would return instances of your decorator (or, better yet, a decorated WebDriver that would return your instances from usual findElement() and findElemens() methods). Do it like this:

   public class NeverStaleWebElement implements WebElement {
       private WebElement element;
       private final WebDriver driver;
       private final By foundBy;
   
       public NeverStaleWebElement(WebElement element, WebDriver driver, By foundBy) {
           this.element = element;
           this.driver = driver;
           this.foundBy = foundBy;
       }
   
       @Override
       public void click() {
           try {
               element.click();
           } catch (StaleElementReferenceException e) {
               // log exception
   
               // assumes implicit wait, use custom findElement() methods for custom behaviour
               element = driver.findElement(foundBy);
   
               // recursion, consider a conditioned loop instead
               click();
           }
       }
   
       // ... similar for other methods, too
   
   }
   

   Note that while I think that the foundBy info should be accessible from the generic WebElements to make this easier, Selenium developers consider it a mistake to try something like this and have chosen not to make this information public. It's arguably a bad practice to re-find on stale elements, because you're re-finding elements implicitly without any mechanism for checking whether it's justified. The re-finding mechanism could potentially find a completely different element and not the same one again. Also, it fails horribly with findElements() when there are many found elements (you either need to disallow re-finding on elements found by findElements(), or remember the how-manyeth your element was from the returned List).

   I think it would be useful sometimes, but it's true that nobody would ever use options 1 and 2 which are obviously much better solutions for the robustness of your tests. Use them and only after you're sure you need this, go for it.

4. Use a task queue (that can rerun past tasks)

   Implement your whole workflow in a new way!

   • Make a central queue of jobs to run. Make this queue remember past jobs.
   • Implement every needed task ("find an element and click it", "find an element and send keys to it" etc.) via the Command pattern way. When called, add the task to the central queue which will then (either synchronously or asynchronously, doesn't matter) run it.
   • Annotate every task with @LoadsNewPage, @Reversible etc. as needed.
   • Most of your tasks will handle their exceptions by themselves, they should be stand-alone.
   • When the queue would encounter a stale element exception, it would take the last task from the task history and re-run it to try again.

   This would obviously take a lot of effort and if not thought through very well, could backfire soon. I used a (lot more complex and powerful) variant of this for resuming failed tests after I manually fixed the page they were on. Under some conditions (for example, on a StaleElementException), a fail would not end the test right away, but would wait (before finally time-outing after 15 seconds), popping up an informative window and giving the user an option to manually refresh the page / click the right button / fix the form / whatever. It would then re-run the failed task or even give a possibility to go some steps back in history (e.g. to the last @LoadsNewPage job).

Final nitpicks

All that said, your original solution could use some polishing. You could combine the two methods into one, more general (or at least make them delegate to this one to reduce code repetition):

WebElement getStaleElem(By by, WebDriver driver) {
    try {
        return driver.findElement(by);
    } catch (StaleElementReferenceException e) {
        System.out.println("Attempting to recover from StaleElementReferenceException ...");
        return getStaleElem(by, driver);
    } catch (NoSuchElementException ele) {
        System.out.println("Attempting to recover from NoSuchElementException ...");
        return getStaleElem(by, driver);
    }
}

With Java 7, even a single multicatch block would be sufficient:

WebElement getStaleElem(By by, WebDriver driver) {
    try {
        return driver.findElement(by);
    } catch (StaleElementReferenceException | NoSuchElementException e) {
        System.out.println("Attempting to recover from " + e.getClass().getSimpleName() + "...");
        return getStaleElem(by, driver);
    }
}

This way, you can greatly reduce the amount of code you need to maintain.

The ideal way would be to not use selenium to automate gmail but rather use the Gmail API (https://developers.google.com/gmail/api/#how_do_i_find_out_more) to verify the message was successfully sent. If you do not want to learn how to check the messages at the API level I would highly recommend using the HTML version of gmail using this link as the initial url for gmail (https://mail.google.com/mail/?ui=html) using gmail with javascript enabled will make it much harder to have a reliable test script.

Try this. You need to start with a period at the beginning.

By.xpath("(.//td[@class=\"spreadsheet\"])[24]")

https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/WebElement.html#findElements-org.openqa.selenium.By-

I think the idea is for you to use JUnit's Assert.assertXXX() the difference is that verifyXXX will fail during teardown instead of immediately but I think with Selenium tests you usually want to fail as fast as possible (since those tests tend to be slow).

The API documentation for findElements() says:

When using xpath be aware that webdriver follows standard conventions: a search prefixed with "//" will search the entire document, not just the children of this current node. Use ".//" to limit your search to the children of this WebElement.

So you should be able to do something like:

WebElement foo = driver.findElement(By.xpath("//*[@title='" + title + "']"));
WebElement bar = foo.findElement(By.xpath(".//img"));

The leading dot in the second xpath makes all the difference, as it will search only for child elements of foo.

Solution to Example 1

If you load the website, you will observe that the text above the search editbox initially looks like:

After sometime, the same element's text gets changed to:

Only after this happens, try performing the sendkeys operation. Note that you don't have to perform the sendkeys operation on the element with id = 'search-input'. You first need to click on that element and then another element(cssSelector -> div[role='combobox'] [placeholder='Search by address, Neighbourhood, MLS #']) becomes visible on which you have to perform the sendkeys operation.

Please refer the code below for the same(tested on chrome, edge and firefox).

package usecase;

import java.time.Duration;
import java.util.regex.Pattern;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import io.github.bonigarcia.wdm.WebDriverManager;

public class Flexcase {
    public static WebDriver driver;
    private static final int maxtime = 20;

    /**
     * Creating a custom findElement method that uses Explicit wait and waits for an
     * element to be clickable
     */
    public static WebElement findElement(By by) {
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(maxtime));
        return wait.until(ExpectedConditions.elementToBeClickable(by));
    }

    /**
     * Waits for an element to contain a text matched by the given pattern(regex)
     */
    public static Boolean doesElementContainTextPattern(By by, String pattern) {
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(maxtime));
        return wait.until(ExpectedConditions.textMatches(by, Pattern.compile(pattern)));
    }

    public static void main(String[] args) throws InterruptedException {
        WebDriverManager.chromedriver().setup();
        driver = new ChromeDriver();
        driver.manage().window().maximize();
        driver.get("https://condos.ca/");
        // Waiting for the text to achieve a specific pattern - Search followed by 1+
        // occurrences of either digits or commas followed by 0+ occurrences of any
        // character that is not a newline character followed by the word Listings
        if (doesElementContainTextPattern(By.className("dPFjpJ"), "Search [\\d,]+.*Listings")) {
            findElement(By.id("search-input")).click();
            findElement(By.cssSelector("div[role='combobox'] [placeholder='Search by address, Neighbourhood, MLS #']"))
                    .sendKeys("Test");
        }

        Thread.sleep(10000); // Pls let me see what's happening on the screen
        driver.quit();
    }
}

Demo:

Solution for Example 2

Instead of using javascriptexecutor, I have used the Actions class to build a composite action made up by following actions:

• move the mouse to leftmost position of the slider by finding the slider width in pixels
• move the mouse to the desired position on the slider proportiante to the percentage value passed

Code:(sliding accuracy can be improved)

package usecase;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.interactions.Actions;

import io.github.bonigarcia.wdm.WebDriverManager;

public class Slider {

    public static WebDriver driver;

    public static void moveSliderToPosition(By by, int percentage) throws InterruptedException {
        WebElement slider = driver.findElement(by);
        Actions builder = new Actions(driver);
        builder.moveToElement(slider, (int) (-slider.getRect().width / 2), 0) // move to the leftmost position of the
                                                                                // element
                .moveByOffset((int) (percentage * slider.getRect().width / 100), 0) // move the cursor by an offset
                                                                                    // proportionate to the value passed
                .click().build().perform();
        Thread.sleep(3000); // let me see what has happened on the screen
    }

    public static void main(String[] args) throws InterruptedException {
        WebDriverManager.chromedriver().setup();
        driver = new ChromeDriver();
        driver.manage().window().maximize();
        driver.get("https://demoqa.com/slider");
        By slider = By.cssSelector("input.range-slider");
        moveSliderToPosition(slider, 47);
        moveSliderToPosition(slider, 83);
        moveSliderToPosition(slider, 35);
        driver.quit();
    }
}

Demo: