How to use FindAll.FindByBuilder class of org.openqa.selenium.support package

You can do it by using Chrome DevTools Protocol, and here is the python code:

driver = webdriver.Chrome()
tz_params = {'timezoneId': 'America/New_York'}
driver.execute_cdp_cmd('Emulation.setTimezoneOverride', tz_params)

In order to use ChromeDriver undetectable to Distil checkpoints (which are described nicely in this stackoverflow post), you will need to ensure that your browser does not contain any variable in its window or document prototypes that reveal that you are using a webdriver, as the one you mention.

You can use software as Selenium along with ChromeDriver and Chrome as long as you take some precautions and make some fixes to the binaries. This method will apply only to headed version, if you wish to use headless, you would need to take additional measurements to pass window/rendering tests, described here.

1. Fix Chrome binary, or use an old version

First, lets deal with that navigator.webdriver set to True. This is defined by W3C protocol here as part of 'NavigatorAutomationInformation' of browsers, which extends the Navigator interface. How to remove it? That project has a lot of files, third party stuff, blink web runtimes, etc. So, instead of becoming crazy trying to figure out how this works, as Chromium is open-source, just be clever and search google for the commit which incorporated that. Here is the link. Pay attention to these files:

• third_party/WebKit/Source/core/frame/Navigator.h, which holds the line of code:

   `bool webdriver() const { return true; }` 
  
  This method is supossed to always return true, as you can see.
  

• third_party/WebKit/Source/core/frame/Navigator.idl, which contains the extensions of Navigators, included our

  `Navigator implements NavigatorAutomationInformation;` 
  
  which is being commited. Interesting, isn't it?
  

• third_party/WebKit/Source/core/frame/NavigatorAutomationInformation.idl contains the extension itself, with a read-only variable, which is webdriver:

       `[
           NoInterfaceObject, // Always used on target of 'implements'
           Exposed=(Window),
           RuntimeEnabled=AutomationControlled
       ] interface NavigatorAutomationInformation {
           readonly attribute boolean webdriver;
       };`
  

To get rid of this functionality, it should be enough commenting the line in Navigator.idl which extends Navigator with this functionality, and compile the source (compiling in linux here). However, this is a laborious task for almost any computer and can take several hours. If you look the date of the commit, it was on Oct 2017, so an option is to download any version of Chrome released before that date. To search for mirrors, you can google for inurl:/deb/pool/main/g/google-chrome-stable/.

2. Fix ChromeDriver

Distil checks the regex rule '/\$[a-z]dc_/' against window variables, and ChromeDriver adds one as mentioned here which satisfies that condition. As they mention, you have to edit call_function.js amongst the source code, and redefine the variable var key = '$cdc_asdjflasutopfhvcZLmcfl_';. with something else. Also, probably easier, you can use an hex editor to update the existing binary.

If you decided to use an older version of Chrome -I guess you did-, you will need to use an appropiate version of ChromeDriver. You can know which one is fine for your Chrome version in the ChromeDriver downloads webpage. For example, for Chrome v61 (which fits your needs), you could use ChromeDriver 2.34. Once done, just put the ChromeDriver binary on '/usr/bin/local'.

3. Take other precautions

• Pay attention to your user-agent.
• Don't perform too many repeated requests.
• Use a (random) delay between requests.
• Use the Chrome arguments used here to mimic a normal user profile.

Check the below links:

• Implicit Wait - It instructs the web driver to wait for some time by poll the DOM. Once you declared implicit wait it will be available for the entire life of web driver instance. By default the value will be 0. If you set a longer default, then the behavior will poll the DOM on a periodic basis depending on the browser/driver implementation.

• Explicit Wait + ExpectedConditions - It is the custom one. It will be used if we want the execution to wait for some time until some condition achieved.

This is what worked for me using Explicit Wait from here WebDriver: Advanced Usage

public void checkAlert() {
    try {
        WebDriverWait wait = new WebDriverWait(driver, 2);
        wait.until(ExpectedConditions.alertIsPresent());
        Alert alert = driver.switchTo().alert();
        alert.accept();
    } catch (Exception e) {
        //exception handling
    }
}

Each time, before you change page, retrieve the text, i.e, the number of records. Then, Click on the Next button to navigate to the next page. And then, wait for that to be invisible using explicit wait.

Below code might help you out:

String retrieved_text = driver.findElement(By.xpath("//xpath of the element related to records")).getText(); //Retrieving text, i.e., the innerHtml representing number of records

/*Click on the next button to navigate to the next page*/
driver.findElement(By.xpath("//next button's xpath")).click();

WebDriverWait wait = new WebDriverWait(driver,30);
wait.until(ExpectedConditions.invisibilityOfElementWithText(By.xpath("//xpath of the element related to records"), retrieved_text)); 

NOTE: In the last line of code use xpath that locates the element that shows text 'displaying 1-10 of 2100', i.e, the element that shows display per page information and use the retrieve text to infer that the concerned element with that certain text is invisible or not in the next page.

This will help in identifying the invisibility of the concerned element easily.

OTHER WAY: In case you want to navigate to next pages till you detect a certain element

(PUT EVERYTHING IN A DO-WHILE LOOP)

• Wait for the element to appear in the page.
• If found, then write code to act on the element and break out of loop.
• Else, click on the Next button
• CONDITION will be till the NEXT button gets disabled.

What is happening here is the creation of an anonymous class that inherits from ExpectedCondition. In the body of this class he is then overriding the method apply(...).

You can also check pageloaded using following code

IWait<IWebDriver> wait = new OpenQA.Selenium.Support.UI.WebDriverWait(driver, TimeSpan.FromSeconds(30.00));

 wait.Until(driver1 => ((IJavaScriptExecutor)driver).ExecuteScript("return document.readyState").Equals("complete"));

File pathToBinary = new File("C:\\user\\Programme\\FirefoxPortable\\App\\Firefox\\firefox.exe");
FirefoxBinary ffBinary = new FirefoxBinary(pathToBinary);
FirefoxProfile firefoxProfile = new FirefoxProfile();       
WebDriver driver = new FirefoxDriver(ffBinary,firefoxProfile);

Please Find the below code, you can scroll and take screenshots as many as you want. Note the elements Webelement. Store them and scroll relatively. You can Scroll depending upon how many screenshots you want.

driver.get("http://www.bbc.com");       
driver.manage().window().maximize();
System.out.println(driver.getTitle());
File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(scrFile, new File("D:/chromedriver/scr3.png"));  
WebElement elements = driver.findElement(By.xpath(".//*[@id='page']/section[6]/div/div/div[1]/ul/li[3]/div/div[2]/h3/a"));    
Thread.sleep(3000L);
JavascriptExecutor js = (JavascriptExecutor) driver;
int yPosition = elements.getLocation().getY();
js.executeScript("window.scroll (0, " + yPosition + ") ");       
Thread.sleep(3000L);         
File scrFile1 = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(scrFile1, new File("D:/chromedriver/scr4.png"));
driver.close();