How to use getKind method of org.openqa.selenium.devtools.idealized.log.model.LogEntry class

You need random string generator. This answer I stole from here.

protected String getSaltString() {
        String SALTCHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
        StringBuilder salt = new StringBuilder();
        Random rnd = new Random();
        while (salt.length() < 10) { // length of the random string.
            int index = (int) (rnd.nextFloat() * SALTCHARS.length());
            salt.append(SALTCHARS.charAt(index));
        }
        String saltStr = salt.toString();
        return saltStr;

    }

Call it as getSaltString()+"@gmail.com" in you code

If anyone actually knew a general and always-applicable answer, it would have been implemented everywhere ages ago and would make our lives SO much easier.

There are many things you can do, but every single one of them has a problem:

1. As Ashwin Prabhu said, if you know the script well, you can observe its behaviour and track some of its variables on window or document etc. This solution, however, is not for everyone and can be used only by you and only on a limited set of pages.

2. Your solution by observing the HTML code and whether it has or hasn't been changed for some time is not bad (also, there is a method to get the original and not-edited HTML directly by WebDriver), but:

   • It takes a long time to actually assert a page and could prolong the test significantly.
   • You never know what the right interval is. The script might be downloading something big that takes more than 500 ms. There are several scripts on our company's internal page that take several seconds in IE. Your computer may be temporarily short on resources - say that an antivirus will make your CPU work fully, then 500 ms may be too short even for a noncomplex scripts.
   • Some scripts are never done. They call themselves with some delay (setTimeout()) and work again and again and could possibly change the HTML every time they run. Seriously, every "Web 2.0" page does it. Even Stack Overflow. You could overwrite the most common methods used and consider the scripts that use them as completed, but ... you can't be sure.
   • What if the script does something other than changing the HTML? It could do thousands of things, not just some innerHTML fun.

3. There are tools to help you on this. Namely Progress Listeners together with nsIWebProgressListener and some others. The browser support for this, however, is horrible. Firefox began to try to support it from FF4 onwards (still evolving), IE has basic support in IE9.

And I guess I could come up with another flawed solution soon. The fact is - there's no definite answer on when to say "now the page is complete" because of the everlasting scripts doing their work. Pick the one that serves you best, but beware of its shortcomings.

I believe you can do this with Python, however it isn't packaged as simply as a FluentWait class. Some of this was covered in the documentation you provided by not extensively.

The WebDriverWait class has optional arguments for timeout, poll_frequency, and ignored_exceptions. So you could supply it there. Then combine it with an Expected Condition to wait for elements for appear, be clickable, etc... Here is an example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import *

driver = webdriver.Firefox()
# Load some webpage
wait = WebDriverWait(driver, 10, poll_frequency=1, ignored_exceptions=[ElementNotVisibleException, ElementNotSelectableException])
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//div")))

Obviously you can combine the wait/element into one statement but I figured this way you can see where this is implemented.

There are 2 way to do this that will give you meaningful data.

1. Use Browsermob Proxy with Selenium. This is an example in python but its pretty much the same in Java

   from browsermobproxy import Server
   server = Server("path/to/browsermob-proxy")
   server.start()
   proxy = server.create_proxy()
   
   from selenium import webdriver
   profile  = webdriver.FirefoxProfile()
   profile.set_proxy(proxy.selenium_proxy())
   driver = webdriver.Firefox(firefox_profile=profile)
   
   proxy.new_har("google")
   driver.get("http://www.google.co.uk")
   proxy.har # returns a HAR JSON blob
   
   proxy.stop()
   driver.quit()
   

The HAR file that is returned from proxy.har, which is just a JSON blob, will give you the information that you need. I blogged about it earlier this year

2. The other approach is to use the navigations timings spec available in modern browsers. All that you need to do is execute some javaScript and you will get details of page load etc.

   ((JavascriptExecutor)driver).executeScript("var performance = window.performance || {};" + 
               "var timings = performance.timing || {};"+
               "return timings;");
   
   /* The hashmap returned will contain something like the following.
    * The values are in milliseconds since 1/1/1970
    *
    * connectEnd: 1280867925716
    * connectStart: 1280867925687
    * domainLookupEnd: 1280867925687
    * domainLookupStart: 1280867925687
    * fetchStart: 1280867925685
    * legacyNavigationStart: 1280867926028
    * loadEventEnd: 1280867926262
    * loadEventStart: 1280867926155
    * navigationStart: 1280867925685
    * redirectEnd: 0
    * redirectStart: 0
    * requestEnd: 1280867925716
    * requestStart: 1280867925716
    * responseEnd: 1280867925940
    * responseStart: 1280867925919
    * unloadEventEnd: 1280867925940
    */ 
   

Many GWT elements comes with ensureDebugId (method on UIObject) to allow you to explicitly set Ids to elements for testing and debugging purpose. You also need to inherit the module

 <inherits name="com.google.gwt.user.Debug"/> 

to make it work. The advantage of this is, you can remove the trace from the production deployment by removing the inherited module in during prod mode compile. Hence there wont be code changes to remove unnecessary Ids.

Using an explicit wait worked for me (tried both chrome and firefox):

WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement element = wait.until(
    ExpectedConditions.visibilityOfElementLocated(By.id("ctl62_txtSearchLastName"))
);

element.sendKeys("test");

Resulted into:

You will want to write a custom ExpectedCondition that queries angular to see if it is done. Here's one I prepared earlier:

    public static ExpectedCondition angularHasFinishedProcessing() {
        return new ExpectedCondition<Boolean>() {
            @Override
            public Boolean apply(WebDriver driver) {
                String hasAngularFinishedScript = "var callback = arguments[arguments.length - 1];\n" +
                        "var el = document.querySelector('html');\n" +
                        "if (!window.angular) {\n" +
                        "    callback('false')\n" +
                        "}\n" +
                        "if (angular.getTestability) {\n" +
                        "    angular.getTestability(el).whenStable(function(){callback('true')});\n" +
                        "} else {\n" +
                        "    if (!angular.element(el).injector()) {\n" +
                        "        callback('false')\n" +
                        "    }\n" +
                        "    var browser = angular.element(el).injector().get('$browser');\n" +
                        "    browser.notifyWhenNoOutstandingRequests(function(){callback('true')});\n" +
                        "}";

                JavascriptExecutor javascriptExecutor = (JavascriptExecutor) driver;
                String isProcessingFinished = javascriptExecutor.executeAsyncScript(hasAngularFinishedScript).toString();

                return Boolean.valueOf(isProcessingFinished);
            }
        };
    }

This is assuming that your have defined ng-app on your html element. If you define it elsewhere in the DOM you will need to update the code above. This is pretty much stolen from protractor, it's checking to see if there are any outstanding ajax requests and checking to see if angular thinks it's in a testable state.

The important part is to execute this JavaScript snippet as an async script because angular is using promises to trigger a callback when it is ready. If you don't run this as an async script it will not work.

You will also need to set a script timeout on the driver object, this is the amount of time selenium will wait for the callback to complete before throwing a timeout exception. You can set it as follows:

driver.manage().timeouts().setScriptTimeout(15, TimeUnit.SECONDS);

It's then as simple as plugging this ExpectedCondition into a wait like so:

WebDriverWait wait = new WebDriverWait(driver, 15, 100);
wait.until(angularHasFinishedProcessing());

Hope this helps.

I have checked the div structure and the way linkedin is showing the results. So, if you hit the url directly and check the by following xpath: //li[contains(@class,'search-result')] You would find out that all the results are already loaded on the page, but linkedin are showing only 5 results in one go and on scrolling, it shows the next 5 results, however all the results are already loaded on the page and can be found out by the mentioned xpath.

Refer to this image which highlights the div structure and results when you find the results on entering the xpath on hitting the url: https://imgur.com/Owu4NPh and
Refer to this image which highlights the div structure and results after scrolling the page to the bottom and then finding the results using the same xpath: https://imgur.com/7WNR830

You could see the result set is same however there is an additional search-result__occlusion-hint part in the < li > tag in the last 5 results and through this linkedin is hiding the next 5 results and showing only the first 5 results on the first go.

Now comes the implementation part, i have checked "Next" button comes only when you scroll through whole results on the page, so instead of scrolling to a definite coordinates because that can be changed for different screensizes and windows, you can take the results in a list of webelement and get it's size and then scroll to the last element of that list. In this case, if there are total 10 results then the page will be scrolled to the 10th results and if there are only 4 results then the page will be scrolled to the 4th result and after scrolling you can check if the Next button is present on the page or not. For this, you can check the list size of the "Next" button web element list, if the list size is greater than 0, it means the next button is present on the page and if its not greater than 0, that means the Next button is not present on the list and you can stop your execution there.

So to implement it, i have taken a boolean which has an initial value as true and the code will be run in a loop till that boolean becomes false and it will become false when the Next button list size becomes equal to 0.

Please refer to the code below:

public class App 
{    
    WebDriver driver;

  // For initialising javascript executor
  public Object executeScript(String script, Object... args) {
    JavascriptExecutor exe = (JavascriptExecutor) driver;
    return exe.executeScript(script, args);
  }

  // Method for scrolling to the element
  public void scrollToElement(WebElement element) {
    executeScript("window.scrollTo(arguments[0],arguments[1])", element.getLocation().x, element.getLocation().y);

    }

  public static void main(String[] args) {
    // You can change the driver to bot according to your usecase
    driver = new FirefoxDriver();
    // Add your direct URL here and perform the login after that, if necessary
    driver.get(url);
    // Wait for the URL to load completely
    Thread.sleep(10000);
    // Initialising the boolean
    boolean nextButtonPresent = true;
    while (nextButtonPresent) {
        // Fetching the results on the page by the xpath
        List<WebElement> results = driver.findElements(By.xpath("//li[contains(@class,'search-result')]"));
        // Scrolling to the last element in the list
        scrollToElement(results.get(results.size() - 1));
        Thread.sleep(2000);

        // Checking if next button is present on the page
        List<WebElement> nextButton = driver.findElements(By.xpath("//button[@class='next']"));
        if (nextButton.size() > 0) {
            // If yes then clicking on it
            nextButton.get(0).click();
            Thread.sleep(10000);
        } else {
            // Else setting the boolean as false
            nextButtonPresent = false;
            System.out.println("Next button is not present, so ending the script");
        }
      }
   }
}

My guess is that what you are seeing is due to a maven dependency issue. I ran into a similar problem at one point where the Chrome WebDriver would just crash due to a NumberFormatException whenever we tried to access it's cookies. The root there was an incompatible transitive dependency (e.g. it needed a particular version of a dependency and our project was pulling in an older version and maven thought it was all ok). In our case the fix was to explicitly add the dependency on the correct version in our test project that was running the selenium tests. Unfortunately, I can't be more clear than that since I don't know the maven dependency structure of the projects in question. Try looking at:

mvn dependency:tree

and then check the versions of the web driver dependencies.

Also it looks like you are using chrome web driver 2.21 and chrome 51. The current chrome web driver is 2.22 and the release notes state it fixes a lot of issues with chrome 51

https://sites.google.com/a/chromium.org/chromedriver/downloads

Edit (adding): Also, not sure if the code execution is getting to perform any code at all, but it sounds like that exception can happen if the dev tools are open as that will disconnect the Chrome WebDriver from the Chrome browser. getting selenium error - disconnected: received Inspector.detached event using chrome driver version 2.20.x