How to use prepareToInterceptTraffic method of org.openqa.selenium.devtools.idealized.Network class

I would argue storing the files in the page class itself. Loading from properties file would incur additional overhead or parsing a large file. Maintaining such a file would also be harder, even with a good tool support you would be forced to use CTRL + F more than you should.

Even on a more conceptual level it feels wrong. A good fit for storing in properties files are configurable parameters, and especially the ones that are good candidates to be tweaked in runtime.

Locators don't have this nature. If the benefit you're seeking is declaring in a central place you should instead use dedicated constant class, which will give you much better refactoring options.

Simply use single quotes instead of double quotes, it should work.

String cssFormatString = "div[aria-label='2018-10-17 Shared Google Drive Folder']"

So your string concatination will be simple,

String cssFormatString = "div[aria-label='" + strDate +  " Shared Google Drive Folder']"

First the update ¹

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.

Preventing Detection ²

To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:

• Adding the argument --disable-blink-features=AutomationControlled

  from selenium import webdriver
  
  options = webdriver.ChromeOptions() 
  options.add_argument('--disable-blink-features=AutomationControlled')
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get("https://www.website.com")
  

You can find a relevant detailed discussion in Selenium can't open a second page

• Rotating the user-agent through execute_cdp_cmd() command as follows:

  #Setting up Chrome/83.0.4103.53 as useragent
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
  

• Change the property value of the navigator for webdriver to undefined

  driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
  

• Exclude the collection of enable-automation switches

  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  

• Turn-off useAutomationExtension

  options.add_experimental_option('useAutomationExtension', False)
  

Sample Code ³

Clubbing up all the steps mentioned above and effective code block will be:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

History

As per the W3C Editor's Draft the current implementation strictly mentions:

The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.

Further,

Navigator includes NavigatorAutomationInformation;

It is to be noted that:

The NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.

The NavigatorAutomationInformation interface is defined as:

interface mixin NavigatorAutomationInformation {
    readonly attribute boolean webdriver;
};

which returns true if webdriver-active flag is set, false otherwise.

Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.

Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.

Update (6-Nov-2019)

As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions() class to add a couple of arguments to:

• Exclude the collection of enable-automation switches
• Turn-off useAutomationExtension

through an instance of ChromeOptions as follows:

• Java Example:

  System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
  ChromeOptions options = new ChromeOptions();
  options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
  options.setExperimentalOption("useAutomationExtension", false);
  WebDriver driver =  new ChromeDriver(options);
  driver.get("https://www.google.com/");
  

• Python Example

  from selenium import webdriver
  
  options = webdriver.ChromeOptions()
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
  driver.get("https://www.google.com/")
  

• Ruby Example

    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("--disable-blink-features=AutomationControlled")
    driver = Selenium::WebDriver.for :chrome, options: options
  

Legends

¹: Applies to Selenium's Python clients only.

²: Applies to Selenium's Python clients only.

³: Applies to Selenium's Python clients only.

It works, but it's not very useful, because I want to make an external .js which contains all the JavaScript functions and call them from there, not in a String.

You can achieve this only by loading your external js file into the DOM

var addscript=window.document.createElement('script');addscript.type='text/javascript';addscript.src='http://localhost/somescript.js';document.getElementsByTagName('body')[0].appendChild(addscript);

Note : Most browsers do not allow you to load local resources so put your external js file in local webserver and then access it like http://localhost/somescript.js

After loading the js file into DOM now you can call the javascript functions in the external js file

Example

Lets say we have a external js file named somescript.js which contains the below function

//simple function which sets the value "test" to the search box

window.somefunc = function () {document.getElementsByName("s")[0].value='test';}

Webdriver code :

     driver.get("http://www.jquery.com");

     //Load the External js file into DOM

     ((JavascriptExecutor) driver)
      .executeScript("var addscript=window.document.createElement('script');addscript.type='text/javascript';addscript.src='http://localhost/somescript.js';document.getElementsByTagName('body')[0].appendChild(addscript);");

     //wait for the js to be loaded to the DOM

     ((JavascriptExecutor) driver)
      .executeScript("return typeof(somefunc)").toString().equals("function");


     //Now you call the JavaScript functions in the JS file

     ((JavascriptExecutor) driver)
      .executeScript("somefunc();");

Note : Behind the scenes Selenium wraps your JavaScript code in an anonymous function. So your somefunc function is local to this anonymous function.And due to JavaScript's scoping rules, somefunc doesn't exist outside of that anonymous function. so we have made it a global function by assigning it to window.

EDIT :

And I don't really understand why you use the window statement. And I was searching something like ((JavascriptExecutor) driver).executeScript("here the .js"); But I don't know if it is possible

This is how executeScript method executes the provided javascript

The script fragment provided will be executed as the body of an anonymous function.

Example if we used the below code

((JavascriptExecutor) driver)
      .executeScript("somefunc = function () {document.getElementsByName("s")[0].value='test';}");

((JavascriptExecutor) driver)
      .executeScript("somefunc();");

(function() {
        somefunc = function () {document.getElementsByName("s")[0].value='test';}
    })();

    (function() {
        somefunc();
    });

What do you mean where you say that you want to put the external .js into the DOM?

By DOM i mean Document object model of the page constructed as a tree of Objects (in short your webpage).We use javascript to load the external js to the webpage and then call the functions in the js file and execute them(like in the above example).

In the code that you put in your edit. Both functions are the same?

I just gave an example by that what i meant was each script provided in execute script will be executed in the body of an anonymous function.In our case we haven't used executescript to create the somefunc function rather used it from the external js file in dom we only called it using the executescript method so you can do it with or without the window object

  //simple function which sets the value "test" to the search box

somefunc = function () {document.getElementsByName("s")[0].value='test';}//this will also work

Hope this helps you.Kindly get back if you have any queries.

Now there's a native solution for that, the or method, check the doc.

You use it like so:

driverWait.until(ExpectedConditions.or(
    ExpectedConditions.presenceOfElementLocated(By.cssSelector("div.something")),
    ExpectedConditions.presenceOfElementLocated(By.cssSelector("div.anything"))));

Go for xpath text based search. It allows you to find an element based on the text

// with * we are doing tag indepenedent search. If you know the tag, say it's a `div`, then //div[contains(text(),'Text To find')] can be done
By byXpath = By.xpath("//*[contains(text(),'Text To find')]"); 
WebElement myDynamicElement = (new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(byXpath));

Try this:

 driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

The browser cannot upload a file without an <input> element, unless the file is dropped from the desktop. It would be a security breach to be able to upload a file by code.

So in your case, the<input> is probably created once the user has clicked the link.

One way to handle this case is to silence the click event, click the link and then set the file to the <input>:

// disable the click event on an `<input>` file
((JavascriptExecutor)driver).executeScript(
    "HTMLInputElement.prototype.click = function() {                     " +
    "  if(this.type !== 'file') HTMLElement.prototype.click.call(this);  " +
    "};                                                                  " );

// trigger the upload
driver.findElement(By.id("Dialogs_Dialogs_lbAttachUpload"))
      .click();

// assign the file to the `<input>`
driver.findElement(By.cssSelector("input[type=file]"))
      .sendKeys("filepath\\Test PDF.pdf");

Note that you may also need to wait for the <input> to be created.

Element.scrollIntoView()

Element.scrollIntoView() method scrolls the element on which it's called into the Viewport of the browser window.

Syntax

• element.scrollIntoView()
• element.scrollIntoView(alignToTop) // Boolean parameter
• element.scrollIntoView(scrollIntoViewOptions) // Object parameter

Parameters

The parameters for this method are:

• alignToTop (Optional): Is a Boolean value, if true, the top of the element will be aligned to the top of the visible area of the scrollable ancestor. This is the default value. If false, the bottom of the element will be aligned to the bottom of the visible area of the scrollable ancestor. Corresponds to scrollIntoViewOptions: {block: "end", inline: "nearest"}.
• scrollIntoViewOptions (Optional): Is an Object with the following properties:
• behavior (Optional): Defines the transition animation. One of "auto" or "smooth". Defaults to "auto".
• block (Optional): Defines vertical alignment. One of "start", "center", "end", or "nearest". Defaults to "start".
• inline (Optional): Defines horizontal alignment. One of "start", "center", "end", or "nearest". Defaults to "nearest".

This usecase

As per your line of code:

((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element);

the argument true refers to boolean value for alignToTop. Hence the issue.

Solution

To automate horizontal scrolling you need to pass the argument for the inline parameter either among the following:

• start
• center
• end
• nearest

Alternative

As an alternative you can use either of the following options

• scrollLeft(): Element.scrollLeft() property gets or sets the number of pixels that an element's content is scrolled from its left edge. If the element's direction is rtl (right-to-left), then scrollLeft is 0 when the scrollbar is at its rightmost position (at the start of the scrolled content), and then increasingly negative as you scroll towards the end of the content.
• scrollWidth(): Element.scrollWidth() read-only property is a measurement of the width of an element's content, including content not visible on the screen due to overflow.

Outro

You can find a couple of relevant detailed discussion in:

• What is the difference between the different scroll options?
• Selenium python Error: element could not be scrolled into view