How to use encoded method of ClassMethods Package

Apart from mentioning which folder to use for download, you need to tell Firefox that it needs to use custom download location. That you can do using the code below:-

profile['browser.download.folderList'] = 2
profile['browser.download.dir'] = download_dir

I know I am a bit late, but I had a similar situation and I found how to overcome this. The answer is also here but for convenience, I write it below. So when you open chrome from selenium the --disable-sync argument is by default attached to the instance. So you have to exclude it first like this:

options.add_experimental_option(
    'excludeSwitches',
    ['disable-sync']) 

and then to re-enable it like this:

options.add_argument('--enable-sync')

Note that in order to have something to sync you need to run selenium with a user profile using the --user-data-dir=PATH-TO-PROFILE argument.

Selenium is not one of the best options to use to attempt to build a "web-crawler". It can be too flakey at times, especially when it comes across unexpected scenarios. Selenium WebDriver is a great tool for automating and testing expectancies and user interactions. Instead, good old fashioned curl would probably be a better option for web-crawling. Also, I am pretty sure there are some ruby gems that might help you web-crawl, just Google search it!

But To answer the actual question if you were to use Selenium WebDriver:

I'd work out a filtering algorithm where you can add the HTML of an element that you interact with to an variable array. Then, when you go on to the next window/tab/link, it would check against the variable array and skip the element if it finds a matching HTML value.

Unfortunately, SWD does not support getting request headers and responses with its API. The common work-around is to use a third party proxy to intercept the requests.

============

Now I'd like to address a few issues with your code.

I'd suggest before iterating over the links, add a @default_current_window = @driver.window_handle. This will allow you to always return back to the correct window at the end of your script when you call @driver.switch_to.window(@default_current_window).

In your @links iterator, instead of iterating over all the possible windows that could be displayed, use @driver.switch_to.window(@driver.window_handles.last). This will switch to the most recently displayed new window (and it only needs to happen once per link click!).

You can DRY up your inputs and form code by doing something like this:

inputs = []
inputs << @driver.find_elements(:tag_name => "input")
inputs << @driver.find_elements(:tag_name => "form")
inputs.flatten
inputs.each do |i|
  begin
    i.send_keys "value"
    i.submit
  rescue e
    puts "ERROR: #{e.message}"
  end
end

Please note how I just added all of the elements you wanted SWD to find into a single array variable that you iterate over. Then, when something bad happens, a single rescue is needed (I assume you don't want to automatically quit from there, which is why you just want to print the message to the screen).

Learning to DRY up your code and use external gems will help you achieve a lot of what you are trying to do, and at a faster pace.

You could do this two ways in Ruby. One is to follow @Saifur's solution, which would look like this:

# Returns true if no element is found
#
webdriver.find_elements(:xpath, element).empty?

The other way is to rescue the exception:

# Returns false if no element is found
#
webdriver.find_element(:xpath, element) rescue false

I'd suggest Saifur's solution.

Simple:

Before do
  Capybara::Selenium::Driver.new(app, :browser => :firefox, :profile => profile)
end

rmagick can be used to compare two images for identical pixels, using the compare_channel method:

img1 =  Magick::Image.read('image1.png')
img2 =  Magick::Image.read('image2.png')

diff_img, diff_metric  = img1[0].compare_channel( img2[0], Magick::MeanSquaredErrorMetric )

# diff_img is a cool feature - an image showing which pixels are different

# diff_metric == 0.0 for "no difference"

See #compare_channel for more details.

You can set --start-maximizedby following for Chrome. See this post for more details.

Capybara.register_driver :chrome_maximize do |app|
  caps = Selenium::WebDriver::Remote::Capabilities.chrome( 
    'chromeOptions' => {
      "args" => [ "--start-maximized", "--otherthings" ]
    } 
  )

  $driver = Capybara::Selenium::Driver.new(app, {:browser => :chrome, :desired_capabilities => caps})
end

How about you try locating the First link using CSS selectors, something like this:

driver.find_element(:css, "#rso li:nth-child(1) div > h3 > a").click

where the 1 in the brackets (after nth-child) refers to the first search result.

Also I may be wrong, but try :link_text instead of :link, something like this :

resultlink = driver.find_element(:link_text, "Google Translate")
resultlink.click

I used driver.action.move_to(el).perform which differs ever so slightly from the other answers, so I thought I would include it for completeness sake.