How to use generate_as_json method of Selenium.WebDriver Package

The methods to set the execution speed in WebDriver were deprecated for all language bindings some time ago. It is no longer possible to modify the execution speed of the running WebDriver code.

First solution is to run your scraper in the same process as Rails, but in a different thread:

require 'C:\Users\Admin\Dropbox\dev\watir-scripts\ds-vin.rb'
thr = Thread.new do 
    b = Watir::Browser.new :ie
    ...
end

Second is to fix your environment if you are going to use Process.spawn - assuming your gems are installed system-wide you can do something like

ENV['GEM_PATH'] = "c:/Ruby220/lib/ruby/gems/2.2.0"

at the top of your ds-vin.rb script. To find the exact gem path you should use, make sure you run gem which watir-webdriver and gem environment and copy from there.

Here is some hints to your problem:

begin

    wait = Selenium::WebDriver::Wait.new(:timeout => 10) # seconds
    wait.until { driver.title.include? "page title" }
    driver.find_element(:xpath, ".//*[@id='subTabHeaders']/div[3]")}.click

rescue Selenium::WebDriver::Error::TimeOutError

    exit(1)

end

You can use Selenium to interact with the webpage - fill form fields, click buttons etc. You can even execute your own javascript code.

Selenium cheat sheet

Edit:

Using pry to stop the code execution so you can manually manipulate the web page.

# Code for starting Selenium session and opening the web page
...

# Use pry to stop the code execution.
# Resume the program using command 'exit' in the pry context
require 'pry'; binding.pry

# Code to get results after you manually manipulate the web page
...

Separate Databases.

Not sure how you are running your tests in parallel. Parallel tests should each have their own databases, otherwise, yes, you'll get massive data conflicts.

If your parallel test runners are sharing the same database and you call delete_all on one of your runners, it will cause the other runners to fail because their test databases (which are all the same) will now be empty.

Gems like parallel_tests use separate databases for each test runner by suffixing the test database name with the runner number, e.g. test_database1, test_database2, etc.

So, you should use one of these pre-built libraries that helps take care of this but if you're rolling your own, then make sure you run each runner in a separate database to avoid database conflicts.

This fixed it for me. I added the bottom portion of this snippet to my spec_helper.rb after registering the remote driver.

Capybara.register_driver :selenium do |app|
  # ...

  Capybara::Selenium::Driver.new(app,
                                 browser: :remote,
                                 url: url,
                                 desired_capabilities: capabilities)
end

# Add file upload capability on remote driver
page.driver.browser.file_detector = lambda do |args|
  str = args.first.to_s
  str if File.exist?(str)
end

Be sure to only do this when using a remote driver otherwise the regular, local selenium driver won't know how to handle the file_detector method.

You could get the window height, as well as the body content height, to approximate how many screenshots you need. I say "approximate" since the methods for getting the heights have various caveats, eg handling scrollbars. Unless you really need pixel perfect screenshots, this approximation should be good enough.

# Determine how many screenshots are needed to get all of the content
window_height, content_height = b.execute_script('return [window.innerHeight, document.body.clientHeight];')
screenshot_pieces = content_height / window_height.to_f

# Always take a screenshot of the top
b.scroll.to :top
b.screenshot.save('top.png')

# If there are more than 2 pieces, take a screenshot of the center
if screenshot_pieces > 2 
  b.scroll.to :center
  b.screenshot.save('center.png')
end

# If there are more than 1 piece, take a screenshot of the bottom
if screenshot_pieces > 1
  b.scroll.to :bottom
  b.screenshot.save('bottom.png')
end

You can tweak the above conditions to your liking. For example, if you don't want your screenshots to have overlapping parts (ie don't care about fractional parts of the page), you can change the conditions to screenshot_pieces >= 3 and screenshot_pieces >= 2 respectively.

The error is thrown because i.attribute('Andorra') returned nil which ruby failed to convert to a string. Here are a few examples that should get you what you want:

# print the name attribute
puts 'Name is: %s' % i.attribute('name')

# print the value attribute
puts 'Value is: %s' % i.attribute('value')

# print the text content
puts 'Text is: %s' % i.text

The XPath suggested by Saurabh Gaur, can be written in a more readable Watir-like fashion using:

$browser.ul(class: 'dropdown-menu').link(text: 'Mr').click

Note that this assumes that there is only one ul element with class dropdown-menu. If there are multiple, you will need to scope the search to the specific dropdown using an element that likely exists higher in the DOM.

However, given there is likely only one link with text "Mr", you can probably get away with simply:

$browser.link(text: 'Mr').click

Given the link is a dialog that switches from hidden to visible, you may need to also wait:

$browser.link(text: 'Mr').when_present.click