Develop Reference

Selenium won't work unless I actually look at the Web page (perhaps anti-crawler mechanism by JavaScript?)

The following code works fine ONLY when I look at the Web page (aka the Chrome tab being manipulated by Selenium).
Is there a way to make it work even when I'm browsing another tab/window?
(I wonder how the website knows I'm actually looking at the web page or not...)
#This is a job website in Japanese
login_url = "https://mypage.levtech.jp/"
driver = selenium.webdriver.Chrome("./chromedriver")
#Account and password are required to log in.
#I logged in and got to the following page, which displays a list of companies that I have applied for:
#https://mypage.levtech.jp/recruits/screening
#Dictionary to store company names and their job postings
jobs = {}
for i, company in enumerate(company_names):
time.sleep(1)
element = driver.find_elements_by_class_name("ScreeningRecruits_ListItem")[i]
while element.text == "":
#While loops and time.sleep() are there because the webpage seems to take a while to load
time.sleep(0.1)
element = driver.find_elements_by_class_name("ScreeningRecruits_ListItem")[i]
td = element.find_element_by_tag_name("td")
while td.text == "":
time.sleep(0.1)
td = element.find_element_by_tag_name("td")
if td.text == company:
td.click()
time.sleep(1)
jobs[company] = get_job_desc(driver) #The get_job_desc function checks HTML tags and extract info from certain elements
time.sleep(1)
driver.back()
time.sleep(1)
print(jobs)
By the way, I have tried adding a user agent and scroll down the page using the following code, in the hope that the Web page would believe that I'm "looking at it." Well, I failed :(
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

So, I think the answer to your question is due to window_handles. Whenever we open a new tab, Selenium changes the window's focus on us ( obviously ). Because the focus is on another page, we need to use the driver.switch_to.window(handle_here) method. This way, we can switch to our proper tab. In order to do this, I found a website that has a similar functionality ( also in Japanese / Kanji? ) that might help you out.
MAIN PROGRAM - For Reference
from selenium import webdriver
from selenium.webdriver.chrome.webdriver import WebDriver as ChromeDriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as DriverWait
from selenium.webdriver.support import expected_conditions as DriverConditions
from selenium.common.exceptions import WebDriverException
import time
def get_chrome_driver():
"""This sets up our Chrome Driver and returns it as an object"""
path_to_chrome = "F:\Selenium_Drivers\Windows_Chrome85_Driver\chromedriver.exe"
chrome_options = webdriver.ChromeOptions()
# Browser is displayed in a custom window size
chrome_options.add_argument("window-size=1500,1000")
return webdriver.Chrome(executable_path = path_to_chrome,
options = chrome_options)
def wait_displayed(driver : ChromeDriver, xpath: str, int = 5):
try:
DriverWait(driver, int).until(
DriverConditions.presence_of_element_located(locator = (By.XPATH, xpath))
)
except:
raise WebDriverException(f'Timeout: Failed to find {xpath}')
# Gets our chrome driver and opens our site
chrome_driver = get_chrome_driver()
chrome_driver.get("https://freelance.levtech.jp/project/search/?keyword=&srchbtn=top_search")
wait_displayed(chrome_driver, "//div[#class='l-contentWrap']//ul[#class='asideCta']")
wait_displayed(chrome_driver, "//div[#class='l-main']//ul[#class='prjList']")
wait_displayed(chrome_driver, "//div[#class='l-main']//ul[#class='prjList']//li[contains(#class, 'prjList__item')][1]")
# Click on the first item title link
titleLinkXpath = "(//div[#class='l-main']//ul[#class='prjList']//li[contains(#class, 'prjList__item')][1]//a[contains(#href, '/project/detail/')])[1]"
chrome_driver.find_element(By.XPATH, titleLinkXpath).click()
time.sleep(2)
# Get the currently displayed window handles
tabs_open = chrome_driver.window_handles
if tabs_open.__len__() != 2:
raise Exception("Failed to click on our Link's Header")
else:
print(f'You have: {tabs_open.__len__()} tabs open')
# Switch to the 2nd tab and then close it
chrome_driver.switch_to.window(tabs_open[1])
chrome_driver.close()
# Check how many tabs we have open
tabs_open = chrome_driver.window_handles
if tabs_open.__len__() != 1:
raise Exception("Failed to close our 2nd tab")
else:
print(f'You have: {tabs_open.__len__()} tabs open')
# Switch back to our main tab
chrome_driver.switch_to.window(tabs_open[0])
chrome_driver.quit()
chrome_driver.service.stop()
For scrolling, you could use this method
def scroll_to_element(driver : ChromeDriver, xpath : str, int = 5):
try:
webElement = DriverWait(driver, int).until(
DriverConditions.presence_of_element_located(locator = (By.XPATH, xpath))
)
driver.execute_script("arguments[0].scrollIntoView();", webElement)
except:
raise WebDriverException(f'Timeout: Failed to find element using xpath {xpath}\nResult: Could not scroll')

Capybara/Selenium gets a Net::ReadTimeout randomly on location.reload()

I'm using Capybara, the selenium-webdriver gem, and chromedriver in order to drive my javascript enabled tests.
The problem is that about 50% of our builds fail due to a Net::ReadTimeout error. At first this was manifesting as a 'could not find element' error, but after I upped Capybara's default max wait time to 30 seconds, I started seeing the timeout.
I examined the screenshots of when the timeout happens, it's stuck on a 'Successfully logged in' modal that we show briefly before using the Javascript function, location.reload(), to reload the page.
I've ran the test locally and can sometimes reproduce it, also randomly. Sometimes it zips by this modal and does the reload so fast you can barely see it, and other times it just hangs forever.
I don't feel like it's an asset compilation issue, since the site has already loaded at that point in order for the user to access the login form.
Wondering if anyone has seen this before and knows a solution.
The specific code:
visit login_path
page.within '#sign-in-pane__body' do
fill_in 'Email', with: user.email
click_button 'Submit'
end
expect(page).to have_content 'Enter Password'
page.within '#sign-in-pane__body' do
fill_in 'Password', with: user.password
click_button 'Submit'
end
expect(page).to have_text 'Home page landing text'
The hang up happens between click_button 'Submit' and expecting the home page text.
The flow of the logic causing the timeout is the user submits the login form, we wait for the server to render a .js.erb template that triggers a JS event upon successful login. When that trigger happens we show a modal saying that login was successful, then execute a location.reload().

It turned out this wasn't exclusive to doing a location.reload() in JS. It sometimes happened just visiting a page.
The solution for me was to create an HTTP client for the selenium driver and specify a longer timeout:
Capybara.register_driver :chrome do |app|
client = Selenium::WebDriver::Remote::Http::Default.new
client.read_timeout = 120
Capybara::Selenium::Driver.new(app, {browser: :chrome, http_client: client})
end

Solved similar problem by using my own version of visit method:
def safe_visit(url)
max_retries = 3
times_retried = 0
begin
visit url
rescue Net::ReadTimeout => error
if times_retried < max_retries
times_retried += 1
puts "Failed to visit #{url}, retry #{times_retried}/#{max_retries}"
retry
else
puts error.message
puts error.backtrace.inspect
exit(1)
end
end
end

Here is what you need to do if you need to configure it for headless chrome
Capybara.register_driver :headless_chrome do |app|
client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 120 # instead of the default 60
options = Selenium::WebDriver::Chrome::Options.new
options.headless!
Capybara::Selenium::Driver.new(app, {
browser: :chrome,
http_client: client,
options: options
})
end
Capybara.default_driver = :headless_chrome
Capybara.javascript_driver = :headless_chrome
Passing headless argument in capabilities was not working for me.
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
chromeOptions: { args: %w[headless disable-gpu] }
)
Here is more details about why headless in capabilities was not working.

Infinite Scroll on Quora using Selenium in Python and Javascript

I try to handle the "infinite scrolling" in Quora website.
I use selenium lib with Python after trying to use the send_keys methods i try to run Javascript command in order to scroll down the page.
It doesn't working when i run the code, but if i try to run in the firefox console it's work.
How can i fix this problem? and it's possibile use PhantomJs?
def scrapying(self):
print platform.system()
browser = webdriver.Firefox()
#browser = webdriver.PhantomJS(executable_path='/usr/local/bin/node_modules/phantomjs/lib/phantom/bin/phantomjs')
browser.get("https://www.quora.com/C-programming-language")
#browser.get("https://answers.yahoo.com/dir/index?sid=396545660")
time.sleep(10)
#elem = browser.find_element_by_class_name("topic_page content contents main_content fixed_header ContentWrapper")
no_of_pagedowns = 500
while no_of_pagedowns:
#elem.send_keys(Keys.SPACE)
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(0.5)
no_of_pagedowns -= 1
browser.quit()
myClassObject = getFrom()
myClassObject.scrapying()

One of the options would be to recursively scroll into view of the last loaded post on a page:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.quora.com/C-programming-language")
NUM_POSTS = 200
posts = driver.find_elements_by_css_selector("div.pagedlist_item")
while len(posts) < NUM_POSTS:
driver.execute_script("arguments[0].scrollIntoView();", posts[-1])
posts = driver.find_elements_by_css_selector("div.pagedlist_item")
print(len(posts))
And it would scroll the page down until, at least, NUM_POSTS posts are loaded.

I'm also not able to trigger the infinite scroll to work using this while using Firefox. The gist of the code works in the console, however:
for i in range(0, 5):
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(3)

Rails capybara visiting unrelated page causes test to fail

I've got got a page with some AJAX code that I'm trying to build a spec for using Capybara. This test works:
context 'from the JobSpec#index page' do
scenario 'clicks the Run now link' do
visit job_specs_path
within( find('tr', text: #job_spec.name) ) { click_link 'run_now' }
visit current_path
expect(find('tr', text: #job_spec.name)).to have_content(/(running|success)/)
end
end
After clicking the link 'run_now', a job is launched and the user is redirected to the launched_jobs_path, which has a Datatable that uses some AJAX to pull currently running jobs. This test passes. However, I wanted to add to the test to check to make sure that the job wasn't already running before clicking the 'run_now' button (which would give me a false positive). I started playing around with it, but then I noticed that even simply putting visit launched_jobs_path before visit job_specs_path would cause the test to fail with the error
Failure/Error: expect(find('tr', text: #job_spec.name)).to have_content(/(running|success)/)
Capybara::ElementNotFound:
Unable to find css "tr" with text "job_spec_collection-1"
I'm fairly certain that this is an issue with the Javascript not running at the right time, but I don't quite know how to resolve it. Any ideas? This is Rails 4.1 with Capybara 2.4, and Capybara-webkit 1.3.
Thanks!
Update
In the parent context of the spec, I do have :js => true
feature 'User runs a JobSpec now', :js => true do
before do
Capybara.current_driver = :webkit
#job_spec = FactoryGirl.create(:job_spec, :with_client_spaces)
end
after { Capybara.use_default_driver }
context 'from the JobSpec#index page' do
...
end
end

I think it might be as simple as adding :js => true to your scenario header so that it knows to launch webkit / selenium to test the js / ajax.
scenario 'clicks the Run now link', :js => true do
Hope that helps!
UPDATE
Well here's one more idea. You could try adding this to your 'rails_helper' or 'spec_helper' file to set the :js driver.
Capybara.javascript_driver = :webkit
Do you have other :js tests that are working properly with :webkit? Have you tried using save_and_open_page to see if the element you are looking for is actually on the page and Capybara just isn't finding it?

Ok, pretty sure I've got something that will at least work for now. If anyone has more elegant solutions I'd love to hear them.
I think the issue is in the complexity of what happens in the 'run_now' controller. It adds a job to a Rufus scheduler process, which then has to start the job (supposed to start immediately). Only after the launched job is running via Rufus does an entry in the launched_jobs table get created.
While developing, I would see the job just launched as soon as I clicked 'Run now'. However, I'm guessing that WebKit is so fast that the job hasn't been launched by the time it gets to the launched_jobs#index page. The extra visit current_path there seem to help at first because it would refresh the page, but it only worked occasionally. Wrapping these last two steps in a synchronize block seems to work reliably (I've run it about 20 times now without any failures):
context 'from the JobSpec#index page' do
scenario 'clicks the Run now link' do
# First make sure that the job isn't already running
visit launched_jobs_path
expect(page).to have_no_selector('tr', text: #job_spec.name)
# Now launch the job
visit job_specs_path
within( find('tr', text: #job_spec.name) ) { click_link 'run_now' }
page.document.synchronize do
visit current_path
expect(find('tr', text: #job_spec.name)).to have_content(/(running|success)/)
end
end
end

Capybara with :js => true causes test to fail

I'm new to Capybara and testing on Rails in general, so please forgive me if this is a simple answer.
I've got this test
it "should be able to edit an assignment" do
visit dashboard_path
select(#project.client + " - " + #project.name, :from => "assignment_project_id")
select(#team_member.first_name + " " + #team_member.last_name, :from => "assignment_person_id")
click_button "Create assignment"
page.should have_content(#team_member.first_name)
end
it passes as is, but if I add :js => true it fails with
cannot select option, no option with text 'Test client - Test project' in select box 'assignment_project_id'
I'm using FactoryGirl to create the data, and as the test passes without JS, I know that part is working.
I've tried with the default JS driver, and with the :webkit driver (with capybara-webkit installed)
I guess I don't understand enough what turning on JS for Capybara is doing.
Why would the test fail with JS on?

I've read the Capybara readme at https://github.com/jnicklas/capybara and it solved my issue.
Transactional fixtures only work in the default Rack::Test driver, but
not for other drivers like Selenium. Cucumber takes care of this
automatically, but with Test::Unit or RSpec, you may have to use the
database_cleaner gem. See this explanation (and code for solution 2
and solution 3) for details.
But basically its a threading issue that involves Capybara having its own thread when running the non-Rack driver, that makes the transactional fixtures feature to use a second connection in another context. So the driver thread is never in the same context of the running rspec.
Luckily this can be easily solve (at least it solved for me) doing a dynamic switching in th DatabaseCleaner strategy to use:
RSpec.configure do |config|
config.use_transactional_fixtures = false
config.before :each do
if Capybara.current_driver == :rack_test
DatabaseCleaner.strategy = :transaction
else
DatabaseCleaner.strategy = :truncation
end
DatabaseCleaner.start
end
config.after do
DatabaseCleaner.clean
end
end

A variation of brutuscat's answer that fixed our feature specs (which all use Capybara):
config.before(:suite) do
DatabaseCleaner.clean_with(:truncation)
end
config.before(:each) do
# set the default
DatabaseCleaner.strategy = :transaction
end
config.before(:each, type: :feature) do
DatabaseCleaner.strategy = :truncation
end
config.before(:each) do
DatabaseCleaner.start
end
config.append_after(:each) do
DatabaseCleaner.clean
end

There is another way to deal with this problem now described and discussed here: Why not use shared ActiveRecord connections for Rspec + Selenium?