I am attempting to download a file from a website using Selenium and Python 3. This requires pressing a confirmation button on an overlay window. The overlay window is not within an iFrame - the HTML is simply dynamically added when the overlay appears - but Selenium is not able to find the button by xPath, returning a NoSuchElementException. Am I missing anything that would cause Selenium not to be able to see the element as it appears in the page source? So far as I can tell, Selenium should be able to locate the button with no issue.
#Initialize Driver
driver = webdriver.Safari()
cmd = "osascript -e 'tell application \"Safari\" to set bounds of front window to {0, 22, 1500, 1022}'"
os.system(cmd)
#Call up seach link
driver.get(data_url)
wait_a = WebDriverWait(driver, 15)
element = wait_a.until(EC.presence_of_element_located((By.ID, "md-input-3")))
#Initialize and send login information (defined above)
username = driver.find_element_by_id("md-input-3")
password = driver.find_element_by_id("md-input-6")
username.send_keys(crunchbase_username)
password.send_keys(crunchbase_password)
#Click login button
password.send_keys(Keys.ENTER)
#Wait for results page to finish loading
wait = WebDriverWait(driver, 15)
element = wait.until(EC.title_contains("Signals"))
time.sleep(2)
#Press Download Button
driver.find_element_by_xpath("//button[#aria-label='Export your results']").click()
time.sleep(2)
#Press csv button
driver.find_element_by_xpath("//button[#aria-label='Export to CSV']").click()
time.sleep(2)
#Confirm downlaod
driver.find_element_by_xpath("//*[#id='cdk-overlay-36']/md-dialog-container/confirmation-dialog/dialog-layout/div/md-dialog-actions/div/button[2]").click()
#Close driver
#driver.close()
The page source is overly complicated and highly stylized so I will not include it here, but a screenshot of the relevant section of the code in my browser's web inspector is below. The element which I'm trying to click is highlighted in blue.
Web Inspector Screenshot
I appreciate any help with this.
It is hard to tell without having access to the page under question and being able to see what's going. Few general points:
Try css selectors instead of xpath. They are more robust, easier to work with and fast.
Acvoid using dynamically generated IDs. In your screenshot I can't even see an id that appears in your code.
When you have more than one element of the same kind (like buttons in your case) try getting all webelements under a certain parent and test all of them for having an attribute value that you are looking for.
For example:
elemItems = driver.find_elements_by_css_selector(menuItemSelector)
for element in elements:
if element.text == "export":
elemItems[1].click()
Here, you find all the elements of a certain type (buttons for example) and select one that has "export" text in it.
Before clicking on the element, execute the following lines:
WebElement element = driver.findElement(By.xpath(" path_of_your_module "));
((JavaScriptExecutor) driver). executeScript("argument[0].scrollIntoView(true);", element);
I faced the same issue on a dialogOverlay and I fixed it. I realized that on clicking the overlay button that would bring the overlay, selenium was searching for the element before the overlay loads the dynamic content. So I did this:
def download():
global browser
notFound = True
while(notFound):
try:
elem = browser.find_element(By.ID, 'btnFormat2')
elem.click()
notFound = False
except BaseException:
print("----Error Download Element not found")
download()
The code will continuously look for the element until its loaded on the overlay.
I am using Selenium driver to automatically display webpages from StackOverflow. I want all the comments to be displayed in the browser, so I have code to find the elements at the bottom of the comments lists that are like this:
<a class="js-show-link comments-link " title="expand to show all comments on this post" href="#" onclick="">show <b>7</b> more comments</a>
Once I locate this element, I try a .click() method on it, but nothing happens until about the third click.
(One more thing: after I posted the question, I realize that the .click() method works if I employ it after I manually click on the link, and I reload the page. This is strange. Maybe the page needs to be in the cache?)
This used to work fine a few months ago, but now there is an event listener (a little dark icon with the letters "ev" in it) right next to this element, and the onclick attribute is an empty string.
Would the event listening bind affect the behavior of .click()? Is there a solution? Thanks.
I am using selenium in Python 2.7. Here is my code:
link = u'http://stackoverflow.com/questions/549/the-definitive-guide-to-form-based-website-authentication'
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
driver = webdriver.Firefox(capabilities=firefox_capabilities)
driver.get(link)
myelement = driver.find_element_by_css_selector('[title="expand to show all comments on this post"]')
myelement.click()
TL,DR:
For whatever reason, my selenium python script can't seem to "click" on the buttons I need.
Context:
Hello. My task is one many are probably familiar with: I'd like to automate the process of opening a website, logging in, and clicking on a few drop-down menu links within the website, which will ultimately lead me to a page where I can download a spreadsheet. I'm able to open the web page and log in. To proceed, I have to:
Click on the drop down menu header, and
in the drop down menu, click on the appropriate option.
Here's a snapshot of the pertinent HTML code from the website:
<td class="x-toolbar-cell" id="ext-gen45">
<table id="ext-comp-1045" class="x-btn x-btn-noicon" style="width: auto;" cellspacing="0">
<tbody class="x-btn-small x-btn-icon-small-left">
<tr>
<td class="x-btn-ml"><i> </i></td>
<td class="x-btn-mc">
<em class="x-btn-arrow" unselectable="on">
<button type="button" id="ext-gen46" class=" x-btn-text">Reports</button>
</em>
</td>
<td class="x-btn-mr"><i> </i></td>
</tr>
</tbody>
</table>
</td>
The item I need to "click" has a button tag, specifically:
<button type="button" id="ext-gen46" class=" x-btn-text">Reports</button>
To select it with selenium, I've tried the following:
reports_click_element = browser.find_element_by_id('ext-gen46').click()
and when that failed,
reports_element = browser.find_element_by_xpath("//button[contains(text(), 'Reports')]").click()
That one actually executed without an ExceptionMessage error, but I found out it was selecting other elements in the page that had "Reports" text, as opposed to the particular button I need.
When I've tried to zero in on the button I need clicked, the interpreter returned an error message indicating that the html attributes could not be found.
How can I proceed from here? (Should I be focusing on the unselectable="on" tag in the element right above the button I need clicked?)
Please let me know if I can add anything to the question. Thanks in advance.
Update:
I have switched into an iframe that I believe the menu is a part of- but I still cannot select the button. So far, here is my Python code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import time
binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\Firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary)
browser.get("https://app.website.com")
login_entry(username, password) # this works fine; it's just a user-created function to login. Ignore.
time.sleep(10) # wait for website's markup to load
browser.switch_to.frame(browser.find_element_by_tag_name("iframe"))
time.sleep(10)
# This is the point where I'm trying to click on the "Reports" button
reports_element = browser.find_element_by_xpath("//*[contains(text(), 'Reports')]") #this refers to other elements
reports_element = browser.find_element_by_xpath("//button[contains(text(), 'Reports')][1]") #no luck here either
A couple of cases which came to mind.
More than one element exists but not all are visible
elements = browser.find_elements_by_xpath("//button[contains(text(), 'Reports')]")
for element in elements:
if element.is_displayed():
print "element is visible"
element.click()
else:
print("element is not visible")
print(element)
The element exists, it would be visible but is out of screen.
from selenium.webdriver.common.action_chains import ActionChains
elements = browser.find_elements_by_xpath("//button[contains(text(), 'Reports')]")
for element in elements:
ActionChains(driver).move_to_element(element).perform()
try:
element.click()
except:
print("couldn't click on {}".format(element))
Can you also try to record your clicks and keyboard entries with
Selenium IDE for Firefox? Then save it as a python script and post it as a comment here?
Some ideas:
Print the DOM source of the iframe and check if the html is what you expected:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import time
binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\Firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary)
browser.get("https://app.website.com")
login_entry(username, password) # this works fine; it's just a user-created function to login. Ignore.
time.sleep(10) # wait for website's markup to load
browser.switch_to.frame(browser.find_element_by_tag_name("iframe"))
time.sleep(10)
print browser.page_source
It could be, for example, that you switched to the wrong iframe, in case there is more than one in the page.
If the html is correct, then you could try to use the id of the button instead of trying to get it by text:
reports_element = browser.find_element_by_xpath("//button[#id='ext-gen46']")
If that still doesn't work, you can try to click the button with javascript:
browser.execute_script("""
(function() {
document.getElementById("ext-gen46").click();
})()
""")
You can even include jQuery by saving jQuery in a local file, then storing the contents to a variable, and running, like:
with open(JQUERY_PATH) as f:
jquery = f.read()
browser.execute_script(jquery)
Then you could use it to click the button with:
driver.execute_script("""
(function() {
jQuery(document).ready(function($) {
$('#ext-gen46').click();
}, jQuery)
})()
""")
The "page source" is only what comes with the document request. It will not show any DOM elements created via javascript after the page loads. It does sound like your elements are within some sort of iframe. In the browser console, try this and see if it returns any elements:
document.querySelectorAll('iframe')
EDIT for your update:
Once again, the page source is only what is available at document load. Everything that comes afterward that is loaded dynamically can only be seen by using the browser inspector or by getting parts of the document w/ javascript. The link being basically the same html is probably because it is a link that acts with javascript and isn't meant to lead to an actual html document page. You probably need to do in your code is:
browser.switch_to.frame(browser.find_element_by_css_selector('iframe'))
reports_element = browser.find_element_by_link_text('Reports')
reports_element.click()
I would suggest the following things to try:
Switch to the correct frame.
There may be a chance of no frame or one frame or more than one nested frames, where your element can be as a child of another. Therefore, you must switch to the right frame before you find the element. My detailed answer is here.
Use a more restricted XPATH. For example:
report_element =browser.find_element_by_xpath("//td/em/button[text()='Reports']")
You should check whether there is only one iframe on current page. You can use
len(browser.find_elements_by_tag_name('iframe'))
If output is greater than 1 and you use browser.switch_to.frame(browser.find_element_by_tag_name("iframe")), it means that you're trying to switch to first iframe found by webdriver, but you actually might need to switch to another iframe
Good solution is to find required iframe in button's ancestors with F12 or right-click on button + inspect element and use specific attributes of iframe in XPath that will match exact iframe, e.g.
browser.switch_to.frame(browser.find_element_by_xpath('//iframe[#class="iframe_class_name"]'))
Bad solution is to define index of required iframe by exhaustive search like:
browser.switch_to.frame(browser.find_elements_by_tag_name("iframe")[0])
reports_element = browser.find_element_by_xpath("//button[text()='Reports'][contains(#id, 'ext-gen')]")
reports_element.click()
...
browser.switch_to.frame(browser.find_elements_by_tag_name("iframe")[1])
reports_element = browser.find_element_by_xpath("//button[text()='Reports'][contains(#id, 'ext-gen')]")
reports_element.click()
....
I'm doing something similar with Siebel OneView, which puts most of its control buttons and menus inside Java screen elements. These elements prevent me from finding the HTML objects to activate with Selenium, so I have fallen back to using pyautogui:
import pyautogui
import time
#look for and click the Activites menu
try:
x,y = pyautogui.locateCenterOnScreen('acts_button.png')
try:
pyautogui.click(x, y)
except PermissionError:
#allow load - note this is AFTER the click permission error
time.sleep(7)
pass
except TypeError:
<code to deal with NOT finding the button>
This looks for a copy of a previously taken screenshot, stored in the same file location as the python script. In this section the image is called acts_button.png.
[Quick note: There are two try: except: statements in here. The inner one deals with any problems clicking the button as Windows will often throw out permission errors. The outer one is more important and tells your script what to do if it can't find the button. In my code, I try clicking on a preset x,y location with pyautogui.click(958, 169); if that fails, I ask the user for input (Failures detected by the next screen not loading).
The screenshots themselves are created by using commands like this
acts_button = pyautogui.screenshot('acts_button.png', region=(928,162,63,15))
Where the region is a tuple of the following parts of your target button, measured as pixels
left hand edge
top edge
width
height
Now all you need is a way to find the (x,y) values for when you take the screenshot. Fortunately I have code for that too:
#! python3
import pyautogui, sys, time
print('Press Ctrl-C to quit.')
try:
while True:
x, y = pyautogui.position()
positionStr = 'X: ' + str(x).rjust(4) + ' Y: ' + str(y).rjust(4)
print(positionStr, end='')
print('\b' * len(positionStr), end='', flush=True)
time.sleep(1)
except KeyboardInterrupt:
print('\n')
This will pop up a small window with the (x,y) coordinates of the mouse, updating once per second.
The routine to set up a new click event is therefore
Open the site you want to navigate
Use the mouse locator script to find the settings for the screenshot tuple
Use pyautogui.screenshot to save an image of the button
Use the main script to look for and click that image
After a couple of goes you should be able to set up a new click event in less than a minute. You can either use this method where you know Selenium will fail, or as a way of catching and managing exceptions when Selenium might fail.
Why not use python request (with session) and BeautifulSoup modules to do this kind of job (user interact on a website) ?
Problem: Hi Guys, I'm testing a CMS tool using selenium c# but problem is to find a selector for a tiny drop down button because of random ID(all selectors). While it is generating HTML codes but i can not take the help of it as the next time when script runs it changes the IDs (Class name and all other identifiers).
Tried : i tried storing Xpaths of all drop down button on page in an array and next time clicking on the array position of the element but it didnt store any element xpath in array.
please suggest what can i do in this case, possibly its a case of java script enabled page.
HTML Code of element:
<span class="epi-extraIcon epi-pt-contextMenu epi-iconContextMenu" role="presentation" title="Display menu" data-dojo-attach-point="iconNodeMenu" _dijitmenuuniqname_51_43="1"/>
Recently I used selenium in C# and had a few problems like that.
My solution was to use XPath.
I inspected the elements that I needed with firebug (on Mozilla Firefox) to get the Xpath.
After that, I used HtmlAgilityPack nuget to load the page source and select the nodes and then I was able to get the elements.
I also disabled the JQuery animations of the page to avoid some problems.
So, my code for the selection of the nodes was something like that:
var document = new HtmlDocument();
document.LoadHtml(pageSource);
var htmlLoaded = DocumentParsing(document.DocumentNode.SelectNodes(
"/html/body/table[2]/tbody/tr/td/table[2]/tbody/tr/td[1]/font[2]/b[1] |" +
"/html/body/table[2]/tbody/tr/td/table[2]/tbody/tr/td[1]/font[2]/b[2]));
And my code for disable JQuery animations:
try
{
var js = DriverService as IJavaScriptExecutor;
js.ExecuteScript("$.fx.off = !$.fx.off;");
return true;
}
catch (Exception)
{
return false;
}
Hope that helps.
Using selenium (with python bindings), I'm dealing with a webpage that is almost entirely AJAX; including hyperlinks. Instead of using the element.click() method, I wish to execute the javascript in the "onclick" attribute of the tag:
The tag:
<a onclick="javascript:setEvent(event);requisition_openRequisitionDescription('requisitionListInterface','actOpenRequisitionDescription',_ftl_api.lstVal('requisitionListInterface', 'requisitionListInterface.listRequisition', 'requisitionListInterface.d327682e687', this),'requisitionList');return ftlUtil_followLink(this);" href="#" title="View this job description" id="requisitionListInterface.reqTitleLinkAction.row1">
The code:
from selenium import webdriver
firefox = webdriver.Firefox()
firefox.get("some_url")
elem = firefox.find_element_by_id("requisitionListInterface.reqTitleLinkAction.row1")
jcode = elem.get_attribute("onclick")
firefox.execute_script(jcode)
The Error:
WebDriverException: Message: u'event is not defined' ; Stacktrace:
Disclaimer:
I don't understand Javascript. As far as I can tell, it's expecting the 'event' variable, however I guess it has something to do with a callback ?
EDIT:
I've assumed that the javascript is modifying the href attribute but is it possible for javascript to redirect the browser without modifying the hyperlink?
You can just click it using this.
elem.click()
The onclick event will automatically get triggered.
Unless selenium does something crazy, which i doubt, onclick="javascript:setEvent(event) is not valid javascript. the rest of the string however is. When this code block gets executed, it tries to invoke a function called setEvent with the variable event, which remains undefined.
<a onclick="requisition_openRequisitionDescription('requisitionListInterface','actOpenRequisitionDescription',_ftl_api.lstVal('requisitionListInterface', 'requisitionListInterface.listRequisition', 'requisitionListInterface.d327682e687', this),'requisitionList');return ftlUtil_followLink(this);" href="#" title="View this job description" id="requisitionListInterface.reqTitleLinkAction.row1">
since event is nowhere else needed in your script, just remove it?
from selenium.webdriver.common.action_chains import ActionChains
firefox = webdriver.Firefox()
# You need a mouse to hover the span elements here, and please try this way :)
self.mouse = webdriver.ActionChains(firefox)
# You need get the span element from its xpath:
elem = firefox.find_element_by_id("requisitionListInterface.reqTitleLinkAction.row1")
# Then you hover on span element by mouse and click on it:
self.mouse.move_to_element(elem).click().perform()