Selenium cannot find element in main window - javascript

I am attempting to download a file from a website using Selenium and Python 3. This requires pressing a confirmation button on an overlay window. The overlay window is not within an iFrame - the HTML is simply dynamically added when the overlay appears - but Selenium is not able to find the button by xPath, returning a NoSuchElementException. Am I missing anything that would cause Selenium not to be able to see the element as it appears in the page source? So far as I can tell, Selenium should be able to locate the button with no issue.
#Initialize Driver
driver = webdriver.Safari()
cmd = "osascript -e 'tell application \"Safari\" to set bounds of front window to {0, 22, 1500, 1022}'"
os.system(cmd)
#Call up seach link
driver.get(data_url)
wait_a = WebDriverWait(driver, 15)
element = wait_a.until(EC.presence_of_element_located((By.ID, "md-input-3")))
#Initialize and send login information (defined above)
username = driver.find_element_by_id("md-input-3")
password = driver.find_element_by_id("md-input-6")
username.send_keys(crunchbase_username)
password.send_keys(crunchbase_password)
#Click login button
password.send_keys(Keys.ENTER)
#Wait for results page to finish loading
wait = WebDriverWait(driver, 15)
element = wait.until(EC.title_contains("Signals"))
time.sleep(2)
#Press Download Button
driver.find_element_by_xpath("//button[#aria-label='Export your results']").click()
time.sleep(2)
#Press csv button
driver.find_element_by_xpath("//button[#aria-label='Export to CSV']").click()
time.sleep(2)
#Confirm downlaod
driver.find_element_by_xpath("//*[#id='cdk-overlay-36']/md-dialog-container/confirmation-dialog/dialog-layout/div/md-dialog-actions/div/button[2]").click()
#Close driver
#driver.close()
The page source is overly complicated and highly stylized so I will not include it here, but a screenshot of the relevant section of the code in my browser's web inspector is below. The element which I'm trying to click is highlighted in blue.
Web Inspector Screenshot
I appreciate any help with this.

It is hard to tell without having access to the page under question and being able to see what's going. Few general points:
Try css selectors instead of xpath. They are more robust, easier to work with and fast.
Acvoid using dynamically generated IDs. In your screenshot I can't even see an id that appears in your code.
When you have more than one element of the same kind (like buttons in your case) try getting all webelements under a certain parent and test all of them for having an attribute value that you are looking for.
For example:
elemItems = driver.find_elements_by_css_selector(menuItemSelector)
for element in elements:
if element.text == "export":
elemItems[1].click()
Here, you find all the elements of a certain type (buttons for example) and select one that has "export" text in it.

Before clicking on the element, execute the following lines:
WebElement element = driver.findElement(By.xpath(" path_of_your_module "));
((JavaScriptExecutor) driver). executeScript("argument[0].scrollIntoView(true);", element);

I faced the same issue on a dialogOverlay and I fixed it. I realized that on clicking the overlay button that would bring the overlay, selenium was searching for the element before the overlay loads the dynamic content. So I did this:
def download():
global browser
notFound = True
while(notFound):
try:
elem = browser.find_element(By.ID, 'btnFormat2')
elem.click()
notFound = False
except BaseException:
print("----Error Download Element not found")
download()
The code will continuously look for the element until its loaded on the overlay.

Related

How can we determine when to use JavaScriptExecutor in selenium c#?

TestInChrome1 throws me an exception - "OpenQA.Selenium.ElementNotInteractableException: element not interactable" However, when JavaScriptExecutor is used in TestInChrome2, it works well.
My questions are:
why does Click() method is not working in TestInChrome1?
how do can we determine that JavaScriptExecutor is necessary without trial and errors?
[TestMethod]
public void TestInChrome1()
{
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://ultimateqa.com/");
IWebElement element = driver.FindElement(By.TagName("title"));
element.Click();
driver.Quit();
}
[TestMethod]
public void TestInChrome2()
{
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://ultimateqa.com/");
IWebElement element = driver.FindElement(By.TagName("title"));
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
string title = (string)js.ExecuteScript("return document.title");
driver.Quit();
}
The <title> tag is required in all HTML documents and it defines the title of the document. The element:
Defines a title in the browser toolbar.
Provides a title for the page when it is added to favorites.
Displays a title for the page in search-engine results.
Note A: You can NOT have more than one element in an HTML document.
Note B: If you omit the <title> tag, the document will not validate as HTML.
If you observe the HTML DOM of any website e.g. https://ultimateqa.com/ you will observe the <title> resides within the <head>. Hence the information of this tag is visible and readable but not interactable.
TestInChrome1()
So as per the discussion above, in TestInChrome1():
You won't be able to invoke Click() on the title tag.
To extract the title you can use the Title property from IWebDriver Interface and you can use the following solution:
Console.WriteLine(driver.Title);
TestInChrome2()
By now you must be aware that Selenium beside allowing users to simulate common activities performed by end-users, entering text into fields, selecting drop-down values and checking boxes, and clicking links in documents, it also provides many other controls such as arbitrary JavaScript execution. To extract the <title> you can use the ExecuteScript() method from IJavaScriptExecutor interface as follows:
Console.WriteLine((string)((IJavaScriptExecutor)driver).ExecuteScript("return document.title"));
When we use selenium, it has methods written in such a way that it tries to emulate how user will interact with the webpage.
So, lets take the case there is a button on the screen but it never displays on the visible area of the screen. Maybe it is not scrollable (or maybe it is always hidden) and will be never be available to user. Now this is a valid bug.
If you use javascript executor, it will click on the buttob and your script won't be able to catch this issue
If you use selenium method for click, script will fail giving you some exception
In my case, I have encountered Element not interactable exception in mostly false positive (invalid, non-bug) scenarios
if some field on screen was inactive earlier, you performed some action it became active. but due to faster execution, it was still inactive, when script interacted with it
Say you have dropdown on screen, you expand the dropdown and click on some field. Now you click some other field when dropdown was closed. While execution dropdown does not close immediately and element you want to access next is obsecured by that dropdown (can happen for popups, or some element that expands on screen, combobox, searchbox)
If you see too many issues due to element not interactable , just catch this exception, take screenshot for your reference, generate a warning in logs for yourself and you can implement direct click using javascript executor in catch block.
Atleast by generating warning for yourself, you can check if you are not missing out on an actual issue.
Hope this helps.

Selenium Webdriver-python, unable to find the dynamic element, tried all the possible ways

I am successfully able to navigate to the desired page after logging in. Static elements are located and printed, peacefully. The page makes JavaScript call and the contents are updated after about 4-5 seconds, I'm unable to locate the dynamic elements.
I'm attaching the image of Inspect element before and after loading of Javascript elements.
Please, have a look at the code below and suggest the possible solution.
P.S. Out of 100 times this worked worked for about 2-3 times.
layer = "https://desired.website"
driver.get(layer)
driver.find_element_by_id("email").send_keys('my#email.com')
driver.find_element_by_id("password").send_keys('myPassword')
driver.find_element_by_class_name("css-173kae7").click()
#NOW I'M SUCCESSFULLY LOGGED IN
#Opening the Desired Page, This is a static element
wait = WebDriverWait(driver, 30)
element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "css-1nga9gi")))
driver.execute_script('''window.open("desired.page","_blank");''')
#Successfully opened desired page and switched to newly opened tab
#Trying to access the element present in <tbody> tag, please refer "Inspect Element after JavaScript elements are loaded"- image.
wait = WebDriverWait(driver, 30)
element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "css-167dqxg")))
check = driver.find_elements_by_class_name("ccss-13ozsj7")
for e in check:
print(e.text)
print("ALL DATA SUCCESSFULLY PRINTED")
Nothing happens for 30 seconds and I get time-out error and "ALL DATA SUCCESSFULY PRINTED" is displayed.
ERROR CODE I GET IS:
element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "css-167dqxg")))
File "C:\Python\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Please have a look at the Inspect element.
Inspect Element at the beginning of Page Load
Inspect Element after JavaScript elements are loaded
You may not be switching to the new content on the tab opened, please try to add this:
#after this line
driver.execute_script('''window.open("desired.page","_blank");''')
#try adding this
#this will switch to the newest tab
driver.switch_to_window(driver.window_handles[-1])
You may be stuck looking for your desired elements on the old tab.
when using Python use either 'starts-with' instead of 'undefined-object'
EMAIL ADDRESS XPATH .//*[#id='email-undefined-objectObject-26149']
driver.find_element_by_xpath(".//input[starts-with(#id,'email')]".clear()
driver.find_element_by_xpath(".//input[starts-with(#id,'email')]".send_keys("scotty.mitch#gmail.com"))
....or 'contains'
driver.find_element_by_xpath(".//input[contains(#id,'email')]".clear()

selenium click not working on element with onclick="" and bound event listener

I am using Selenium driver to automatically display webpages from StackOverflow. I want all the comments to be displayed in the browser, so I have code to find the elements at the bottom of the comments lists that are like this:
<a class="js-show-link comments-link " title="expand to show all comments on this post" href="#" onclick="">show <b>7</b> more comments</a>
Once I locate this element, I try a .click() method on it, but nothing happens until about the third click.
(One more thing: after I posted the question, I realize that the .click() method works if I employ it after I manually click on the link, and I reload the page. This is strange. Maybe the page needs to be in the cache?)
This used to work fine a few months ago, but now there is an event listener (a little dark icon with the letters "ev" in it) right next to this element, and the onclick attribute is an empty string.
Would the event listening bind affect the behavior of .click()? Is there a solution? Thanks.
I am using selenium in Python 2.7. Here is my code:
link = u'http://stackoverflow.com/questions/549/the-definitive-guide-to-form-based-website-authentication'
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
driver = webdriver.Firefox(capabilities=firefox_capabilities)
driver.get(link)
myelement = driver.find_element_by_css_selector('[title="expand to show all comments on this post"]')
myelement.click()

Selenium in Python, Unable to Click 'button': Tag not found

TL,DR:
For whatever reason, my selenium python script can't seem to "click" on the buttons I need.
Context:
Hello. My task is one many are probably familiar with: I'd like to automate the process of opening a website, logging in, and clicking on a few drop-down menu links within the website, which will ultimately lead me to a page where I can download a spreadsheet. I'm able to open the web page and log in. To proceed, I have to:
Click on the drop down menu header, and
in the drop down menu, click on the appropriate option.
Here's a snapshot of the pertinent HTML code from the website:
<td class="x-toolbar-cell" id="ext-gen45">
<table id="ext-comp-1045" class="x-btn x-btn-noicon" style="width: auto;" cellspacing="0">
<tbody class="x-btn-small x-btn-icon-small-left">
<tr>
<td class="x-btn-ml"><i> </i></td>
<td class="x-btn-mc">
<em class="x-btn-arrow" unselectable="on">
<button type="button" id="ext-gen46" class=" x-btn-text">Reports</button>
</em>
</td>
<td class="x-btn-mr"><i> </i></td>
</tr>
</tbody>
</table>
</td>
The item I need to "click" has a button tag, specifically:
<button type="button" id="ext-gen46" class=" x-btn-text">Reports</button>
To select it with selenium, I've tried the following:
reports_click_element = browser.find_element_by_id('ext-gen46').click()
and when that failed,
reports_element = browser.find_element_by_xpath("//button[contains(text(), 'Reports')]").click()
That one actually executed without an ExceptionMessage error, but I found out it was selecting other elements in the page that had "Reports" text, as opposed to the particular button I need.
When I've tried to zero in on the button I need clicked, the interpreter returned an error message indicating that the html attributes could not be found.
How can I proceed from here? (Should I be focusing on the unselectable="on" tag in the element right above the button I need clicked?)
Please let me know if I can add anything to the question. Thanks in advance.
Update:
I have switched into an iframe that I believe the menu is a part of- but I still cannot select the button. So far, here is my Python code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import time
binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\Firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary)
browser.get("https://app.website.com")
login_entry(username, password) # this works fine; it's just a user-created function to login. Ignore.
time.sleep(10) # wait for website's markup to load
browser.switch_to.frame(browser.find_element_by_tag_name("iframe"))
time.sleep(10)
# This is the point where I'm trying to click on the "Reports" button
reports_element = browser.find_element_by_xpath("//*[contains(text(), 'Reports')]") #this refers to other elements
reports_element = browser.find_element_by_xpath("//button[contains(text(), 'Reports')][1]") #no luck here either
A couple of cases which came to mind.
More than one element exists but not all are visible
elements = browser.find_elements_by_xpath("//button[contains(text(), 'Reports')]")
for element in elements:
if element.is_displayed():
print "element is visible"
element.click()
else:
print("element is not visible")
print(element)
The element exists, it would be visible but is out of screen.
from selenium.webdriver.common.action_chains import ActionChains
elements = browser.find_elements_by_xpath("//button[contains(text(), 'Reports')]")
for element in elements:
ActionChains(driver).move_to_element(element).perform()
try:
element.click()
except:
print("couldn't click on {}".format(element))
Can you also try to record your clicks and keyboard entries with
Selenium IDE for Firefox? Then save it as a python script and post it as a comment here?
Some ideas:
Print the DOM source of the iframe and check if the html is what you expected:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import time
binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\Firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary)
browser.get("https://app.website.com")
login_entry(username, password) # this works fine; it's just a user-created function to login. Ignore.
time.sleep(10) # wait for website's markup to load
browser.switch_to.frame(browser.find_element_by_tag_name("iframe"))
time.sleep(10)
print browser.page_source
It could be, for example, that you switched to the wrong iframe, in case there is more than one in the page.
If the html is correct, then you could try to use the id of the button instead of trying to get it by text:
reports_element = browser.find_element_by_xpath("//button[#id='ext-gen46']")
If that still doesn't work, you can try to click the button with javascript:
browser.execute_script("""
(function() {
document.getElementById("ext-gen46").click();
})()
""")
You can even include jQuery by saving jQuery in a local file, then storing the contents to a variable, and running, like:
with open(JQUERY_PATH) as f:
jquery = f.read()
browser.execute_script(jquery)
Then you could use it to click the button with:
driver.execute_script("""
(function() {
jQuery(document).ready(function($) {
$('#ext-gen46').click();
}, jQuery)
})()
""")
The "page source" is only what comes with the document request. It will not show any DOM elements created via javascript after the page loads. It does sound like your elements are within some sort of iframe. In the browser console, try this and see if it returns any elements:
document.querySelectorAll('iframe')
EDIT for your update:
Once again, the page source is only what is available at document load. Everything that comes afterward that is loaded dynamically can only be seen by using the browser inspector or by getting parts of the document w/ javascript. The link being basically the same html is probably because it is a link that acts with javascript and isn't meant to lead to an actual html document page. You probably need to do in your code is:
browser.switch_to.frame(browser.find_element_by_css_selector('iframe'))
reports_element = browser.find_element_by_link_text('Reports')
reports_element.click()
I would suggest the following things to try:
Switch to the correct frame.
There may be a chance of no frame or one frame or more than one nested frames, where your element can be as a child of another. Therefore, you must switch to the right frame before you find the element. My detailed answer is here.
Use a more restricted XPATH. For example:
report_element =browser.find_element_by_xpath("//td/em/button[text()='Reports']")
You should check whether there is only one iframe on current page. You can use
len(browser.find_elements_by_tag_name('iframe'))
If output is greater than 1 and you use browser.switch_to.frame(browser.find_element_by_tag_name("iframe")), it means that you're trying to switch to first iframe found by webdriver, but you actually might need to switch to another iframe
Good solution is to find required iframe in button's ancestors with F12 or right-click on button + inspect element and use specific attributes of iframe in XPath that will match exact iframe, e.g.
browser.switch_to.frame(browser.find_element_by_xpath('//iframe[#class="iframe_class_name"]'))
Bad solution is to define index of required iframe by exhaustive search like:
browser.switch_to.frame(browser.find_elements_by_tag_name("iframe")[0])
reports_element = browser.find_element_by_xpath("//button[text()='Reports'][contains(#id, 'ext-gen')]")
reports_element.click()
...
browser.switch_to.frame(browser.find_elements_by_tag_name("iframe")[1])
reports_element = browser.find_element_by_xpath("//button[text()='Reports'][contains(#id, 'ext-gen')]")
reports_element.click()
....
I'm doing something similar with Siebel OneView, which puts most of its control buttons and menus inside Java screen elements. These elements prevent me from finding the HTML objects to activate with Selenium, so I have fallen back to using pyautogui:
import pyautogui
import time
#look for and click the Activites menu
try:
x,y = pyautogui.locateCenterOnScreen('acts_button.png')
try:
pyautogui.click(x, y)
except PermissionError:
#allow load - note this is AFTER the click permission error
time.sleep(7)
pass
except TypeError:
<code to deal with NOT finding the button>
This looks for a copy of a previously taken screenshot, stored in the same file location as the python script. In this section the image is called acts_button.png.
[Quick note: There are two try: except: statements in here. The inner one deals with any problems clicking the button as Windows will often throw out permission errors. The outer one is more important and tells your script what to do if it can't find the button. In my code, I try clicking on a preset x,y location with pyautogui.click(958, 169); if that fails, I ask the user for input (Failures detected by the next screen not loading).
The screenshots themselves are created by using commands like this
acts_button = pyautogui.screenshot('acts_button.png', region=(928,162,63,15))
Where the region is a tuple of the following parts of your target button, measured as pixels
left hand edge
top edge
width
height
Now all you need is a way to find the (x,y) values for when you take the screenshot. Fortunately I have code for that too:
#! python3
import pyautogui, sys, time
print('Press Ctrl-C to quit.')
try:
while True:
x, y = pyautogui.position()
positionStr = 'X: ' + str(x).rjust(4) + ' Y: ' + str(y).rjust(4)
print(positionStr, end='')
print('\b' * len(positionStr), end='', flush=True)
time.sleep(1)
except KeyboardInterrupt:
print('\n')
This will pop up a small window with the (x,y) coordinates of the mouse, updating once per second.
The routine to set up a new click event is therefore
Open the site you want to navigate
Use the mouse locator script to find the settings for the screenshot tuple
Use pyautogui.screenshot to save an image of the button
Use the main script to look for and click that image
After a couple of goes you should be able to set up a new click event in less than a minute. You can either use this method where you know Selenium will fail, or as a way of catching and managing exceptions when Selenium might fail.
Why not use python request (with session) and BeautifulSoup modules to do this kind of job (user interact on a website) ?

Python and Selenium: What's a good way to access links hidden in dropdown menus built by Javascript?

I'm building a webcrawler in Python using Selenium. Below is the function that searches for links. It works fine, except for the part that's commented out. The idea is to hover over each link that was found on the page being examined, and if that hovering action reveals more links (like in a dropdown menu built with Javascript, for example), then add those links using a recursive call to getLinksFromHTML (the "if code != 1" part is just there to make sure there's only one recursive call).
However, the recursive call doesn't pick up any new links when I test it on a page that has links inside JS dropdown menus (the page I'm looking at is http://wilmingtontaxesandaccounting.com). All visible links are picked up fine.
What can I do so that Selenium sees those dropdown links? I need a general solution, i.e., no specific element IDs or anything else page-specific harcoded into the code. Thanks for reading!
def getLinksFromHTML(currUrl, code):
ListlinkerHref = browser.find_elements_by_xpath("//*[#href]")
links1 = []
links2 = []
for link in ListlinkerHref:
url = link.get_attribute("href")
#hov = ActionChains(browser).move_to_element(link)
#hov.perform()
#if code != 1:
#links1 = self.getLinksFromHTML(currUrl, 1)
if url not in links1:
links2.append(url)
return links1 + links2
It turns out the reason it wasn't working was that I was using a Firefox driver. Apparently with Firefox, move_to_element doesn't actually hover on the element, it just "focuses" there. When I switched to a Chrome driver, it actually hovered over the menu items and showed the submenus. (Note the actual function has some errors in it, but that's not the point of this question.)
TL;DR: If you're using Selenium Webdriver and you want to hover over links to reveal content like submenus, use Chrome and not Firefox.

Categories

Resources