My script scrapes the page and if there is a new element according to my requirements, it clicks the button, when there is only one element everything is perfect but problem is when it clicks the button, a new page opens. So if there are more than one element, I need to get back to previous page to continue process.
I tried browser.back() for sure, but when it comes back to previous page it doesn't remember the elements and gives this error as I expected: selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up with the line if len(css(".Total", parent=new, nowait=True)) > 0: which is the first line, I made it to recognise the new elements.
I also tried to open a new tab/window by clicking the button but button doesn't have this feature because it is javascript. Is there a efficent way to solve this issue?
first_window_handler = driver.current_window_handle
driver.find_element_by_css_selector("body").send_keys(Keys.CONTROL + 't')
second_window_handler = driver.window_handles[1]
# from second page
driver.switch_to.window(second_window_handler)
element_from_second = driver.find_element_by_css_selector('something')
# from first page
driver.switch_to.window(first_window_handler)
element_from_first = driver.find_element_by_css_selector('something else')
now you can switch between windows, and the elements will still be interactable
As per exception
selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up
we came to know that whenever page loads webdriver loosing references to webelements which it holds previously.
So the best thing is always pass locators dynamically on calling predefined keywords/methods in java prospective, so that webdriver looks for that web element at that instance and perform action.
Some times we may receive same exception in looping list of web elements, because in loop due to actions webdriver may loose reference, so in loop also we need to specify locators so that it will not fail. example if i need to click on links, i will give path like this "//a["+i+"]"
Thank You,
Murali
First I would like to describe the motivation for my question.
I have a complex web page to test with Selenium + HtmlUnit, which launches diverse javascript scripts. The problem which I describe should be quite common.
On the page there is a button to which jQuery binds a click callback (click event handler) after the page is loaded. There is an explicit Wait (this is a Selenium term) for the button to become clickable in the test code. So as soon as the button becomes clickable, it gets clicked by Selenium. Often, however, this happens before jQuery manages to attach to the button the click event handler. In this case the Selenium test fails.
What I thought to do is to preprocess the web page accessed by HtmlUnit before javascript starts executing on the page, injecting some <script>myownscript()</script>at the beginning of the page (so that it executes before any other script on that page). Then I would be able to know, controlling certain conditions in the Selenium test code, when exactly the attaching of the click event handler has happened (how I exactly do this, depends on the details of the application). If I make Selenium click the button then, the presence of the click event handler will be guaranteed, and the test would proceed further as planned - with no errors due to the missing click event handler.
Let us leave apart the question whether the idea is a good or a bad one (a much simpler one, of course, would be just introducing a large enough delay in the Selenium test code before trying to click the problematic button, but then there might be a problem with the overall duration of tests, because the problem I described is present on many pages of the application being tested).
Are there some hooks in Selenium/HtmlUnit which permit to preprocess the page fetched from the server, injecting a script as I described, before javascript starts executing on the page?
In this case, you can use JavaScriptExecutor. You can add a function to do anything you want in the String script.
WebElement button = driver.findElement(By.id("my-button"));
JavascriptExecutor jsExe = ((JavascriptExecutor) driver);
String script = "console.log(arguments[0].id); return arguments[0].id";
Object oj = jsExe.executeScript(script, button);
String txt = oj.toString();
System.out.println(txt);
Please be careful if you want to use aycn such as setTimeout(), it will return immediately. See an example for async method in my answer at: method execute_script don't wait end of script to return value with selenium in python
The Story:
Here on StackOverflow, I've seen users reporting that they cannot click an element via selenium WebDriver "click" command and can work around it with a JavaScript click by executing a script.
Example in Python:
element = driver.find_element_by_id("myid")
driver.execute_script("arguments[0].click();", element)
Example in WebDriverJS/Protractor:
var elm = $("#myid");
browser.executeScript("arguments[0].click();", elm.getWebElement());
The Question:
Why is clicking "via JavaScript" works when a regular WebDriver click does not? When exactly is this happening and what is the downside of this workaround (if any)?
I personally used this workaround without fully understanding why I have to do it and what problems it can lead to.
Contrarily to what the currently accepted answer suggests, there's nothing specific to PhantomJS when it comes to the difference between having WebDriver do a click and doing it in JavaScript.
The Difference
The essential difference between the two methods is common to all browsers and can be explained pretty simply:
WebDriver: When WebDriver does the click, it attempts as best as it can to simulate what happens when a real user uses the browser. Suppose you have an element A which is a button that says "Click me" and an element B which is a div element which is transparent but has its dimensions and zIndex set so that it completely covers A. Then you tell WebDriver to click A. WebDriver will simulate the click so that B receives the click first. Why? Because B covers A, and if a user were to try to click on A, then B would get the event first. Whether or not A would eventually get the click event depends on how B handles the event. At any rate, the behavior with WebDriver in this case is the same as when a real user tries to click on A.
JavaScript: Now, suppose you use JavaScript to do A.click(). This method of clicking does not reproduce what really happens when the user tries to click A. JavaScript sends the click event directly to A, and B will not get any event.
Why a JavaScript Click Works When a WebDriver Click Does Not?
As I mentioned above WebDriver will try to simulate as best it can what happens when a real user is using a browser. The fact of the matter is that the DOM can contain elements that a user cannot interact with, and WebDriver won't allow you to click on these element. Besides the overlapping case I mentioned, this also entails that invisible elements cannot be clicked. A common case I see in Stack Overflow questions is someone who is trying to interact with a GUI element that already exists in the DOM but becomes visible only when some other element has been manipulated. This sometimes happens with dropdown menus: you have to first click on the button the brings up the dropdown before a menu item can be selected. If someone tries to click the menu item before the menu is visible, WebDriver will balk and say that the element cannot be manipulated. If the person then tries to do it with JavaScript, it will work because the event is delivered directly to the element, irrespective of visibility.
When Should You Use JavaScript for Clicking?
If you are using Selenium for testing an application, my answer to this question is "almost never". By and large, your Selenium test should reproduce what a user would do with the browser. Taking the example of the drop down menu: a test should click on the button that brings up the drop down first, and then click on the menu item. If there is a problem with the GUI because the button is invisible, or the button fails to show the menu items, or something similar, then your test will fail and you'll have detected the bug. If you use JavaScript to click around, you won't be able to detect these bugs through automated testing.
I say "almost never" because there may be exceptions where it makes sense to use JavaScript. They should be very rare, though.
If you are using Selenium for scraping sites, then it is not as critical to attempt to reproduce user behavior. So using JavaScript to bypass the GUI is less of an issue.
The click executed by the driver tries to simulate the behavior of a real user as close as possible while the JavaScript HTMLElement.click() performs the default action for the click event, even if the element is not interactable.
The differences are:
The driver ensures that the element is visible by scrolling it into the view and checks that the element is interactable.
The driver will raise an error:
when the element on top at the coordinates of the click is not the targeted element or a descendant
when the element doesn't have a positive size or if it is fully transparent
when the element is a disabled input or button (attribute/property disabled is true)
when the element has the mouse pointer disabled (CSS pointer-events is none)
A JavaScript HTMLElement.click() will always perform the default action or will at best silently fail if the element is a disabled.
The driver is expected to bring the element into focus if it is focusable.
A JavaScript HTMLElement.click() won't.
The driver is expected to emit all the events (mousemove, mousedown, mouseup, click, ...) just like like a real user.
A JavaScript HTMLElement.click() emits only the click event.
The page might rely on these extra events and might behave differently if they are not emitted.
These are the events emitted by the driver for a click with Chrome:
mouseover {target:#topic, clientX:222, clientY:343, isTrusted:true, ... }
mousemove {target:#topic, clientX:222, clientY:343, isTrusted:true, ... }
mousedown {target:#topic, clientX:222, clientY:343, isTrusted:true, ... }
mouseup {target:#topic, clientX:222, clientY:343, isTrusted:true, ... }
click {target:#topic, clientX:222, clientY:343, isTrusted:true, ... }
And this is the event emitted with a JavaScript injection:
click {target:#topic, clientX:0, clientY:0, isTrusted:false, ... }
The event emitted by a JavaScript .click() is not trusted and the default action may not be invoked:
https://developer.mozilla.org/en/docs/Web/API/Event/isTrusted
https://googlechrome.github.io/samples/event-istrusted/index.html
Note that some of the drivers are still generating untrusted events. This is the case with PhantomJS as of version 2.1.
The event emitted by a JavaScript .click() doesn't have the coordinates of the click.
The properties clientX, clientY, screenX, screenY, layerX, layerY are set to 0. The page might rely on them and might behave differently.
It may be ok to use a JavaScript .click() to scrap some data, but it is not in a testing context. It defeats the purpose of the test since it doesn't simulate the behavior of a user. So, if the click from the driver fails, then a real user will most likely also fail to perform the same click in the same conditions.
What makes the driver fail to click an element when we expect it to succeed?
The targeted element is not yet visible/interactable due to a delay or a transition effect.
Some examples :
https://developer.mozilla.org/fr/docs/Web (dropdown navigation menu)
http://materializecss.com/side-nav.html (dropdown side bar)
Workarrounds:
Add a waiter to wait for the visibility, a minimum size or a steady position :
// wait visible
browser.wait(ExpectedConditions.visibilityOf(elem), 5000);
// wait visible and not disabled
browser.wait(ExpectedConditions.elementToBeClickable(elem), 5000);
// wait for minimum width
browser.wait(function minimumWidth() {
return elem.getSize().then(size => size.width > 50);
}, 5000);
Retry to click until it succeeds :
browser.wait(function clickSuccessful() {
return elem.click().then(() => true, (ex) => false);
}, 5000);
Add a delay matching the duration of the animation/transition :
browser.sleep(250);
The targeted element ends-up covered by a floating element once scrolled into the view:
The driver automatically scrolls the element into the view to make it visible. If the page contains a floating/sticky element (menu, ads, footer, notification, cookie policy..), the element may end-up covered and will no longer be visible/interactable.
Example: https://twitter.com/?lang=en
Workarounds:
Set the size of the window to a larger one to avoid the scrolling or the floating element.
Mover over the element with a negative Y offset and then click it:
browser.actions()
.mouseMove(elem, {x: 0, y: -250})
.click()
.perform();
Scroll the element to the center of the window before the click:
browser.executeScript(function scrollCenter(elem) {
var win = elem.ownerDocument.defaultView || window,
box = elem.getBoundingClientRect(),
dy = box.top - (win.innerHeight - box.height) / 2;
win.scrollTo(win.pageXOffset, win.pageYOffset + dy);
}, element);
element.click();
Hide the floating element if it can't be avoided:
browser.executeScript(function scrollCenter(elem) {
elem.style.display = 'none';
}, element);
NOTE: let's call 'click' is end-user click. 'js click' is click via JS
Why is clicking "via JavaScript" works when a regular WebDriver click does not?
There are 2 cases for this to happen:
I. If you are using PhamtomJS
Then this is the most common known behavior of PhantomJS . Some elements are sometimes not clickable, for example <div>. This is because PhantomJS was original made for simulating the engine of browsers (like initial HTML + CSS -> computing CSS -> rendering). But it does not mean to be interacted with as an end user's way (viewing, clicking, dragging). Therefore PhamtomJS is only partially supported with end-users interaction.
WHY DOES JS CLICK WORK? As for either click, they are all mean click. It is like a gun with 1 barrel and 2 triggers. One from the viewport, one from JS. Since PhamtomJS great in simulating browser's engine, a JS click should work perfectly.
II. The event handler of "click" got to bind in the bad period of time.
For example, we got a <div>
-> We do some calculation
-> then we bind event of click to the <div>.
-> Plus with some bad coding of angular (e.g. not handling scope's cycle properly)
We may end up with the same result. Click won't work, because WebdriverJS trying to click on the element when it has no click event handler.
WHY DOES JS CLICK WORK? Js click is like injecting js directly into the browser. Possible with 2 ways,
Fist is through devtools console (yes, WebdriverJS does communicate with devtools' console).
Second is inject a <script> tag directly into HTML.
For each browser, the behavior will be different. But regardless, these methods are more complicating than clicking on the button. Click is using what already there (end-users click), js click is going through backdoor.
And for js click will appear to be an asynchronous task. This is related a with a kinda complex topic of 'browser asynchronous task and CPU task scheduling' (read it a while back can't find the article again). For short this will mostly result as js click will need to wait for a cycle of task scheduling of CPU and it will be ran a bit slower after the binding of the click event.
(You could know this case when you found the element sometimes clickable, sometimes not.
)
When exactly is this happening and what is the downside of this
workaround (if any)?
=> As mention above, both mean for one purpose, but about using which entrance:
Click: is using what providing by default of browser.
JS click: is going through backdoor.
=> For performance, it is hard to say because it relies on browsers. But generally:
Click: doesn't mean faster but only signed higher position in schedule list of CPU execution task.
JS click: doesn't mean slower but only it signed into the last position of schedule list of CPU task.
=> Downsides:
Click: doesn't seem to have any downside except you are using PhamtomJS.
JS click: very bad for health. You may accidentally click on something that doesn't there on the view. When you use this, make sure the element is there and available to view and click as the point of view of end-user.
P.S. if you are looking for a solution.
Using PhantomJS? I will suggest using Chrome headless instead. Yes, you can set up Chrome headless on Ubuntu. Thing runs just like Chrome but it only does not have a view and less buggy like PhantomJS.
Not using PhamtomJS but still having problems? I will suggest using ExpectedCondition of Protractor with browser.wait() (check this for more information)
(I want to make it short, but ended up badly. Anything related with theory is complicated to explain...)
Thank you for the good explanation, I was hitting the same issue and your explanation helped to resolve my issue.
button = driver.wait.until(EC.presence_of_element_located(
(By.XPATH, "//div[#id='pagination-element']/nav[1]/span[3]/button[1]/span[1]/i[1]")
))
driver.execute_script("arguments[0].click();", button)
if (theElement.Enabled)
{
if (!theElement.Selected)
{
var driver = (IJavaScriptExecutor)Driver;
driver.ExecuteScript("arguments[0].click();", theElement); //ok
//theElement.Click();//action performed on theElement, then pops exception
}
}
I do not agree we will "almost never" use JS to simulate the click action.
Above theElement.Click(), we'll check the Radio button but then Exception pops as above image.
Actually, this is no page load action after the Click, the click is just to select the Radio button, and I don't know why the WebDriver Click() will cause this exception, can anyone explain why this exception occurred.
I use the Webdriver 3.141.59 and IE 11 and selenium-server-standalone-3.141.59.jar for a remote test.
I don't know if this is the effects of an update panel or what, but I basically have a drop down list that allows a user to select an item as a filter. When the item is selected it should bring back only one item into a grid view. That is this specific filter will at most bring back the record you are looking for. This works fine if the user clicks an "apply" link to apply the filter. Behind the apply link is some server-side code (C# within an ASP.NET Web Forms application).
We had a request by a user with something to the effect of:
"Why do I have to click the apply button if I make a selection in this
one drop down filter...it should simply get that one record I am
searching for. This helps me because I don't have to click the
"Apply" button."
I agreed with him and thought what is the easiest way to do this...I thought: Simple, I will have an on change event handler of the drop down such that when a selection is made I'll trigger a click event. Something to this effect:
$("#MainContent_ddlCompany").on("change", function() {
var companyId = $("#MainContent_ddlCompany").val();
$("#MainContent_hdnCompanyValue").val(companyId);
$("#<%=ddlCompany.ClientID %>").trigger("chosen:updated");
if (companyId.length > 0) {
$(".apply").click();
$(".apply").removeClass("applyButton");
$(".apply").addClass("resetButton");
} else {
//cleared selection of a company
$(".apply").removeClass("resetButton");
$(".apply").addClass("applyButton");
}
});
At first this didn't work, and I couldn't tell why, but then after some serious googling I changed this line:
$(".apply").click();
To this:
$('.apply')[0].click();
That worked great...so I decided to test it some more. As I kept selecting one filter value after another I noticed the page started to slow down. In fact by the 6th or 7th time it was pretty unusable. I don't know why it's happening, but I suspect again it has to do with the fact that this linkbutton with the class name .apply is inside an update panel.
But still I thought to myself, it was inside of an update panel before I changed my jQuery code to simulate the click event. So why does the page slow down and drag with this little piece of code? Is calling the event from jQuery code rendering something else in the HTML that could be causing this?
If I change my code back and force the user to click the apply button then we are back to a good normal speed. Why is it if I tell jQuery to simulate clicking the button my page slow down? It's doing the same thing, the simulation of the click of this link button is calling its server-side code method whether the user clicks it or I have jQuery click it.
For now I'm at a loss as to why this is happening because this button is in an update panel in either case, yet when I have jQuery click it via $('.apply')[0].click(); the page slows down after several attempts. Yet when I have the user simply click this button (without the jQuery click event) then it works fine?
What am I missing here?
Ugh, well, I found my issue. Because I was using updatepanels I had to wrap my jQuery code to include an add_endRequest. That is, you have something to the effect of:
$(document).ready(function() {
//Some initial event/triggers
var prm = Sys.WebForms.PageRequestManager.getInstance();
prm.add_endRequest(function () {
//Copy of some initial event/triggers
});
});
Why do I use the endRequest you ask? Well, because updatepanels basically throw away all your events after an asynchronous postback because the HTML at that point (after an update) is rendered again and at that point all events associated with any control inside an update panel are wiped away. At this point of course document.ready() does not run, so I have to resubscribe to these events inside of endRequest. Enter my issue...
I had a huge brain fart where I basically took everything, literally everything inside document ready and copied it into endRequest. In fact, if I remember correctly, I read articles which stated
Whatever you have in document ready simply copy paste into endRequest
That's fine, but you have to be careful here. I was throwing in events that were not wrapped around inside of an updatepanel into endRequest. The result is disastrous...at least for me.
These events would be attached then multiple times..or based on the number of asynchronous postbacks made. In my case, as I was testing I mentioned after the 6th or 7th time performance starts degrading. Well, by that time my controls were being attached that many times to events. For instance, my .apply button along with my dropdownlist were both outside of my updatepanel. But my jQuery code was attaching the change event of my dropdownlist in both document ready and endRequest.
The result is initially it's pretty fast, because it's only in document ready. But as I make asynchronous postbacks these events are being attached every time. For n tests I would have n attached events...in my case the test of 7 yields 7 on change event handlers!
Case in point, do not place any event handlers such as jQuery's on() event for any controls that are NOT inside an update panel. Otherwise you will run into what I ran into which was poor performance as events are happening.
Im using Python + Selenium + Splinter + Firefox to create an interactive web crawler.
The python script offers the options, then Selenium opens Firefox and sends some orders.
Right now, I need to let the python script know the web element that the user wants to interact with.
The method I currently use is:
Right-click the item in the website (Firefox), click 'inspect
element', then click in the Firefox inspector, click 'copy HTML', then
feed it manually to the script, which will then be able to go on.
But for obvious reasons I feel this process is far from perfection.
I know nothing of javascript, but after reading other questions I get the feeling that javascript could actually be the solution.
Splinter allows to run javascript and pick up the returning values into the python script, so, theoretically:
Would it be possible to run a javascript code that would return the html code of the next element the user clicks? So the named method would only be right-clicking the desired element?
Clarification for Amey's comment:
The python script opens a Firefox window, which control is still retained from the script.
And with splinter, javascript code can be executed and waited upon completion / information return.
This means that the python script can ask the user to click or right-click in the Firefox window that it owns, so the aim would be to launch a javascript that would "catch" which element the user clicks on.
Would that be enough for javascript to catch the desired element?
This was an interesting question. My strategy was to use Javascript to add listeners to the elements you're targeting. Since you didn't specify what types of elements, I used links. This could easily be adapted though.
When an element is clicked, the listener creates a new page element with an ID you specify and sets the value attribute to the relevant information.
Then, assuming you've set driver.implicitly_wait, you can just wait for the element to appear.
driver.execute_script("for(var i = 0; i < document.links.length; i++){document.links[i].onclick = function clicked(){var e = document.createElement('a'); e.setAttribute('id','myUniqueID'); e.setAttribute('value', this); document.getElementsByTagName('body')[0].appendChild(e);};}")
clicked = driver.find_element_by_id('myUniqueID').get_attribute('value')