Develop Reference

Selenium-Java-geckodriver : Test Execution halts with JavaScript error as "document.getElementById(...) is null"

I had been encountering this issue for a while now. Browsed through a lot of threads on SO & other forums but still clueless.
While automating a simple flow on a Web Application with Selenium 3.4.0, geckodriver v0.16.1 & Mozilla Firefox 53.0 within Eclipse Neon v2 IDE, I am facing an error intermittently on the console as:
JavaScript error: https://www.url.com/my, line 1715: TypeError: document.getElementById(...) is null
Though using chromedriver v2.29/Google Chrome 58.0 or using Python I don't face any such issue.
Once this error appears, the Test Execution halts and finally shows TimeoutException as follows:
Exception in thread "main" org.openqa.selenium.TimeoutException: Timeout loading page after 300000ms
URL of the website is: https://www.shareinvestor.com/my
The HTML DOM is:
<div id="sic_sitemap">
<div id="sic_container">
<div id="sic_header">
<h1 id="sic_siteTitle">
<div id="sic_headerMembershipLink">
<a id="sic_mobileEdition" href="/mobile">
<div id="sic_loginContainer" class="sic_logIn" style="">
<div class="sic_login-wrap">
<div class="sic_logIn-subscribe">
<div class="sic_logIn-bg">
<a href="/user/login.html">
</div>
</div>
</div>
</div>
<div id="sic_subHeader">
<div id="sic_mainNav" class="sic_withRightCorner">
<div id="sic_sideBar" class="sic_expanded { selected_market_suffix: 'MY'}">
<div class="sic_superBanner sic_superBannerTop">
<div id="sic_content" class="sic_collapsed">
<div id="sic_footer" class="si_fixed">
</div>
As of now, I have tried out the following options but of no avail:
Java Click
JavascriptExecutor Click
Actions Click
Here is my code:
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.interactions.Actions;
import org.openqa.selenium.remote.DesiredCapabilities;
public class 78644072022 {
public static void main(String[] args) {
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
DesiredCapabilities dc = DesiredCapabilities.firefox();
dc.setCapability("marionette", true);
WebDriver driver = new FirefoxDriver(dc);
driver.manage().window().maximize();
driver.get("https://www.shareinvestor.com/my");
WebElement login_button = driver.findElement(By.xpath("//div[#id='sic_loginContainer']/div/div[#class='sic_logIn-bg']/a"));
//Java Click
login_button.click();
//JavascriptExecutor Click
/*JavascriptExecutor jse = (JavascriptExecutor)driver;
jse.executeScript("arguments[0].click();", login_button);*/
//Actions Click
/*Actions act = new Actions(driver);
act.moveToElement(login_button).click().build().perform();*/
driver.findElement(By.xpath("//input[#id='sic_login_header_username']")).sendKeys("debanjan");
driver.findElement(By.xpath("//input[#id='sic_login_header_password']")).sendKeys("5786");
driver.findElement(By.xpath("//input[#id='sic_login_submit']")).click();
}
}
I am looking for a Java Solution with geckodriver to overcome the error JavaScript error:TypeError: document.getElementById(...) is null
In one of the SO threads I saw a solution like:
You need to do a null check in updateHTML like this:
function updateHTML(elmId, value) {
var elem = document.getElementById(elmId);
if(typeof elem !== 'undefined' && elem !== null) {
document.getElementById(elmId).innerHTML = value;
}
}
Can we implement this?
Any suggestions & pointers will be helpful.

I am looking for a Java Solution with geckodriver to overcome the error JavaScript error:TypeError: document.getElementById(...) is null
To answer your question, I don't believe there's anything you can do to "fix" this via Java/Selenium. This is a JavaScript error, which originates from the website that you are trying to access. You might want to consider contacting their support team, maybe one of their developers can look at this issue.
Instead of clicking on the login button, maybe consider navigating directly to the login page?
driver.get("https://www.shareinvestor.com/user/login.html");

Firstly, Those javascript errors are not triggered due to any of the selenium code. Ofcourse, the timeout has been triggered by the selenium(will discuss on this on a later point).
You will get that javascript error irrespective of any kind of browser you launch the URL with. But in case of the gecko, you are notified in the eclipse console with but not in case of Chrome. If you need to see the java script error in chrome, just launch the url in chrome and go to devtools/console(F12). You can also see the same in firefox console too.
Chrome Img:
Secondly, We are getting timeout exception because the site is really taking too much time to load. I have been waited for 7 minutes and the page is still loading even now. Selenium won't executes its script unless the page has been completely launched. As a result we are getting timeout exception(not sure about the default page launch time permitted). I thought of bypassing directly to the login page ("https://www.shareinvestor.com/user/login.html") and that's also not taking any finite time to load completely.
Intepolation: Those java script errors are not an issue for automation But those page loads are really. This site doesn't seems like a good candidate for automation with this issue.
Update1: otherwise we can also stop the page loading via another thread like sending "esc" key sequence using Action class.
Update2: I tried the same code today and it works fine. Below is the code snippet that i have tried(There is no change at all)
public static void main(String[] args) throws InterruptedException
{
System.setProperty("webdriver.gecko.driver", "F:\\Softwares\\Selenium\\Webdriver\\geckodriver.exe");
WebDriver driver = new FirefoxDriver();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
driver.get("https://www.shareinvestor.com/my");
WebElement login_button = driver.findElement(By.xpath("//div[#id='sic_loginContainer']/div/div[#class='sic_logIn-bg']/a"));
//Java Click
login_button.click();
System.out.println("In Login PAge");
driver.findElement(By.xpath("//input[#id='sic_login_header_username']")).sendKeys("debanjan");
driver.findElement(By.xpath("//input[#id='sic_login_header_password']")).sendKeys("5786");
System.out.println("Entered password");
driver.findElement(By.xpath("//input[#id='sic_login_submit']")).click();
}
Selenium version - 3.4.0
Gecko driver - v0.16.1(mine is 32 bit)
Mozilla - 51.0.1 (Update=>It's working on 53.02 also)
Hope this helps you. Thanks.

I think I've managed to find what is causing this uproar in your script.
I inspected your HTML and it seems javascript method function showRemaining() is causing this problem; because showRemaining() contains this statement
document.getElementById('countdown').innerHTML = '';
where it tries to set innerHTML attribute for element having id as countdown to ''.
But countdown doesn't exist anywhere on your web page hence the error.
TypeError: document.getElementById(...) is null
and somehow selenium isn't able to look past this error. So I think getting it fixed from developers should help you.
UPDATE :
Basically you need to wait all elements to load using implicit wait, once all elements are loaded, then your Javascript error gets resolved and ultimately interaction with the webpage is not hindered by that javascript error:
driver.get("https://www.shareinvestor.com/my");
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
driver.manage().window().maximize();
/*String script = "function updateHTML(countdown, value) {" +
"var elem = document.getElementById('countdown');"+
"if(typeof elem !== 'undefined' && elem !== null) {"+
" document.getElementById('countdown').innerHTML = value;"+
"}"+
"}";
((JavascriptExecutor) driver).executeScript(script);*/
WebElement login_button = driver.findElement(By.xpath("//div[#id='sic_loginContainer']/div/div[#class='sic_logIn-bg']/a"));
login_button.click();
driver.findElement(By.xpath("//input[#id='sic_login_header_username']")).sendKeys("debanjan");
driver.findElement(By.xpath("//input[#id='sic_login_header_password']")).sendKeys("5786");
driver.findElement(By.xpath("//input[#id='sic_login_submit']")).click();

It looks like the XPath for the login_button is incorrect:
By.xpath("//div[#id='sic_loginContainer']/div/div[#class='sic_logIn-bg']/a");
Should be:
By.xpath("//div[#id='sic_loginContainer']/div/div/div[#class='sic_logIn-bg']/a");
This may explain the TimeoutException, since Selenium is unable to locate the non-existent element. EDIT: My mistake, you should see a NoSuchElementException if the element cannot be located.
As for the JavaScript error, unless you were trying to use Selenium to access the web element being manipulated by the JavaScript code (id='countdown' element), Selenium should just ignore the error. Otherwise, there's the potential for other Selenium errors, such as StaleElementReferenceException, but that does not seem to be the case.

This is due to your application HTML page using the async javascript and it uses some variable reference that is not available at the time of execution. we had same issue and asked developer to follow some best practices for javascript in the HTML like putting the script tag at end of the page. I have checked the HTML page source of the site and it contains many script tags in between the code. This will block the DOM rendering. Better, ask the developer to follow some best practices to include script tag in HTML. you refer the link Where should I put <script> tags in HTML markup? for the best practices.

javascript + Selenium WebDriver cannot load list of followers in instgram

I am learning JavaScript,node.js and Selenium Web Driver.
As part of my education process I am developing simple bot for Instagram.
To emulate browser I use Chrome web driver.
Faced problem when trying to get list of followers and amount of followers for the account:
This code opens instagram page, enters credentials, goes to some account and opens followers for this account.
Data like username and password I take from the settings.json.
var webdriver = require('selenium-webdriver'),
by = webdriver.By,
Promise = require('promise'),
settings = require('./settings.json');
var browser = new webdriver
.Builder()
.withCapabilities(webdriver.Capabilities.chrome())
.build();
browser.manage().window().setSize(1024, 700);
browser.get('https://www.instagram.com/accounts/login/');
browser.sleep(settings.sleep_delay);
browser.findElement(by.name('username')).sendKeys(settings.instagram_account_username);
browser.findElement(by.name('password')).sendKeys(settings.instagram_account_password);
browser.findElement(by.xpath('//button')).click();
browser.sleep(settings.sleep_delay);
browser.get('https://www.instagram.com/SomeAccountHere/');
browser.sleep(settings.sleep_delay);
browser.findElement(by.partialLinkText('followers')).click();
This part should open all followers, but not working:
var FollowersAll = browser.findElement(by.className('_4zhc5 notranslate _j7lfh'));
Tried also by xpath:
var FollowersAll = browser.findElement(by.xpath('/html/body/div[2]/div/div[2]/div/div[2]/ul/li[3]/div/div[1]/div/div[1]/a'));
When I run in the browser's console:
var i = document.getElementsByClassName('_4zhc5 notranslate _j7lfh');
it is working fine.
I run code in debug mode (use WebStorm) and it shows in each case that variable "FollowersAll" is undfined.
The same happens when I try to check amount of followers for the account.
Thanks in advance.
example of the selected element

In DOM, class names may be used multiple time. In this case, findElement by className wont work.
Xpath should be Relative and should not be Absolute.
Try Xpath with unique HTML Attribute. For example:
1. //div[#id/text()='value']
In chrome browser, open Developer Tools(press F12). If you framed an Xpath, just press Ctrl+F and paste that Xpath. If it states 1 of 1, then you can surely use that Xpath.
If it states 1 of many, then you need to dig deeper to take exact Xpath.

Infinite Scroll on Quora using Selenium in Python and Javascript

I try to handle the "infinite scrolling" in Quora website.
I use selenium lib with Python after trying to use the send_keys methods i try to run Javascript command in order to scroll down the page.
It doesn't working when i run the code, but if i try to run in the firefox console it's work.
How can i fix this problem? and it's possibile use PhantomJs?
def scrapying(self):
print platform.system()
browser = webdriver.Firefox()
#browser = webdriver.PhantomJS(executable_path='/usr/local/bin/node_modules/phantomjs/lib/phantom/bin/phantomjs')
browser.get("https://www.quora.com/C-programming-language")
#browser.get("https://answers.yahoo.com/dir/index?sid=396545660")
time.sleep(10)
#elem = browser.find_element_by_class_name("topic_page content contents main_content fixed_header ContentWrapper")
no_of_pagedowns = 500
while no_of_pagedowns:
#elem.send_keys(Keys.SPACE)
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(0.5)
no_of_pagedowns -= 1
browser.quit()
myClassObject = getFrom()
myClassObject.scrapying()

One of the options would be to recursively scroll into view of the last loaded post on a page:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.quora.com/C-programming-language")
NUM_POSTS = 200
posts = driver.find_elements_by_css_selector("div.pagedlist_item")
while len(posts) < NUM_POSTS:
driver.execute_script("arguments[0].scrollIntoView();", posts[-1])
posts = driver.find_elements_by_css_selector("div.pagedlist_item")
print(len(posts))
And it would scroll the page down until, at least, NUM_POSTS posts are loaded.

I'm also not able to trigger the infinite scroll to work using this while using Firefox. The gist of the code works in the console, however:
for i in range(0, 5):
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(3)

Controlling a Firefox Extension via Javascript

Is it possible, using javascript, to control an overlay firefox extension? I've extracted the contents of the extension and have identified what functions/methods I need to run, but they are not accessible within the scope of the console.
Thanks in advance for any ideas.

Yes it possible to interact with other add-ons, given the right circumstances.
My test case here will be com.googlecode.sqlitemanager.openInOwnWindow(), which is part of the SqliteManager addon.
In newer builds (I'm using Nightly), there is the Browser Toolbox. With it is is as simple as opening a toolbox and executing com.googlecode.sqlitemanager.openInOwnWindow() in the Console.
You may instead use the Browser Console (or any chrome enabled WebDev Console for that matter, e.g. the Console of "about:newtab"). But you need some boilerplate code to first find the browser window. So here is the code you can execute there: var bwin = Services.wm.getMostRecentWindow("navigator:browser"); bwin.com.googlecode.sqlitemanager.openInOwnWindow()
Again, enable chrome debugging. Then open a Scratchpad and switch to Chrome in the Environment menu. Now executing com.googlecode.sqlitemanager.openInOwnWindow() in our Scratchpad will work.
You may of course write your own overlay add-on.
As a last resort, patch the add-on itself.
Bootstrapped/SDK add-ons: you can load XPIProvider.jsm (which changed location recently) and get to the bootstrapped scope (run environment of bootstrap.js) via XPIProvider.bootstrapScopes[addonID], and take it from there (use whatever is in the bootstrap scope, e.g. the SDK loader).
Now about the right circumstances: If and how you can interact with a certain add-on depends on the add-on. Add-ons may have global symbols in their overlay and hence browser window, such as in the example I used. Or may use (to some extend) JS code modules. Or have their own custom loader stuff (e.g. AdBlock Plus has their own require()-like stuff and SDK add-ons have their own loader, which isn't exactly easy to infiltate)...
Since your question is rather unspecific, I'll leave it at this.
Edit by question asker: This is correct, however I figured I'd add an example of the code I ended up using in the end, which was in fact taken directly from mozilla's developer network website:
In my chrome js:
var myExtension = {
myListener: function(evt) {
IprPreferences.setFreshIpStatus(true); // replace with whatever you want to 'fire' in the extension
}
}
document.addEventListener("MyExtensionEvent", function(e) { myExtension.myListener(e); }, false, true);
// The last value is a Mozilla-specific value to indicate untrusted content is allowed to trigger the event.
In the web content:
var element = document.createElement("MyExtensionDataElement");
element.setAttribute("attribute1", "foobar");
element.setAttribute("attribute2", "hello world");
document.documentElement.appendChild(element);
var evt = document.createEvent("Events");
evt.initEvent("MyExtensionEvent", true, false);
element.dispatchEvent(evt);

Update for Firefox 47 and up
Things changed drastically in Firefox 47. This is the new way to access it.
var XPIScope = Cu.import('resource://gre/modules/addons/XPIProvider.jsm');
var addonid = 'Profilist#jetpack';
var scope = XPIScope.XPIProvider.activeAddons.get(addonid).bootstrapScope
Old way for < Firefox 47
Update for methods of today
Typically you will do so like this:
If i wanted to get into AdBlocks scope, I check AdBlock id, it is {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} so I would go:
var XPIScope = Cu.import('resource://gre/modules/addons/XPIProvider.jsm');
var adblockScope = XPIScope.XPIProvider.bootstrapScopes['{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}'];
You can now tap into anything there.
Another example, I have an addon installed with id NativeShot#jetpack
I would tap into it like this:
var XPIScope = Cu.import('resource://gre/modules/addons/XPIProvider.jsm');
var nativeshotScope = XPIScope.XPIProvider.bootstrapScopes['NativeShot#jetpack'];
if you do console.log(nativeshotScope) you will see all that is inside.

Unable to sendkeys using webdriverjs specifically F11 to maximise the browser

With the below code block it opens a chrome browser fine it just won't full screen the browser using F11. i used to use C# and selenium and that worked fine using this method on chrome and different browsers. It finds the element 'body' but then does not send the key press. Am I doing something wrong here that i should be requiring some other library?
the documentation for webdriverjs is pathetic and there is very few examples, I am seriously considering dumping it for something else possibly python.
var webdriver = require('selenium-webdriver');
var driver = new webdriver.Builder().
withCapabilities(webdriver.Capabilities.chrome()).
build();
driver.get('https://www.google.co.uk/');
driver.wait(function () {
return driver.getTitle().then(function (title) {
return title === 'Google';
});
}, 1000);
driver.findElement(webdriver.By.xpath('/html/body')).sendKeys("F11");
why are we doing this. we are developing a website that will change depending on size 800x600 + with and without the toolbar depending on how the screen is used different items will be displayed. i can maximise the window using,
driver.manage().window().maximize();
This however still leaves the toolbar present and doesn't act as if the user has pressed the F11 key.

it tooks some time to find it but you should have all the Keys in webdriver.Key
driver.findElement(webdriver.By.xpath('/html/body')).sendKeys(webdriver.Key.F11);
Hope it helps!

A co-worker has just discovered that it works well in C# with:
Driver.Instance.Manage().Window.FullScreen();

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
ActionChains(driver).send_keys(Keys.F11).perform()
I use a similar command to toggle JS via Firefox's NoScript add-on
edit: I just tested, and it works!

This should work for you:
driver.findElement(webdriver.By.xpath('/html/body')).sendKeys(Keys.F11);

You can maximise the window using:
driver.manage().window().maximize();

For me the one emulating correctly the F11 on startup is:
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-fullscreen");
WebDriver driver = new ChromeDriver(options);
Or if the kiosk mode is preferred:
ChromeOptions options = new ChromeOptions();
options.addArguments("--kiosk");
WebDriver driver = new ChromeDriver(options);