Develop Reference

Selenium will not show javascript loaded table in webpage with Python

I am trying to scrape the following webpage: https://steamdb.info/app/730/graphs/
(I have gained permission from the website)
The problem is that the "Monthly Breakdown" table seems to be loaded by Javascript, and BeautifulSoup does not work. When using Selenium to open the webpage, it says that to see the table "You must have Javascript enabled.", which should be enabled when using Selenium. Here is my code:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--enable-javascript")
browser = webdriver.Chrome(options=options)
browser.maximize_window()
url = "https://steamdb.info/app/730/graphs/"
browser.get(url)
Any ways to solve this problem?
How the page should look:
How it looks on Selenium:

Try this and see if you no longer get that error message:
options.add_argument("javascript.enabled", True)
You may also need to look into "waits" here to make sure the async operation on the webpage has time to load.

Update:
To enable or disable javascript :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
#value 1 enables it , if you set to 2 it disables it
chrome_options.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 1})
driver =webdriver.Chrome(r".\chromedriver.exe",options=chrome_options)
driver.get("https://www.google.com")
THis wont solve your issue
you can check if the javascript is enabled by typing below in your address bar:
chrome://settings/content/javascript?search=javascript
you can see that even if its enabled the website won't load properly.
it seems they have enabled security to avoid using selenium in thier website
Previous answer:
There is no command line argument called --enable-javascript chromium project try the above javascript-harmony flag instead.
below are the full list of supported chrome flags:
https://peter.sh/experiments/chromium-command-line-switches/#login-profile
please add screenshots and other information if more help is needed

How to disable screenshots and javascript for PhantomJS in python selenium?

I am scraping in a python/selenium framework using phantomJS on windows. First, I tried to disable javascript and screenhsots with selenium:
driver = webdriver.PhantomJS("phantomjs.exe", desired_capabilities = dcap)
webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.page.settings.javascriptEnabled"] = False
webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.takesScreenshot"] = False
webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.page.clearMemoryCash"] = False
However, when I have a look at ghostdriver.log, Session.negotiatedCapabilities includes:
browserName:phantomjs
version:2.1.1
driverName:ghostdriver
driverVersion:1.2.0
platform:windows-7-32bit
javascriptEnabled:true # Should be false
takesScreenshot:true # Should be false
Therefore, I think I need to disable both parameters during onInitialized=function(), similar to the below code snippet:
phantom_exc_uri='/session/$sessionId/phantom/execute'
driver.command_executor._commands['executePhantomScript'] = ('POST', phantom_exc_uri)
initScript="""
this.onInitialized=function() {
var page=this;
### disable javascript and screenshots here ###
}
"""
driver.execute('executePhantomScript',{'script': initScript, 'args': []})
Q1: How come I can set some phantomJS specs in webdriver.DesiredCapabilities, but others not? Is this my mistake or some bug?
Q2: Is it reasonable to accomplish this during onInitialized or am I on the wrong way?
Q2: If so, how to disable JS and screenshots during onInitialized?

You have raised quite a few queries in your question. Let me try to address them all. A simple workflow with Selenium v3.8.1, ghostdriver v1.2.0 and phantomjs v2.1.1 Browser shows us that the following Session.negotiatedCapabilities are passed by default :
"browserName":"phantomjs"
"version":"2.1.1"
"driverName":"ghostdriver"
"driverVersion":"1.2.0"
"platform":"windows-8-32bit"
"javascriptEnabled":true
"takesScreenshot":true
"handlesAlerts":false
"databaseEnabled":false
"locationContextEnabled":false
"applicationCacheEnabled":false
"cssSelectorsEnabled":true
"webStorageEnabled":false
"rotatable":false
"acceptSslCerts":false
"nativeEvents":true
"proxy":{"proxyType":"direct"}}
So by default it was mandated that to establish a successful session through PhantomJSDriver and Ghost Browser combination the following Capabilities were a minimum requirement.
Then the users had the DesiredCapabilities class at their disposal to tweak the capabilities. But there are certain capabilities which are minimum requirement to create a successful Ghost Browser session.
javascriptEnabled is such a property which is mandatory. Till a few releases back Selenium did allow to tweak the javascriptEnabled attribute to false. But now WebDriver being a W3C Recommendation Candidate the mandatory capabilities cannot be over-ridden anymore through DesiredCapabilities at user level.
Even if you try to tweak them at user level, WebDriver will override them to default while configuring the capabilities.
So, though you have tried the following :
webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.page.settings.javascriptEnabled"] = False
webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.takesScreenshot"] = False
The properties javascriptEnabled and takesScreenshot defaults to required mandatory configuration.
Update
As you mentioned in your comment What about changing those AFTER the Ghostdriver session is established, i.e. page.onInitialized the straight answer is No.
Once the capabilities are freezed and negotiated to initialize a Browsing Session the capabilities holds true till the particular session is active. So you can't change any of the capabilities once the session is established. To change the capabilities you have to configure the WebDriver instance again.

python : disable download popup when using firefox with selenium

I have script that using selenium and firefox to automating download action.
The problem is whenever I run script I always get pop up from firefox keep asking what kinds of action I would like to do, even though I set download path in firefox preference. I checked files and folders to create master mimeTypes.rdf for all users, but I couldn't find mine.(I'm using ubuntu). I found ~/.mozilla/firefox but there was no file for directory of my profile name nor any file has an extension like .rdf
here is the criminal's pic that making me crazy
firefox download popup
below is what I've done to disable the popup.
profile = FirefoxProfile()
profile.set_preference("browser.download.panel.shown", False)
profile.set_preference("browser.helperApps.neverAsk.openFile", 'application/zip')
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", 'application/zip')
profile.set_preference("browser.download.folderList", 2)
profile.set_preference("browser.download.dir", "/home/i-06/Downloads")
driver = webdriver.Firefox(firefox_profile=profile)

I have spent many hours trying to suppress that "save or open" pop-up that appears when downloading a file using the firefox driver with selenium (python 3.x). None of the many suggestions involving various values for profile.set_preference worked for me. Maybe I missed something.
Still, I finally got it working by the other method that is recommended : using an existing firefox profile.
You can tweak your default (or custom) profile to the file save behaviour you want. Type the following in the firefox address bar and make changes here :
about:preferences#applications
Then the only setting up you need to do to download the file into your current working directory is :
from selenium import webdriver
fp = webdriver.FirefoxProfile(<your firefox profile directory>)
fp.set_preference("browser.download.folderList",2)
fp.set_preference("browser.download.dir", os.getcwd())
driver = webdriver.Firefox(firefox_profile=fp)
If you have a typical ubuntu setup, you can find your default firefox profile dir by viewing ~/.mozilla/firefox/profile.ini
In that .ini file, look for Path under [Profile0]

I doubt you need to define both. Remove the below line from your code
profile.set_preference("browser.helperApps.neverAsk.openFile", 'application/zip')
Also sometime the MIME type of zip file can be different based on the server. It could be any of below
application/octet-stream
multipart/x-zip
application/zip
application/zip-compressed
application/x-zip-compressed
So in Network tab check what is the content type you are getting and add that to your profile to make sure the dialog doesn't come

I removed profile.set_preference("browser.helperApps.neverAsk.openFile", 'application/zip') as Tarun Lalwani suggest and it still work. But my problem was that I put application/mp4 instead of video/mp4. You could check MIME type here.

How to test inline installation of Chrome extension in Protractor? [duplicate]

I am using selenium for some browser automation. I need to install an extension in the browser for my work. I am doing it as follows:
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
executable_path = "/usr/bin/chromedriver"
options = Options()
options.add_extension('/home/TheRookie/Downloads/extensionSamples/abhcfceiempjmchhhdhbnkbimnfpckgl.crx')
browser = webdriver.Chrome(executable_path=executable_path, chrome_options=options)
The browser is starting fine but I am prompted with a pop-up to confirm that I want to add the extension as follows:
and after I get this pop-up, Python soon returns with the following exception:
selenium.common.exceptions.WebDriverException: Message: u'unknown
error: failed to wait for extension background page to load:
chrome-extension://abhcfceiempjmchhhdhbnkbimnfpckgl/toolbar.html\nfrom
unknown error: page could not be found:
chrome-extension://abhcfceiempjmchhhdhbnkbimnfpckgl/toolbar.html\n
(Driver info: chromedriver=2.12.301324
(de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux
3.13.0-39-generic x86_64)'
I tried handling the popup as a regular JavaScript alert using the following code:
alert = browser.switch_to_alert()
alert.accept()
However, this doesn't help. Could anyone please tell me how do I install this extension without the popup or a way to accept the popup? Any help would be greatly appreciated. Thanks!

Usually, you cannot test inline installation of a Chrome extension with just Selenium, because of that installation dialog. There are a few examples in the wild that show how to use external tools outside Selenium to solve this problem, but these are not very portable (i.e. platform-specific) and rely on a state of Chrome's UI, which is not guaranteed to be consistent.
But that does not mean that you cannot test inline installation. If you replace chrome.webstore.install with a substitute that behaves like the chrome.webstore.install API (but without the dialog), then the end-result is the same for all intents and purposes.
"Behaves like chrome.webstore.install" consists of two things:
Same behavior in error reporting and callback invocation.
An extension is installed.
I have just set up such an example on Github, which includes the source code of the helper extension/app and a few examples using Selenium (Python, Java). I suggest to read the README and the source code to get a better understanding of what happens: https://github.com/Rob--W/testing-chrome.webstore.install.
The sample does not require the tested extension to be available in the Chrome Web store. It does not even connect to the Chrome Web store. In particular, it does not check whether the site where the test runs is listed as a verified website, which is required for inline installation to work.

I had some really big code which I would have to re-write if I had to use Java. Luckily, python has a library for automating GUI events called ldtp. I used that to automate the clicking on the "Add" button. I did something on the following lines:
from ldtp import *
from threading import Thread
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def thread_function():
for i in range(5):
if activatewindow('Confirm New Extension'):
generatekeyevent('<left><space>')
break
time.sleep(1)
def main():
executable_path = "/usr/bin/chromedriver"
options = Options()
options.add_extension('/home/TheRookie/Downloads/extensionSamples/abhcfceiempjmchhhdhbnkbimnfpckgl.crx')
thread.start()
browser = webdriver.Chrome(executable_path=executable_path, chrome_options=options)
Hope it helps somebody.

is it possible to develop a browser toolbar that accepts selenium automation

usually selenium does not identify elements on a browser toolbar. But this is for pre-build browser toolbars. Can a browser toolbar ( for FF, IE & Chrome ) be developed in such a way that selenium can be used to identify elements on the toolbar and then perform control actions on it , like clicking on a radio button on the toolbar.

Because Selenium can only control things that JavaScript can control, you need to find out whether JavaScript can control the toolbar. JavaScript has 3 main features. I controls the JavaScript processing, the DOM (document object model) and the BOM (browser object model). There might have been more control given to JavaScript when HTML 5 and CS3 came out, but this is my knowledge of it.
I think you may need to use a cross-browser extension to make your sidebar/toolbar. google-gears/silverlight/adobe-air/jnext. You might need to make the UI using the DOM. I'm not sure. You will need to research.. never done this before.
If the toolbar is part of the BOM (and/or DOM), then you can. Just create a custom command, called Selenium.prototype.doControlToolbar or something and put your JavaScript logic inside of that function (similar to what I have below). Good luck!
Option #1 - if using Selenium IDE:
Specify the user-extensions.js file under Selenium IDE > Options (menu) > Options (menu option) > General Tab, then browse to your file under "Selenium Core Extensions".
Option #2 - if using Selenium RC Server:
If you're not using the IDE and using Selenium RC server with a client driver (like JUnit for example), you must specify the path of the *.js file with the -userExtensions parameter when you start the Selenium RC Server on the command line. But you said you just wanted to use the IDE, so I'd ignore this. It takes quite a bit of other setup to use the Selenium RC server.
java -jar selenium-server.jar -userExtensions user-extensions.js
=======================
I made the following custom command (JavaScript function) in my custom user-extensions.js file.. I had to exit and restart the IDE before it found it. Type everything after the "do" in the "Command" field in the IDE to find the custom command. It looks like it also added a "customAlertAndWait" to the IDE as well.
Code in user-extensions.js file:
Selenium.prototype.doCustomAlert = function(sTarget, sValue) { alert('Target: ' + sTarget + ' ... Value: ' + sValue); };
Selenium IDE command details:
Command: customAlert
Target: custom alert target
Value: custom alert value