I am following a Django Tutorial by Marina Mele, which is pretty good but a bit outdated since it was last updated in 2016, I believe. I am now trying the Selenium testing and ran into the problem that I can send my e-mail address via Selenium but not the password. My code is
self.get_element_by_id("identifierId").send_keys(credentials["Email"])
self.get_button_by_id("identifierNext").click()
self.get_element_by_tag('input').send_keys(credentials["Passwd"])
self.get_button_by_id("passwordNext").click()
with these functions being defined as:
def get_element_by_id(self, element_id):
return self.browser.wait.until(EC.presence_of_element_located(
(By.ID, element_id)))
def get_element_by_tag(self, element_tag):
return self.browser.wait.until(EC.presence_of_element_located(
(By.TAG_NAME, element_tag)))
def get_button_by_id(self, element_id):
return self.browser.wait.until(EC.element_to_be_clickable(
(By.ID, element_id)))
Most advices that I read to this issue circled around waiting until the element appears. However, this is covered through these functions. And I am using by_tag since the current version of Google Authentication is using an input for the password field that has not an ID but is a div/div/div child of the div with the "passwordIdentifier"-id. I have also tried using Xpath but it seems that this does not make a difference.
Also, it seems like Selenium is capable of finding the elements...at least when I check with print commands. So, locating the element seems not to be the problem. However, Selenium fails to send the keys from what I can see when I look at what happens in the Firefox browser, while Selenium is testing. What could be the issue? Why is Selenium struggling to send the password keys to the Authentication form?
Thanks to everyone in advance!
When you search an input on google registration page, you will find 8 WebElements. I think it is the origin of your problem.
I would use another localizer such as an xpath = //input[#name='password'] or a By on the name instead of the tag name, as implemented below:
def get_element_by_name(self, element_tag):
return self.browser.wait.until(EC.presence_of_element_located(
(By.NAME, element_tag)))
and:
self.get_element_by_id("identifierId").send_keys(credentials["Email"])
self.get_button_by_id("identifierNext").click()
self.get_element_by_name('password').send_keys(credentials["Passwd"])
self.get_button_by_id("passwordNext").click()
Related
I'm looking for a way to automatically extract information from a web page, more specifically an online game (https://www.virtualregatta.com/fr/offshore-jeu/).
In the game, I want to extract/copy the position of the boat. With Mozilla and its debug tools, I used the network debugger and I saw an HTML POST request containing what I want.
It seems that we receive as a response a json containing a structure with latitude/longitude.
This is perfect to me, but I want a more user friendly way to get it and I would need advices. Problem is that I'm really a beginner in web development haha.
Is it possible to do this using a script ? (But I suppose it will be complicated to first log into the game)
Is it possible to create a basic Mozilla plugin which would be able to catch the request/response and copy the position to clipboard for me ?
anything else ?
EDIT:
I've tried using a Mozilla plugin, and I achieved to add a listener on POST request. I see the request to get the boat information but I can't find a way to get the json response in JS.
function logURL(responseDetails) {
console.log(responseDetails);
}
browser.webRequest.onResponseStarted.addListener(
logURL,
{urls: ["*://*.virtualregatta.com/getboatinfos"]}
);
In Chrome I use Broomo for this purposes. It helps you to add scripts in web pages, you can console.log the POST you found, and of course you can create functions and Use the webpage Backend.
In firefox I found this one js-injector. But I didn't use it before.
Update:
Now there are a new extension for both browsers:
Chrome: ABC JS-CSS Injector
Firefox: ABC JS-CSS Injector
I've finally found a thread on newbie help on this subject but I am no way forward with resolving this issue, partly because I'm a newbie at programming :)
The thread is:
Newbie: How to overcome Javascript "onclick" button to scrape web page?
I have a similar issue. The site I would like to scrape from has lots of information of a lot of parts, but I would like to only scrape certain part information (company, part number, etc). I have two issues:
How to grab such information from this site without the need to put in search information? Use a Crawler?
A part number has most of the information on a page but there is on page Javascript 'onclick()' function, when it is clicked opens up a small window displaying information that, in addition to, I would like to scrape. How can I scrape the information in this additional window?
I'm using import.io but have been advised to switch to Selenium and PhantomJS. I would welcome other suggestions, and not too complicated (or instructions provided, which would be awesome!), of other tools. I would really appreciate if someone can help me overcome this issue or provide instructions. Thank you.
If you are a newbie and you want to create a web crawler for data extraction then I would recommend selenium however, selenium webdriver is slower than scrapy (a python framework for coding web crawlers)
As you have been advised to use selenium, I will only focus on selenium using python.
For your first issue : "How to grab such information from this site"
Suppose the website from which you want to extract data is www.fundsupermart.co.in (selected this to show how to handle new window pop ups)
using selenium you can crawl by writing:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://www.fundsupermart.co.in/main/fundinfo/mutualfund-AXIS-BANKING-DEBT-FUND--GROWTH-AXS0077.html')
This will open the firefox browser webdriver and will load the page of the link provided in the get() method
Now suppose if you want to extract a table then you can extract by using its tag_name, xpath or class_name by using functions provided by selenium. Like here if I want to extract table under "Investment Objective" :
Then for this I will:
right click -> inspect element -> find the appropriate tag from console -> right click -> copy xpath
Here I found that <tbody> tag was the one from which I can extract the table so I right clicked that and clicked on copy xpath so I got the xpath of that tag i.e. :
xpath=/html/body/table/tbody/tr[2]/td/table/tbody/tr[3]/td/table[2]/tbody/tr/td/table/tbody/tr[1]/td/font/table/tbody/tr[1]/td/table/tbody/tr[5]/td/table/tbody
then, in the code add this line:
driver.find_element_by_xpath(xpath).text
Similarly you can extract other data from any website also see selenium's doc here
For you second issue : "How can I scrape the information in this additional window?"
For clicking the link you can use click() function provided by selenium. Suppose here I want to click the link : Click here for price history then I will get the xpath(as done previously) and add line :
driver.find_element_by_xpath(xpath).click()
I will open a new window like this :
Now to extract data from new window you will have to switch to new window which you can do by adding this line:
windows = driver.window_handles
driver.switch_to_window(windows[1])
Now, by doing this I have switched the webdriver to the new window and now I can extract data as I did earlier and to close this window and switch back to original window just add :
driver.close()
driver.switch_to_window(windows[0])
This was a very basic and naive approach of web crawlers using selenium. The tutorial given here is really good and will help you a lot.
I'm writing a kde plasmoid using qml. It's a widget, displaying mobile usage for one of largets mobile priveders in our country, using the api provided by the operater. In order to get the data one must do a request using phone number + password and I'd like to use kwallet to store "accounts" in some kwallet's folder for this widget.
The question is, how do I use kwallet in qml/javascript based widget, if it is even possible? I can't find any info on the web. I found this plasmoid using kwallet: http://kde-look.org/content/show.php/gmail-plasmoid?content=101229
but this one is written in python and is importing some python kde libs, so I can't really use that. Any suggestions or even links to some usefull api would be great.
Kwallet can be accessed using qdbus on the command-line. And apparently there is a way to make command-line calls in Javascript plasmoids using the extension LaunchApp, like this:
Button {
onButtonClick: plasmoid.runCommand("qdbus",
["<add-missing-parameters-here>"]);
}
For the extension to work, you need to add this line to your desktop file:
X-Plasma-RequiredExtensions=LaunchApp
The exact command-line calls go something like this:
Make a call to open the wallet
qdbus org.kde.kwalletd /modules/kwalletd org.kde.KWallet.open <wallet name> 0 "<your application name>"
Use the returned ID to acess a password
qdbus org.kde.kwalletd /modules/kwalletd readPasswordList <wallet-id> kmail "<entry name>" "<your application name>"
I haven't tried any of this, but theoretically it could work.
Links:
Example using runCommand: http://server.ericsbinaryworld.com/blog/2012/06/06/developing-my-first-plasmoid-the-qml-code/
LaunchApp documentation: http://techbase.kde.org/Development/Tutorials/Plasma/JavaScript/API-LaunchApp
Wallet access using the command-line: http://learnonthejob.blogspot.de/2009/11/accessing-kde-wallet-from-cmdline.html
I have a webpage that contains some information that I am interested in. However, those information are generated by Javascript.
If you do something similar like below:
browser = webdriver.Chrome()
browser.set_window_size(1000, 1000)
browser.get('https://www.xxx.com') # cannot make the web public, sorry
print browser.page_source
It only print out a few javascript functions and some headers which doesn't contain that information that I want - Description of Suppliers, etc... So, when I try to collect those information using Selenium, the browser.find_element_by_class_name would not find the element I want successfully either.
I tried the code below assuming it would has the same effect as typing document in the javascript console, but obviously not.
result = browser.execute_script("document")
print result
and it returns NULL...
However, if I open up the page in Chrome, right click the element and inspect element. I could see the populated source code. See the attached picture.
Also, I was inspired by this commend that helps a lot.
I could open up the javascript console in Chrome, and if I type in
document
I could see the complete html sitting there, which is exactly what I want. I am wondering is there a way to store the js populated source code using selenium?
I've read some posts saying that it requires some security work to store the populated document to client's side.
Hope I have made myself clear and appreciates any suggestion or correction.
(Note, I have zero experience with JS so detailed explaination would be gratefully appreciated!)
I'm facing problems by testing the chrome webstore - using WebdriverJS and my own node.js script.
When trying to query the result using css selectors, I get allmost everytime "no such element" errors from the WebDriver Server.
var client = require('webdriverjs').remote();
client.init().url('https://chrome.google.com/webstore',function(result{console.log(result);})
.setValue('#searchbox-input','sampleapp \n')
.click('**?what should be here?**', function(result){console.log(result);}).end();
I'm struggling with the part - ?what should be here? so i can automate clicks on a displayed result.
CSS selectors have strange formats and can't be accessed.
Since the chrome webstore doens't seem to have button to click and submit the search, you could, instead, send the carriage return character and get it to trigger the search result.
Try something like this:
var client = require('webdriverjs').remote();
client.init()
.url('https://chrome.google.com/webstore')
.setValue('#searchbox-input','sampleapp \u000D')
.end();
The "\u000D" is just a carriage return unicode code.