I am learning JavaScript,node.js and Selenium Web Driver.
As part of my education process I am developing simple bot for Instagram.
To emulate browser I use Chrome web driver.
Faced problem when trying to get list of followers and amount of followers for the account:
This code opens instagram page, enters credentials, goes to some account and opens followers for this account.
Data like username and password I take from the settings.json.
var webdriver = require('selenium-webdriver'),
by = webdriver.By,
Promise = require('promise'),
settings = require('./settings.json');
var browser = new webdriver
.Builder()
.withCapabilities(webdriver.Capabilities.chrome())
.build();
browser.manage().window().setSize(1024, 700);
browser.get('https://www.instagram.com/accounts/login/');
browser.sleep(settings.sleep_delay);
browser.findElement(by.name('username')).sendKeys(settings.instagram_account_username);
browser.findElement(by.name('password')).sendKeys(settings.instagram_account_password);
browser.findElement(by.xpath('//button')).click();
browser.sleep(settings.sleep_delay);
browser.get('https://www.instagram.com/SomeAccountHere/');
browser.sleep(settings.sleep_delay);
browser.findElement(by.partialLinkText('followers')).click();
This part should open all followers, but not working:
var FollowersAll = browser.findElement(by.className('_4zhc5 notranslate _j7lfh'));
Tried also by xpath:
var FollowersAll = browser.findElement(by.xpath('/html/body/div[2]/div/div[2]/div/div[2]/ul/li[3]/div/div[1]/div/div[1]/a'));
When I run in the browser's console:
var i = document.getElementsByClassName('_4zhc5 notranslate _j7lfh');
it is working fine.
I run code in debug mode (use WebStorm) and it shows in each case that variable "FollowersAll" is undfined.
The same happens when I try to check amount of followers for the account.
Thanks in advance.
example of the selected element
In DOM, class names may be used multiple time. In this case, findElement by className wont work.
Xpath should be Relative and should not be Absolute.
Try Xpath with unique HTML Attribute. For example:
1. //div[#id/text()='value']
In chrome browser, open Developer Tools(press F12). If you framed an Xpath, just press Ctrl+F and paste that Xpath. If it states 1 of 1, then you can surely use that Xpath.
If it states 1 of many, then you need to dig deeper to take exact Xpath.
Related
I'm running a custom function in App Scripts which utilizes the Youtube (YouTube Data API v3) advanced service. When running, I get the following error:
GoogleJsonResponseException: API call to youtube.videos.list failed with error: Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup. (line 15).
I'm not sure how to authenticate my application. I've added it to a cloud project and enabled the API's.
Update: Here's what my code looks like:
function getYoutubeData(youtubeId) {
// Don't run on empty
if(!youtubeId){return null}
// Make the request
var vidData = YouTube.Videos.list("statistics, snippet", {id: youtubeId}).items;
if (!vidData|vidData.length<1){return null}
// Get the first item
vidData = vidData[0];
return vidData.statistics
}
I believe your goal as follows.
You want to put the value of vidData.statistics in your script to the cell.
You want to achieve this using custom function like =getYoutubeData(youtubeId).
For this, how about this answer?
Issue and workaround:
Unfortunately, when YouTube Data API of Advanced Google services is used in the custom function, the access token is not used. From your script, I think that the reason of your issue is this. For example, when the function of const sample = () => ScriptApp.getOAuthToken(); is used as the custom function like =sample(), no value is returned. I think that this is the current specification of Google side because of the security.
In order to achieve your goal under above situation, how about the following workarounds?
Workaround 1:
In this workaround, at first, the youtube ID is set to the cells in Google Spreadsheet. And the value of vidData.statistics are retrieved by the Google Apps Script which is not the custom function and replace the youtube ID with the result values.
Sample script:
Please set the range of cells of youtube IDs to sourceRange and the sheet name. At the sample, it supposes that the youtube IDs are put to the cells "A1:A10". And please run getYoutubeData() at the script editor. Of course, you can also set this to the custom menu.
function getYoutubeData() {
const sourceRange = "A1:A10"; // Please set the range of cells of youtube IDs.
const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1"); // Please set the sheet name.
const range = sheet.getRange(sourceRange);
const youtubeIds = range.getValues();
const values = youtubeIds.map(([youtubeId]) => {
// This is your script.
if(!youtubeId){return [null]}
var vidData = YouTube.Videos.list("statistics, snippet", {id: youtubeId}).items;
if (!vidData|vidData.length<1){return [null]}
vidData = vidData[0];
return [JSON.stringify(vidData.statistics)];
});
range.setValues(values);
}
Workaround 2:
In this workaround, the custom function is used. But, in this case, the Web Apps is used as the wrapper. By this, the authorization process is done at the Web Apps. So the custom function can be run without the authorization. Please do the following flow.
1. Prepare script.
When your script is used, it becomes as follows. Please copy and paste the following script to the script editor.
Sample script:
// This is your script.
function getYoutubeData_forWebApps(youtubeId) {
// Don't run on empty
if(!youtubeId){return null}
// Make the request
var vidData = YouTube.Videos.list("statistics, snippet", {id: youtubeId}).items;
if (!vidData|vidData.length<1){return null}
// Get the first item
vidData = vidData[0];
return vidData.statistics
}
// Web Apps using as the wrapper.
function doGet(e) {
const res = getYoutubeData_forWebApps(e.parameter.youtubeId)
return ContentService.createTextOutput(JSON.stringify(res));
}
// This is used as the custom function.
function getYoutubeData(youtubeId) {
const url = "https://script.google.com/macros/s/###/exec?youtubeId=" + youtubeId; // Please set the URL of Web Apps after you set the Web Apps.
return UrlFetchApp.fetch(url).getContentText();
}
2. Deploy Web Apps.
On the script editor, Open a dialog box by "Publish" -> "Deploy as web app".
Select "Me" for "Execute the app as:".
By this, the script is run as the owner.
Select "Anyone, even anonymous" for "Who has access to the app:".
In this case, no access token is required to be request. I think that I recommend this setting for testing this workaround.
Of course, you can also use the access token. But, in this case, when the access token is used, this sample script cannot be directly used as the custom function.
Click "Deploy" button as new "Project version".
Automatically open a dialog box of "Authorization required".
Click "Review Permissions".
Select own account.
Click "Advanced" at "This app isn't verified".
Click "Go to ### project name ###(unsafe)"
Click "Allow" button.
Click "OK".
Copy the URL of Web Apps. It's like https://script.google.com/macros/s/###/exec.
When you modified the Google Apps Script, please redeploy as new version. By this, the modified script is reflected to Web Apps. Please be careful this.
Please set the URL of https://script.google.com/macros/s/###/exec to url of above script. And please redeploy Web Apps. By this, the latest script is reflected to the Web Apps. So please be careful this.
4. Test this workaround.
Please put =getYoutubeData("###youtubeId###") to a cell. By this, the youtube ID is sent to the Web Apps and the Web Apps returns the values of vidData.statistics.
Note:
These are the simple sample scripts for explaining the workarounds. So when you use this, please modify it for your actual situation.
References:
Custom Functions in Google Sheets
Web Apps
Taking advantage of Web Apps with Google Apps Script
I want to automate some searching stuff for myself, but I have a bit of a problem here.
On this website:
https://shop.orgatop.de/
The program can't find the search bar, and I don't really know why.
driver = webdriver.Firefox()
driver.get('https://shop.orgatop.de/')
input_search = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[#id="solrSearchTerm"]')))
input_search.click()
input_search.send_keys('asd')
input_search.send_keys(Keys.RETURN)
The element is present inside nested iframe like innerFrame>catalog>content>input.You need to switch those frame first inorder to access the input search box.
Induce WebDriverWait() and frame_to_be_available_and_switch_to_it()
driver = webdriver.Firefox()
driver.get('https://shop.orgatop.de/')
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"innerFrame")))
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"catalog")))
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"content")))
input_search = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, '//*[#id="solrSearchTerm"]')))
input_search.click()
input_search.send_keys('asd')
input_search.send_keys(Keys.RETURN)
Browser snashot:
I am trying to work out a comparable command to use in jmeter webdriver sampler (JavaScript) how to do a waitForPopUp command. There must be a way. I have something that works for waiting for an element, but I can't work it out for a popup.
Update
I am using this code for waiting for an element:
var wait = new support_ui.WebDriverWait(WDS.browser, 5000)
WaitForLogo = function() {
var logo = WDS.browser.findElement(org.openqa.selenium.By.xpath("//img[#src='/images/power/ndpowered.gif']"))
}
wait.until(new com.google.common.base.Function(WaitForLogo))
And this works, but I can't work out how reuse this to wait for a popup, that has no name, in Java I have used:
selenium.waitForPopUp("_blank", "30000");
selenium.selectWindow("_blank");
And that works, but I can't work out an comparable JavaScript that will work in Jmeter for performance, as I can't get Java working in Jmeter.
I was able to get this working using:
var sui = JavaImporter(org.openqa.selenium.support.ui)
and:
wait.until(sui.ExpectedConditions.numberOfWindowsToBe(2))
In WebDriver Sampler you have the following methods:
WDS.browser.switchTo.frame('frame name or handle') - for switching to a frame
WDS.browser.switchTo.window('window name or handle') - for switching to a window
WDS.browser.switchTo.alert() - for switching to a modal dialog
WDS.browser.getWindowHandles() - for getting all open browser window handles
See JavaDoc on WebDriver.switchTo method and The WebDriver Sampler: Your Top 10 Questions Answered guide for more details.
This is a follow up question to the query which I had about scraping web pages.
My earlier question: Pin down exact content location in html for web scraping urllib2 Beautiful Soup
This question is regarding doing the same, but the issue is to do the same recursively over multiple page s/views.
Here is my code
from selenium.webdriver.firefox import web driver
driver = webdriver.WebDriver()
driver.get('http://www.walmart.com/ip/29701960?page=seeAllReviews')
for review in driver.find_elements_by_class_name('BVRRReviewDisplayStyle3Main'):
title = review.find_element_by_class_name('BVRRReviewTitle').text
rating =review.find_element_by_xpath('.//div[#class="BVRRRatingNormalImage"]//img').get_attribute('title')
print title, rating
From the url, you'll see that no change is seen if we navigate to the second page, otherwise it wouldn't have been an issue. In this case, the next page clicker calls in a javascript from the server. Is there a way we can still scrape this using selenium in python just by some slight modification of my presented code ? Please let me know if there is.
Thanks.
Just click Next after reading each page:
from selenium.webdriver.firefox import webdriver
driver = webdriver.WebDriver()
driver.get('http://www.walmart.com/ip/29701960?page=seeAllReviews')
while True:
for review in driver.find_elements_by_class_name('BVRRReviewDisplayStyle3Main'):
title = review.find_element_by_class_name('BVRRReviewTitle').text
rating = review.find_element_by_xpath('.//div[#class="BVRRRatingNormalImage"]//img').get_attribute('title')
print title,rating
try:
driver.find_element_by_link_text('Next').click()
except:
break
driver.quit()
Or if you want to limit the number of pages that you are reading:
from selenium.webdriver.firefox import webdriver
driver = webdriver.WebDriver()
driver.get('http://www.walmart.com/ip/29701960?page=seeAllReviews')
maxNumOfPages = 10; # for example
for pageId in range(2,maxNumOfPages+2):
for review in driver.find_elements_by_class_name('BVRRReviewDisplayStyle3Main'):
title = review.find_element_by_class_name('BVRRReviewTitle').text
rating = review.find_element_by_xpath('.//div[#class="BVRRRatingNormalImage"]//img').get_attribute('title')
print title,rating
try:
driver.find_element_by_link_text(str(pageId)).click()
except:
break
driver.quit()
I think this would work. Although the python might be a little off, this should give you a starting point:
continue = True
while continue:
try:
for review in driver.find_elements_by_class_name('BVRRReviewDisplayStyle3Main'):
title = review.find_element_by_class_name('BVRRReviewTitle').text
rating =review.find_element_by_xpath('.//div[#class="BVRRRatingNormalImage"]//img').get_attribute('title')
print title, rating
driver.find_element_by_name('BV_TrackingTag_Review_Display_NextPage').click()
except:
print "Done!"
continue = False
I'm trying to read a node from a website that contains java script.
In VB .NET I just use the following code:
Dim listSpan As IHTMLElementCollection = bodyel.getElementsByTagName("span")
For Each spanItem As IHTMLElement In listSpan
If spanItem.className & "" = "span_name" Then
If Not spanItem.innerText Is Nothing Then
str_result = spanItem.innerText.ToString
Console.WriteLine("Found it: " & str_result)
Else
str_result = "NO"
Console.WriteLine("Not Found")
Console.Beep(500, 500)
End If
End If
Next
But I just can't find a way to convert this code to work in Android service. (Java).
I tried Jsoup but Jsoup is only reading the "view source code" elements and not the javascript results as html.
try {
Document doc = Jsoup.connect(str_link).get();
Elements links = doc.select("span_name");
for(Element link : links) {
String result = link.text();
Log.d("TMA Service","result: " + result);
list.add(title);
}
I mean. This code in VB can find everything. (just like if I right click in an element using google chrome and select "Inspect Element". This shows everything and I'd like to know how to get this data with Android.
CAN SOME ONE GIVE ME AN EXAMPLE?
Thanks.
Unfortunately you can't handle Javascript and dynamic content with Jsoup. Please see my answer here for more information and some examples of Java libraries, that may help you here.
Edit:
HtmlUnit - Getting started (section Getting started)
HtmlUnit: A Simple Example: Check Yahoo Email
How to use HtmlUnit in Java?
HtmlUnit: A Quick Introduction
HtmlUnit – A quick introduction
Getting started with HtmlUnit