So I am trying to scrape some content with node.js x-ray scraping framework. While I can get the content from a single page but for exemple only for one employee I can't get my head around on how to get for all the employees.
Working Exemple but return me the first employee:
const request =require('request');
const Xray=require('x-ray');
var x = Xray();
x('http://www.viadeo.com/fr/company/unicef',
'.pan',[{
name:'.pan-emp-name',
job:'.pan-emp-pos',
since:'.pan-emp-age'
// job:'#profile #overview-summary-current ol'
}]).write('result.json')
Thank you so much
x('http://www.viadeo.com/fr/company/unicef',
'#pan-emp .pan-employees .pan-empployee',[{
company:'#company-info .company-logo-picture',
nom:'.pan-emp-name',
job:'.pan-emp-pos',
depuis:'.pan-emp-age'
// job:'#profile #overview-summary-current ol'
}]).write('result.json')
Working like a charm,
So now my problem is to get the company info
Related
I'm trying to parse a stringified JSON output from a web app created from a google sheets script. I thought it couldn't be that complicated, but I've tried everything I could think of or find out online... so now asking for help if that's OK!
on the web app / Google Sheets side, the code is:
function doGet(e) {
var spreadsheet = SpreadsheetApp.openById('spreadsheetID');
var worksheet = spreadsheet.getSheetByName('Rankings C/U');
var output = JSON.stringify({ data: worksheet.getDataRange().getValues() });
return HtmlService.createHtmlOutput(output);
}
I've published the script, the web app works, I'm OK with that bit.
I've put random values on the spreadsheet: [[1,2],[3,4]] if we speak in matrix format.
on the other end, I've tried a bunch of stuff including .fetch, JSON.parse() to get the data in a usable format within the Google Sites embedded code, but the real issue is that I think I can't get to allocate the payload to a variable?
I'm using Google Sites to fetch the data.
with the basic module "<> embed", with the "by URL" option, with the following code:
https://script.google.com/macros/s/scriptID/exec
I get the following output - that looks what it should be:
{"data":[[1,2],[3,4]]}
but when trying to include this in a script module ("embed code") - no chance!
<form name="get-images">
<input name="test" id="test" value="we'll put the contents of cell A1 here">
</form>
<script>
const form = document.forms['get-images']
var usableVariable = JSON.parse("https://script.google.com/macros/s/scriptID/exec"); // here I'm trying to allocate the stringified JSON to a variable
form.elements['test'].value = usableVariable[1,1]; //allocating the first element of the parsed array
</script>
I'm pretty sure I'm missing something obvious - but now I ran out of ideas!
Thanks for any help :)
I believe your goal as follows.
In your situation, the bottom script is embedded to the Google site.
You want to retrieve the values from doGet and want to put the value of cell "B2" to the input tag.
The settings of Web Apps is Execute the app as: Me and Who has access to the app: Anyone, even Anonymous.
Modification points:
In your case, I think that return ContentService.createTextOutput(output); is suitable instead of return HtmlService.createHtmlOutput(output); in Google Apps Script.
In order to retrieve the values from doGet, in this modification, fetch is used.
You want to retrieve the cell "B2" from usableVariable[1,1];, please modify it to usableVariable[1][1];
When above points are reflected to your script, it becomes as follows.
Modified script:
Google Apps Script side:
function doGet(e) {
var spreadsheet = SpreadsheetApp.openById('spreadsheetID');
var worksheet = spreadsheet.getSheetByName('Rankings C/U');
var output = JSON.stringify({ data: worksheet.getDataRange().getValues() });
return ContentService.createTextOutput(output);
}
HTML & Javascript side:
<form name="get-images">
<input name="test" id="test" value="we'll put the contents of cell A1 here">
</form>
<script>
let url = "https://script.google.com/macros/s/###/exec";
fetch(url)
.then((res) => res.json())
.then((res) => {
const usableVariable = res.data;
const form = document.forms['get-images'];
form.elements['test'].value = usableVariable[1][1]; // usableVariable[1][1] is the cell "B2".
});
</script>
Note:
When you modified the Google Apps Script of Web Apps, please redeploy the Web Apps as new version. By this, the latest script is reflected to the Web Apps. Please be careful this.
In my environment, I could confirm that above HTML & Javascript worked in the Google site by embedding.
References:
Class ContentService
Using Fetch
Web Apps
I am building a web application in NodeJS with Express, Angular JS and Google App-Engine Datastore.
I am learning Node. I created a form that "sanitizes" (escapes) user input before inserting into the database. I followed the NodeJS tutorial on the MDN website to create this code:
//Trim and escape all inputs
req.sanitize('requester').escape();
req.sanitize('requester').trim();
req.sanitize('dataowner').escape();
req.sanitize('dataowner').trim();
req.sanitize('requested_filepath_list').escape();
req.sanitize('requested_filepath_list').trim();
The 'requested_filepath_list' is a list of UNIX file paths.
So when a user submits the request, it is stored in the database in the "Escaped" format.
//Escaped data
/top/example/test123.txt
Question: How can I "unescape" the data for display purposes?
//Desired output
/top/example/test123.txt
I tried the unescape function but it does not seem to work, it just returns the same output.
let escape_str = '/top/example/test123.txt';
let unescaped_str = unescape(escape_str);
console.log('unescaped_str: ' + unescaped_str);
//Output
unescaped_str: /top/example/test123.txt
//Desired output
/top/example/test123.txt
I was able to use the 'he' library to achieve this requirement.
Here's a post with the details: What's the right way to decode a string that has special HTML entities in it?
Here's the library. I was able to install it using npm install.
https://www.npmjs.com/package/he
Example solution:
const he = require('he');
let escape_str = '/top/example/test123.txt';
let unescaped_str = he.decode(escape_str);
console.log('unescaped_str ' + unescaped_str);
You can try this :)
const querystring = require('querystring');
querystring.unescape(escape_str);
Hi I am building a web app with Flask Python. I got a problem here:
#app.route('/analytics/signals/<ticker_url>')
def analytics_signals_com_page(ticker_url):
all_ticker = full_list
ticker_name = com_name
ticker = ticker_url.upper()
pricerec = sp500[ticker_url.upper()].tolist()
timerec = sp500[ticker_url.upper()].index.tolist()
return render_template('company.html', all_ticker=all_ticker, ticker_name=ticker_name, ticker=ticker, pricerec=pricerec, timerec=timerec)
Here I am defining company pages based on the a page will contain different content. The problem is that everything is fine upto ticker = ticker_url.upper(). It works perfectly fine. But for pricerec and timerec, they make problems.
sp500 is a pandas DataFrame columns being companies like "AAPL", "GOOG","MSFT", and so forth 505 companies and the index are timestamps, and values are the prices at each time.
So what I am doing for the pricerec, I am taking the ticker_url and use it to take the specific company's price and make it as a list. And timerec is to take the index (timestamps) and make it as a list. And I am passing these two variables into the company.html page.
But it makes internal server error. I do not know why it happens.
My expectation was that when a user click a button that href to "~/analytics/signals/aapl" then the company.html page will contain the pricerec and timerec for me to draw a graph. But it didn't work like that. It makes internal server error. I defined those two variables in the javascript also like I did for the other variables(all_ticker, ticker_name, and ticker)
Can anyone help me with this issue?
Thanks!
Hoping one of you guys can help me here. I'm trying to import data into excel from this webpage as well as some others that are the same website: https://re.po.st/qZZYQJ - basically I want to take the total clicks figure: 322,627 and have that in a column beside the URL in excel. I tried using importhtml and importxml in google spreadsheets but could not get it to work. I believe it is because this element is rendered in javascript? Does anyone know how to get this element? Is there a VBA code I could use. I'm not the most technical person so my apologies if I'm overseeing some things.
Mike
Function test(byval strLocation as string) as string
Dim driver As SeleniumWrapper.WebDriver
Set driver = New SeleniumWrapper.WebDriver
driver.Start "chrome", "https://re.po.st"
driver.Open strLocation
Dim data1
data1 = driver.FindElementById("sguidtotaltable").FindElementByTagName("span").Text
test = data1
driver.stop 'Stops the browser
End Function
I'm having massive problems with databinding to a ListView in a Windows 8 app using Javascript.
Inside the "activated" event on default.js I have written some code to get some data from a web service and push it into an array. This bit works OK and the array is populated.
The problem I have is that the app won't recognise the data. I have this code in a page called inspections.html:
data-win-options="{itemTemplate: select('#imageTextListCollectionTemplate'),
itemDataSource: dataList.dataSource,
layout: {type: WinJS.UI.ListLayout}}
and then in the "activated" event I declare:
var dataList = new Array();
and push the data from the web service into this array. But at runtime I get an error that says something along the lines of "can't find dataSource on undefined dataList".
I've done some of the examples on the MS website and in one of them it creates a dummy dataset and references it from a namespace. I kinda think that what I'm missing here is a namespace too but I don't know what the namespace for default.js is. Or maybe I'm wrong and it's something totally different.
Please help - this is so fundamental (and should be easy) but I can't get my head around it.
Do you want to create datalist in HTML or javascript?
It seems you want to create it from JavaScript. Assuming that you have already pushed your data into array from your webservice, you only need to call:
var dataList = new WinJS.Binding.List(array);
now accessing dataList.dataSource is perfectly valid.
Also, to create the datalist you don't always need an array. You could probably start with an empty list and then keep inserting data directly into the data list from web services, like:
var dataList = new WinJS.Binding.List([]);
dataList.push(value1);
dataList.push(value2);
...
Hope it helps. Let me know if you have any more questions.
If you are getting troubled by assigning datasource in HTML side
Prefer js side like
var list = new WinJS.Binding.List(array here);
listView.itemDataSource = list.dataSource;
by using this you can easily go through the data which you are binding to ListView.