I'm new to Javascript and even newer to google script, so please be comprehensive :)
I'm trying to build a little script to scrap some data from a bunch of URL. I' using Parser library. Here is what I've:
function getArray() {
var newData = new Array();
var sheet = SpreadsheetApp.openById('my_id').getSheetByName('Sheet4');
var urls = sheet.getRange(1,1,5,5).getValues();
var fromText = '<span class="nb-shares">';
var toText = '</span>';
for(i in urls){
var url = urls[i];
var content = UrlFetchApp.fetch(url).getContentText();
var scraped = Parser
.data(content)
.from(fromText)
.to(toText)
.build();
newData.push(scraped);}
var sheet2 = SpreadsheetApp.openById('my_id').getSheetByName('Sheet5');
sheet2.getRange(5, 1, newData.length, newData[1].length).setValues(newData);
}
It return me the following error : Cannot convert Array to Object
What I'm trying to do is looping on an URLs array so I can scrap some data from each one of these URL and return the results in my sheet.
Try changing newData.push(scraped) to newData.push([scraped])
Related
I can fetch the data using two different urls depending whether I want to file to be csv or json.
CSV edition
In the case of the csv url I do the following:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName("data")
get_url="https://opendata.rdw.nl/resource/m9d7-ebf2.csv?
datum_tenaamstelling=20200404%20&$limit=20000";
var result_data = UrlFetchApp.fetch(get_url);
var raw_data = result_data.getContentText()
Logger.log(raw_data);
I end up with a string that I can not store it to the spreadsheet file without using a loop.
JSON edition
Using the json url, I am doing the following:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName("data")
get_url="https://opendata.rdw.nl/resource/m9d7-ebf2.json?
datum_tenaamstelling=20200404%20&$limit=20000";
var result_data = UrlFetchApp.fetch(get_url, options);
var raw_data = result_data.getContentText()
var json_list = JSON.parse(raw_data)
for(var i = 0; i < json_list.length; i++) {
var obj = json_list[i];
Logger.log(obj.merk);
sheet.appendRow([obj.kenteken,
obj.merk,
obj.catalogusprijs,
obj.datum_eerste_afgifte_nederland,
obj.datum_eerste_toelating,
obj.datum_tenaamstelling
]);
}
Which again I end up using an iteration which is not practical if I want to store 20000 rows to the spreadsheet. After sometime it stops and returns a runtime error.
Is there any way I can directly copy the data to the spreadsheet file?
For example to do something like that:
sheet.getRange(1, 1,raw_data.length,raw_data[0].length).setValues(raw_data);
How to store specific from a JSON object response
Looping through all data and call the method sheet.appendRow() during each iteration would indeed be slow and inefficient due to the number of calls - see Apps Script Best Practices.
A significantly more efficient way to do it, would be to create an array to store the data of interest before writing it a single time with setValues().
The sample below still incorporates a for loop, however given that no calls to external services are made during each iteration, the whole script took only around 10 s to execute and write 12582 rows of data into the sheet.
Sample:
function myFunction() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName("data")
get_url="https://opendata.rdw.nl/resource/m9d7-ebf2.json?datum_tenaamstelling=20200404%20&$limit=20000";
var result_data = UrlFetchApp.fetch(get_url);
var raw_data = result_data.getContentText()
var json_list = JSON.parse(raw_data)
var array = [];
for(var i = 0; i < json_list.length; i++) {
var obj = json_list[i];
array.push([obj.kenteken,
obj.merk,
obj.catalogusprijs,
obj.datum_eerste_afgifte_nederland,
obj.datum_eerste_toelating,
obj.datum_tenaamstelling
]);
}
sheet.getRange(1, 1,array.length,array[0].length).setValues(array);
}
Ok...not sure how to do this. Right now I 4 sheets and 4 scripts for each sheet producing 4 json feeds. What I am trying to experiment with is having one script that will produce 1 json that I can use in a web page and just call the type of class. They are all formatted the same with columns etc.
Here is the Google Script App code I have.
function doGet(e){
// Change Spread Sheet url
var ss = SpreadsheetApp.openByUrl("https://docs.google.com/spreadsheets/d/SpreadsheetID/edit#gid=0");
// Sheet Name, Change Sheet to whatever name you used to name your sheet
var sheet = ss.getSheetByName("Class Sheet 1");
return getClasses(sheet);
}
function getClasses(sheet){
var dataObj = {};
var dataArray = [];
// collecting data starting from 2nd Row , 1st column to last row and last column
var rows = sheet.getRange(2,1,sheet.getLastRow()-1, sheet.getLastColumn()).sort([{column: 1, ascending: true}, 1]).getValues();
for(var i = 0, l= rows.length; i<l ; i++){
var dataRow = rows[i];
var record = {};
record['Name'] = dataRow[0];
record['Note'] = dataRow[1];
record['Address'] = dataRow[2];
record['StreetAddress'] = dataRow[3];
record['City'] = dataRow[4];
record['State'] = dataRow[5];
record['ZipCode'] = dataRow[6];
record['ContactName'] = dataRow[7];
record['EMailAddress'] = dataRow[8];
record['CustomerServicePhone'] = dataRow[9];
dataArray.push(record);
}
dataObj = dataArray;
var result = JSON.stringify(dataObj);
return ContentService.createTextOutput(result).setMimeType(ContentService.MimeType.JSON);
}
Scratching my head on this a little bit....I'm sure its something simple and I am probably over thinking things, but any help would be appreciated.
Possible Solution:
The e object in your doGet(e) provides a way to send parameters to your script. You can access different sheets with different url parameters. You can then easily get the requested SheetName through e.parameter. Use
https://script.google.com/.../exec?sheet=ClassSheet1 //for ClassSheet1
https://script.google.com/.../exec?sheet=ClassSheet2 //for ClassSheet2
Code.gs:
function doGet(e){
var ss = SpreadsheetApp.openByUrl("https://docs.google.com/spreadsheets/d/SpreadsheetID/edit#gid=0");
var sheetName = e.parameter.sheet;
var sheet = ss.getSheetByName(sheetName);
return getClasses(sheet);
}
You can also provide UI in your web-app to select a sheet.
// save to CSV
var csv = document.getElementById('a');
var csvContent = new Array();
csvContent.push(["COL_A","COL_B","COL_C"]);
var fileName = "file.csv";
var blob = new Blob(["\ufeff", csvContent]);
var url = URL.createObjectURL(blob);
csv.href = url;
csv.setAttribute('download', fileName);
csv.click();
// load CSV file
..
var lines = loadedCsvFile.split(/\r\n|\n/);
var result = [];
var headers = lines[0].split(",");
for(var i=0;i<lines.length;i++){
var obj = {};
var currentLine = lines[i].split(",");
for(var j=0;j<headers.length;j++){
obj[headers[j]] = currentLine[j];
}
result.push(obj);
}
this is my csv save/load code.
But, this code can use only first sheet.
how can I use second sheet?
I should be very grateful to you if you might help me.
I assume you're talking about a CSV from Excel (or the likes) which 'had' multiple sheets?
CSVs don't use the sheet concept, they're are saved as a flat file. You will either need to save each sheet as a CSV individually and read separately, or concatenate the sheets.
I need create Javascript array from xml ..
I`v get xml data from poly php with ajax. Everything is ok.
I must create array like that:
point = [
new google.maps.LatLng(40.9921196514,47.8604733650 ),
new google.maps.LatLng(40.9922511293,47.8606186245 ),
];
Code
downloadUrl("poly.php", function(data) {
var xml = data.responseXML;
var polys = xml.documentElement.getElementsByTagName("poly");
for (var i = 0; i < polys.length; i++) {
var pid = polys[i].getAttribute("pid");
point = [
new google.maps.LatLng(parseFloat(polys[i].getAttribute("plat")), parseFloat(polys[i].getAttribute("plng")) );
];
i`ve do that but it does not work.. ((
P.S. I get data from MySQL.
...
Xml:
<polys>
<poly pid="1" pstatus="status1" plat="40.992638" plng="47.860474"/>
<poly pid="2" pstatus="status2" plat="40.992252" plng="47.860619"/>
</polys>
May I assume you use the function downloadUrl from googles util.js ?
When yes: data is already a document, there is no need to access data.responseXML
Each attempt to access a property of xml will result in an error, because xml is undefined
Replace this:
var xml = data.responseXML;
var polys = xml.documentElement.getElementsByTagName("poly");
with:
var polys = data.documentElement.getElementsByTagName("poly");
There is an syntax-error:
point = [
new google.maps.LatLng(parseFloat(polys[i].getAttribute("plat")), parseFloat(polys[i].getAttribute("plng")) );
];
remove the semicolon:
("plng")) );
//---------^
But to get the desired result you must create the point-array outside the loop:
var point=[];
and add the LatLng's to point inside the loop:
point.push(new google.maps.LatLng(parseFloat(polys[i].getAttribute("plat")),
parseFloat(polys[i].getAttribute("plng"))));
what I want to do is access an excel file, take the contents of a column and then store it in an array. I'm guessing this can be done by iterating through each element of the range object and copying it to an array. My current code is shown below:
function GetData(){
var excel = new ActiveXObject("Excel.Application");
var excel_file = excel.Workbooks.Open("C:\\Data.xls");
var excel_sheet = excel.Worksheets("Sheet2");
//returns a RANGE object
var myrow = excel_sheet.Columns(1);
//doing this puts a single range object in myarray
var myarray = new Array(myrow);
//try printing the contents of the range object
for (mycell in myrow){
document.getElementById('div1').innerHTML = mycell;
}
}
Please check the following code to get data from an excel file. Hope this helps you:
CODE:
function GetData(){
var excel = new ActiveXObject("Excel.Application");
var excel_file = excel.Workbooks.Open("C:\\Data.xls");
var excel_sheet = excel.Worksheets("Sheet2");
for(var i=2;i<20;i++){
var myrow = excel_sheet.Range("A"+i); //to read values in row A
document.getElementById('div1').innerHTML = myrow;
}
}
Tested in IE9.