Develop Reference

Fetch paginated data without knowing the page number

I have an API that only allows fetching 1000 rows/fetch.
So for instance, if I want to retrieve all the data from the API, the idea is to loop through the response data each time fetching and check the length of it (if responseData.length !== 0, then continue fetching, stop when responseData.length === 0, also increase the firstRow every time it starts the new loop until it reaches to the end (responseData.length === 0)
const fetchDataByRowCount = async (url, token, rowCount = 2, firstRow = 0) => {
// firstRow is the value where the next fetch starts (E.g: 0-999, 1000-1999, etc.).
// rowCount is the value for total rows fetched (E.g: 1000 rows for each fetching time).
const data = await axios({
method: "get",
url: `${url}?rowCount=${rowCount}&firstRow=${firstRow}`,
headers: {
client_id: "",
Authorization: `Bearer ${token}`,
},
});
return data.data;
};
export const paginatedFetch = async (url, type, rowCount = 2, firstRow = 0) => {
let newResponse;
let total = [];
let token = await getToken(type); // stored to reuse token within an hour
do {
if (!token) {
const newToken = await getToken(type);
newResponse = await fetchDataByRowCount(url, newToken);
} else {
newResponse = await fetchDataByRowCount(
url,
token,
(rowCount = 2),
(firstRow = 0)
);
}
// console.log(total, "total");
total = [...total, ...newResponse];
// newResponse = [];
let newFirstRow = firstRow + 1000;
newResponse = await fetchDataByRowCount(
url,
token,
(rowCount = 2),
newFirstRow
);
total = [...total, ...newResponse];
} while (newResponse.length !== 0);
return total;
};
But the problem is that my function didn't exit the do while loop, the newResponse always returns value !==0.
Also, the function only runs once.
Could you guys help me check this, please?

From the code you posted, there's still a thing I can't figure out, and it's rowCount, so I let it as it is in the following "remastered" code:
export const paginatedFetch = async (url, type, rowCount = 2, firstRow = 0) => {
let newResponse;
let total = [];
let token;
let numberOfRows = firstRow;
do {
if (!token) token = await getToken(type);
newResponse = await fetchDataByRowCount(
url,
token,
(rowCount = 2),
numberOfRows
);
total = [...total, ...newResponse];
numberOfRows += 1000;
} while (newResponse.length !== 0);
return total;
};
I got rid of a few things that were redundant, and made the code a tiny bit more efficient with variable assignments etc.
You also mention this:
the newResponse always returns value !==0.
Be careful doing this, as newResponse is initially undefined. Now I never used do...while loops so I don't know exactly what could happen, but it could for example not run at all. Hence the Also, the function only runs once if you're speaking about the paginatedFetch function.
Now if I were to re-write it, I would do it like so:
export const paginatedFetch = async (url, type, rowCount = 2, firstRow = 0) => {
let total = [];
let token;
let numberOfRows = firstRow;
while (true) {
if (!token) token = await getToken(type);
let res = await fetchDataByRowCount(
url,
token,
(rowCount = 2),
numberOfRows
);
total = [...total, ...res];
numberOfRows += 1000;
if (res.length <= 0) break;
}
return total;
};
Again, be careful with while (true), you have to be absolutely sure of what the API returns and res is indeed an array.
The better solution would be the API (if you are the developer) giving an endpoint to count the total number of rows. That way, you would have a way to be absolutely sure how many rows there are and write your code around that.

Math.max.apply method and spread (...) function returning Infinity/NaN when processing an array

I am using the Math.max function to process an array using the spread function, but am still receiving a NaN error. Any help into this matter would be greatly appreciated.
import finnhub from 'https://cdn.skypack.dev/finnhub';
const api_key = finnhub.ApiClient.instance.authentications['api_key'];
api_key.apiKey = "c5cam0iad3ib55bb0ajg"
const finnhubClient = new finnhub.DefaultApi()
let stockTicker = prompt("Please enter the symbol of the stock to chart:");
//Finance API Function Call Below
finnhubClient.stockCandles(stockTicker, "D", ((Math.trunc(Date.now() / 1000)) - 8640000), Date.now(), (error, data, response) => {
let dataSetDayHigh = []
let dataSetDayRange = 1
for (dataSetDayRange = 1; dataSetDayRange < 41; dataSetDayRange = dataSetDayRange + 1){
dataSetDayHigh[dataSetDayRange] = Math.trunc(data.h[dataSetDayRange])
}
let yAxisRangeUpper = Math.max.apply(null, dataSetDayHigh)
console.log("Data Set Day High Values = " + yAxisRangeUpper)

How to push an entire array in javascript based on conditional content of single array element?

I have created a Google script that pushes data every hour from the Capital Bikeshare API to a Google Sheet, but I have noticed that the way I am currently pulling the data doesn't maintain consistency over time. Here's the code I'm using:
function myFunction() {
// Set the active spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
var currentData = ss.getSheetByName("Current");
var historicData = ss.getSheetByName("Historic");
// Fetch API
var stationInfo = UrlFetchApp.fetch('https://gbfs.capitalbikeshare.com/gbfs/en/station_information.json');
var stationStatus = UrlFetchApp.fetch('https://gbfs.capitalbikeshare.com/gbfs/en/station_status.json');
// Get the current date and time
var today = new Date();
var date = today.getFullYear()+'-'+(today.getMonth()+1)+'-'+today.getDate();
var time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
var dateTime = date+' '+time;
// Parse the JSON reply
var jsonInfo = stationInfo.getContentText();
var dataInfo = JSON.parse(jsonInfo);
var jsonStatus = stationStatus.getContentText();
var dataStatus = JSON.parse(jsonStatus);
// Create the data frame for every BID station
var stationInfo72 = dataInfo["data"]["stations"][69];
var stationStatus72 = dataStatus["data"]["stations"][69];
var stationInfo87 = dataInfo["data"]["stations"][83];
var stationStatus87 = dataStatus["data"]["stations"][83];
var stationInfo330 = dataInfo["data"]["stations"][311];
var stationStatus330 = dataStatus["data"]["stations"][311];
var stationInfo153 = dataInfo["data"]["stations"][143];
var stationStatus153 = dataStatus["data"]["stations"][143];
var stationInfo226 = dataInfo["data"]["stations"][213];
var stationStatus226 = dataStatus["data"]["stations"][213];
var stationInfo365 = dataInfo["data"]["stations"][342];
var stationStatus365 = dataStatus["data"]["stations"][342];
var stationInfo473 = dataInfo["data"]["stations"][446];
var stationStatus473 = dataStatus["data"]["stations"][446];
var outputStationsInfo = [stationInfo72, stationInfo87, stationInfo330, stationInfo153, stationInfo226, stationInfo365, stationInfo473]
var outputStationsStatus = [stationStatus72, stationStatus87, stationStatus330, stationStatus153, stationStatus226, stationStatus365, stationStatus473]
Logger.log(outputStationsInfo, outputStationsStatus)
// Create lists of each element
var outputHead = [];
var outputTail = [];
outputStationsInfo.forEach(function(elem,i) {
outputHead.push([elem["station_id"],elem["name"],elem["capacity"], elem["lat"], elem["lon"]]);
});
outputStationsStatus.forEach(function(elem,i) {
outputTail.push([elem["num_bikes_available"], elem["num_ebikes_available"], dateTime]);
});
// Publish arrays in the Current sheet
currentData.getRange(2,1,7,5).setValues(outputHead);
currentData.getRange(2,6,7,3).setValues(outputTail);
// Publish arrays in the Historic sheet
historicData.getRange(historicData.getLastRow() + 1,1,7,5).setValues(outputHead);
historicData.getRange(historicData.getLastRow() - 6,6,7,3).setValues(outputTail);
}
Essentially, I am drilling into the 69th item in the indexes of the JSONs to get the data that I need from two different APIs, and then I merge them together to create a data frame of everything I need to push to the sheet. However, sometimes the API does not report them in the normal order and I end up getting bikeshare stations that aren't in my study area. For example, 99% of the time the 69th item in the array is station_id = 72, but occasionally it's station_id = 73 or something.
Is there a way to conditionally pull a specific array based on the station_id number within the array? I feel like the answer might allow me to do a loop as well to clean this up. Any advice is helpful, as I'm super new to this.

You have to check if the element's station_id is as expected. If not, check through the surrounding parts of the array using a custom iterator.
Snippet:
/**
* #return indexes of the surrounding ``i`` in batches of 5
*/
function* checkSurroundings(i, lastIndex) {
let j = i;
function* check(ct, border, reverse = true, limit = border < 5 ? border : 5) {
const margin = reverse ? ct - limit : ct + limit;
while (ct - margin !== 0) yield reverse ? --ct : ++ct;
return ct;
}
while (i !== 0 || j < lastIndex) {
if (i !== 0) i = yield* check(i, i);
if (j < lastIndex) j = yield* check(j, lastIndex - j, false);
//console.log({ i, j });
}
}
var stations = dataInfo["data"]["stations"];
var stationInfo72 = stations[69];
const iter = checkSurroundings(69,stations.length-1)
//if station_id is not 72, loop through the surrounding indexes
while(stationInfo72["station_id"] !== 72){
const next = iter.next();
if(next.done) {
console.error("station id 72 not found");
break;
}
stationInfo72 = stations[next.value]
}
Snippet showing how checkSurroundings iterates:
/**
* #return indexes of the surrounding ``i`` in batches of 5
*/
function* checkSurroundings(i, lastIndex) {
let j = i;
function* check(ct, border, reverse = true, limit = border < 5 ? border : 5) {
const margin = reverse ? ct - limit : ct + limit;
while (ct - margin !== 0) yield reverse ? --ct : ++ct;
return ct;
}
while (i !== 0 || j < lastIndex) {
if (i !== 0) i = yield* check(i, i);
if (j < lastIndex) j = yield* check(j, lastIndex - j, false);
console.log({ i, j });
}
}
console.log("Order of iteration",[...checkSurroundings(50, 100)])

Conditionally picking elements: filter
For conditionally picking elements from an array in JavaScript, Array.prototype.filter should always be a consideration.
Create a predicate function that matches the shape of your data and checks for certain station IDs.
Here is a function that returns a predicate function. You put in the IDs you want in an array, and it returns the required function for filter.
function byStationId(stationIds) {
return function (obj) {
return stationIds.indexOf(obj.station_id) > -1;
};
}
var myStationFilter = byStationId([72, 73, 74]);
var outputStationsInfo = dataInfo.data.stations.filter(myStationFilter);
Transforming data: map
The pattern
var newArray = [];
oldArray.forEach(function (item) {
newArray.push(/* something based on item */);
});
can usually be replaced with Array.prototype.map
var newArray = oldArray.map(function (item) { return /* something based on item */});
Think of this as the "adapter" from one data shape to another.
function cleanInfo(info) {
return [info.station_id, info.name, info.capacity, info.lat, info.lon];
}
var outputHead = outputStationsInfo.map(cleanInfo);
For the dateTime injection, just do the same trick demonstrated above with the station IDs: have a function that takes a date string and returns the appropriate adapter function.
(Also note the provided date formatting utility Apps Scripts provides, Utilities.formatDate())
var dateTime = Utilities.formatDate(
new Date(),
ss.getSpreadsheetTimeZone(),
"yyyy-MM-dd HH:mm:ss"
);
function cleanStatus(dateTime) {
return function (status) {
return [status.num_bikes_available, status.num_bikes_available, dateTime];
};
}
var outputTail = outputStationsStatus.map(cleanStatus(dateTime));
Here's everything together, untested, just for inspiration. You must at the very least update the line with the station IDs to match your desired station IDs. Note that the helper functions for map and filter are at the bottom, taking advantage of JavaScript's hoisting feature.
function myFunction() {
// Set the active spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
// Get the current date and time
var dateTime = Utilities.formatDate(
new Date(),
ss.getSpreadsheetTimeZone(),
"yyyy-MM-dd HH:mm:ss"
);
// Fetch API
var stationInfo = UrlFetchApp.fetch(
"https://gbfs.capitalbikeshare.com/gbfs/en/station_information.json"
);
var stationStatus = UrlFetchApp.fetch(
"https://gbfs.capitalbikeshare.com/gbfs/en/station_status.json"
);
// Parse the JSON reply
var dataInfo = JSON.parse(stationInfo.getContentText());
var dataStatus = JSON.parse(stationStatus.getContentText());
// Create the data frame for every BID station
var myStationFilter = byStationId([72, 73, 74]); //!! UPDATE THESE NUMBERS
var outputStationsInfo = dataInfo.data.stations.filter(myStationFilter);
var outputStationsStatus = dataStatus.data.station.filter(myStationFilter);
// Create lists of each element
var outputHead = outputStationsInfo.map(cleanInfo);
var outputTail = outputStationsStatus.map(cleanStatus(dateTime));
// Publish arrays in the Current sheet
var currentData = ss.getSheetByName("Current");
currentData.getRange(2, 1, 7, 5).setValues(outputHead);
currentData.getRange(2, 6, 7, 3).setValues(outputTail);
// Publish arrays in the Historic sheet
var historicData = ss.getSheetByName("Historic");
historicData
.getRange(historicData.getLastRow() + 1, 1, 7, 5)
.setValues(outputHead);
historicData
.getRange(historicData.getLastRow() - 6, 6, 7, 3)
.setValues(outputTail);
//-------- helper functions ------------
function byStationId(stationIds) {
return function (obj) {
return stationIds.indexOf(obj.station_id) > -1;
};
}
function cleanInfo(info) {
return [info.station_id, info.name, info.capacity, info.lat, info.lon];
}
function cleanStatus(dateTime) {
return function (status) {
return [status.num_bikes_available, status.num_bikes_available, dateTime];
};
}
}

G-script : List and Filter G-Drive Files from Variables Stored in a G-Sheet

I try to list files from G-Drive with filters. Filters variables are stored in a G-Sheet.
Here comes the code. It does not work. But it does when I put text string instead of variables.
Thanks a lot,
function Lister_fichiers() {
// Log the name of every file in the user's Drive that modified after date = 'Var!'A1,
// whose name contains "MARQUE = 'Var!'B1".
var app = SpreadsheetApp;
var classeur = app.getActiveSpreadsheet();
const Formsheet = classeur.getSheetByName('Var');
const CsvM = classeur.getSheetByName('CsvM');
var Date1 = Formsheet.getRange('A1').getValue();
var Date2 = new Date(Date.UTC(Date1));
var Marque = Formsheet.getRange('B1').getValue();
var Pays = Formsheet.getRange('C1').getValue();
// this line here does not work and causes an error
var sffiles = DriveApp.searchFiles("modifiedDate > '"+Date2+"' and title contains '"+Marque+"' ");
}
Also, it's without errors but still not working without the date filter.
In the sheet, A1 = "2019-01-01" and B1 = "Marque1"
// Lines below works fine
// var sffiles = DriveApp.searchFiles('modifiedDate > "2019-01-01" and title contains "Marque1" and mimeType = "application/vnd.google-apps.spreadsheet" and trashed=false');
while (sffiles.hasNext()){
var sffile = sffiles.next();
var sfname = sffile.getName();
var sfdate = sffile.getLastUpdated();
var sfsize = sffile.getSize();
var sfurl = sffile.getUrl();
var fileParents = sffile.getParents();
while (fileParents.hasNext()) {
var sffolder = fileParents.next();
Logger.log(sffolder.getName());
}
var sfid = sffile.getId();
Logger.log(sffile.getId());
Logger.log(sffile.getName());
var sflist = [sfname,sfdate,sfsize,sfurl,sfid,sffolder];
CsvM.appendRow(sflist);
}
CsvM.appendRow(["A-Name", "Date", "Size", "URL", "ID", "Dossier Parent"]);
CsvM.sort(1);
}

What is the exact error message? I suspect the issue is with this line:
var Date2 = new Date(Date.UTC(Date1));
If var Date1 A1 is "2019-01-01", you should just be able to do:
var Date2 = new Date(Date1)
Which gives you the datetime object.
I think the confusion is that you can pass a string like "2019-01-01" to new Date() [0] but not to Date.UTC() [1]
[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date
[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/UTC

How to get all items list on NetSuite?

I am just starting with NetSuite and trying to pull all items with details using Restlet. With some research, I am able to pull all the items but the way I am doing now is not straightforward. I first pull the all ids of item using nlapiSearchRecord and loop through each id to get details of each item using nlapiLoadRecord and added to array. This way, it is taking to much time. Is there other way to pull all items with their details? Below is my code.
function getAllIDs() {
return nlapiSearchRecord('item', null, null, null);
}
function getRecord() {
var all_IDs = getAllIDs();
var len=all_IDs.length;
var result =new Array();
for(var i=0;i<all_IDs.length;i++) {
if(all_IDs[i].getRecordType()==="inventoryitem")
result[i]=nlapiLoadRecord(all_IDs[i].getRecordType(),all_IDs[i].id)
}
return result;
}

You can use what #Krypton suggested but you will always get 1000 results at max.
Try following if you have requirement to get more than 1000 (using Suitescript 2.0):
var columns = [];
var filters = [['isinactive', 'is', 'F']];
columns.push(search.createColumn({ name: "itemid"}));
columns.push(search.createColumn({ name: "displayname"}));
columns.push(search.createColumn({ name: "salesdescription"}));
columns.push(search.createColumn({ name: "baseprice"}));
var inventoryitemSearch = search.create({
type: search.Type.INVENTORY_ITEM, //Change the type as per your requirement
filters: filters,
columns: columns
});
var arrResults = [];
var count = 1000;
var startIndex = 0;
var endIndex = 1000;
var resultSet= inventoryitemSearch.run();
while (count == 1000) {
var results = resultSet.getRange(startIndex, endIndex);
arrResults = arrResults.concat(results);
startIndex = endIndex;
endIndex += 1000;
count = results.length;
}
log.debug({title: 'arrResults ', details: arrResults });

You can include the details you want in the search. So, for example, you can include an nlobjSearchFilter so that the search only returns inventory items, and add an nlobjSearchColumn for each field you want to see in the details. This way all the details you want to see are returned with the search and you can loop through the results to do what you want with them without loading every record individually - which will be where most of the performance hit is happening.
An example:
var inventoryitemSearch = nlapiSearchRecord("inventoryitem",null,
[
["type","anyof","InvtPart"]
],
[
new nlobjSearchColumn("itemid",null,null).setSort(false),
new nlobjSearchColumn("displayname",null,null),
new nlobjSearchColumn("salesdescription",null,null),
new nlobjSearchColumn("baseprice",null,null)
]
);
Then you can loop through the results to get details:
var name, displayName, description, price;
for ( var i = 0; inventoryitemSearch != null && i < searchresults.length; i++ ) {
var searchresult = inventoryitemSearch[ i ];
name = searchresult.getValue( 'itemid' );
displayName = searchresult.getValue( 'displayname' );
description = searchresult.getValue( 'salesdescription' );
price = searchresult.getValue( 'baseprice' );
}
There is a lot to learn about scripted searches in NetSuite, so I'd recommend starting here (NetSuite login required) and follow the links and keep reading / experimenting until your eyes glaze over.

I just like to use a generic function that accepts a search object...
const getAllResults = searchObj => {
try {
const Resultset = searchObj.run()
const maxResults = searchObj.runPaged().count
let ResultSubSet = null
let index = 0
const maxSearchReturn = 1000
let AllSearchResults = []
do {
let start = index
let end = index + maxSearchReturn
if (maxResults && maxResults <= end) {
end = maxResults
}
ResultSubSet = Resultset.getRange(start, end)
if (ResultSubSet.length === 0) {
break
}
// we could intriduce a record processor to lighetn up the load
AllSearchResults = AllSearchResults.concat(ResultSubSet)
index = index + ResultSubSet.length
if (maxResults && maxResults == index) {
break
}
} while (ResultSubSet.length >= maxSearchReturn)
return AllSearchResults
} catch (e) {
log.error(`getAllResults()`, `error : ${e}`)
}
}