Apps Script - Need Advice with Optimization or Maximum Runtime Bypass - javascript

I'm trying to run a script in Apps Script, but the problem is that with my current code it would take up to 6 hours to complete the function/execution. With the one request maximum runtime of 6 minutes and daily runtime limit of 1 hour it would take me 6 days do get the needed data.
My 3 questions would be:
Is there any way I can optimize my code for it to have a quicker execution/runtime?
Is there a way to bypass the single maximum runtime of 6 minutes?
If there is a way to bypass the 6 minutes, is there a way to bypass the daily maximum runtime of 1 hour?
The code I'm running is the following:
function scrapeBuff163() {
var startIndex = 33858;
var endIndex = 60000;
var response;
for(var i = startIndex; i <= endIndex; i++){
response = UrlFetchApp.fetch("https://buff.163.com/api/market/goods/sell_order?game=csgo&goods_id="+i,
{muteHttpExceptions: true });
if(response.getContentText().length >= 1500){
spliceAndInsert(response);
}
}
}
function spliceAndInsert(response){
const ss = SpreadsheetApp.getActiveSpreadsheet()
const sheet = ss.getSheetByName("Buff163");
var marketHashSubLen = 21;
var goodsIdSubLen = 12;
var idStart = response.getContentText().search('"goods_id"');
var goodsId = response.getContentText().slice(idStart,idStart+17).substring(goodsIdSubLen);
var marketHashStart = response.getContentText().search('"market_hash_name"');
let marketHashName = response.getContentText().slice(marketHashStart,marketHashStart+100).substring(marketHashSubLen);
marketHashName = marketHashName.substring(0, marketHashName.search('"'));
sheet.appendRow([goodsId, marketHashName]);
}
Thank you in advance! :D
p.s. I know my code isn't using the best practices.

Related

loadTable() function is p5.js only working as expected for 2 out of the 5 .csv files given whilst using same procedure each time

I'm developing in Google Chrome browser using node.js and p5.js. I want to read 5 .csv files and place the values into arrays, which has worked successfully for 2 out of the 5 files. To do this I have used the loadtable function to place the data into p5 table objects to then loop through to retrieve the values and place them into arrays. To loop through, my for loop is using the getrowcount function as a bound which returns the correct number of rows for the two working instances, however for the 3 that don't work, the row count returned is zero, even though when I look at the table objects I've loaded the csv data into, they are clearly populated with the records. I can see that the "columns" and "rows" value when expanding the table objects do read 0, however when traversing down another layer, the "columns" and "rows" values are correctly stored and I'm assuming the function is returning the first incorrect case of rows and columns. Please find attached links to screenshots as I'm sure my explanation is very confusing. The method I've used is very simple and almost identical each time so I'm unsure where I could have made a mistake, any ideas would be great.
console log screenshot https://imgur.com/HmcZmw9
code screenshot https://imgur.com/Ouv6OLX
code screenshot https://imgur.com/cxEkWVc
code screenshot https://imgur.com/3sPGXgb
code screenshot https://imgur.com/LlRdEhw
The first line in the console log shows a successful occurrence where the data table is loaded in fine, the second line shows the array being populated correctly and the third line prints the correct row count.
The second data table shows an incorrect time where the table is populated with values but the rows and columns count is incorrect, but when expanding the object once, you can see the correct rows and columns count is there. Below shows the empty array and row count of 0.
This is my first post so apologies if I've asked in the wrong way. Happy to give more information. Thanks.
The following is the code used. Most of the operations take place in function setup():
// This file retrieves and formats the data from smartgrid readings
// Create p5 Table objects
let wind_energy_table = new p5.Table();
let solar_energy_table = new p5.Table();
let weather_10_table = new p5.Table();
let weather_60_table = new p5.Table();
let weather_forecast_table = new p5.Table();
// Create arrays to handle and access values easier
const wind_energy = [];
const solar_energy = [];
const weather_10 = [];
const weather_60 = [];
const weather_forecast = [];
// Populate tables with values, preload runs before setup
function preload() {
wind_energy_table = loadTable("/data/wind_energy.csv", "csv");
solar_energy_table = loadTable("/data/solar_energy.csv", "csv");
weather_10_table = loadTable("/data/weather_data_10.csv", "csv");
weather_60_table = loadTable("/data/weather_data_60.csv", "csv");
weather_forecast_table = loadTable("/data/weather_forecast.csv", "csv");
}
function setup() {
let we_date, we_time, we_p1, we_p2, we_p3, we_pt;
// pi indicates power of ith wind farm (1,2,3), pt = power exchange (Megawatts)
let se_date, se_time, se_ps;
// ps = power in watts of subgrid
let w10_date, w10_time, w10_airtemp, w10_wd, w10_ws, w10_sd, w10_ap, w10_precip;
// wd = wind direction (in decadegrees Celsius), ws = wind speed (in 0.1 m/s), sd = sunshine duration in last 10 minutes,
// air pressure (in 0.1 hPa), precip = amount of precipitation in last 10 minutes
// decadegrees = 10 degrees
// hPa = hectopascal i.e. 100 pascal
let w60_date, w60_time, w60_cc, w60_wd, w60_ws, w60_at, w60_ap, w60_sd, w60_precip;
// cc = cloud coverage (in 1/8), wd = wind direction (in decadegrees Celsius), ws = wind speed (in 0.1 m/s),
// at = air temperature (in 0.1 degrees), ap = air pressure (in 0.1 hPa), sd = sunshine duration in last hour,
// precip = precipitation in last hour
// cloud coverage 8/8 = whole sky covered
let wf_date, wf_time, wf_validDate, wf_temp, wf_dp, wf_ws, wf_gs, wf_ap, wf_precipProb, wf_cc, wf_si, wf_wd, wf_ah, wf_ad
// validDate = date the forecast is valid for, dp = dew point (in Celsius), ws = wind speed (in m/s), gs = gust speed (in m/s),
// ap = air pressure (in hPa), precipProb = probability of precipitation (in %), cc = cloud coverage, si = solar irradiance (in kJ/m^2),
// wd = wind direction (in degrees), ah = air humidity (in %), ad = air density (in J/(kg*K))
// temp (in degrees Celsius), J = Joule, K = Kelvin
// Populate arrays with arrays for each values by looping through the tables in csv files
for (let i = 0; i < wind_energy_table.getRowCount(); i++) {
we_date = wind_energy_table.getRow(i).arr[0];
we_time = wind_energy_table.getRow(i).arr[1];
we_p1 = wind_energy_table.getRow(i).arr[2];
we_p2 = wind_energy_table.getRow(i).arr[3];
we_p3 = wind_energy_table.getRow(i).arr[4];
we_pt = wind_energy_table.getRow(i).arr[5];
wind_energy[i] = [we_date, we_time, we_p1, we_p2, we_p3, we_pt];
}
console.log(wind_energy_table);
console.log(wind_energy);
console.log(wind_energy_table.getRowCount());
for (let j = 0; j < solar_energy_table.getRowCount(); j++) {
se_date = solar_energy_table.getRow(j).arr[0];
se_time = solar_energy_table.getRow(j).arr[1];
se_ps = solar_energy_table.getRow(j).arr[2];
solar_energy[j] = [se_date, se_time, se_ps];
}
console.log(solar_energy_table);
console.log(solar_energy);
console.log(solar_energy_table.getRowCount());
console.log(solar_energy_table.getRow(0));
for (let k = 0; k < weather_10_table.getRowCount(); k++) {
w10_date = weather_10_table.getRow(k).arr[0];
w10_time = weather_10_table.getRow(k).arr[1];
w10_airtemp = weather_10_table.getRow(k).arr[2];
w10_wd = weather_10_table.getRow(k).arr[3];
w10_ws = weather_10_table.getRow(k).arr[4];
w10_sd = weather_10_table.getRow(k).arr[5];
w10_ap = weather_10_table.getRow(k).arr[6];
w10_precip = weather_10_table.getRow(k).arr[7];
weather_10[k] = [w10_date, w10_time, w10_airtemp, w10_wd, w10_ws, w10_sd, w10_ap, w10_precip];
}
console.log(weather_10_table);
console.log(weather_10);
for (let l = 0; l < weather_60_table.getRowCount(); l++) {
w60_date = weather_60_table.getRow(l).arr[0];
w60_time = weather_60_table.getRow(l).arr[1];
w60_cc = weather_60_table.getRow(l).arr[2];
w60_wd = weather_60_table.getRow(l).arr[3];
w60_ws = weather_60_table.getRow(l).arr[4];
w60_at = weather_60_table.getRow(l).arr[5];
w60_ap = weather_60_table.getRow(l).arr[6];
w60_sd = weather_60_table.getRow(l).arr[7];
w60_precip = weather_60_table.getRow(l).arr[8];
weather_60[l] = [w60_date, w60_time, w60_cc, w60_wd, w60_ws, w60_at,
w60_ap, w60_sd, w60_precip
];
}
console.log(weather_60_table);
console.log(weather_60);
for (let m = 0; m < weather_forecast_table.getRowCount(); m++) {
wf_date = weather_forecast_table.getRow(m).arr[0];
wf_time = weather_forecast_table.getRow(m).arr[1];
wf_validDate = weather_forecast_table.getRow(m).arr[2];
wf_temp = weather_forecast_table.getRow(m).arr[3];
wf_dp = weather_forecast_table.getRow(m).arr[4];
wf_ws = weather_forecast_table.getRow(m).arr[5];
wf_gs = weather_forecast_table.getRow(m).arr[6];
wf_ap = weather_forecast_table.getRow(m).arr[7];
wf_precipProb = weather_forecast_table.getRow(m).arr[8];
wf_cc = weather_forecast_table.getRow(m).arr[9];
wf_si = weather_forecast_table.getRow(m).arr[10];
wf_wd = weather_forecast_table.getRow(m).arr[11];
wf_ah = weather_forecast_table.getRow(m).arr[12];
wf_ad = weather_forecast_table.getRow(m).arr[13];
weather_forecast[m] = [wf_date, wf_time, wf_validDate, wf_temp, wf_dp, wf_ws,
wf_gs, wf_ap, wf_precipProb, wf_cc, wf_si, wf_wd, wf_ah, wf_ad
];
}
console.log(weather_forecast_table);
console.log(weather_forecast);
}

Determining time remaining until bus departs

For our digital signage system, I'd like to show how long until the next bus departs. I've built the array that holds all the times and successfully (maybe not elegantly or efficiently) gotten it to change all that to show how much time is remaining (positive or negative) until each listed departure.
I need a nudge in the right direction as to how to determine which bus is next based on the current time. If there is a bus in 7 minutes, I only need to display that one, not the next one that leaves in 20 minutes.
I was thinking perhaps a for loop that looks at the array of remaining times and stops the first time it gets to a positive value. I'm concerned that may cause issues that I'm not considering.
Any assistance would be greatly appreciated.
UPDATE: Unfortunately, all the solutions provided were throwing errors on our signage system. I suspect it is running some limited version of Javascript, but thats beyond me. However, the different solutions were extremely helpful just in getting me to think of another approach. I think I've finally come on one, as this seems to be working. I'm going to let it run over the holiday and check it on Monday. Thanks again!
var shuttleOrange = ["09:01", "09:37", "10:03", "10:29", "10:55", "11:21", "11:47", "12:13", "12:39", "13:05", "13:31", "13:57", "14:23", "14:49", "15:25", "15:51", "16:17", "16:57", "17:37", "18:17"];
var hFirst = shuttleOrange[0].slice(0,2);
var mFirst = shuttleOrange[0].slice(3,5);
var hLast = shuttleOrange[shuttleOrange.length-1].slice(0,2);
var mLast = shuttleOrange[shuttleOrange.length-1].slice(3,5);
var theTime = new Date();
var runFirst = new Date();
var runLast = new Date();
runFirst.setHours(hFirst,mFirst,0);
runLast.setHours(hLast,mLast,0);
if ((runFirst - theTime) >= (30*60*1000)) {
return "The first Orange Shuttle will depart PCN at " + shuttleOrange[0] + "."
} else if (theTime >= runLast) {
return "Orange Shuttle Service has ended for the day."
} else {
for(var i=0, l=shuttleOrange.length; i<l; i++)
{
var h = shuttleOrange[i].slice(0,2);
var m = shuttleOrange[i].slice(3,5);
var departPCN = new Date();
departPCN.setHours(h,m,0);
shuttleOrange[i] = departPCN;
}
for(var i=shuttleOrange.length-1; i--;)
{
//var theTime = new Date();
if (shuttleOrange[i] < theTime) shuttleOrange.splice(i,1)
}
var timeRem = Math.floor((shuttleOrange[0] - theTime)/1000/60);
if (timeRem >= 2) {
return "Departing in " + timeRem + " minutes."
} else if (timeRem > 0 && timeRem < 2) {
return "Departing in " + timeRem + " minute."
} else {
return "Departing now."
}
}
You only need to search once to find the index of the next scheduled time. Then as each time elapses, increment the index to get the next time. Once you're at the end of the array, start again.
A sample is below, most code is setup and helpers. It creates a dummy schedule for every two minutes from 5 minutes ago, then updates the message. Of course you can get a lot more sophisticated, e.g. show a warning when it's in the last few minutes, etc. But this shows the general idea.
window.addEventListener('DOMContentLoaded', function() {
// Return time formatted as HH:mm
function getHHmm(d) {
return `${('0'+d.getHours()).slice(-2)}:${('0'+d.getMinutes()).slice(-2)}`;
}
var sched = ["09:01", "09:37", "10:03", "10:29", "10:55", "11:21", "11:47",
"12:13", "12:39", "13:05", "13:31", "13:57", "14:23", "14:49",
"15:25", "15:51", "16:17", "16:57", "17:37", "18:17","21:09"];
var msg = '';
var msgEl = document.getElementById('alertInfo');
var time = getHHmm(new Date());
var index = 0;
// Set index to next scheduled time, stop if reach end of schedule
while (time.localeCompare(sched[index]) > 0 && index < sched.length) {
++index;
}
function showNextBus(){
var time = getHHmm(new Date());
var schedTime;
// If run out of times, next scheduled time must be the first one tomorrow
if (index == sched.length && time.localeCompare(sched[index - 1]) > 0) {
msg = `Current time: ${time} - Next bus: ${sched[0]} tomorrow`;
// Otherwise, show next scheduled time today
} else {
// Fix index if rolled over a day
index = index % sched.length;
schedTime = sched[index];
msg = `Current time: ${time} - Next bus: ${schedTime}`;
if (schedTime == time) msg += ' DEPARTING!!';
// Increment index if gone past this scheduled time
index += time.localeCompare(schedTime) > 0? 1 : 0;
}
msgEl.textContent = msg;
// Update message each second
// The could be smarter, using setInterval to schedule running at say 95%
// of the time to the next sched time, but never more than twice a second
setInterval(showNextBus, 1000);
}
showNextBus();
}, false);
<div id="alertInfo"></div>
Edit
You're right, I didn't allow for the case where the current time is after all the scheduled times on the first running. Fixed. I also changed all the string comparisons to use localeCompare, which I think is more robust. Hopefully the comments are sufficient.
I have used filter for all shuttle left after the right time and calculated how much time left for the first one.
var shuttleOrange = ["09:01", "09:37", "10:03", "10:29", "10:55", "11:21", "11:47", "12:13", "12:39", "13:05", "13:31", "13:57", "14:23", "14:49", "15:25", "15:51", "16:17", "16:57", "17:37", "18:17"];
var d = new Date();
var h = d.getHours();
var m = d.getMinutes();
var remainShuttle = shuttleOrange.filter(bus => bus.substring(0,2) > h || (bus.substring(0,2) == h && bus.substring(3,5) > m));
var leftMinutes = (parseInt(remainShuttle[0].substring(0,2))*60 + parseInt(remainShuttle[0].substring(3,5)) - (parseInt(h) *60 + parseInt(m)));
console.log(parseInt(leftMinutes / 60) + " hours and " + leftMinutes % 60 +" minutes left for next shuttle");

how to optimize a JavaScript random normal generation algo

Here is my code. This function returns a number that follows the standard normal distribution.
var rnorm = function() {
var n = 1000;
var x = 0;
var ss = 0;
for (var i = 0; i < n; i++) {
var a = Math.random();
x += a;
ss += Math.pow(a-0.5, 2);
}
var xbar = x/n;
var v = ss/n;
var sd = Math.sqrt(v);
return (xbar-0.5)/(sd/Math.sqrt(n));
};
It is simple and exploits the central limit theorem. Here is a jsfiddle running this thing 100,000 times and printing some info to the console.
https://jsfiddle.net/qx61fqpn/
It looks about right (I haven't written code to draw a histogram yet). The right proportions appear between (-1, 1), (-2, 2), (-3, 3) and beyond. The sample mean is 0, var & sd are 1, skew is 0 and kurtosis is 3.
But it's kind of slow.
I intend to write more complex functions using this one as a building block (like the chi-sq distribution).
My question is:
How can I rewrite this code to improve its performance? I'm talking speed. Right now, it's written to be clear to someone with a bit of JavaScript and statistics knowledge what I'm doing (or trying to do - in the event I've done it wrong).

How could I correctly total the points?

I am javascript learner struggling to design a small javascript game for my kids (5 to 10 years old) in which points are based on time elapsed. But, I cannot figure out a way to total the points. I have managed the code below but the result is not accurate. Probably the program is totalling all the items in array with each click. Can anybody help please? I am a newbie and there will be many mistakes or absurdities in here, I request you to be helpful politely while correcting me. Any help is appreciated..
document.getElementById("box1").onclick = function() {
clickT = Date.now();
reactT = (clickT - createdT) / 1000; //gets the time difference for reaction.
points = reactT * 1000;
points = 2000 - points;
pRecord.push(points); //add points to array.
for (i = 0; i < pRecord.length; i++) {
totalpoints += pRecord[i];
}
document.getElementById("time").innerHTML = reactT;
this.style.display = "none";
document.getElementById("score").innerHTML = totalpoints;
}
Just set totalpoints to zero before you sum the points:
document.getElementById("box1").onclick = function() {
var clickT = Date.now();
var reactT = (clickT - createdT) / 1000; //gets the time difference for reaction.
var points = reactT * 1000;
points = 2000 - points;
pRecord.push(points); //add points to array.
var totalpoints = 0;
for (var i = 0; i < pRecord.length; i++){
totalpoints += pRecord[i];
}
document.getElementById("time").innerHTML = reactT;
this.style.display = "none";
document.getElementById("score").innerHTML = totalpoints;
}
And also I don't know if you defined your variables in the outer scope, but I guess you did not, so I added var before every variable creation.
Here is an improved version of your code that also properly registers the reaction times, capping the maximum allowed reaction time to a configured value.
In your original implementation you could get bad readings if the reaction time was greater than 2 seconds.
Also, in your original code, you don`t need to divide by 1000 and then multiply back, since you end up with milliseconds anyway.
This is it:
document.getElementById("box1").addEventListener("click", function() {
clickT = Date.now();
// Gets the time difference in milliseconds for reaction.
reactT = clickT - createdT;
// Maximum allowed reaction time after which we give no more points.
var maxPoints = 2000;
// We cap the registered reaction time to the maximum allowed.
points = Math.max(reactT, maxPoints);
// We score the reaction time based
points = maxPoints - points;
// Add points to array.
pRecord.push(points);
// Compute the total points.
var totalpoints = 0;
for (i = 0; i < pRecord.length; i++){
totalpoints += pRecord[i];
}
document.getElementById("time").innerHTML = reactT;
this.style.display = "none";
document.getElementById("score").innerHTML = totalpoints;
}
You can notice that I have defined the totalpoints variable (and initialized it with 0), as otherwise, at each click, all your scores were re-added, not just the last one.
I have made the assumption that totalpoints was not already defined before the code you have pasted. Should this assumption be wrong and you have already initialized totalpoints before, in your code, then you need to replace the following piece from my code:
// Compute the total points.
var totalpoints = 0;
for (i = 0; i < pRecord.length; i++){
totalpoints += pRecord[i];
}
...with:
// Add the new points to the total.
totalpoints += points;

Javascript: For loop extremely slow, any way to speed it up?

I have a for-loop from 0 to 8,019,000,000 that is extremely slow.
var totalCalcs = 0;
for (var i = 0; i < 8019000000; i++)
totalCalcs++;
window.alert(totalCalcs);
in chrome this takes between 30-60secs.
I also already tried variations like:
var totalCalcs = 0;
for (var i = 8019000000; i--; )
totalCalcs++;
window.alert(totalCalcs);
Didn't do too much difference unfortunately.
Is there anything I can do to speed this up?
Your example is rather trivial, and any answer may not be suitable for whatever code you're actually putting inside of a loop with that many iterations.
If your work can be done in parallel, then we can divide the work between several web workers.
You can read a nice introduction to web workers, and learn how to use them, here:
http://www.html5rocks.com/en/tutorials/workers/basics/
Figuring out how to divide the work is a challenge that depends entirely on what that work is. Because your example is so small, it's easy to divide the work among inline web workers; here is a function to create a worker that will invoke a function asynchronously:
var makeWorker = function (fn, args, callback) {
var fnString = 'self.addEventListener("message", function (e) {self.postMessage((' + fn.toString() + ').apply(this, e.data))});',
blob = new Blob([fnString], { type: 'text/javascript' }),
url = URL.createObjectURL(blob),
worker = new Worker(url);
worker.postMessage(args);
worker.addEventListener('message', function (e) {
URL.revokeObjectURL(url);
callback(e.data);
});
return worker;
};
The work that we want done is adding numbers, so here is a function to do that:
var calculateSubTotal = function (count) {
var sum = 0;
for (var i = 0; i < count; ++i) {
sum++;
}
return sum;
};
And when a worker finishes, we want to add his sum to the total AND tell us the result when all workers are finished, so here is our callback:
var total = 0, count = 0, numWorkers = 1,
workerFinished = function (subTotal) {
total += subTotal;
count++;
if (count == numWorkers) {
console.log(total);
}
};
And finally we can create a worker:
makeWorker(calculateSubTotal, [10], workerFinished); // logs `10` to console
When put together, these pieces can calculate your large sum quickly (depending on how many CPUs your computer has, of course).
I have a complete example on jsfiddle.
Treating your question as a more generic question about speeding up loops with many iterations: you could try Duff's device.
In a test using nodejs the following code decreased the loop time from 108 seconds for your second loop (i--) to 27 seconds
var testVal = 0, iterations = 8019000000;
var n = iterations % 8;
while (n--) {
testVal++;
}
n = parseInt(iterations / 8);
while (n--) {
testVal++;
testVal++;
testVal++;
testVal++;
testVal++;
testVal++;
testVal++;
testVal++;
}

Categories

Resources