Struggle to commit more than 500 document in firestore batch - javascript

I spent the last 2 days trying to find a way to modify a virtually unlimited amount of data in my cloud function.
I am aware it's a topic already heavily discussed but despite trying every single solution I found, I always end up with a similar scenario so I guess it's not only the way iI treat my batches.
The goal here is to be able to scale up our activity and anticipate the growth of users so I'm running some stress tests.
I'm skipping the entire app idea because it would take a while but here is the problem :
1) I am feeding an array with the list of the desired actions for the batch :
tempFeeObject["star"]=FieldValue.increment(restartFee)
let docRefFees = db.collection(collectedFeesDb).doc(element); // log action
finalArray.push([docRefFees,tempFeeObject,"set"])
2) I try to resolve all the final arrays. In this case, for the stress test, we talk about 6006 documents.
Here is the code for it :
try
{
const batches = []
finalArray.forEach((data, i) => {
if (i % 400 === 0) {
let newBatch = db.batch();
batches.push(newBatch)
}
const batch = batches[batches.length - 1]
let tempCommit = {...data[1]}
Object.keys(tempCommit).forEach(key => {
if(data[1][key] && data[1][key].seconds && !Number.isNaN(data[1][key].seconds)){
let sec = 0
let nano = 0
if(Object.keys(data[1][key])[1].includes("nano")){
sec = parseInt(data[1][key][Object.keys(data[1][key])[0]],10)
nano = parseInt(data[1][key][Object.keys(data[1][key])[1]],10)
}else{
sec = parseInt(data[1][key][Object.keys(data[1][key])[1]],10)
nano = parseInt(data[1][key][Object.keys(data[1][key])[0]],10)
}
let milliseconds = (sec*1000)+(nano/1000000)
tempCommit[key] = admin.firestore.Timestamp.fromMillis(milliseconds)
}
});
if(data[2]==="set"){
batch.set(data[0], tempCommit, {merge: true});
}else if(data[2]==="delete"){
batch.delete(data[0]);
}
})
await Promise.all(batches.map(batch => batch.commit()))
console.log(`${finalArray.length} documents updated`)
return {
result:"success"
}
}
catch (error) {
console.log(`***ERROR: ${error}`)
return {
result:"error"
}
}
The middle part might seem confusing but it is made to recreate the timestamp (i had errors when instead of a timestamp I had the array of values instead. And I test the array to find the seconds and nanoseconds because I also had cases where my script found seconds either in the first or second position).
So my script checks each key of the committed document and recreates the timestamp if he finds one.
3) Results :
The results are my main problem :
On one side it is working since my database is being updated with the correct values.
But it is not happening the way it should.
Here are the function logs :
4:04:01.790 PM
autoStationEpochMakerTestNet
Function execution started <=== Function START
4:04:02.576 PM
autoStationEpochMakerTestNet
Auto epoch function treating : 1 actions
4:04:02.577 PM
autoStationEpochMakerTestNet
atation : Arcturian Space Station has 1 actions
4:04:02.582 PM
autoStationEpochMakerTestNet
Function execution took 791 ms. Finished with status: ok <== function END
4:04:02.682 PM
autoStationEpochMakerTestNet
list used : 1 <== But the functions called by the previous one are still printing
4:04:04.926 PM
autoStationEpochMakerTestNet
***ERROR: Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
4:04:05.726 PM
autoStationEpochMakerTestNet
result : { result: 'error' }
4:04:05.726 PM
autoStationEpochMakerTestNet
Arcturian Space Station function epoch generator failed. testmode = true
4:07:02.107 PM
autoStationEpochMakerTestNet
Function execution started <== Second function Trigger START
4:07:02.156 PM
autoStationEpochMakerTestNet
6006 documents updated <== Receiving logs from what i assume is the previous function trigger
4:07:02.158 PM
autoStationEpochMakerTestNet
result : { result: 'success' }
4:07:02.210 PM
autoStationEpochMakerTestNet
Auto epoch function treating : 0 actions <== As expected there is no more action to treat since it was actually processed on the previous call
4:07:02.212 PM
autoStationEpochMakerTestNet
Function execution took 104 ms. Finished with status: ok
So my comprehension of what is happening is as follows :
The function triggers and builds the batches but is somehow instantly finished without waiting for the function or returning anything.
Meantimes there are still logs coming from the function including this one :
***ERROR: Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
So I feel my batch array isn't working as expected.
Then on the following function triggers (scheduled function, every 3 min). All the functions seem to finish to execute properly.
Once again, my database is being updated as it should be.
A few elements might be noticing:
Some of the data being committed are timestamps which I think counts for more than 1 action ?
In this case, one document has more than 2000 fields updated (simple int value)
await promise.all() is new to me so it's also one of my leads.
I set the function ram at 2Go and timeout at 540sec.
Also here is the code for the main function (the scheduled one, which is here to dispatch actions and wait for the answer) :
Object.keys(finalObject).forEach(async (stationName) => {
if(finalObject[stationName].finalList.length>0){
let epochDbUsed = stationsEpochDb[stationName]
let resultGenerator = await generateEpochsForStation(finalObject[stationName].finalList,finalObject[stationName].actionsToRemove,actionsDb,epochDbUsed,testMode)
if(resultGenerator.result==="success"){
await db.collection(logsDb).add({
"user":"cloudFunction",
"type": `createdEpochs${stationName}`,
"station":stationName,
"testMode":testMode,
"actionQty":actionCount,
"note":`New epoch has been created on ${stationName}. testmode : ${testMode}`,
"date":FieldValue.serverTimestamp()
});
}else{
console.log(`${stationName} function epoch generator failed. testmode = ${testMode}`)
await db.collection(logsDb).add({
"user":"cloudFunction",
"type": `createdEpochs${stationName}Failure`,
"station":stationName,
"testMode":testMode,
"actionQty":actionCount,
"note":`New epoch has been created on ${stationName}. testmode : ${testMode}, FAILED`,
"date":FieldValue.serverTimestamp()
});
}
}
});
return null
I am not sure if the return null at the end could mess things up, since its a scheduled function, I'm not sure what I'm supposed to return (actually never really thought of that, and realizing as I write, I'm going to look that up after posting that :) ).
Well, I think this is enough data to understand the situation, feel free to ask for any more details of course.
Thanks a lot to whoever will take the time to help :)
Edit 1 :
I think I might have found a solution :
const _datawait = [];
finalArray.forEach( data => {
let docRef = data[0];
_datawait.push( docRef.set(data[1],{merge: true}) );
})
const _dataloaded = await Promise.all( _datawait );
console.log(`${actionCount} documents updated`)
// update date from last epoch
await updateLastEpochDateEnd(stationEpochDb,lastEpoch.epochId)
// add epoch
await db.collection(stationEpochDb).add(finalEpoch)
return _dataloaded
Doing so wait for all requests stored in the finalArray ([docref],[data]) to receive their promise before returning anything.
Took me a while to get there but it feels good to see the logs in order :
8:46:59.311 PM
stationRestarter
Function execution started
8:46:59.355 PM
stationRestarter
docID : PO3UstEZt1lV1IHwxBIR
8:47:10.667 PM
stationRestarter
6004 documents updated
8:47:11.201 PM
stationRestarter
Function execution took 11890 ms. Finished with status: ok
I'm done for today, I will update tomorrow to see if this is a final solution :)

Related

Javascript: sync with server

I want to sync with my server and found some code (which may be a starting point for my own):
var serverDateTimeOffset
function getServerDateTime() {
if (serverDateTimeOffset === undefined || serverDateTimeOffset === null) {
serverDateTimeOffset = 0; //while syncing with the server, do not start a new sync, but return localtime
var clientTimestamp = (new Date()).valueOf();
$.getJSON('/getDateTime.ashx?ct=' + clientTimestamp, function (data) {
var nowTimeStamp = (new Date()).valueOf();
var serverTimestamp = data.serverTimestamp;
var serverClientRequestDiffTime = data.diff;
var serverClientResponseDiffTime = nowTimeStamp - serverTimestamp;
var responseTime = (serverClientRequestDiffTime - nowTimeStamp + clientTimestamp - serverClientResponseDiffTime) / 2
var syncedServerTime = new Date((new Date()).valueOf() - (serverClientResponseDiffTime - responseTime) / 2);
serverDateTimeOffset = (serverClientResponseDiffTime - responseTime) / 2;
console.log("synced time with server:", syncedServerTime, serverDateTimeOffset);
})
}
return new Date((new Date()).valueOf() - serverDateTimeOffset);
}
But I do not understand what is the meaning of the variable responseTime (and the logical reasoning behind it). What does it represent? Anyone has suggestions? It appears that it combines some relative with absolute times.
Example in my reasoning:
10 start at client ==+5 (req in transit) ==> 15 (arrival) ==> (server processing +5) ==> before sending (20) ==+7 (resp in transit) ==> 27 (back at client). Calculation for responseTime would result in -19/2 = -9.5.
When receive time is same as sent time at server (server processing would be 0) then 5 -22 + 10 -7 responseTime resulsts in -14/2 = -7.
And what about the variable serverDateTimeOffset?
Do I miss something? I found a description of NTP which may be helpful:
Synchronizing a client to a network server consists of several packet exchanges where each exchange is a pair of request and reply. When sending out a request, the client stores its own time ( ) into the packet being sent. When a server receives such a packet, it will in turn store its own time ( ) into the packet, and the packet will be returned after putting a transmit timestamp into the packet. When receiving the reply, the receiver will once more log its own receipt time to estimate the travelling time of the packet. The travelling time (delay) is estimated to be half of "the total delay minus remote processing time", assuming symmetrical delays.
Edit:
The function is polled regularly (it is part of the update() function, so the first time it may indeed be 0)
$(document).ready(function () {
...
setTimeout(function () { update() }, 2000); }
..
});
Some tests resulted in the following examples:
responseTime= (serverClientRequestDiffTime( -578 )-nowTimeStamp( 1621632913398 )+clientTimestamp( 1621632913081 )-serverClientResponseDiffTime( 895 ))/2= -895
serverDateTimeOffset=(serverClientResponseDiffTime( 895 )-responseTime( -895 ))/2= 895
added req delay via proxy:
responseTime= (serverClientRequestDiffTime( 2456 )-nowTimeStamp( 1621632992161 )+clientTimestamp( 1621632988800 )-serverClientResponseDiffTime( 905 ))/2= -905
serverDateTimeOffset=(serverClientResponseDiffTime( 905 )-responseTime( -905 ))/2= 905

How to measure total time when async function is involved

I have a javascript function being called this way
do_work();
setTimeout(do_some_other_work);
do_work();
setTimeout(do_some_other_work);
do_work();
How do i output the total time spent in this case ?
You can use global variables: one with the starting time (to compare later), and one that gets incremented by do_work() and do_some_other_work() using a function that checks if the counter is at the desired number and then calculates the time difference.
So, something like:
var start = new Date(), call_counter = 0;
Then at the end of do_work() and do_some_other_work():
incrementCounter();
And have:
function incrementCounter() {
if ( ++call_counter == 5 ) {
console.log( ((new Date()) - start) );
}
}

Javascript inheritance - asking experts oppinion on the following code

A friend of mine is attending a JavaScript course and thinks that the code he is submitting for grading is correct. However, the grader support keeps reporting it as not correct. He asked for my help and I tested the code on several IDEs and editors, online and offline, and I also got back every time a correct evaluation.
However I don't use often JavaScript ans I'm hesitating to answer my friend that he is right.
I would be most grateful if someone with more experience could tell me if the code evaluates correctly or not. Thank you.
"Determines which day of the week had the most nnumber of people visiting the pet store.
If more than one day of the week has the same, highest amount of traffic, an array containing the days (in any order) should be returned.
(ex. ["Wednesday", "Thursday"]).
If the input is null or an empty array, the function should return null.
#param week an array of Weekday objects
#return a string containing the name of the most popular day of the week if there is only one most popular day, and an array of the strings containing the names of the most popular days if there are more than one that are most popular"
function Weekday (name, traffic) {
this.name = name;
this.traffic = traffic;
}
function mostPopularDays(week) {
// IMPLEMENT THIS FUNCTION!
this.week = week;
if (typeof week !== 'object' || week === null || week === undefined || week.length === 0) {
return null;
}
var maxTr = 0;
var maxTrDay = [];
for (var i = 0; i < this.week.length; i++) {
if (this.week[i].traffic > maxTr) {
maxTrDay = [this.week[i].name];
//maxTrDay = this.week[i].name;
maxTr = this.week[i].traffic;
} else if (this.week[i].traffic === maxTr) {
//maxTrDay = [this.week[i].name];
maxTrDay.push(this.week[i].name);
} else if (this.week.length > 7) {
this.week.shift();
}
}
if (maxTrDay.length === 1) {
console.log("The most popular day of the week was:")
return maxTrDay[0];
} else if (maxTrDay > 1) {
console.log("The most popular days of the week were:")
return maxTrDay;
}
return null;
}
The test case that the grader reports as failed are the following:
1. mostPopularDays should return an array of days when more than one day has most popular traffic
I used the following lines for testing, and the output was always the last (commented) line below:
var week = [];
var sun = new Weekday('Sunday', 100); week.push(sun);
var mon = new Weekday('Monday', 90); week.push(mon);
var tue = new Weekday('Tuesday', 100); week.push(tue);
mostPopularDays(week);
// [Sunday, Tuesday]
The issue is (maxTrDay > 1) is comparing an array object with the number 1. This will be false for all array inputs except for, confusingly, e.g. ([2] > 1), but that's JS for you.
Running your code as-is with the provided driver (with added quotes to Tuesday to avoid a ReferenceError) yields the output of null.
Your friend probably means (maxTrDay.length > 1), which compares based on length and yields the correct output:
The most popular days of the week were:
=> [ 'Sunday', 'Tuesday' ]

Interpreting/parsing data from Bluetooth heart rate monitor (Cordova)

I am creating an application using Cordova which requires interpreting data from a Bluetooth HR monitor (capable of recording raw RR intervals, such as the Polar H7). I am using the cordova-plugin-ble-central
I am having a difficult time making sense of the data received from the monitor, despite trawling the internet for answers and reading the Bluetooth Heart Rate Service Characteristic specification numerous times.
Here is my function which runs each time data is received:
onData: function(buffer) {
console.log(buffer);
// var data8 = new Uint8Array(buffer);
var data16 = new Uint16Array(buffer);
var rrIntervals = data.slice(1);
for (i=0; i<rrIntervals.length; i++) {
rrInterval = rrIntervals[i];
heartRate.addReading(rrInterval); // process RR interval elsewhere
}
},
When I log the data received in buffer, the following is output to the console:
console output
I know how to extract RR intervals (highlighted in yellow), but I don't really understand what the other values represent, which I require as users might be connecting with other monitors which don't transmit RR intervals etc.
A quick plain English explanation of what the data received means and how to parse it would be much appreciated. For example, what number constitutes the flags field, and how to convert this to binary to extract the sub-fields (ie. to check if RR intervals present - I know this is determined by the 5th bit in the flags field.)
The plugin also states that 'Raw data is passed from native code to the success callback as an ArrayBuffer' but I don't know how to check the flags to determine if the data from the specific HR monitor is in 8 or 16 bit format. Below is another console log of when I create both Uint8 and Uint16 arrays from the data received. Again, I have highlighted the heart rate and RR intervals, but I need to know what the other values represent and how to parse them correctly.
console log with Uint8 and Uint16 output
The whole code is below:
var heartRateSpec = {
service: '180d',
measurement: '2a37'
};
var app = {
initialize: function() {
this.bindEvents();
},
bindEvents: function() {
document.addEventListener('deviceready', this.onDeviceReady, false);
},
onDeviceReady: function() {
app.scan();
},
scan: function() {
app.status("Scanning for Heart Rate Monitor");
var foundHeartRateMonitor = false;
function onScan(peripheral) {
// this is demo code, assume there is only one heart rate monitor
console.log("Found " + JSON.stringify(peripheral));
foundHeartRateMonitor = true;
ble.connect(peripheral.id, app.onConnect, app.onDisconnect);
}
function scanFailure(reason) {
alert("BLE Scan Failed");
}
ble.scan([heartRateSpec.service], 5, onScan, scanFailure);
setTimeout(function() {
if (!foundHeartRateMonitor) {
app.status("Did not find a heart rate monitor.");
}
}, 5000);
},
onConnect: function(peripheral) {
app.status("Connected to " + peripheral.id);
ble.startNotification(peripheral.id, heartRateSpec.service, heartRateSpec.measurement, app.onData, app.onError);
},
onDisconnect: function(reason) {
alert("Disconnectedz " + reason);
beatsPerMinute.innerHTML = "...";
app.status("Disconnected");
},
onData: function(buffer) {
var data = new Uint16Array(buffer);
if (heartRate.hasStarted() == false) {
heartRate.beginReading(Date.now());
} else {
var rrIntervals = data.slice(1);
for (i=0; i<rrIntervals.length; i++) {
rrInterval = rrIntervals[i];
heartRate.addReading(rrInterval);
}
}
},
onError: function(reason) {
alert("There was an error " + reason);
},
status: function(message) {
console.log(message);
statusDiv.innerHTML = message;
}
};
app.initialize();
Many thanks in advance for any help or advice.
UPDATE for a more in depth explanation check out this post I wrote on the subject.
I've figured it out - here's a quick explanation for anyone who encounters a similar problem:
The data passed into onData(buffer) is just binary data, so whether we convert it into a Uint8Array or a Uint16Array it still represents the same binary data. Of course, the integers in Uint16 will likely be larger as they comprise 16 bits rather than 8.
The flags field is always represented by the first byte, so we can get this by converting the data (passed in as buffer) to a Uint8Array and access the first element of this array which will be the element with index 0.
We can then check the various bit fields using bitwise operations. For example, the Bluetooth Heart Rate Service Characteristic specification tells us that the fifth bit represents whether the reading contains RR intervals (1) or doesn't contain any (0).
Below we can see that the fifth bit is the number 16 in binary:
128 64 32 16 8 4 2 1
0 0 0 1 0 0 0 0
Therefore the operation 16 & flag (where flag is the byte containing the flags field) will return 16 (which can be hoisted to true) if the reading contains RR intervals and 0 (hoisted to false) if it doesn't.

Collection Boolean isn't being set to false - Meteor

So in short, the app that i'm developing is a bus timetable app using Meteor, as a practice project.
inside my body.js, I have an interval that runs every second, to fetch the current time and compare to items in a collection.
To show relevant times, I have added an isActive boolean, whenever the current time = sartTime of the collection, it sets it to true, and that is working fine.
But when I do the same thing for endTime and try to set it to false, so I can hide that timeslot, it just doesn't work. Even consoles don't show up. What am I missing? I have recently just started doing meteor, so excuse the redundancies.
Worth noting that the times that I'm comparing to are times imported from an CSV file, so they have to be in the 00:00 AM/PM format.
Thank you guys so much for your time.
Body.js code:
Template.Body.onCreated(function appBodyOnCreated() {
Meteor.setInterval(() => {
var h = (new Date()).getHours();
const m = ((new Date()).getMinutes() <10?'0':'') + ((new Date()).getMinutes());
var ampm = h >= 12 ? ' PM' : ' AM';
ampmReac.set(ampm);
if (h > 12) {
h -= 12;
} else if (h === 0) {
h = 12;
}
const timeAsAString = `${h}${m}`;
const timeAsAStringFormat = `${h}:${m}`;
whatTimeIsItString.set(timeAsAStringFormat + ampm); // convert to a string
const timeAsANumber = parseInt(timeAsAString); // convert to a number
whatTimeIsIt.set(timeAsANumber); // update our reactive variable
if (Timetables.findOne({TimeStart: whatTimeIsItString.get()}).TimeStart == whatTimeIsItString.get())
{
var nowTimetable = Timetables.findOne({TimeStart: whatTimeIsItString.get() });
Timetables.update({_id : nowTimetable._id },{$set:{isActive : true}});
console.log('I am inside the START statement');
}
else if (Timetables.findOne({TimeEnd: whatTimeIsItString.get()}).TimeEnd == whatTimeIsItString.get())
{
var nowTimetable = Timetables.findOne({TimeEnd: whatTimeIsItString.get() });
Timetables.update({_id : nowTimetable._id },{$set:{isActive : false}});
console.log('I am inside the END statement');
}
}, 1000); //reactivate this function every second
});
})
Very probably it is just that your if / else blocks does what you ask it:
It tries to find a document in Timetables, with specified TimeStart. If so, it makes this document as "active".
If no document is previously found, i.e. there is no timeslot which TimeStart is equal to current time, then it tries to find a document with specified TimeEnd.
But your else if block is executed only if the previous if block does not find anything.
Therefore if you have a next timeslot which starts when your current timeslot ends, your if block gets executed for that next timeslot, but the else if block is never executed to de-activate your current timeslot.
An easy solution would be to transform your else if block in an independent if block, so that it is tested whether the previous one (i.e. TimeStart) finds something or not.
Ok so I got it to work eventually. My problem was that it was never going to the second IF statement.
What I have done is set up a whole new Meteor.interval(() >) function, and placed that second IF in there, as is.
I think the problem was that it was it checks the first IF statement and gets stuck there, no matter what the outcome of the parameters is.

Categories

Resources