Develop Reference

Asyncronicity in a reduce() function WITHOUT using async/await

I am patching the exec() function to allow subpopulating in Mongoose, which is why I am not able to use async/await here -- my function will be chained off a db call, so there is no opportunity to call await on it, and within the submodule itself, there I can't add async/await outside of an async function itself.
With that out of the way, let's look at what I'm trying to do. I have two separate arrays (matchingMealPlanFoods and matchingMealPlanRecipeFoods) full of IDs that I need to populate. Both of them reside on the same array, foods. They each require a db call with aggregation, and the problem in my current scenario is that only one of the arrays populates because they are happening asynchronously.
What I am trying to do now is use the reduce function to return the updated foods array to the next run of reduce so that when the final result is returned, I can replace the entire foods array once on my doc. The problem of course is that my aggregate/exec has not yet returned a value by the time the reduce function goes into its next run. Is there a way I can achieve this without async/await here? I'm including the high-level structure here so you can see what needs to happen, and why using .then() is probably not viable.
EDIT: Updating code with async suggestion
function execute(model, docs, options, lean, cb) {
options = formatOptions(options);
let resolvedCount = 0;
let error = false;
(async () => {
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (let option of options) {
const path = option.path.split(".");
// ... various things happen here to prep the data
const aggregationOptions = [
// // $match, then $unwind, then $replaceRoot
];
await rootRefModel
.aggregate(aggregationOptions)
.exec((err, refSubDocuments) => {
// more stuff happens
console.log('newFoodsArray', newFoodsArray); // this is to check whether the second iteration is using the updated newFoods Array
const arrToReturn = newFoodsArray.map((food) => {
const newMatchingArray = food[nests[1]].map((matchingFood) => {
//more stuff
return matchingFood;
});
const updatedFood = food;
updatedFood[`${nests[1]}`] = newMatchingArray;
return updatedFood;
});
console.log('arrToReturn', arrToReturn);
newFoodsArray = [...arrToReturn];
});
}
};
console.log('finalNewFoods', newFoodsArray); // this should log after the other two, but it is logging first.
const document = doc.toObject();
document.foods = newFoodsArray;
if (resolvedCount === options.length) cb(null, [document]);
}
})()
EDIT: Since it seems it will help, here is the what is calling the execute function I have excerpted above.
/**
* This will populate sub refs
* #param {import('mongoose').ModelPopulateOptions[]|
* import('mongoose').ModelPopulateOptions|String[]|String} options
* #returns {Promise}
*/
schema.methods.subPopulate = function (options = null) {
const model = this.constructor;
if (options) {
return new Promise((resolve, reject) => execute(model, [this], options, false, (err, docs) => {
if (err) return reject(err);
return resolve(docs[0]);
}));
}
Promise.resolve();
};
};

We can use async/await just fine here, as long as we remember that async is the same as "returning a Promise" and await is the same as "resolving a Promise's .then or .catch".
So let's turn all those "synchronous but callback-based" calls into awaitables: your outer code has to keep obeying the API contract, but since it's not meant to a return a value, we can safely mark our own version of it as async, and then we can use await in combination with promises around any other callback based function calls in our own code just fine:
async function execute(model, docs, options, lean, andThenContinueToThis) {
options = formatOptions(options);
let option, resolvedCount = 0;
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (option of options) {
// ...things happen here...
const aggregationOptions = [/*...data...*/];
try {
const refSubDocuments = await new Promise((resolve, reject) => rootRefModel
.aggregate(aggregationOptions)
.exec((err, result) => err ? reject(err) : resolve(result));
// ...do some work based on refSubDocuments...
}
// remember to forward errors and then stop:
catch (err) {
return andThenContinueToThis(err);
}
}
// remember: bind newFoodsArray somewhere so it doesn't get lost next iteration
}
// As our absolutely last action, when all went well, we trigger the call forwarding:
andThenContinueToThis(null, dataToForward);
}

How to use a promise with Firestore Firebase

I'm trying to read some data from Firestore using an array of keys and a map function, then, when all the data is back from Firestore, run a function on the resulting data set. This code works, but not in the right order; I'm struggling to make doSomethingUsefulWithData() delay until arrayOfDataFromFirestore is complete.
I've tried await but I get an error message saying "Uncaught syntax error: await is only valid in async functions and async generators". I think the awaits shown below are in the async function but apparently not.
If I run the version shown here, doSomethingUsefulWithData() runs before getData(), despite following it with a .then.
var arrayOfFirestoreKeys = [key0, key1, key3];
function doSomethingUsefulWithData(){
//Do something useful with the data here
};
function dataGet(){
var arrayOfDataFromFirestore = [];
arrayOfDataFromFirestore = Promise.all(arrayOfFirestoreKeys.map(async (dataGet, index) => {
var docRef = db.collection('dataInFirestore').doc(arrayOfFirestoreKeys[index]).get().then((doc) => {
if (doc.exists) {
console.log("Document data from async:", doc.data());
//doSomethingUsefulWithData here works but runs n times for n items in arrayOfFirestoreKeys
};
//Trying await doSomethingUsefulWithData here produces warning 'await must be in async function or function generator'
});
//Trying await doSomethingUsefulWithData here produces warning 'await must be in async function or function generator'
}))
return(arrayOfDataFromFirestore);
};
arrayOfDataFromFirestore = dataGet().then(doSomethingUsefulWithData(arrayOfDataFromFirestore));
//With this version of the code,
//doSomethingUsefulWithData runs first and produces a complaint that arrayOfDataFromFirestore is undefined.
//Then the contents of arrayOfDataFromFirestore follow into console.log

If I correctly understand your question, the following should do the trick:
var arrayOfFirestoreKeys = [key0, key1, key3];
function doSomethingUsefulWithData(arrayOfSnapshots) {
// arrayOfSnapshots is an Array of DocumentSnapshot
arrayOfSnapshots.forEach(docSnap => {
if (docSnap.exists) {
// Do something
}
});
}
function dataGet() {
const arrayOfDataFromFirestore = arrayOfFirestoreKeys.map((k) =>
db.collection('dataInFirestore').doc(k).get()
);
return Promise.all(arrayOfDataFromFirestore);
}
dataGet().then(array => {
doSomethingUsefulWithData(Array);
});

Reading snapshot of other user in nested firebase function

I am trying to loop through my database that is structured like this:
Realtime Structure
I want to use the RBUserID (RB: Referred By) key which references another user. I'm using the code below to cycle through the UserIDs and find which userID matches the RBUserID.
function readOnceandFindRef() {
refRead.once("value", function (snapshot) {
snapshot.forEach(function (childSnapshot) {
var childKey = childSnapshot.key;
var childData = childSnapshot.val();
var RBUserID = childData.RBUserID;
console.log(RBUserID);
debugger; // 1
refRead.child(RBUserID).once("value", function (snapshot) {
if (snapshot.exists()) {
console.log(snapshot.key);
debugger; // 2
}
debugger; // 3
});
});
});
}
The forEach loop reads the users and console.log(RBUserID) works, but then it skips through
refRead.child(RBUserID).once("value", function (snapshot) {
and doesn't runs Debugger 2 and Debugger 3 until after the forEach loop has ended. The chrome debugger shows this:
Chrome Debugger
Any help of why it's skipping the read function inside would be appreciated

That's because once() is asynchronous and returns immediately, before the query is complete. The callback you pass to it is invoked some time later, while the code immediately after it continues to execute.
Note that once() also returns a promise that resolves with the snapshot object, and it's more common to use that instead of the callback argument in order to make it easier to do asynchronous programming.

As #Doug rightly pointed out in his answer, since once() is an asynchronous operation, you might need to tweak your function readOnceandFindRef for checking all values synchronously.
One of the solution would be using for-of loop instead of forEach loop.
async function readOnceandFindRef() {
const snapshots = await refRead.once("value");
for (const snapshot of snapshots) {
var childKey = snapshot.key;
var childData = snapshot.val();
var RBUserID = childData.RBUserID;
console.log(RBUserID);
debugger; // 1
const childSnapshot = await refRead.child(RBUserID).once("value");
if(childSnapshot.exists()){
console.log(snapshot.key);
debugger; // 2
}
debugger; // 3
}
}
However, in this case, you will be checking through each user-id one after another.
But using the solution of Promise.all, you can execute all such asynchronous operations in parallel.
Solution 2
async function readOnceandFindRef() {
const snapshots = await refRead.once("value");
// Following part allows us to execute all the asynchronous operations
// at once and still store their outcomes synchronously in the results array.
const results = await Promise.all(snapshots.map(snapshot => {
var childKey = snapshot.key;
var childData = snapshot.val();
var RBUserID = childData.RBUserID;
console.log(RBUserID);
debugger; // 1
return refRead.child(RBUserID).once("value").then(snapshot => {
const doesRefReadExist = false;
if(childSnapshot.exists()){
console.log(snapshot.key);
doesRefReadExist = true;
}
return { rbUserID: RBUserID, doesRefReadExist}
}) ;
}))
}
The results array would look something like:
[
{
rbUserID: RBUserID-1,
doesRefReadExist: true/false
},
{
rbUserID: RBUserID-2,
doesRefReadExist: true/false
},
.
.
.
]

How to wait for a Firebase retrieve value and only then exit the function?

I have a Firebase query.
Because Firebase works asynchronously, the function continue to run without waiting for the Firebase retrieve value.
Is there a way to wait for the result from the Firebase query and only then to make the return from the function?
function CheckBuyingCondition(prefix){
var Res= "";
var Current_Price_Open_ref = firebase.database().ref("dailyT/Current_Price_Open/"+nextDayTrading).orderByChild("Prefix").equalTo(prefix)
Current_Price_Open_ref.once("value").then(function(snapshot) {
if(snapshot.exists()){
snapshot.forEach(function(childSnapshot) {
var val = childSnapshot.val();
res =""+ val.Current_Price_Open;
});
}else{
res = "NA";
}
});
return res; //(Here i got res = "" instead of the correct value from Firebase query
}

Use async/await:
async function checkBuyingCondition(prefix) {
var res = '';
var currentPriceOpenRef = firebase.database()
.ref(`dailyT/currentPriceOpen/${nextDayTrading}`)
.orderByChild('prefix')
.equalTo(prefix);
var snapshot = await currentPriceOpenRef.once('value');
if(snapshot.exists()) {
snapshot.forEach(function(childSnapshot) {
var val = childSnapshot.val();
res = `${val.currentPriceOpen}`;
});
} else {
res = 'NA';
}
return res;
}
Take note that this does not make your function synchronous at all, thus the async keyword at the beginning of your function declaration; it just makes your function look like one.
On the 3rd line inside the function you'll notice the await keyword. This waits for your promise to resolve then returns the result which in your case, is the snapshot from Firebase. You can only use await inside async functions.
More Reading: Javascript Async/Await

What you're proposing is making the Firebase SDK asynchronous call into an synchronous call. This is not a good idea, and to be honest, not even possible in JavaScript. If you need to make a helper function that deals with Firebase APIs, that function should instead accept a callback function to be invoked when the work completes, or return a promise so that the caller of the function can decide what to do next.
Read here to learn more about why Firebase APIs are asynchronous.

Try this:
function CheckBuyingCondition(prefix){
var Res= "";
var Current_Price_Open_ref = firebase.database().ref("dailyT/Current_Price_Open/"+nextDayTrading).orderByChild("Prefix").equalTo(prefix)
return Current_Price_Open_ref.once("value").then(function(snapshot) {
if(snapshot.exists()){
snapshot.forEach(function(childSnapshot) {
var val = childSnapshot.val();
res =""+ val.Current_Price_Open;
});
return res;
}else{
res = "NA";
}
});
}
Firebase queries are promises, so you because you can return the result from the promise and get it with another promise.

node-mysql timing

i have a recursive query like this (note: this is just an example):
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
database.query('select * from users where owner='+data.id,function(err,result,fields)
{
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
this don't work because the timmings are all wrong, the code don't wait for the query.
i've tried to do this:
var MyQuery = function(QueryString){
var Data;
var Done = false;
database.query(QueryString, function(err, result, fields) {
Data = result;
Done = true;
});
while(Done != true){};
return Data;
}
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
result= MyQuery('select * from users where owner='+data.id);
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
but he just freezes on the while, am i missing something?

The first hurdle to solving your problem is understanding that I/O in Node.js is asynchronous. Once you know how this applies to your problem the recursive part will be much easier (especially if you use a flow control library like Async or Step).
Here is an example that does some of what you're trying to do (minus the recursion). Personally, I would avoid recursively loading a possibly unknown number/depth of records like that; Instead load them on demand, like in this example:
var User = function(data) {
this.data = data
this.minions;
};
User.prototype.getMinions = function(primaryCallback) {
var that = this; // scope handle
if(this.minions) { // bypass the db query if results cached
return primaryCallback(null, this.minions);
}
// Callback invoked by database.query when it has the records
var aCallback = function(error, results, fields) {
if(error) {
return primaryCallback(error);
}
// This is where you would put your recursive minion initialization
// The problem you are going to have is callback counting, using a library
// like async or step would make this party much much easier
that.minions = results; // bypass the db query after this
primaryCallback(null, results);
}
database.query('SELECT * FROM users WHERE owner = ' + data.id, aCallback);
};
var user = new User(someData);
user.getMinions(function(error, minions) {
if(error) {
throw error;
}
// Inside the function invoked by primaryCallback(...)
minions.forEach(function(minion) {
console.log('found this minion:', minion);
});
});
The biggest thing to note in this example are the callbacks. The database.query(...) is asynchronous and you don't want to tie up the event loop waiting for it to finish. This is solved by providing a callback, aCallback, to the query, which is executed when the results are ready. Once that callback fires and after you perform whatever processing you want to do on the records you can fire the primaryCallback with the final results.

Each Node.js process is single-threaded, so the line
while(Done != true){};
takes over the thread, and the callback that would have set Done to true never gets run because the thead is blocked on an infinite loop.
You need to refactor your program so that code that depends on the results of the query is included within the callback itself. For example, make MyQuery take a callback argument:
MyQuery = function(QueryString, callback){
Then call the callback at the end of your database.query callback -- or even supply it as the database.query callback.

The freezing is unfortunately correct behaviour, as Node is single-threaded.
You need a scheduler package to fix this. Personally, I have been using Fibers-promise for this kind of issue. You might want to look at this or another promise library or at async