Develop Reference

Asyncronicity in a reduce() function WITHOUT using async/await

I am patching the exec() function to allow subpopulating in Mongoose, which is why I am not able to use async/await here -- my function will be chained off a db call, so there is no opportunity to call await on it, and within the submodule itself, there I can't add async/await outside of an async function itself.
With that out of the way, let's look at what I'm trying to do. I have two separate arrays (matchingMealPlanFoods and matchingMealPlanRecipeFoods) full of IDs that I need to populate. Both of them reside on the same array, foods. They each require a db call with aggregation, and the problem in my current scenario is that only one of the arrays populates because they are happening asynchronously.
What I am trying to do now is use the reduce function to return the updated foods array to the next run of reduce so that when the final result is returned, I can replace the entire foods array once on my doc. The problem of course is that my aggregate/exec has not yet returned a value by the time the reduce function goes into its next run. Is there a way I can achieve this without async/await here? I'm including the high-level structure here so you can see what needs to happen, and why using .then() is probably not viable.
EDIT: Updating code with async suggestion
function execute(model, docs, options, lean, cb) {
options = formatOptions(options);
let resolvedCount = 0;
let error = false;
(async () => {
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (let option of options) {
const path = option.path.split(".");
// ... various things happen here to prep the data
const aggregationOptions = [
// // $match, then $unwind, then $replaceRoot
];
await rootRefModel
.aggregate(aggregationOptions)
.exec((err, refSubDocuments) => {
// more stuff happens
console.log('newFoodsArray', newFoodsArray); // this is to check whether the second iteration is using the updated newFoods Array
const arrToReturn = newFoodsArray.map((food) => {
const newMatchingArray = food[nests[1]].map((matchingFood) => {
//more stuff
return matchingFood;
});
const updatedFood = food;
updatedFood[`${nests[1]}`] = newMatchingArray;
return updatedFood;
});
console.log('arrToReturn', arrToReturn);
newFoodsArray = [...arrToReturn];
});
}
};
console.log('finalNewFoods', newFoodsArray); // this should log after the other two, but it is logging first.
const document = doc.toObject();
document.foods = newFoodsArray;
if (resolvedCount === options.length) cb(null, [document]);
}
})()
EDIT: Since it seems it will help, here is the what is calling the execute function I have excerpted above.
/**
* This will populate sub refs
* #param {import('mongoose').ModelPopulateOptions[]|
* import('mongoose').ModelPopulateOptions|String[]|String} options
* #returns {Promise}
*/
schema.methods.subPopulate = function (options = null) {
const model = this.constructor;
if (options) {
return new Promise((resolve, reject) => execute(model, [this], options, false, (err, docs) => {
if (err) return reject(err);
return resolve(docs[0]);
}));
}
Promise.resolve();
};
};

We can use async/await just fine here, as long as we remember that async is the same as "returning a Promise" and await is the same as "resolving a Promise's .then or .catch".
So let's turn all those "synchronous but callback-based" calls into awaitables: your outer code has to keep obeying the API contract, but since it's not meant to a return a value, we can safely mark our own version of it as async, and then we can use await in combination with promises around any other callback based function calls in our own code just fine:
async function execute(model, docs, options, lean, andThenContinueToThis) {
options = formatOptions(options);
let option, resolvedCount = 0;
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (option of options) {
// ...things happen here...
const aggregationOptions = [/*...data...*/];
try {
const refSubDocuments = await new Promise((resolve, reject) => rootRefModel
.aggregate(aggregationOptions)
.exec((err, result) => err ? reject(err) : resolve(result));
// ...do some work based on refSubDocuments...
}
// remember to forward errors and then stop:
catch (err) {
return andThenContinueToThis(err);
}
}
// remember: bind newFoodsArray somewhere so it doesn't get lost next iteration
}
// As our absolutely last action, when all went well, we trigger the call forwarding:
andThenContinueToThis(null, dataToForward);
}

Firebase Firestore - Common function for queries but with different outcomes

I'll try to explain myself the best I can.
What I'm trying to do is to set a common function which I can pass the name of the collection and do something with that. The problem is that since it is an asynchronous request I can't just do var a = getFirestoreData(); so i'll have to use this piece of code over and over again:
const db = firebase.firestore();
myCollection = 'SomeRandomCollectionName';
//Repetitive piece of code
db.collection(myCollection).get().then((snapshot) => {
var dataFromCollection = [];
snapshot.docs.forEach(doc => {
dataFromCollection.push(doc.data());
})
//TODO -> Do something with that data.
});
I want to be able to do different things with different collections with that same function (only one collection each time).
Basically what I want is something like a Utils class and use it for example:
Utils.getUsers("users");
Utils.getUsersAndUpdate("users", fieldToUpdate, dataToInsert);
Both of those functions use that same previous piece of code but with different continuity. How do I do this taking into account that it is an asynchronous request?
This is kind of a solution I made but it lacks the possibility to add parameters to the targetFunction since the amount of parameters each function takes may vary.
function getFromDatabaseToFunction(targetFunction, collection){
db.collection(collection).get().then((snapshot) => {
var dataToTargetFunction = [];
snapshot.docs.forEach(doc => {
dataToTargetFunction.push(doc.data());
})
targetFunction(dataToTargetFunction);
})
}
P.S. This may likely be a duplicate but since I don't know what to search for I can't say for sure

In JavaScript, the completion or failure of an asynchronous function is managed with a Promise. In your case, you can simply return a promise from getFromDatabaseToFunction and chain new actions with then():
function getFromDatabase(collection){
return db.collection(collection).get().then((snapshot) => {
var result = [];
snapshot.docs.forEach(doc => {
result.push(doc.data());
});
return result;
});
}
Here is how to use it:
var targetFunction = ...;
getFromDatabase("collectionName").then(collectionData => {
// do whatever you want with 'targetFunction' and 'collectionData'
});

Build out a .promise like:
function fetchCollection(collectionPath){
return new Promise(function (resolve, reject) {
var ref = firebase.firestore().collection(collectionPath);
ref.get().then(function(querySnapshot) {
resolve(querySnapshot);
}).catch(function (error) {
reject(error);
});
});
then use it like:
fetchCollection("users").then(function (querySnapshot) {
...
});

Call function after two promises resolve in angularJS, one using result from other

In a controller function, I make some operations:
Get a list of organizations with a promise
In the then of this promise, I loop through each of them to extract some data and populate some of my controller attributes.
One of this operation is to call another promise to gather all users attached to this organization, with a loop inside of it to extract name and other stuff.
When I get ALL of it, so every organization has been parsed, and within them all users too, I must call a function to update my view.
I got it working by setting some flags (orgParsed and usersParsed) but I find it to be... a code shame.
I heard about a way of maybe doing this by using $q to wait for the two promises and maybe loops inside their "then" to be resolve before calling my view function. But I struggle applying this code change since the second promise use the result of the first to gather the organization ID.
Here is my current code:
this.getOrgData = function () {
return Service.getList().then(function (result) {
var orgCount = result.Objects.length;
var orgParsed = 0;
_.forEach(result.Objects, function (org) {
org.Users = [];
// Some logic here using 'org' data
Service.getUsers(org.Id, 0, 0).then(function (userResult) {
usersParsed = 0;
_.forEach(userResult.Objects, function (user) {
// Some Logic here using 'user.Name'
usersParsed++;
});
orgParsed++;
if (orgParsed === orgCount && usersParsed === userResult.Objects.length) {
self.sortMenuList(); // My view Function
}
});
});
$scope.$broadcast("getOrgData");
});
};
Do you see any way to trigger my self.sortMenuList() function only when I can be sure I got all users of every companies parsed in more elegant/efficient/safe way?

Yes, that counting should definitely be replaced by $q.all, especially as you did not bother to handle any errors.
this.getOrgData = function () {
return Service.getList().then(function (result) {
$scope.$broadcast("getOrgData"); // not sure whether you want that here before the results from the loop
return $q.all(_.map(result.Objects, function (org) {
org.Users = [];
// Some logic here using 'org' data
return Service.getUsers(org.Id, 0, 0).then(function (userResult) {
_.forEach(userResult.Objects, function (user) {
// Some Logic here using 'user.Name'
});
});
}));
}).then(function() {
self.sortMenuList(); // My view Function;
})
};

The problem you describe sounds like you want to wait until a certain amount of promises are all resolved, and then do something with the result. That's really easy when you use Promise.all():
this.getOrgData = function () {
return Service.getList().then(function (result) {
var promises = [];
_.forEach(result.Objects, function (org) {
org.Users = [];
// Some logic here using 'org' data
// Store the promise for this user in the promises array
promises.push(Service.getUsers(org.Id, 0, 0));
});
// userResults is an array of all the results of the promises, in the same order as the getUsers was called
Promise.all(promises).then(function (userResults) {
_.forEach(userResults, function(userResult) {
_.forEach(userResult.Objects, function (user) {
// Some Logic here using 'user.Name'
});
});
self.sortMenuList();
});
$scope.$broadcast("getOrgData");
});
};

node.js: structure multiple API requests, work on them and combine them

currently I am struggeling a little bit with node.js (I am new to it) doing different API requests (Usabilla API), work on the results and then combine them in order to work on the whole set (e.g. export).
Requesting the API is not the problem but I can't get the results out to do some other stuff on it (asynchronous code drives me crazy).
Attached please find a overview how I thought to do this. Maybe I am totally wrong about this or maybe you have other more elegant suggestions.
My code works until I have to request the two different API "adresses" (they are provided) and then extract the results to do some other stuff.
My problem here is that there are nested functions with a promise and I cant figure out how to pass this through the parent function inside waterfall to get handled by the next function.
In the code, of course there is nothing parallel as shown in the diagram.
Thats another point, how to do that ? Simply nest parallel and series/ another waterfall inside waterfall ?
I am a little bit confused because that gets more and more complex for a simple problem when this would be done with synchronous code.
Here I build up all my request querys (at the moment 4):
function buildQuery(IDs,callback){
var i = 0;
var max = Object.keys(IDs).length;
async.whilst(
function(){return i < max},
function(callback){
FeedbackQuery[i] =
{
identifier: IDs[i].identifier,
query:
{id: IDs[i].id,
params: {since:sinceDate,}
}
};
i++;
callback(null,i);
})
console.log(FeedbackQuery);
callback (null,FeedbackQuery);
};
I then have to decide which type of query it is and add it to an object which should contain all the items of this identifier type:
function FeedbackRequest(FeedbackQuery,callback)
{
var i = 0;
var max = Object.keys(FeedbackQuery).length;
async.whilst(
function(){return i < max},
function (callback){
identifier = FeedbackQuery[i].identifier;
APIquery = FeedbackQuery[i].query;
switch(identifier)
{
case 'mobilePortal':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.websites.buttons.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsA[item] = feedback;
callback(null, feedbackResultsA);
})
break;
case 'apps':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.apps.forms.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsB[item] = feedback;
callback(null, feedbackResultsB);
})
break;
}
i++;
callback(null,i);
})
};
Currently the functions are bundled in an async waterfall:
async.waterfall([
async.apply(buildQuery,IDs2request),
FeedbackRequest,
// a function to do something on the whole feedbackResults array
],function (err, result) {
// result now equals 'done'
if (err) { console.log('Something is wrong!'); }
return console.log('Done!');
})
How it actually should be:
Structure
Thank you very much for any tips or hints!

I'm not proficient with async, and I believe if you'r new to this, it's harder than a simple Promise library like bluebird combine with lodash for helpers.
What I would do based on your schemas :
var firstStepRequests = [];
firstStepRequests.push(buildQuery());// construct your first steps queries, can be a loop, goal is to have firstStepRequests to be an array of promise.
Promise.all(firstStepRequests)
.then((allResults) => {
var type1 = _.filter(allResults, 'request_type_1');
var type2 = _.filter(allResults, 'request_type_2');
return {
type1: type1,
type2: type2
};
})
.then((result) => {
result.type1 = //do some work
result.type2 = //do some work
return result;
})
.then((result) => {
//export or merge or whatever.
});
Goal is to have a simple state machine.
UPDATE
If you want to keep identifier for a request, you can use props to have :
var props = {
id_1:Promise,
id_2:Promise,
id_3:Promise
};
Promise.props(props).then((results) => {
// results is {
id_1:result of promise,
id_2:result of promise,
etc...
}
})
You could do something like :
var type1Promises = getType1Requests(); //array of type 1
var type2Promises = getType2Requests(); // array of type 2
var props = {
type_1: Promise.all(type1Promises),
type_2: Promise.all(type2Promises)
}
Promise.props(props).then((result) => {
//result is : {
type_1: array of result of promises of type 1
type_2: array of result of promises of type 2
}
})

javascript/jquery: Iterative called function; wait till the previous call is finished

I've some problem with a library calling a function on each item. I've to check the state for this item via an ajax request and don't want to call one request per item, but get a range of item states.
Because these items are dates I can get some range pretty easy - that's the good part :)
So to to give some code ...
var itemStates = {};
var libraryObj = {
itemCallback: function(item) {
return checkState(item);
}
}
function checkState(item) {
if(!itemStates.hasOwnProperty(item)) {
$.get('...', function(result) {
$.extend(true, itemStates, result);
});
}
return itemStates[item];
}
The library is now calling library.itemCallback() on each item, but I want to wait for the request made in checkState() before calling checkState() again (because the chance is extremly high the next items' state was allready requested within the previous request.
I read about the defer and wait(), then() and so on, but couldn't really get an idea how to implement this.
Many thanks to everybody who could help me with this :)

You can achieve this by using jQuery.Deferred or Javascript Promise. In the following code, itemCallback() will wait for previous calls to finish before calling checkState().
var queue = [];
var itemStates = {};
var libraryObj = {
itemCallback: function(item) {
var def = $.Deferred();
$.when.apply(null, queue)
.then(function() {
return checkState(item);
})
.then(function(result) {
def.resolve(result);
});
queue.push(def.promise());
return def.promise();
}
}
function checkState(item) {
var def = $.Deferred();
if (!itemStates.hasOwnProperty(item)) {
$.get('...', function(result) {
$.extend(true, itemStates, result);
def.resolve(itemStates[item]);
});
} else
def.resolve(itemStates[item]);
return def.promise();
}
//these will execute in order, waiting for the previous call
libraryObj.itemCallback(1).done(function(r) { console.log(r); });
libraryObj.itemCallback(2).done(function(r) { console.log(r); });
libraryObj.itemCallback(3).done(function(r) { console.log(r); });
libraryObj.itemCallback(4).done(function(r) { console.log(r); });
libraryObj.itemCallback(5).done(function(r) { console.log(r); });
Same example built with Javascript Promises
var queue = [];
var itemStates = {};
var libraryObj = {
itemCallback: function(item) {
var promise = new Promise(resolve => {
Promise.all(queue)
.then(() => checkState(item))
.then((result) => resolve(result));
});
queue.push(promise);
return promise;
}
}
function checkState(item) {
return new Promise(resolve => {
if (item in itemStates)
resolve(itemStates[item]);
else {
$.get('...', function(result) {
$.extend(true, itemStates, result);
resolve(itemStates[item]);
});
}
});
}
//these will execute in order, waiting for the previous call
libraryObj.itemCallback(1).then(function(r) { console.log(r); });
libraryObj.itemCallback(2).then(function(r) { console.log(r); });
libraryObj.itemCallback(3).then(function(r) { console.log(r); });
libraryObj.itemCallback(4).then(function(r) { console.log(r); });
libraryObj.itemCallback(5).then(function(r) { console.log(r); });

The library is now calling library.itemCallback() on each item, but I want to wait for the request made in checkState() before calling checkState() again (because the chance is extremely high the next items' state was already requested within the previous request.
One thing I can think of doing is making some caching function, depending on the last time the function was called return the previous value or make a new request
var cached = function(self, cachingTime, fn){
var paramMap = {};
return function( ) {
var arr = Array.prototype.slice.call(arguments);
var parameters = JSON.stringify(arr);
var returning;
if(!paramMap[parameters]){
returning = fn.apply(self,arr);
paramMap[parameters]={timeCalled: new Date(), value:returning};
} else {
var diffMs = Math.abs(paramMap[parameters].timeCalled - new Date());
var diffMins = ( diffMs / 1000 ) / 60;
if(diffMins > cachingTime){
returning = fn.apply(self,arr);
paramMap[parameters] = {timeCalled: new Date(), value:returning};
} else {
returning = paramMap[parameters].value;
}
}
return returning;
}
}
Then you'd wrap the ajax call into the function you've made
var fn = cached(null, 1 , function(item){
return $.get('...', function(result) {
$.extend(true, itemStates, result);
});
});
Executing the new function would get you the last promise called for those parameters within the last request made at the last minute with those parameters or make a new request

simplest and dirty way of taking control over the library is to override their methods
But I don't really know core problem here so other hints are below
If you have the control over the checkState then just collect your data and change your controller on the server side to work with arrays that's it
and if you don't know when the next checkState will be called to count your collection and make the request use setTimeout to check collection after some time or setIterval to check it continuously
if you don't want to get same item multiple times then store your checked items in some variable like alreadyChecked and before making request search for this item in alreadyChecked
to be notified when some library is using your item use getter,
and then collect your items.
When you will have enough items collected then you can make the request,
but when you will not have enought items then use setTimeout and wait for some time. If nothing changes, then it means that library finishes the iteration for now and you can make the request with items that left of.
let collection=[];// collection for request
let _items={};// real items for you when you don't want to perfrom actions while getting values
let itemStates={};// items for library
let timeoutId;
//instead of itemStates[someState]=someValue; use
function setItem(someState,someValue){
Object.defineProperty(itemStates, someState, { get: function () {
if(typeof timeoutId=="number")clearTimeout(timeoutId);
//here you can add someState to the collection for request
collection.push(_items[someState]);
if(collection.length>=10){
makeRequest();
}else{
timeoutId=setTimeout(()=>{...checkCollectionAndMakeRequest...},someTime);
}
return someValue;
} });
}