Develop Reference

Asyncronicity in a reduce() function WITHOUT using async/await

I am patching the exec() function to allow subpopulating in Mongoose, which is why I am not able to use async/await here -- my function will be chained off a db call, so there is no opportunity to call await on it, and within the submodule itself, there I can't add async/await outside of an async function itself.
With that out of the way, let's look at what I'm trying to do. I have two separate arrays (matchingMealPlanFoods and matchingMealPlanRecipeFoods) full of IDs that I need to populate. Both of them reside on the same array, foods. They each require a db call with aggregation, and the problem in my current scenario is that only one of the arrays populates because they are happening asynchronously.
What I am trying to do now is use the reduce function to return the updated foods array to the next run of reduce so that when the final result is returned, I can replace the entire foods array once on my doc. The problem of course is that my aggregate/exec has not yet returned a value by the time the reduce function goes into its next run. Is there a way I can achieve this without async/await here? I'm including the high-level structure here so you can see what needs to happen, and why using .then() is probably not viable.
EDIT: Updating code with async suggestion
function execute(model, docs, options, lean, cb) {
options = formatOptions(options);
let resolvedCount = 0;
let error = false;
(async () => {
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (let option of options) {
const path = option.path.split(".");
// ... various things happen here to prep the data
const aggregationOptions = [
// // $match, then $unwind, then $replaceRoot
];
await rootRefModel
.aggregate(aggregationOptions)
.exec((err, refSubDocuments) => {
// more stuff happens
console.log('newFoodsArray', newFoodsArray); // this is to check whether the second iteration is using the updated newFoods Array
const arrToReturn = newFoodsArray.map((food) => {
const newMatchingArray = food[nests[1]].map((matchingFood) => {
//more stuff
return matchingFood;
});
const updatedFood = food;
updatedFood[`${nests[1]}`] = newMatchingArray;
return updatedFood;
});
console.log('arrToReturn', arrToReturn);
newFoodsArray = [...arrToReturn];
});
}
};
console.log('finalNewFoods', newFoodsArray); // this should log after the other two, but it is logging first.
const document = doc.toObject();
document.foods = newFoodsArray;
if (resolvedCount === options.length) cb(null, [document]);
}
})()
EDIT: Since it seems it will help, here is the what is calling the execute function I have excerpted above.
/**
* This will populate sub refs
* #param {import('mongoose').ModelPopulateOptions[]|
* import('mongoose').ModelPopulateOptions|String[]|String} options
* #returns {Promise}
*/
schema.methods.subPopulate = function (options = null) {
const model = this.constructor;
if (options) {
return new Promise((resolve, reject) => execute(model, [this], options, false, (err, docs) => {
if (err) return reject(err);
return resolve(docs[0]);
}));
}
Promise.resolve();
};
};

We can use async/await just fine here, as long as we remember that async is the same as "returning a Promise" and await is the same as "resolving a Promise's .then or .catch".
So let's turn all those "synchronous but callback-based" calls into awaitables: your outer code has to keep obeying the API contract, but since it's not meant to a return a value, we can safely mark our own version of it as async, and then we can use await in combination with promises around any other callback based function calls in our own code just fine:
async function execute(model, docs, options, lean, andThenContinueToThis) {
options = formatOptions(options);
let option, resolvedCount = 0;
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (option of options) {
// ...things happen here...
const aggregationOptions = [/*...data...*/];
try {
const refSubDocuments = await new Promise((resolve, reject) => rootRefModel
.aggregate(aggregationOptions)
.exec((err, result) => err ? reject(err) : resolve(result));
// ...do some work based on refSubDocuments...
}
// remember to forward errors and then stop:
catch (err) {
return andThenContinueToThis(err);
}
}
// remember: bind newFoodsArray somewhere so it doesn't get lost next iteration
}
// As our absolutely last action, when all went well, we trigger the call forwarding:
andThenContinueToThis(null, dataToForward);
}

Javascript: Making sure one async function doesn't run until the other one is complete; working with promises

I'm working with fetching information from a github repository. I want to get the list of pull requests within that repo, get the list of commits associated with each pull request, then for each commit I want to get information such as the author of the commit, the number of files associated with each commit and the number of additions and deletions made to each file. I'm using axios and the github API to accomplish this. I know how to work with the API, but the promises and async functions are keeping me from accomplishing my task. I have the following code:
const axios = require('axios');
var mapOfInformationObjects = new Map();
var listOfCommits = [];
var listOfSHAs = [];
var gitApiPrefix = link I'll use to start fetching data;
var listOfPullRequestDataObjects = [];
var listOfPullRequestNumbers = [];
var mapOfPullNumberToCommits = new Map();
function getAllPullRequests(gitPullRequestApiLink) {
return new Promise((resolve, reject) => {
axios.get(gitPullRequestApiLink).then((response) =>{
listOfPullRequestDataObjects = response['data'];
var k;
for (k = 0; k < listOfPullRequestDataObjects.length; k++){
listOfPullRequestNumbers.push(listOfPullRequestDataObjects[k]['number']);
}
resolve(listOfPullRequestNumbers);
}).catch((error) => {
reject(error);
})
})
}
function getCommitsForEachPullRequestNumber(listOfPRNumbers) {
var j;
for (j = 0; j < listOfPRNumbers.length; j++) {
currPromise = new Promise((resolve, reject) => {
currentGitApiLink = gitApiPrefix + listOfPRNumbers[j] + "/commits";
axios.get(currentGitApiLink).then((response) => {
mapOfPullNumberToCommits.set(listOfPRNumbers[j], response['data']);
resolve("Done with Pull Request Number: " + listOfPRNumbers[j]);
}).catch((error) => {
reject(error);
})
})
}
}
function getListOfCommits(gitCommitApiLink){
return new Promise((resolve, reject) => {
axios.get(gitCommitApiLink).then((response) => {
resolve(response);
}).catch((error) => {
reject(error);
})
})
}
So far, I made some functions that I would like to call sequentially.
First I'd like to call getAllPullRequestNumbers(someLink)
Then I'd like to call getCommitsForEachPullRequestNumber(listofprnumbers)
Then getListOfCommits(anotherLink)
So it would look something like
getAllPullRequestNumbers(someLink)
getCommitsForEachPullRequestNumber(listofprnumbers)
getListOfCommits(anotherlink)
But two problems arise:
1) I'm not sure if this is how you would call the functions so that the first function in the sequence completes before the other.
2) Because I'm not familiar with Javascript, I'm not sure, especially with the getCommitsForEachPullRequestNumber function since you run a loop and call axios.get() on each iteration of the loop, if this is how you work with promises within the functions.
Would this be how you would go about accomplishing these two tasks? Any help is much appreciated. Thanks!

When you a number of asynchronous operations (represented by promises) that you can run all together and you want to know when they are all done, you use Promise.all(). You collect an array of promises and pass it to Promise.all() and it will tell you when they have all completed or when one of them triggers an error. If all completed, Promise.all() will return a promise that resolves to an array of results (one for each asynchronous operation).
When you're iterating an array to do your set of asynchronous operations, it then works best to use .map() because that helps you create a parallel array of promises that you can feed to Promise.all(). Here's how you do that in getCommitsForEachPullRequestNumber():
function getCommitsForEachPullRequestNumber(listOfPRNumbers) {
let mapOfPullNumberToCommits = new Map();
return Promise.all(listOfPRNumbers.map(item => {
let currentGitApiLink = gitApiPrefix + item + "/commits";
return axios.get(currentGitApiLink).then(response => {
// put data into the map
mapOfPullNumberToCommits.set(item, response.data);
});
})).then(() => {
// make resolved value be the map we created, now that everything is done
return mapOfPullNumberToCommits;
});
}
// usage:
getCommitsForEachPullRequestNumber(list).then(results => {
console.log(results);
}).catch(err => {
console.log(err);
});
Then, in getListOfCommits(), since axios already returns a promise, there is no reason to wrap it in a manually created promise. That is, in fact, consider a promise anti-pattern. Instead, just return the promise that axios already returns. In fact, there's probably not even a reason to have this as a function since one can just use axios.get() directly to achieve the same result:
function getListOfCommits(gitCommitApiLink){
return axios.get(gitCommitApiLink);
}
Then, in getAllPullRequests() it appears you are just doing one axios.get() call and then processing the results. That can be done like this:
function getAllPullRequests(gitPullRequestApiLink) {
return axios.get(gitPullRequestApiLink).then(response => {
let listOfPullRequestDataObjects = response.data;
return listOfPullRequestDataObjects.map(item => {
return item.number;
});
});
}
Now, if you're trying to execute these three operations sequentially in this order:
getAllPullRequests(someLink)
getCommitsForEachPullRequestNumber(listofprnumbers)
getListOfCommits(anotherlink)
You can chain the promises from those three operations together to sequence them:
getAllPullRequests(someLink)
.then(getCommitsForEachPullRequestNumber)
.then(mapOfPullNumberToCommits => {
// not entirely sure what you want to do here, perhaps
// call getListOfCommits on each item in the map?
}).catch(err => {
console.log(err);
});
Or, if you put this code in an async function, then you can use async/awit:
async function getAllCommits(someLink) {
let pullRequests = await getAllPullRequests(someLink);
let mapOfPullNumberToCommits = await getCommitsForEachPullRequestNumber(pullRequests);
// then use getlistOfCommits() somehow to process mapOfPullNumberToCommits
return finalResults;
}
getAllCommits.then(finalResults => {
console.log(finalResults);
}).catch(err => {
console.log(err);
});

not as clean as jfriend00 solution,
but I played with your code and it finally worked
https://repl.it/#gui3/githubApiPromises
you get the list of commits in the variable listOfCommits
I don't understand the purpose of your last function, so I dropped it

Firebase not receiving data before view loaded - empty array returned before filled

In the following code I save each item's key and an email address in one table, and to retrieve the object to fetch from the original table using said key. I can see that the items are being put into the rawList array when I console.log, but the function is returning this.cartList before it has anything in it, so the view doesn't receive any of the data. How can I make it so that this.cartList waits for rawList to be full before it is returned?
ionViewWillEnter() {
var user = firebase.auth().currentUser;
this.cartData.getCart().on('value', snapshot => {
let rawList = [];
snapshot.forEach(snap => {
if (user.email == snap.val().email) {
var desiredItem = this.goodsData.findGoodById(snap.val().key);
desiredItem.once("value")
.then(function(snapshot2) {
rawList.push(snapshot2);
});
return false
}
});
console.log(rawList);
this.cartList = rawList;
});
}
I have tried putting the this.cartList = rawList in a number of different locations (before return false, even inside the .then statement, but that did not solve the problem.

The following function call is asynchronous and you're falling out of scope before rawList has a chance to update because this database call takes a reasonably long time:
desiredItem.once("value").then(function(snapshot2) {
rawList.push(snapshot2);
});
You're also pushing the snapshot directly to this list, when you should be pushing snapshot2.val() to get the raw value.
Here's how I would fix your code:
ionViewWillEnter() {
var user = firebase.auth().currentUser;
this.cartData.getCart().on('value', snapshot => {
// clear the existing `this.cartList`
this.cartList = [];
snapshot.forEach(snap => {
if (user.email == snap.val().email) {
var desiredItem = this.goodsData.findGoodById(snap.val().key);
desiredItem.once("value")
.then(function(snapshot2) {
// push directly to the cartList
this.cartList.push(snapshot2.val());
});
}
return false;
});
});
}

The problem is the Promise (async .once() call to firebase) inside the forEach loop (sync). The forEach Loop is not gonna wait for the then() statement so then on the next iteration the data of the previous iteration is just lost...
let snapshots = [1, 2, 3];
let rawList = [];
snapshots.forEach((snap) => {
console.log(rawList.length)
fbCall = new Promise((resolve, reject) => {
setTimeout(function() {
resolve("Success!");
}, 2500)
});
fbCall.then((result) => {
rawList.push(result);
});
})
You need forEach to push the whole Promise to the rawList and Then wait for them to resolve and do sth with the results.
var snapshots = [1, 2, 3];
var rawList = [];
var counter = 0;
snapshots.forEach((snap) => {
console.log(rawList.length)
var fbCall = new Promise((resolve, reject) => {
setTimeout(function() {
resolve("Success!" + counter++);
}, 1500)
});
rawList.push(fbCall);
})
Promise.all(rawList).then((res) => {
console.log(res[0]);
console.log(res[1]);
console.log(res[2]);
});
The thing is, it is still a bit awkward to assign this.cartList = Promise.all(rawList) as it makes it a Promise. So you might want to rethink your design and make something like a getCartList Service? (dont know what ur app is like :p)

Since you're using angular you should also be using angularfire2, which makes use of Observables which will solve this issue for you. You will still be using the normal SDK for many things but for fetching and binding data it is not recommended to use Firebase alone without angularfire2 as it makes these things less manageable.
The nice things about this approach is that you can leverage any methods on Observable such as filter, first, map etc.
After installing it simply do:
public items$: FirebaseListObservable<any[]>;
this.items$ = this.af.database.list('path/to/data');
And in the view:
{{items$ | async}}
In order to wait for the data to appear.

Use AngularFire2 and RxJS this will save you a lot of time, and you will do it in the proper and maintainable way by using the RxJS operators, you can learn about those operators here learnrxjs

Call function after two promises resolve in angularJS, one using result from other

In a controller function, I make some operations:
Get a list of organizations with a promise
In the then of this promise, I loop through each of them to extract some data and populate some of my controller attributes.
One of this operation is to call another promise to gather all users attached to this organization, with a loop inside of it to extract name and other stuff.
When I get ALL of it, so every organization has been parsed, and within them all users too, I must call a function to update my view.
I got it working by setting some flags (orgParsed and usersParsed) but I find it to be... a code shame.
I heard about a way of maybe doing this by using $q to wait for the two promises and maybe loops inside their "then" to be resolve before calling my view function. But I struggle applying this code change since the second promise use the result of the first to gather the organization ID.
Here is my current code:
this.getOrgData = function () {
return Service.getList().then(function (result) {
var orgCount = result.Objects.length;
var orgParsed = 0;
_.forEach(result.Objects, function (org) {
org.Users = [];
// Some logic here using 'org' data
Service.getUsers(org.Id, 0, 0).then(function (userResult) {
usersParsed = 0;
_.forEach(userResult.Objects, function (user) {
// Some Logic here using 'user.Name'
usersParsed++;
});
orgParsed++;
if (orgParsed === orgCount && usersParsed === userResult.Objects.length) {
self.sortMenuList(); // My view Function
}
});
});
$scope.$broadcast("getOrgData");
});
};
Do you see any way to trigger my self.sortMenuList() function only when I can be sure I got all users of every companies parsed in more elegant/efficient/safe way?

Yes, that counting should definitely be replaced by $q.all, especially as you did not bother to handle any errors.
this.getOrgData = function () {
return Service.getList().then(function (result) {
$scope.$broadcast("getOrgData"); // not sure whether you want that here before the results from the loop
return $q.all(_.map(result.Objects, function (org) {
org.Users = [];
// Some logic here using 'org' data
return Service.getUsers(org.Id, 0, 0).then(function (userResult) {
_.forEach(userResult.Objects, function (user) {
// Some Logic here using 'user.Name'
});
});
}));
}).then(function() {
self.sortMenuList(); // My view Function;
})
};

The problem you describe sounds like you want to wait until a certain amount of promises are all resolved, and then do something with the result. That's really easy when you use Promise.all():
this.getOrgData = function () {
return Service.getList().then(function (result) {
var promises = [];
_.forEach(result.Objects, function (org) {
org.Users = [];
// Some logic here using 'org' data
// Store the promise for this user in the promises array
promises.push(Service.getUsers(org.Id, 0, 0));
});
// userResults is an array of all the results of the promises, in the same order as the getUsers was called
Promise.all(promises).then(function (userResults) {
_.forEach(userResults, function(userResult) {
_.forEach(userResult.Objects, function (user) {
// Some Logic here using 'user.Name'
});
});
self.sortMenuList();
});
$scope.$broadcast("getOrgData");
});
};

node.js: structure multiple API requests, work on them and combine them

currently I am struggeling a little bit with node.js (I am new to it) doing different API requests (Usabilla API), work on the results and then combine them in order to work on the whole set (e.g. export).
Requesting the API is not the problem but I can't get the results out to do some other stuff on it (asynchronous code drives me crazy).
Attached please find a overview how I thought to do this. Maybe I am totally wrong about this or maybe you have other more elegant suggestions.
My code works until I have to request the two different API "adresses" (they are provided) and then extract the results to do some other stuff.
My problem here is that there are nested functions with a promise and I cant figure out how to pass this through the parent function inside waterfall to get handled by the next function.
In the code, of course there is nothing parallel as shown in the diagram.
Thats another point, how to do that ? Simply nest parallel and series/ another waterfall inside waterfall ?
I am a little bit confused because that gets more and more complex for a simple problem when this would be done with synchronous code.
Here I build up all my request querys (at the moment 4):
function buildQuery(IDs,callback){
var i = 0;
var max = Object.keys(IDs).length;
async.whilst(
function(){return i < max},
function(callback){
FeedbackQuery[i] =
{
identifier: IDs[i].identifier,
query:
{id: IDs[i].id,
params: {since:sinceDate,}
}
};
i++;
callback(null,i);
})
console.log(FeedbackQuery);
callback (null,FeedbackQuery);
};
I then have to decide which type of query it is and add it to an object which should contain all the items of this identifier type:
function FeedbackRequest(FeedbackQuery,callback)
{
var i = 0;
var max = Object.keys(FeedbackQuery).length;
async.whilst(
function(){return i < max},
function (callback){
identifier = FeedbackQuery[i].identifier;
APIquery = FeedbackQuery[i].query;
switch(identifier)
{
case 'mobilePortal':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.websites.buttons.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsA[item] = feedback;
callback(null, feedbackResultsA);
})
break;
case 'apps':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.apps.forms.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsB[item] = feedback;
callback(null, feedbackResultsB);
})
break;
}
i++;
callback(null,i);
})
};
Currently the functions are bundled in an async waterfall:
async.waterfall([
async.apply(buildQuery,IDs2request),
FeedbackRequest,
// a function to do something on the whole feedbackResults array
],function (err, result) {
// result now equals 'done'
if (err) { console.log('Something is wrong!'); }
return console.log('Done!');
})
How it actually should be:
Structure
Thank you very much for any tips or hints!

I'm not proficient with async, and I believe if you'r new to this, it's harder than a simple Promise library like bluebird combine with lodash for helpers.
What I would do based on your schemas :
var firstStepRequests = [];
firstStepRequests.push(buildQuery());// construct your first steps queries, can be a loop, goal is to have firstStepRequests to be an array of promise.
Promise.all(firstStepRequests)
.then((allResults) => {
var type1 = _.filter(allResults, 'request_type_1');
var type2 = _.filter(allResults, 'request_type_2');
return {
type1: type1,
type2: type2
};
})
.then((result) => {
result.type1 = //do some work
result.type2 = //do some work
return result;
})
.then((result) => {
//export or merge or whatever.
});
Goal is to have a simple state machine.
UPDATE
If you want to keep identifier for a request, you can use props to have :
var props = {
id_1:Promise,
id_2:Promise,
id_3:Promise
};
Promise.props(props).then((results) => {
// results is {
id_1:result of promise,
id_2:result of promise,
etc...
}
})
You could do something like :
var type1Promises = getType1Requests(); //array of type 1
var type2Promises = getType2Requests(); // array of type 2
var props = {
type_1: Promise.all(type1Promises),
type_2: Promise.all(type2Promises)
}
Promise.props(props).then((result) => {
//result is : {
type_1: array of result of promises of type 1
type_2: array of result of promises of type 2
}
})