How to wait for multiple asynchronous calls from forEach loop? - javascript

Trying to make call to multiple asynchronous function, and due to which getting result as undefined.
Tried async.waterfall, however not able to make it work.
Code:
const pendingData = [];
async.waterfall([
function (callback) {
WaitForApproval.find({}, (err,result) => {
callback(null,result);
});
},
function (result, callback) {
result.forEach(data => {
User.findOne({_id: data.uploadedBy}, (err,name) => {
let total = {
id: data._id,
name: name.name,
subject: data.subject,
uploadOn: data.uploadedAt
};
pendingData.push(total);
});
});
callback(null,'done');
}
], function (err,result) {
if(result === 'done') {
console.log(pendingData); // it is giving empty result.
}
});
How to wait for asynchronous function?

The issue you are having is that you are async functions within a non-async forEach loop.
You have a few options here:
Make the mongoDB call recursively - wrap this query in a function that calls itself after the query returns.
Read about mongoDB batch operations - https://docs.mongodb.com/manual/reference/method/Bulk/
Use an async/await pattern with each call by declaring the callback function within the async waterfall as 'async' and then using 'await' inside of the function for each query. Out of the box, forEach is not async. If you want to still use forEach, you can either re-write it async (See below) or use a regular for loop:
async function asyncForEach(array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array)
}
}
** There are additional ways to solve this beyond what is posted here, but here's a few things that will work if implemented correctly.

I would suggest you add the call for callback(null,'done'); immediately pendingData.push(total);
you are right, the async nature is making things hard for you now, but suppose you used Promises and chain them together, that would save you a lot of trouble.
once a time I had a similar problem of asynchronous code running out of order I wanted so I made a little Tweak using a custom promise function(I mead up) and call it order..such Idea can solve your problem if you could apply it to your code properly
https://github.com/lalosh/Ideas/blob/master/promiseOrder.js

Related

How does array.forEach() handle async functions?

I am trying to iterate over a list of keywords, and call mysql.query() for each one of them.
Problem is, I need each query call to end before the next one begins. How do I make that happen?
I have tried making it work with Promise requests but I do not wholly understand how asynchronous calls work.
keywords.forEach(keyword => {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {...}
});
You can do it recursivly
function call(keywords, index){
if(keyworkds[index]){
sql.query("Select Id from Keyword Where Word= ?", [keyworkds[index].keyword],
function (err, ids) {
call(keywords, index + 1)
}
}
}
call(keywords, 0)
forEach will not wait for the asynchronous function to be done. The reason you need to pass a callback is because that's the only real way you have to do something after completion.
For that reason forEach simply will not work. It's possible to rewrite this to have the second iteration happen asynchronously, but it's ugly as hell.
I would recommend (if you can) to switch to async / await and use a mysql client that supports promises. Example:
for (const keyword of keywords) {
const result = await sql.query(
"SELECT Id from Keyword Where Word = ?",
[keyword]
);
}
You can use mysql2/promise from the mysql2 package to use promisified mysql function.
You won't be able to do this with just the forEach. Each call of sql.query will fire as quickly as the loop iterates and will not wait until it finishes. Also, there is no guarantee those queries will resolve in the order you call them in.
At the bare minimum, you can use callback functions, but that would be some really ugly and difficult code to deal with. https://en.wikipedia.org/wiki/Pyramid_of_doom_(programming)
That leaves Promises and async/await. I strongly recommend spending some time on this topic as you will be running into it a lot
https://javascript.info/async
Instead of looking for an async / loop solution, you can solve your original problem with one SQL query:
const args = keywords.map(_ => "?").join();
sql.query("Select Id, Word from Keyword Where Word in (" + args + ")", keywords,
function (err, records) {...}
);
I think async js library (https://caolan.github.io/async) is a good choice for something similar to your problem and with this library you have a cleaner code without nested async calls that produce Pyramid of doom. Whenever you face a problem that has many async calls that will run either in parallel or synchronously you can use it.
as the documentation said you can run only a single async operation at a time with series method like eachSeries.
async.eachSeries(keywords, function(keyword, callback) {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {
if (err){
//if you want to stop reminding queries call callback with err
callback(err)
} else {
callback();
}
}
}, function(err) {
// if any of the queris produced an error, err would equal that error
if( err ) {
// One of the iterations produced an error.
// All processing will now stop.
console.log('A query failed to process');
}else {
console.log('All queries have been processed successfully');
}
});
forEach doesn't do anything special with async functions. It will call an async function, but it won't wait for it to complete before moving on. Therefore, if you need to call async promises sequentially, then you need another approach.
It looks like though you aren't using a library that returns a promise, so you will need to wrap the call in a promise (util.promisefy can help with that, or you can do it manually), or you can use a version of the library that does return promises. It is possible to do this with callbacks, but much harder.
The most straight forward is to use a for … of loop in side of an async function. In order to do this, you need a new enough version of javascript, and async functions are rather new. The basic idea look like this:
async function processCollection(collection) {
for (let item of collection) {
await processItem(item)
}
}
Where processItem would be another async function or a function that returns a promise (which are basically the same thing, just different syntaxes).
Another way, that doesn't require async functions, is to chain together promises. This can be done in a functional way as follows:
collection.reduce((previousPromise, item) => {
return previousPromose.then(() => processItem(item))
}, Promise.resolve(null))
(again, processItem is a function that returns a promise)
Whats goin on here is that by using reduce, you are basically calling promise.then(…).then(…).then(…)… once for each item in the collection. if the callback to then returns a promise, then it will wait for that promise to finish before calling the next then callback. So you get sequential execution. Also, any promise rejections are propagated as well, so errors will stop the next call from happening.

Javascript - wait for async functions with new async functions

Question:
Is it possible to wait for an async function that starts new async functions?
Details:
I have looked up a bunch of ways to wait for an async function to finish before continuing the code, or running a specific function or code block. But one thing have bugged me for a very long time - I do not know whether new async functions started are also being waited for, or if they require their own piece of code to be taken into account.
Pseudocode:
var value = 1;
af1();
alert(value);
async function af1(){
af2();
}
async function af2(){
af3();
}
async function af3(){
value = 2;
}
I dont know if this is a good example (or even the right syntax), but picture the async functions as some ajax requests that takes some time to finish. I have a feeling that if you add a jQuery deferred on af1, it will only wait for af1 and ignore af2 and af3. I am also using an external javascript file for some of the functions, and I dont really have control over what new functions are started in there.
So again, is it possible to just wrap all of this into something and run some code after all is done? Or am I mistaken about jQuery's deferred and .done functions??
No, async functions are not awaited when called. They just return a promise.
Inside an async function - that's their advantage - you can explicitly await promises, including those that are returned from other async functions.
Your code should have been written using return values, like this:
(async function() { // neccessary to use await
value = await af1();
alert(value);
}());
af1().then(alert); // or just using promise syntax
async function af1(){
return af2();
}
async function af2(){
return af3();
}
async function af3(){
return 2; // or maybe rather something like
return $.ajax(…);
}
But you don't need return values, you can use await as well for your closure approach:
(async function() {
var value = 1;
await af1();
// ^^^^^
alert(value);
async function af1(){
await af2();
}
async function af2(){
await af3();
}
async function af3(){
value = 2; // or maybe rather something like
value = await $.ajax(…);
}
}())
Use this git js ASync
How to use
Async provides around 20 functions that include the usual 'functional' suspects (map, reduce, filter, each…) as well as some common patterns for asynchronous control flow (parallel, series, waterfall…). All these functions assume you follow the Node.js convention of providing a single callback as the last argument of your async function.
Quick Examples
async.map(['file1','file2','file3'], fs.stat, function(err, results){
// results is now an array of stats for each file
});
async.filter(['file1','file2','file3'], fs.exists, function(results){
// results now equals an array of the existing files
});
async.parallel([
function(){ ... },
function(){ ... }
], callback);
async.series([
function(){ ... },
function(){ ... }
]);
There are many more functions available so take a look at the docs below for a full list. This module aims to be comprehensive, so if you feel anything is missing please create a GitHub issue for it.
Read More
Apart from above examples, look at below code sample. concept of async and wait would be more clear.
async function doWork(){
try {
const response = await makeRequest('facebook'); //using await will wait until the response returned from the makeRequest function
//console.log('Response Received' + response );
const response2 = await makeRequest('google');
//console.log('Response2 Received' + response2 );
} catch(err) {
alert(err);
}
}
function makeRequest(str){
//function body that takes time to process, eg: server call
return "making request to " + str;
}
doWork();

Returning object to caller function inside callback

I have a strict JavaScript API naming scheme I need to follow, it looks like this:
var Items = function() {
this.items = [];
};
Items.prototype.get() {
db.items.find(function(err, items) {
this.items = items;
});
return this.items;
}
The problem is the async call (db.items.find..) that doesn't have time to finish before the get() method returns an empty this.items..
The client needs to make the calls like this:
items = new Items();
console.log(items.get());
What's best practice to handle async calls here while still strictly following the API naming scheme?
Is there some native way I can let get() wait for a return inside the callback or do I need some kind of async lib for this?
EDIT:
Apparently what you are looking for might be possible using wait.for (https://github.com/luciotato/waitfor). I have not used it though, so I am not sure how well it would suit your needs. The method you would need would be wait.forMethod.
Previous Answer:
There is no way you can write async code in a synchronous manner. Also, it is not a good idea to mix async and sync methods. You are trying to define a synchronous method Item.prototype.get, but inside that you are using an async method db.items.find, which makes Item.prototype.get an asynchronous function. In order to get this to work you will have to define Item.prototype.get as a proper async function with a callback.
Items.prototype.get(fn) {
db.items.find(function(err, items) {
return fn(err, items);
});
}
You could then call it as
items = new Items();
items.get(function(err, items){
console.log(items);
}
I managed to solve this by using SilkJS (http://www.silkjs.net/), which is similar to Node in that it is built atop of V8 but it runs in synchronous mode.
This way I managed to keep the given API spec.

collection.find mongojs synchronous callback

I'm trying to write my own wrapper for the mongojs collection.find method that should return the collection items selected by the specified query (querying isn't implemented yet, it should simply select all results). The problem is that I'm not getting back an array of results. It seems like the find method does some kind of async callback. So how can I force a synchronous call or force my script to wait?
Collection.prototype.find = function () {
var result = new Array;
if (Bridge.isServer) {
db.collection(name).find(function(err, items) {
items.forEach(function(item) {
result.push(item);
});
});
}
return result;
}
I think you should consider making your functions asynchronous, but if you insist on writing synchronous functions, there is a github project for making async functions synchronous.
Here's another SO post dealing with the same topic: Convert an asynchronous function to a synchronous function

When using the await?

I'm using sequelize with typescript. I know that the code is asynchronous, and
Here, I am using promise, and the code works..
I would want to know when I must to use await keyword ?
const promises = []
let tabIdDoc = requestedListIdDoc.toString().split(",")
for (let thisIdDoc of tabIdDoc) {
promises.push(sequelize.models['Document'].findById(thisIdDoc))
}
q.all(promises).then((resultReq) => {
const lstDocs = []
for (let MyDoc of resultReq) {
if (MyDoc.matriculeDoc != "") {
lstDocs.push(sequelize.models['FolderDocContent'].findOrCreate({
where: {
}
}))
}
}
q.all(lstDocs).then(() => {
return response.status(201)
})
}
Is await keyword necessary here ?
You don't ever have to use await as other programming using .then() can always get the job done, but there are numerous times when using await makes your code simpler. This is particularly true when you are trying to sequence a number of asynchronous operations and even more so when you need to use previous results in more than one operation that follows.
Example #1: Serializing operations in a for loop
Suppose you want to save a bunch of items to your database, but for various reasons, you need to save them one by one and you need to save them in order (in other words, you need to sequence them):
async function saveItems(shoppingList) {
for (let item of shoppingList) {
// for loop will pause until this promise resolves
await db.save(item);
}
}
saveItems(myList).then(() => {
// all done
}).catch(err => {
// error here
});
Without using await, you'd have to use a significantly more verbose design pattern using .reduce() or perhaps a recursive function that you call on the completion of the prior operation. This is a lot simpler way to sequence iteration of a loop.
Example #2: Sequencing different operations in a function
Suppose, you need to contact three different outside services. You need to get some data from one, then use that data when you make a second API call, then use both of those pieces of data in a third API call:
const rp = require('request-promise');
async function getPrice(productName) {
// look up productID
const productID = await rp(`http://service1.com/api/getID?name=${productName}`);
// use productID to lookup price
const productPrice = await rp(`http://service1.com/api/getPrice?id=${productID}`);
// put both productID and price into the shopping cart
return rp({uri: 'http://service1.com/api/addToCart', body: {name: productName, id: productID}, json: true);
}
getPrice("Nikon_d750").then(total => {
// all done here, total is new cart total
}).catch(err => {
// error here
});
Both of these examples would require more code in order to properly sequence the asynchronous operations and you'd have to nest logic following inside a .then() handler. Using await instructs the JS interpreter to do that nesting for you automatically.
Some other examples here on MDN.
Several rules to keep in mind with await:
You can only use it inside a function prefixed with the async keyword as in my examples above.
All async functions return a promise. If you have an explicit return value in the function as in return x, then that value becomes the resolved value of the promise when all the asynchronous operations are done. Otherwise, it just returns a promise that has an undefined resolved value. So, to use a result from an async function or to know when it's done, you either have to await the result of that function (inside another async function) or you have to use .then() on it.
await only suspends execution within the containing function. It is as if the rest of the code in the function is placed inside an invisible .then() handler and the function will still return its promise immediately when it hits the first await. It does not block the event loop or block other execution outside the async function. This confuses many people when they first encounter await.
If the promise that you await rejects, then it throws an exception within your function. That exception can be caught with try/catch. If it is not caught, then the async function automatically catches it and rejects the promise that the function returns where the thrown value is the reject reason. It is easy when first using await to completely forget about the error cases when promises you are awaiting can reject.
You never necessarily must use await. You however should use await in every place where you'd otherwise have then with a callback, to simplify your code.
In your example:
const promises = requestedListIdDoc.toString().split(",").map(thisIdDoc =>
sequelize.models['Document'].findById(thisIdDoc)
);
const resultReq = await q.all(promises);
const lstDocs = resultReq.filter(myDoc => MyDoc.matriculeDoc != "").map(myDoc =>
sequelize.models['FolderDocContent'].findOrCreate({
where: {}
})
);
await q.all(lstDocs);
return response.status(201)

Categories

Resources