How does array.forEach() handle async functions? - javascript

I am trying to iterate over a list of keywords, and call mysql.query() for each one of them.
Problem is, I need each query call to end before the next one begins. How do I make that happen?
I have tried making it work with Promise requests but I do not wholly understand how asynchronous calls work.
keywords.forEach(keyword => {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {...}
});

You can do it recursivly
function call(keywords, index){
if(keyworkds[index]){
sql.query("Select Id from Keyword Where Word= ?", [keyworkds[index].keyword],
function (err, ids) {
call(keywords, index + 1)
}
}
}
call(keywords, 0)

forEach will not wait for the asynchronous function to be done. The reason you need to pass a callback is because that's the only real way you have to do something after completion.
For that reason forEach simply will not work. It's possible to rewrite this to have the second iteration happen asynchronously, but it's ugly as hell.
I would recommend (if you can) to switch to async / await and use a mysql client that supports promises. Example:
for (const keyword of keywords) {
const result = await sql.query(
"SELECT Id from Keyword Where Word = ?",
[keyword]
);
}
You can use mysql2/promise from the mysql2 package to use promisified mysql function.

You won't be able to do this with just the forEach. Each call of sql.query will fire as quickly as the loop iterates and will not wait until it finishes. Also, there is no guarantee those queries will resolve in the order you call them in.
At the bare minimum, you can use callback functions, but that would be some really ugly and difficult code to deal with. https://en.wikipedia.org/wiki/Pyramid_of_doom_(programming)
That leaves Promises and async/await. I strongly recommend spending some time on this topic as you will be running into it a lot
https://javascript.info/async

Instead of looking for an async / loop solution, you can solve your original problem with one SQL query:
const args = keywords.map(_ => "?").join();
sql.query("Select Id, Word from Keyword Where Word in (" + args + ")", keywords,
function (err, records) {...}
);

I think async js library (https://caolan.github.io/async) is a good choice for something similar to your problem and with this library you have a cleaner code without nested async calls that produce Pyramid of doom. Whenever you face a problem that has many async calls that will run either in parallel or synchronously you can use it.
as the documentation said you can run only a single async operation at a time with series method like eachSeries.
async.eachSeries(keywords, function(keyword, callback) {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {
if (err){
//if you want to stop reminding queries call callback with err
callback(err)
} else {
callback();
}
}
}, function(err) {
// if any of the queris produced an error, err would equal that error
if( err ) {
// One of the iterations produced an error.
// All processing will now stop.
console.log('A query failed to process');
}else {
console.log('All queries have been processed successfully');
}
});

forEach doesn't do anything special with async functions. It will call an async function, but it won't wait for it to complete before moving on. Therefore, if you need to call async promises sequentially, then you need another approach.
It looks like though you aren't using a library that returns a promise, so you will need to wrap the call in a promise (util.promisefy can help with that, or you can do it manually), or you can use a version of the library that does return promises. It is possible to do this with callbacks, but much harder.
The most straight forward is to use a for … of loop in side of an async function. In order to do this, you need a new enough version of javascript, and async functions are rather new. The basic idea look like this:
async function processCollection(collection) {
for (let item of collection) {
await processItem(item)
}
}
Where processItem would be another async function or a function that returns a promise (which are basically the same thing, just different syntaxes).
Another way, that doesn't require async functions, is to chain together promises. This can be done in a functional way as follows:
collection.reduce((previousPromise, item) => {
return previousPromose.then(() => processItem(item))
}, Promise.resolve(null))
(again, processItem is a function that returns a promise)
Whats goin on here is that by using reduce, you are basically calling promise.then(…).then(…).then(…)… once for each item in the collection. if the callback to then returns a promise, then it will wait for that promise to finish before calling the next then callback. So you get sequential execution. Also, any promise rejections are propagated as well, so errors will stop the next call from happening.

Related

Understanding callbacks and promises. Is this correct?

So, I am currently learning about callbacks and promises and I keep getting tripped up when watching tutorials. So i thought that I would put it in an example form and see if anyone could tell me if my thinking is correct. I am currently working with MongoDB and Mongoose.
Here is an example piece of code taken from a tutorial. I believe that this is considered a callback?
user.save((err) => {
if (err) {
return res.status(400).json({
error: "You are not authorized to perform this action.",
})
});
And then this would be considered a promise?
user.save().catch(err => {
return res.status(400).json({
error: "You are not authorized."
})
})
Are there any performance advantages of using a promise over a callback or vice versa?
Thank You!
Callbacks and promises are different but share some concepts.
A callback is simply a function passed to another function as a parameter where that function can be executed at any time.
for example:
function iNeedACallback(callback)
{
// execute a bunch of code...
// use the callback function provided
callBack("a function called 'iNeedACallback'")
}
const callback = (fromFunction) => console.log("I am a callback function called from: " + fromFunction);
// Execute iNeedACallback by passing in the callback
iNeedACallback(callback);
The above code is a very simplistic instance of a callback function - now this can either be executed sequentially(like in the example above) or it can be an async operation on the event loop. (for more info about the event loop: Why do you need to await AJAX calls in JS but not in C#?)
A Promise makes use of callback functions in order to handle async code in a conventional way. The nature of a Promise infers that its some code that will be placed on the event loop as it executed asynchronously and not block any other code while waiting for an operation like a network call.
for example:
const promise = new Promise((resolve, reject) => {
setTimeout(() => {
const text = "i was only called after 2 seconds :(";
resolve(text);
}, 2000)
})
promise.then((text) => console.log(text))
console.log("i got called immediately :)")
//Output:
//i got called immediately :)
//i was only called after 2 seconds :(
In the above example we give a promise a callback function to execute when an async operation has completed (like a setTimeout) where the most common use cases are network calls like HTTP requests or TCP connections to a database like mongodb.
Back to your original question - your assumption is correct in the sense that your demonstration of a callback and promise is correct, and your question about performance..
Javascript is single threaded so if your callback function is placed on the event loop (which it may or may not be) then it will not block the main execution call stack. Promises are inherently async operation so they will always be placed on the event loop so the performance factor is dependent on the underlying implementation of the callback function - Promises will be async where as some regular callbacks could be sequential and synchronous.

How to "await" (or .then()) a function which is asynchronous, but is wrapped in a module and doesn't return a Promise

(Warning: This question is absolutely enormous because the problem has a really complicated context. Heck, by the time I finish writing this question I may actually end up rubber-ducking myself into actually coming up with the solution...)
(Edit: That didn't happen. I still have no idea what to do. I hope there's not a site rule against gigantic questions like this... here goes...)
I am writing code for a Discord bot (using Node and Discord.js) which interfaces with a database. (Specifically MongoDB.) Of course, this means double the asynchronous behavior. When I write things in the simplest way I can, things work pretty well, and I think I generally understand Promises, callbacks, and await well enough that I can ensure things happen in the right sequence.
However, in refactoring my code to enhance modularity, I've come across a seemingly insurmountable annoyance: I have lost proper error catching, and things print that they have succeeded while in the same breath the module that it executed (correctly) reports that the command has failed.
First, a bit of background.
The bot has a number of commands that use the database; we'll call them "!insult" and "!joke". The idea behind these commands is that they procedurally put together an insult or a joke which is built from components that the users have added to a database. Each command has a separate "collection" (MongoDB term, think SQL table) containing their respective data that was input by users.
The bot was originally written by somebody else, and their solution for adding and removing things to/from each collection was to have four separate commands: "!insultadd", "!insultdelete", "!jokeadd" and "!jokedelete". My first thought on seeing that was "modularity, eat your heart out. Yikes." The codebase contained a lot of code reduplication like this, and so I made it my goal to abstract functionality enough that much of this reduplication could be eliminated and the codebase would be overall much easier to extend and maintain.
So, I have come up with a command called "!db". There is already a layer of modularity to be found: all that !db does, is call "sub-commands" that implement each individual function. These sub-commands are called things like "!dbadd", "!dbdelete", etc. and they are not intended to be called on their own. An important thing to note is that I wrote these sub-commands first, and only once they were all independently functional, I created !db to wrap them in a simplistic manner, just using a case statement. (For example, invoking !db add insultsCollection "ugly" (where insultsCollection is the collection of insulting adjectives) would simply end up calling !dbadd with the appropriate arguments.) So, originally, each sub-command would print out results on its own, using lines like msg.channel.send('Inserted "' + selectedItem + '" into ' + selectedCollection + '.');.
Originally, this worked just fine. !db didn't have to do anything more than simply:
var dbadd = require('../commandsInternal/dbadd.js');
dbadd.execute(msg,args.slice(1),db);
and !dbadd would take care of printing out to the user that the operation was successful, reporting what item was inserted into the DB.
However, a crucial part of this gigantic refactoring is that the external behavior and usage remain largely the same to the end user - that is, !jokeadd and its kin will remain, but their insides will be scooped out and replaced with calls to the relevant !db functions. This is where we start to run into trouble. When I would try calling something like !insultadd, this would happen:
> !insultadd "ugly"
Inserted "ugly" into "insultsCollection". (This is printed by !dbadd.)
The bot can now call you "ugly"! (This is printed by !insultadd.)
This behavior is undesired because fundamentally we want to present to the user as if it were a simple list of adjectives, and so we want to avoid references to e.g. the names of the collections in the DB. So, how did I correct this? The most extensible way, I think, would be to add some sort of flag to the sub-commands, like "beQuiet", to determine whether it prints its own stuff or not. If this were a "normal" codebase, that's probably what I'd do. But...
The commands are written in Node modules that export a couple of things: the name of the command, the cooldown of the command, etc... but most importantly, a function called execute(msg, args, db). This function is how the main flow of the bot calls arbitrary commands. It looks up the name of the command, maps it to an object, and then tries to execute the execute method on the command object. Note that execute takes three args... a Discord.js Message object, the arguments for the command (an array of strings), and a MongoDB Db object. In order to pass a flag like "beQuiet" into !dbadd, I would be forced to add another arg to execute, which I am extremely loathe to to do because it would mean that some commands get "special" args for reasons, and... ugh. It would kinda be a breakdown of consistency, inviting things to become a total free-for-all.
So I can't pass in a flag. Ok, what next? "Well," I thought, "why don't I just move the printing into !db?" So I did that. My switch-case statement now looks like:
switch (choice) {
case "add":
dbadd.execute(msg,args.slice(1),db);
msg.channel.send('Inserted "' + args[2] + '" into ' + args[1] + '.');
break;
case "delete":
dbdelete.execute(msg,args.slice(1),db);
msg.channel.send('"' + args[2] + '" has been removed from ' + args[1] + '.');
break;
// ... etc
}
Alright, cool! So let's execute it... ok, cool, seems to work fine. Now, let's just test it with some invalid input...
> !db delete insultsCollection asdfasdf
Did the user give a collection that exists? : true (Debugging output)
Error: No matches in given collection. (Correct error output from !dbdelete)
"asda" has been removed from hugs. (Erroneous output from !db)
Uh-oh. So, why does this happen? Essentially, it's because of asynchronicity. All calls to database stuff require you to either provide a callback, or handle a Promise. (I much prefer the latter when possible.) So, !dbdelete has stuff like this:
var query = { value: { $eq: selectedItem} };
let numOfFind = await db.collection(selectedCollection)
.find(query)
.count();
// Note that .count() returns a Promise that resolves to an int.
// Hence the await.
if (numOfFind == 0) {
msg.channel.send("Error: No matches in given collection.");
return;
}
Convenient, right? Making the execute() function (which the above code is wrapped in) be an async function made everything a lot easier to write. I use .then() where appropriate, and everything's hunky-dory. But the trouble is essentially in that return...
(Oop, for a minute I thought I had rubber-ducked myself into solving the problem. But apparently just adding throw doesn't work.)
Ok, so... the problem is... whether I use return or throw, !db doesn't care. The way I think about it, making an asynchronous function call (like to db.collection().find()) sort of causes an independent "job" to start. (I am sure I'm very wrong about this, but this model of thinking has worked so far.) By seeing that things like:
db.collection(selectedCollection).deleteMany(query, function(err, result) {
if (err) {
throw err;
console.log('Something went wrong!');
return;
}
console.log('"' + selectedItem + '" has been removed from ' + selectedCollection + '.');
});
console.log("Success! Deleted the thing.");
would actually print "Success!" BEFORE actually deleting the item, I've come to understand that the script carries on its merry way when you call something asynchronous, and if you want it to actually print it afterward, you need to (in the case above) put it inside the callback, or use .then(), or await the result. You have to.
But the problem is... because of the modularity of !dbdelete, I can't do any of those. These don't work:
// Option 1: Callbacks.
// Doesn't work because execute() doesn't take a callback!
case "delete":
dbdelete.execute(msg,args.slice(1),db, function(err, result) {
msg.channel.send('"' + args[2] + '" has been removed from ' + args[1] + '.',msg);
});
break;
// Option 2: .then().
// Doesn't work because execute() doesn't return a Promise!
case "delete":
dbdelete.execute(msg,args.slice(1),db)
.then(function(err, result) {
msg.channel.send('"' + args[2] + '" has been removed from ' + args[1] + '.',msg);
});
break;
// Option 3: await.
// Doesn't work because... I don't really know why but I know it doesn't work.
// Also, again, execute() doesn't return a promise so we can't await it.
case "delete":
await dbdelete.execute(msg,args.slice(1),db);
msg.channel.send('"' + args[2] + '" has been removed from ' + args[1] + '.',msg);
break;
So, I'm at the end of my rope. I have no idea how to resolve this. To be honest I'm seriously considering just making .execute() return a Promise just so I can .then() it. But I really don't want to do that, especially since I don't know how. In short: Is there any way to do .then() on a function that doesn't return a promise? If I could just make it blocking, we'd be fine.
UPDATE: Here is the code for dbdelete.js: https://pastebin.com/LdHm3ybU
UPDATE 2: According to Mark Meyer, because I have used the await keyword, execute() actually does return a Promise! And it turns out, this solves one of the problems:
case "delete":
let throwaway = await dbdelete.execute(msg,args.slice(1),db);
message.channel.send('"' + args[2] + '" has been removed from ' + args[1] + '.');
break;
This code leads to a closer-to-intended result: The print statement still always runs even on failure, but... then I just make dbdelete.execute() return a boolean value that I set to false if !db shouldn't print anything!! So, both problems are now solved! Thank you, everyone, for responding so quickly! You were really helpful! <3
If your .execute() method is asynchronous, the ONLY way the caller can know when it's done or can know what its return value is if you design into the API and asynchronous mechanism for knowing that. A synchronous function will return long before the asynchronous operation inside the function is done so the caller can't know when it's done or know what result it achieved.
So, you will need to create a mechanism for the caller to know when .execute() is done and what it's result is. The common mechanisms are:
Return a promise that resolves/rejects with the final result. The caller uses .then() or await to track it.
Accept a callback that will be called when the final disposition is known.
Use some other mechanism such as an event that is triggered on some known object (streams use this scheme).
You will need to either find some known object that the caller already knows that you can trigger an event on or you will need to change the API to have an asynchronous interface. There is no way in Javascript to convert an asynchronous operation into a synchronous return value so you will need to change the interface.
For a one-shot returned result (not some ongoing event that triggers multiple times), the "modern" way of doing things in Javascript is to return a promise and then the caller can use .then() or await on that promise.
I prefer "beQuiet" this way, reuse of other async logic is more or less the same way
const dbAddSlient = async(..args) => {
//your db insertion
return result
}
const dbAdd = async(...args) => {
//you can use try/catch to wrap your async logic
const result = await dbAddSlient(...args) //reuse
console.log('your log') //additional opts
return result
}
module.exports = {
dbAddSlient,
dbAdd
}
When your struggle with async logic, better convert all callback to promise, then you will feel better. For example dbAddSlient may use mongodriver and async logic with callbacks, and make sure the logics all finished after await or then
const dbinert = (data) => new Promise((resolve, reject) => {
MongoClient.connect("mongodb://localhost:27017/integration_tests", function(err, db) {
if (err) {
reject(err)
}
db.collection('mongoclient_test').insert(data, function(err, result) {
if (err) {
reject(err)
}
db.close()
resolve(result)
})
})
})
const dbAddSlient = async(..args) => {
const result = await dbinert(somedata)
//result is what you resolved, and now all the db operation is surely done
return result
}
// some chained logic with async
(async() => {
try {
await someAsync1()
const result = await dbAddSlient(data)
await someAsync3()
} catch (e) {
//handle error
}
})()
// or use promise
(() => {
someAsync1()
.then(() => dbAddSlient(data))
.then((result) => someAsync3())
.catch(e => {
//handle error
})
})()
The problem you are facing is you are not get used to async logic, I suggest some article to read, some practice better
https://medium.com/#bluepnume/learn-about-promises-before-you-start-using-async-await-eb148164a9c8
There are more complicated async logic you may face, hope this will help when you run into some logic like that
Promise.all
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
Promise.race
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/race
generator
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator
If you got any question, just comment on this answer

Async Frustrations: Should I be using callbacks and how do I pass them through multiple modules?

I am attempting to make a simple text game that operates in a socket.io chat room on a node server. The program works as follows:
Currently I have three main modules
Rogue : basic home of rogue game functions
rogueParser : module responsible for extracting workable commands from command strings
Verb_library: module containing a list of commands that can be invoked from the client terminal.
The Client types a command like 'say hello world'. This triggers the following socket.io listener
socket.on('rg_command', function(command){
// execute the verb
let verb = rogueParser(command);
rogue.executeVerb(verb, command, function(result){
console.log(result);
});
});
Which then in turn invokes the executeVerb function from rogue..
executeVerb: function(verb, data, callback){
verb_library[verb](data, callback);
},
Each verb in verb_library should be responsible for manipulating the database -if required- and then returning an echo string sent to the appropriate targets representing the completion of the action.
EDIT: I chose 'say' when I posted this but it was pointed out afterward that it was a poor example. 'say' is not currently async but eventually will be as will be the vast majority of 'verbs' as they will need to make calls to the database.
...
say: function(data, callback){
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
message = ('you say '+'"'+message.trim()+'"');
response.target = data.user;
response.type = 'echo';
response.message = message;
callback(response);
},
...
My problem is that
1 ) I am having issues passing callbacks through so many modules. Should I be able to pass a callback through multiple layers of modules? Im worried that I'm blind so some scope magic that is making me lose track of what should happen when I pass a callback function into a module which then passes the same callback to another module which then calls the callback. Currently it seems I either end up without access to the callback on the end, or the first function tries to execute without waiting on the final callback returning a null value.
2 ) Im not sure if Im making this harder than it needs to be by not using promises or if this is totally achievable with callbacks, in which case I want to learn how to do it that way before I summon extra code.
Sorry if this is a vague question, I'm in a position of design pattern doubt and looking for advice on this general setup as well as specific information regarding how these callbacks should be passed around. Thanks!
1) Passing callback trough multiple layers doesn't sound like a good idea. Usually I'm thinking what will happen, If I will continiue doing this for a year? Will it be flexible enough so that when I need to change to architecture (let's say customer have new idea), my code will allow me to without rewriting whole app? What you're experiencing is called callback hell. http://callbackhell.com/
What we are trying to do, is to keep our code as shallow as possible.
2) Promise is just syntax sugar for callback. But it's much easier to think in Promise then in callback. So personally, I would advice you to take your time and grasp as much as you can of programming language features during your project. Latest way we're doing asynchronus code is by using async/await syntax which allows us to totally get rid of callback and Promise calls. But during your path, you will have to work with both for sure.
You can try to finish your code this way and when you're done, find what was the biggest pain and how could you write it again to avoid it in future. I promise you that it will be much more educative then getting explicit answear here :)
Asynchronousness and JavaScript go back a long way. How we deal with it has evolved over time and there are numerous applied patterns that attempt to make async easier. I would say that there are 3 concrete and popular patterns. However, each one is very related to the other:
Callbacks
Promises
async/await
Callbacks are probably the most backwards compatible and just involve providing a function to some asynchronous task in order to have your provided function be called whenever the task is complete.
For example:
/**
* Some dummy asynchronous task that waits 2 seconds to complete
*/
function asynchronousTask(cb) {
setTimeout(() => {
console.log("Async task is done");
cb();
}, 2000);
}
asynchronousTask(() => {
console.log("My function to be called after async task");
});
Promises are a primitive that encapsulates the callback pattern so that instead of providing a function to the task function, you call the then method on the Promise that the task returns:
/**
* Some dummy asynchronous task that waits 2 seconds to complete
* BUT the difference is that it returns a Promise
*/
function asynchronousTask() {
return new Promise(resolve => {
setTimeout(() => {
console.log("Async task is done");
resolve();
}, 2000);
});
}
asynchronousTask()
.then(() => {
console.log("My function to be called after async task");
});
The last pattern is the async/await pattern which also deals in Promises which are an encapsulation of callbacks. They are unique because they provide lexical support for using Promises so that you don't have to use .then() directly and also don't have to explicitly return a Promise from your task:
/*
* We still need some Promise oriented bootstrap
* function to demonstrate the async/await
* this will just wait a duration and resolve
*/
function $timeout(duration) {
return new Promise(resolve => setTimeout(resolve, duration));
}
/**
* Task runner that waits 2 seconds and then prints a message
*/
(async function() {
await $timeout(2000);
console.log("My function to be called after async task");
}());
Now that our vocabulary is cleared up, we need to consider one other thing: These patterns are all API dependent. The library that you are using uses callbacks. It is alright to mix these patterns, but I would say that the code that you write should be consistent. Pick one of the patterns and wrap or interface with the library that you need to.
If the library deals in callbacks, see if there is a wrapping library or a mechanism to have it deal in Promises instead. async/await consumes Promises, but not callbacks.
Callbacks are fine, but I would only use them if a function is dependent on some asynchronous result. If however the result is immediately available, then the function should be designed to return that value.
In the example you have given, say does not have to wait for any asynchronous API call to come back with a result, so I would change its signature to the following:
say: function(data){ // <--- no callback argument
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
message = ('you say '+'"'+message.trim()+'"');
response.target = data.user;
response.type = 'echo';
response.message = message;
return response; // <--- return it
}
Then going backwards, you would also change the signature of the functions that use say:
executeVerb: function(verb, data){ // <--- no callback argument
return verb_library[verb](data); // <--- no callback argument, and return the returned value
}
And further up the call stack:
socket.on('rg_command', function(command){
// execute the verb
let verb = rogueParser(command);
let result = rogue.executeVerb(verb, command); // <--- no callback, just get the returned value
console.log(result);
});
Of course, this can only work if all verb methods can return the expected result synchronously.
Promises
If say would depend on some asynchronous API, then you could use promises. Let's assume this API provides a callback system, then your say function could return a promise like this:
say: async function(data){ // <--- still no callback argument, but async!
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
response.target = data.user;
response.type = 'echo';
// Convert the API callback system to a promise, and use AWAIT
await respone.message = new Promise(resolve => someAsyncAPIWithCallBackAsLastArg(message, resolve));
return response; // <--- return it
}
Again going backwards, you would also change the signature of the functions that use say:
executeVerb: function(verb, data){ // <--- still no callback argument
return verb_library[verb](data); // <--- no callback argument, and return the returned promise(!)
}
And finally:
socket.on('rg_command', async function(command){ // Add async
// execute the verb
let verb = rogueParser(command);
let result = await rogue.executeVerb(verb, command); // <--- await the fulfillment of the returned promise
console.log(result);
});

How to wait for multiple asynchronous calls from forEach loop?

Trying to make call to multiple asynchronous function, and due to which getting result as undefined.
Tried async.waterfall, however not able to make it work.
Code:
const pendingData = [];
async.waterfall([
function (callback) {
WaitForApproval.find({}, (err,result) => {
callback(null,result);
});
},
function (result, callback) {
result.forEach(data => {
User.findOne({_id: data.uploadedBy}, (err,name) => {
let total = {
id: data._id,
name: name.name,
subject: data.subject,
uploadOn: data.uploadedAt
};
pendingData.push(total);
});
});
callback(null,'done');
}
], function (err,result) {
if(result === 'done') {
console.log(pendingData); // it is giving empty result.
}
});
How to wait for asynchronous function?
The issue you are having is that you are async functions within a non-async forEach loop.
You have a few options here:
Make the mongoDB call recursively - wrap this query in a function that calls itself after the query returns.
Read about mongoDB batch operations - https://docs.mongodb.com/manual/reference/method/Bulk/
Use an async/await pattern with each call by declaring the callback function within the async waterfall as 'async' and then using 'await' inside of the function for each query. Out of the box, forEach is not async. If you want to still use forEach, you can either re-write it async (See below) or use a regular for loop:
async function asyncForEach(array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array)
}
}
** There are additional ways to solve this beyond what is posted here, but here's a few things that will work if implemented correctly.
I would suggest you add the call for callback(null,'done'); immediately pendingData.push(total);
you are right, the async nature is making things hard for you now, but suppose you used Promises and chain them together, that would save you a lot of trouble.
once a time I had a similar problem of asynchronous code running out of order I wanted so I made a little Tweak using a custom promise function(I mead up) and call it order..such Idea can solve your problem if you could apply it to your code properly
https://github.com/lalosh/Ideas/blob/master/promiseOrder.js

JS basic design when I have naturaly sync code but I need to call a few async functions

Could you please help me to understand javascirpt async hell?
I think I am missing something important ☹ The thing is that js examples and most of the answers on the internet are related to just one part of code – a small snippet. But applications are much more complicated.
I am not going write it directly in JS since I am more interested of the design and how to write it PROPERLY.
Imagine these functions in my application:
InsertTestData();
SelectDataFromDB_1(‘USERS’);
SelectDataFromDB_2(‘USER_CARS’,’USERS’);
FillCollections(‘USER’,’USER_CARS’);
DoTheWork();
DeleteData();
I did not provide any description for the functions but I think it is obvious based on names. They need to go in THIS SPECIFIC ORDER. Imagine that I need to run a select into the db to get USERS and then I need run a select to get USER_CARS for these USERS. So it must be really in this order (consider the same for other functions). The thing is that need to call 6 times Node/Mysql which is async but I need results in specific order. So how can I PROPERLY make that happen?
This could work:
/* not valid code I want to present the idea and keep it short */
InsertTestData(
Mysql.query(select, data, function(err,success)
{
SelectDataFromDB_1(‘USERS’); -- in that asyn function I will call the next procedure
}
));
SelectDataFromDB_1 (
Mysql.query(select, data, function(err,success)
{
SelectDataFromDB_2(‘USERS’); -- in that asyn function I will call the next procedure
}
));
SelectDataFromDB_2 (
Mysql.query(select, data, function(err,success)
{
FillCollections (‘USERS’); -- in that asyn function I will call the next procedure
}
));
etc..
I can “easily” chain it but this looks as a mess. I mean really mess.
I can use some scheduler/timmers to schedule and testing if the previous procedure is done)
Both of them are mess.
So, what is the proper way to do this;
Thank you,
AZOR
If you're using a recent version of Node, you can use ES2017's async/await syntax to have synchronous-looking code that's really asynchronous.
First, you need a version of Mysql.query that returns a promise, which is easily done with util.promisify:
const util = require('util');
// ...
const query = util.promisify(Mysql.query);
Then, wrap your code in an async function and use await:
(async () => {
try {
await InsertTestData();
await SelectDataFromDB_1(‘USERS’);
await SelectDataFromDB_2(‘USER_CARS’,’USERS’);
await FillCollections(‘USER’,’USER_CARS’);
await DoTheWork();
await DeleteData();
} catch (e) {
// Handle the fact an error occurred...
}
})();
...where your functions are async functions, e.g.:
async InsertTestData() {
await query("INSERT INTO ...");
}
Note the try/catch in the async wrapper, it's essential to handle errors, because otherwise if an error occurs you'll get an unhandled rejection notice (and future versions of Node may well terminate the process). (Why "unhandled rejections"? Because async functions are syntactic sugar for promises; an async function returns a promise.) You can either do that with the try/catch shown, or alternate by using .catch on the result of calling it:
(async () => {
await InsertTestData();
// ...
})().catch(e => {
// Handle the fact an error occurred
});

Categories

Resources