I'm trying to use Promise.all() with an array of Promises that is being populated inside of a foreach looop right before but it seems like the Promise.all() is not waiting the promises to be all completed before executing its callback.
What's wrong with the following code? (I tried to simplify it before posting, so parts of it might not make complete sense, but that promised and loops are all there).
class test {
constructor(sql) {
Promise.all([this.sync(sql, 0), this.sync(sql, 1)]).then((data) => {
console.log(data);
});
}
sync(sql, id = 0) {
return new Promise((resolve, reject) => {
request.get('http://localhost/test/' + id, {
json: true
}, (req, res) => {
var promises = [];
res.body['items'].forEach(item => {
promises.push(new Promise((resolve, reject) => {
this.existingRecord(sql, item['id']).then(() => {
resolve(false);
}).catch(() => {
this.add(sql, item).then(resolve(id));
})
}))
});
Promise.all(promises).then((data) => resolve(data));
});
});
}
add(sql, data) {
return new Promise((resolve, reject) => {
console.log('Inserting ' + data['id']);
var request = new sql.Request();
var query = `INSERT INTO test (col1, col2) VALUES (${utils.prepareInsertdata(data)})`;
request.query(query, (err, result) => {
if (err) {
console.log('ERROR INSERTING: ' + data['id']);
console.log(err);
}
resolve();
});
});
}
}
First off, you're making it much harder to write good, clean, error-free code when you have a mix of promises and regular callbacks in your control flow. I find that the best way to write asynchronous code using promises is to first take any asynchronous operations that are not promise based and create promise-based wrappers for them and then write my logic and control flow using only promises. This makes a consistent path for flow of control and for error handling and it removes the mess of promisifying things from the actual main logic.
Then, I see several significant issues in your code.
Asynchronous operations in the constructor
It's almost never a good idea to put asynchronous operations in the constructor. This is because the constructor HAS to return the object itself so that leaves no simple way to communicate back to the code that created your object when the asynchronous operations are actually done and if they succeeded of failed. It is not entirely clear to me what you're trying to accomplish with those async operations, but this is likely a bad design pattern. I favor a factory function that returns a promise that resolves to the new object for combining the creation of an object with asynchronous operations. This gives you everything you need, a fully formed object, knowledge of when the async operations are done and an ability to have error handling for the async operations. You can see more about this factory function option and some other design options here:
Asynchronous operations in constructor
Improve .then() handler construction
When you do this:
this.add(sql, item).then(resolve(id));
You are calling resolve(id) immediately and passing that to .then() rather than waiting for the .then() handler to be called before calling resolve(id). All of this is complicated because you're mixing regular callbacks and promises.
Creating new wrapped promises rather than just returning existing promises
This is related to your mix of regular callbacks and regular promises, but you'd much rather just return an existing promise than wrap it in a new promise that you have to manually resolve and reject. More than half the time, you will miss proper error handling when manually wrapping things in a new promise and it just results in more code than is needed.
Race Conditions
In any sort of multi-user database environment, you can't write database code such as:
if (record exists) {
do one thing
} else {
create new record
}
This is a race condition. If some other database request comes in during the processing of this, it could change the database in the middle of this and you'd be trying to create a record that just got created by another piece of code.
The usual solution varies by database (and you don't say exactly which database library you're using). Usually, you want to let the database manage the creation of unique records making it so that a duplicate record (by whatever key you're managing uniqueness in this table by) isn't allowed in the database and the concurrency of that is managed by the database itself. Some databases have an atomic operation such as findOrCreate() that will find an existing record or create a new one in an atomic fashion. Other databases have other approaches. But, it's important to make sure that adding unique records to the database is an atomic operation that can't ever create unwanted duplicates.
I'd suggest this implementation:
// use promise version of request library (already promisified for us)
const rp = require('request-promise');
class test {
constructor() {
}
init(sql) {
return Promise.all([this.sync(sql, 0), this.sync(sql, 1)]).then((data) => {
console.log(data);
// do something with the data here - probably store it in instance data
});
}
sync(sql, id = 0) {
return rp.get('http://localhost/test/' + id, {json: true}).then(res => {
// process all items
return Promise.all(res.body.items.map(item => {
return this.existingRecord(sql, item.id).then(() => {
return false;
}).catch(() => {
// it's probably bad form here to do this for all possible database errors
// probably this should be looking for a specific error of id not found
// or something like that.
// This is also likely a race condition. You would typically avoid the race
// condition by making the item key unique in the database and just doing an add and letting
// the database tell you the add failed because the item already exists
// This will allow the database to control the concurrency and avoid race conditions
return this.add(sql, item);
});
}));
});
}
}
// factory function that returns promise that resolves to a new object
// don't use new test() elsewhere
function createTestObj(sql) {
let t = new test();
return t.init(sql).then(() => {
// resolve to our new object
return t;
});
}
For your add() method, I'd switch to using the promise interface in your sql database. There should either be one built-in or a 3rd party package that will add one on top of your database interface. This will prevent the manual creation of promises and the incomplete error handling in your add() method.
Related
I'm new to Nodejs and having trouble understand this issue: I tried to run a describe function against an array, and the AWS function seems to run after the main function has finished.
Here's the main function: (loop thru a list of ACM ARNs and check the status)
var checkCertStatus = function(resolveObj){
var promise = new Promise(function(resolve, reject){
console.log('1');
var retObj='';
resolveObj.Items.forEach(function(element) {
var certDescribeParams = {
CertificateArn: element.sslCertId
};
console.log('2');
acm.describeCertificate(certDescribeParams, function(err, data) {
if(err) reject(new Error(err));
else {
console.log(data.Certificate.DomainName + ': ' + data.Certificate.Status);
retObj+=data;
}
});
});
console.log('3');
resolve(retObj);
return promise;
})
}
Based on the debug log, assuming there are 2 items need to be processed, what I got:
1
2
2
3
example.com: ISSUED
example2.com: ISSUED
Basically, I need to pass this result to the next function in the chain (with promise and stuff).
Welcome to Node.js! Speaking generally, it might be helpful to study up on the asynchronous programming style. In particular, you seem to be mixing Promises and callbacks, which may make this example more confusing than it needs to be. I suggest using the AWS SDK's built-in feature to convert responses to Promises.
The first thing I notice is that you are manually constructing a Promise with a resolve/reject function. This is often a red flag unless you are creating a library. Most other libraries support Promises which you can simply use and chain. (This includes AWS SDK, as mentioned above.)
The second thing I notice is that your checkCertStatus function does not return anything. It creates a Promise but does not return it at the end. Your return promise; line is actually inside the callback function used to create the Promise.
Personally, when working with Promises, I prefer to use the Bluebird library. It provides more fully-featured Promises than native, including methods such as map. Conveniently, the AWS SDK can be configured to work with an alternative Promise constructor via AWS.config.setPromisesDependency() as documented here.
To simplify your logic, you might try something along these lines (untested code):
const Promise = require('bluebird');
AWS.config.setPromisesDependency(Promise);
const checkCertStatus = (resolveObj) => {
const items = resolveObj.Items;
console.log(`Mapping ${items.length} item(s)`);
return Promise.resolve(items)
.map((item) => {
const certDescribeParams = {
CertificateArn: item.sslCertId,
};
console.log(`Calling describeCertificate for ${item.sslCertId}`);
return acm.describeCertificate(certDescribeParams)
.promise()
.then((data) => {
console.log(`${data.Certificate.DomainName}: ${data.Certificate.Status}`);
return data;
});
});
};
We're defining checkCertStatus as a function which takes in resolveObj and returns a Promise chain starting from resolveObj.Items. (I apologize if you are not yet familiar with Arrow Functions.) The first and only step in this chain is to map the items array to a new array of Promises returned from the acm.describeCertificate method. If any one of these individual Promises fails, the top-level Promise chain will reject as well. Otherwise, the top-level Promise chain will resolve to an array of the results. (Note that I included an inessential .then step just to log the individual results, but you could remove that clause entirely.)
Hope this helps, and I apologize if I left any mistakes in the code.
This is more of a conceptual question. I understand the Promise design pattern, but couldn't find a reliable source to answer my question about promise.all():
What is(are) the correct scenario(s) to use promise.all()
OR
Are there any best practices to use promise.all()? Should it be ideally used only if all of the promise objects are of the same or similar types?
The only one I could think of is:
Use promise.all() if you want to resolve the promise only if all of the promise objects resolve and reject if even one rejects.
I'm not sure anyone has really given the most general purpose explanation for when to use Promise.all() (and when not to use it):
What is(are) the correct scenario(s) to use promise.all()
Promise.all() is useful anytime you have more than one promise and your code wants to know when all the operations that those promises represent have finished successfully. It does not matter what the individual async operations are. If they are async, are represented by promises and your code wants to know when they have all completed successfully, then Promise.all() is built to do exactly that.
For example, suppose you need to gather information from three separate remote API calls and when you have the results from all three API calls, you then need to run some further code using all three results. That situation would be perfect for Promise.all(). You could so something like this:
Promise.all([apiRequest(...), apiRequest(...), apiRequest(...)]).then(function(results) {
// API results in the results array here
// processing can continue using the results of all three API requests
}, function(err) {
// an error occurred, process the error here
});
Promise.all() is probably most commonly used with similar types of requests (as in the above example), but there is no reason that it needs to be. If you had a different case where you needed to make a remote API request, read a local file and read a local temperature probe and then when you had data from all three async operations, you wanted to then do some processing with the data from all three, you would again use Promise.all():
Promise.all([apiRequest(...), fs.promises.readFile(...), readTemperature(...)]).then(function(results) {
// all results in the results array here
// processing can continue using the results of all three async operations
}, function(err) {
// an error occurred, process the error here
});
On the flip side, if you don't need to coordinate among them and can just handle each async operation individually, then you don't need Promise.all(). You can just fire each of your separate async operations with their own .then() handlers and no coordination between them is needed.
In addition Promise.all() has what is called a "fast fail" implementation. It returns a master promise that will reject as soon as the first promise you passed it rejects or it will resolve when all the promises have resolved. So, to use Promise.all() that type of implementation needs to work for your situation. There are other situations where you want to run multiple async operations and you need all the results, even if some of them failed. Promise.all() will not do that for you directly. Instead, you would likely use something like Promise.settle() for that situation. You can see an implementation of .settle() here which gives you access to all the results, even if some failed. This is particularly useful when you expect that some operations might fail and you have a useful task to pursue with the results from whatever operations succeeded or you want to examine the failure reasons for all the operations that failed to make decisions based on that.
Are there any best practices to use promise.all()? Should it be
ideally used only if all of the promise objects are of the same or
similar types?
As explained above, it does not matter what the individual async operations are or if they are the same type. It only matters whether your code needs to coordinate them and know when they all succeed.
It's also useful to list some situations when you would not use Promise.all():
When you only have one async operation. With only one operation, you can just use a .then() handler on the one promise and there is no reason for Promise.all().
When you don't need to coordinate among multiple async operations.
When a fast fail implementation is not appropriate. If you need all results, even if some fail, then Promise.all() will not do that by itself. You will probably want something like Promise.allSettled() instead.
If your async operations do not all return promises, Promise.all() cannot track an async operation that is not managed through a promise.
Promise.all is for waiting for several Promises to resolve in parallel (at the same time). It returns a Promise that resolves when all of the input Promises have resolved:
// p1, p2, p3 are Promises
Promise.all([p1, p2, p3])
.then(([p1Result, p2Result, p3Result]) => {
// This function is called when p1, p2 and p3 have all resolved.
// The arguments are the resolved values.
})
If any of the input Promises is rejected, the Promise returned by Promise.all is also rejected.
A common scenario is waiting for several API requests to finish so you can combine their results:
const contentPromise = requestUser();
const commentsPromise = requestComments();
const combinedContent = Promise.all([contentPromise, commentsPromise])
.then(([content, comments]) => {
// content and comments have both finished loading.
})
You can use Promise.all with Promise instance.
It's hard to answer these questions as they are the type that tend to answer themselves as one uses the available APIs of a language feature. Basically, it's fine to use Promises any way that suits your use case, so long as you avoid their anti-patterns.
What is(are) the correct scenario(s) to use promise.all()
Any situation in which an operation depends on the successful resolution of multiple promises.
Are there any best practices to use promise.all()? Should it be ideally used only if all of the promise objects are of the same or similar types?
Generally, no and no.
I use promise.all() when I have to do some requests to my API and I don't want to display something before the application loads all the data requested, so I delay the execution flow until I have all the data I need.
Example:
What I want to do I want to load the users of my app and their products (imagine that you have to do multiple requests) before displaying a table in my app with the user emails and the product names of each user.
What I do next I send the requests to my API creating the promises and using promise.all()
What I do when all the data has been loaded Once the data arrives to my app, I can execute the callback of promises.all() and then make visible the table with the users.
I hope it helps you to see in which scenario makes sense to use promises.all()
As #joews mentioned, probably one of the important features of Promise.all that should be explicitly indicated is that it makes your async code much faster.
This makes it ideal in any code that contains independent calls (that we want to return/finish before the rest of the code continues), but especially when we make frontend calls and want the user's experience to be as smooth as possible.
async function waitSecond() {
return new Promise((res, rej) => {
setTimeout(res, 1000);
});
}
function runSeries() {
console.time('series');
waitSecond().then(() => {
waitSecond().then(() => {
waitSecond().then(() => {
console.timeEnd('series');
});
});
});
}
function runParallel() {
console.time('parallel');
Promise.all([
waitSecond(),
waitSecond(),
waitSecond(),
]).then(() => {
console.timeEnd('parallel');
});
}
runSeries();
runParallel();
I tend to use promise all for something like this:
myService.getUsers()
.then(users => {
this.users = users;
var profileRequests = users.map(user => {
return myService.getProfile(user.Id); // returns a promise
});
return Promise.all(profileRequests);
})
.then(userProfilesRequest => {
// do something here with all the user profiles, like assign them back to the users.
this.users.forEach((user, index) => {
user.profile = userProfilesRequest[index];
});
});
Here, for each user we're going off and getting their profile. I don't want my promise chain to get out of hand now that i have x amount of promises to resolve.
So Promise.all() will basically aggregate all my promises back into one, and I can manage that through the next then. I can keep doing this for as long as a like, say for each profile I want to get related settings etc. etc. Each time I create tonnes more promises, I can aggregate them all back into one.
Promise.all-This method is useful for when you want to wait for more than one promise to complete or The Promise.all(iterable) method returns a promise that resolves when all of the promises in the iterable argument have resolved, or rejects with the reason of the first passed promise that rejects.
2.Just use Promise.all(files).catch(err => { })
This throws an error if ANY of the promises are rejected.
3.Use .reflect on the promises before .all if you want to wait for all
promises to reject or fulfill
Syntax -Promise.all(iterable);
Promise.all passes an array of values from all the promises in the iterable object that it was passed.
https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
var isCallFailed = false;
function myEndpoint1() {
return isCallFailed ? Promise.reject("Bohoo!") :Promise.resolve({"a":"a"});
}
function myEndpoint2() {
return Promise.resolve({"b":"b"});
}
Promise.all([myEndpoint1(), myEndpoint2()])
.then(values => {
var data1 = values[0];
var data2 = values[1];
alert("SUCCESS... data1: " + JSON.stringify(data1) + "; data2: " + JSON.stringify(data2));
})
.catch(error => {
alert("ERROR... " + error);
});
you can try another case by making isCallFailed = true.
Use Promise.all only when you need to run a code according to the result of more than one asynchronous operations using promises.
For example:
You have a scenario like, You need to download 2000 mb file from server, and at the same time you are going to free the user storage to make sure it can save the downloaded file.
And you need to save only in case if the file is downloaded successfully and the storage space is created successfully.
you will do like this.
your first asynchronous operation
var p1 = new Promise(function(resolve, reject) {
// you need to download 2000mb file and return resolve if
// you successfully downloaded the file
})
and your second asynchronous operation
var p2 = new Promise(function(resolve, reject) {
// you need to clear the user storage for 2000 mb
// which can take some time
})
Now you want to save only when both of the promises resolved successfully, otherwise not.
You will use promise.all like this.
Promise.all([p1,p2]).then((result)=>{
// you will be here only if your both p1 and p2 are resolved successfully.
// you code to save the downloaded file here
})
.catch((error)=>{
// you will be here if at-least one promise in p1,p2 is rejected.
// show error to user
// take some other action
})
Promise.all can be used in a scenario when there is a routine which is validating multiplerules based on particular criteria and you have to execute them all in parallel and need to see the results of those rules at one point. Promise.all returns the results as an array which were resolved in your rule vaidator routine.
E.g.
const results = await Promise.all([validateRule1, validateRule2, validateRule3, ...]);
then results array may look like (depending upon the conditions) as for example: [true, false, false]
Now you can reject/accept the results you have based on return values. Using this way you won't have to apply multiple conditions with if-then-else.
If you are interested only Promise.all then read below Promise.all
Promise (usually they are called "Promise") - provide a convenient way to organize asynchronous code.
Promise - is a special object that contains your state. Initially, pending ( «waiting"), and then - one of: fulfilled ( «was successful") or rejected ( «done with error").
On the promise to hang callbacks can be of two types:
unFulfilled - triggered when the promise in a state of "completed
successfully."
Rejected - triggered when the promise in the "made in error."
The syntax for creating the Promise:
var promise = new Promise(function(resolve, reject) {
// This function will be called automatically
// It is possible to make any asynchronous operations,
// And when they will end - you need to call one of:
// resolve(result) on success
// reject(error) on error
})
Universal method for hanging handlers:
promise.then(onFulfilled, onRejected)
onFulfilled - a function that will be called with the result with
resolve.
onRejected - a function that will be called when an error reject.
With its help you can assign both the handler once, and only one:
// onFulfilled It works on success
promise.then(onFulfilled)
// onRejected It works on error
promise.then(null, onRejected)
Synchronous throw - the same that reject
'use strict';
let p = new Promise((resolve, reject) => {
// то же что reject(new Error("o_O"))
throw new Error("o_O");
});
p.catch(alert); // Error: o_O
Promisification
Promisification - When taking asynchronous functionality and make it a wrapper for returning PROMIS.
After Promisification functional use often becomes much more convenient.
As an example, make a wrapper for using XMLHttpRequest requests
httpGet function (url) will return PROMIS, which upon successful data loading with the url will go into fulfilled with these data, and in case of error - in rejected with an error information:
function httpGet(url) {
return new Promise(function(resolve, reject) {
var xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.onload = function() {
if (this.status == 200) {
resolve(this.response);
} else {
var error = new Error(this.statusText);
error.code = this.status;
reject(error);
}
};
xhr.onerror = function() {
reject(new Error("Network Error"));
};
xhr.send();
});
}
As you can see, inside the function XMLHttpRequest object is created and sent as usual, when onload / onerror are called, respectively, resolve (at the status 200) or reject.
Using:
httpGet("/article/promise/user.json")
.then(
response => alert(`Fulfilled: ${response}`),
error => alert(`Rejected: ${error}`)
);
Parallel execution
What if we want to implement multiple asynchronous processes simultaneously and to process their results?
The Promise class has the following static methods.
Promise.all(iterable)
Call Promise.all (iterable) receives an array (or other iterable object) and returns PROMIS PROMIS, which waits until all transferred PROMIS completed, and changes to the state "done" with an array of results.
For example:
Promise.all([
httpGet('/article/promise/user.json'),
httpGet('/article/promise/guest.json')
]).then(results => {
alert(results);
});
Let's say we have an array of URL.
let urls = [
'/article/promise/user.json',
'/article/promise/guest.json'
];
To download them in parallel, you need to:
Create for each URL corresponding to PROMIS.
Wrap an array of PROMIS in Promise.all.
We obtain this:
'use strict';
let urls = [
'/article/promise/user.json',
'/article/promise/guest.json'
];
Promise.all( urls.map(httpGet) )
.then(results => {
alert(results);
});
Note that if any of Promise ended with an error, the result will
Promise.all this error.
At the same time the rest of PROMIS ignored.
For example:
Promise.all([
httpGet('/article/promise/user.json'),
httpGet('/article/promise/guest.json'),
httpGet('/article/promise/no-such-page.json') // (нет такой страницы)
]).then(
result => alert("не сработает"),
error => alert("Ошибка: " + error.message) // Ошибка: Not Found
)
In total:
Promise - is a special object that stores its state, the current
result (if any), and callbacks.
When you create a new Promise ((resolve, reject) => ...) function
argument starts automatically, which should call resolve (result) on
success, and reject (error) - error.
Argument resolve / reject (only the first, and the rest are ignored)
is passed to handlers on this Promise.
Handlers are appointed by calling .then / catch.
To transfer the results from one processor to another using Channing.
https://www.promisejs.org/patterns/
I have a function which creates a database object out of three arrays. The arrays are filled in an each loop, one of the arrays relies on the value in the same iteration of the loop.
The dependent array uses the requests library and the cheerio library to grab a string to populate the array with.
Currently the dependent array fills with nulls which I think is because the loop is not waiting for the request to be returned.
I am still learning and would like to get this to work without direct blocking to keep things asynchronous so I'm looking into promises/callbacks.
This is being done server-side but from what I've seen in cheerios docs there is no promises capability.
Here's what I have so far. (getFile() is the function that isn't filling the 'c' array, it also depends on the current value being put into 'b'). I do know that the getFile function gets the correct value with a console log test, so the issue must be in the implementation of filling 'c'.
addToDB() is a function which saves a value into mongoDB, from testing I know that the objects are correctly being put into the db, just the c array is not correct.
function getInfo(path) {
$(path).each(function(i,e) {
a.push(...)
b.push(value)
c.push(getFile(value))
})
var entry = new DB...//(a,b,c)
addToDB(entry);
}
function getFile(...) {
request(fullUrl, function (err, resp, page) {
if (!err && resp.statusCode == 200) {
var $ = cheerio.load(page); // load the page
srcEp = $(this).attr("src");
return srcEp;
} // end error and status code
}); // end request
}
I've been reading about promises/callbacks and then() but I've yet to find anything which works.
First, you have to get your mind around the fact that any process that relies, at least in part, on an asynchronous sub-process, is itself inherently asynchronous.
At the lowest level of this question's code, request() is asynchronous, therefore its caller, getFile() is asynchronous, and its caller, getInfo() is also asynchronous.
Promises are an abstraction of the outcome of asynchronous processes and help enormously in coding the actions to be taken when such processes complete - successfully or under failure.
Typically, low-level asynchronous functions should return a promise to be acted on by their callers, which will, in turn, return a promise to their callers, and so on up the call stack. Inside each function, returned promise(s) may be acted on using promise methods, chiefly .then(), and may be aggregated using Promise.all() for example (syntax varies).
In this question, there is no evidence that request() currently returns a promise. You have three options :
discover that request() does, in fact, return a promise.
rewrite request() to return a promise.
write an adapter function (a "promisifier") that calls request(), and generates/returns the promise, which is later fulfilled or rejected depending on the outcome of request().
The first or second options would be ideal but the safe assumption for me (Roamer) is to assume that an adapter is required. Fortunately, I know enough from the question to be able to write one. Cheerio appears not to include jQuery's promise implementation, so a dedicated promise lib will be required.
Here is an adapter function, using syntax that will work with the Bluebird lib or native js promises:
//Promisifier for the low level function request()
function requestAsync(url) {
return new Promise(function(resolve, reject) {
request(url, function(err, resp, page) {
if (err) {
reject(err);
} else {
if (resp.statusCode !== 200) {
reject(new Error('request error: ' + resp.statusCode));
}
} else {
resolve(page);
}
});
});
}
Now getFile(...) and getInfo() can be written to make use of the promises returned from the lowest level's adapter.
//intermediate level function
function getFile(DOMelement) {
var fullUrl = ...;//something derived from DOMelement. Presumably either .val() or .text()
return requestAsync(fullUrl).then(function (page) {
var $ = cheerio.load(page);
srcEp = $(???).attr('src');//Not too sure what the selector should be. `$(this)` definitely won't work.
return srcEp;
});
}
//high level function
function getInfo(path) {
var a = [], b = [], c = [];//presumably
// Now map the $(path) to an array of promises by calling getFile() inside a .map() callback.
// By chaining .then() a/b/c are populated when the async data arrives.
var promises = $(path).map(function(i, e) {
return getFile(e).then(function(srcEp) {
a[i] = ...;
b[i] = e;
c[i] = srcEp;
});
});
//return an aggregated promise to getInfo's caller,
//in case it needs to take any action on settlement.
return Promise.all(promises).then(function() {
//What to do when all promises are fulfilled
var entry = new DB...//(a,b,c)
addToDB(entry);
}, function(error) {
//What to do if any of the promises fails
console.log(error);
//... you may want to do more.
});
}
I'm working on a script that pings websites and returns the results in a web UI. However, I've run into a problem which I am trying to figure out the best solution for.
This block of code needs to return a array of statuses but due to the asynchronous behaviour of Node.js, it returns an empty array because the code takes time to execute.
Here is what I have:
var ping = require('ping');
function checkConnection(hosts) {
var results = [];
hosts.forEach(function (host) {
ping.sys.probe(host, function (isAlive) {
results.push({"host": host, "status": isAlive});
});
});
return {results: results, timestamp: new Date().getTime()};
}
module.exports.checkConnection = checkConnection;
I know that you could solve this problem with the use of timers but what would be the simples and most ideal solution here?
How to get around the asynchronous Node.js behaviour?
Don't. Instead, embrace it, by having your checkConection accept a callback or return a promise.
Callback example:
function checkConnection(hosts, callback) {
var results = [];
hosts = hosts.slice(0); // Copy
hosts.forEach(function (host) {
ping.sys.probe(host, function (isAlive) {
results.push({"host": host, "status": isAlive});
if (results.length === hosts.length) {
callback({results: results, timestamp: new Date().getTime()});
}
});
});
}
Note the defensive shallow copy of hosts. If you don't do that, then since this code runs asynchronously, the calling code could add to or remove from the hosts array while you were processing responses, and the lengths would never match.
An alternate way to handle that without copying is to simply count how many requests you've initiated:
function checkConnection(hosts, callback) {
var results = [];
var requests = hosts.length;
hosts.forEach(function (host) {
ping.sys.probe(host, function (isAlive) {
results.push({"host": host, "status": isAlive});
if (results.length === requests) {
callback({results: results, timestamp: new Date().getTime()});
}
});
});
}
That looks like it sets up a race condition (what if something modifies hosts after you set requests but before you're done initiating your probe queries?) but it doesn't, because Node runs your JavaScript on a single thread, so no other code can reach in and modify hosts between the requests = hosts.length and hosts.forEach lines.
Like T.J. said, you will need to embrace asynchronous behavior if you are going to program in node.js as that is a fundamental tenet of how it works and how you code a responsive, scalable server using node.js.
T.J.'s answer is a straightforward way of solving this particular problem. But, since async issues will arise over and over again in node.js, promises can be a very useful tool for managing asynchronous behavior and they quickly become indispensable for more complicated multi-operation sequences with robust error handling.
So, here's a solution to your coding issue using Promises:
var ping = require('ping');
var Promise = require('bluebird');
// make a version of ping.sys.probe that returns a promise when done
ping.sys.probeAsync = function(host) {
return new Promise(function(resolve, reject) {
ping.sys.probe(host, function(isAlive) {
resolve({"host": host, "status": isAlive});
});
}
}
function checkConnection(hosts) {
var promises = hosts.map(function(host) {
return ping.sys.probeAsync(host);
});
return Promise.all(promises).then(function(results) {
return {results: results, timestamp: new Date().getTime()};
});
}
module.exports.checkConnection = checkConnection;
Sample Usage:
myModule.checkConnection(myArrayOfHosts).then(function(results) {
// results is the {results: results, timestamp: time} object
});
Step-by-step, here's how this works:
Load the Bluebird promise library.
Create a promisified version of ping.sys.probe called ping.sys.probeAsync that returns a promise that will be resolved when the underlying call is done.
Using .map() on your array, create an array of promises from calling ping.sys.probeAsync on each item in the array.
Using Promise.all(), create a new promise that is the aggregation of all the promises in the array. It will call it's .then() handler only when all the promises in the array have been resolved (e.g. have finished).
Add a .then() handler to Promise.all() so the timestamp can be added to the results.
Return the Promise.all() promise so the caller of checkConnection() gets a promise back they can use.
When calling checkConnection() use a .then() handler to know when all the operations are done and to obtain the results.
Hopefully you can see that once you have a promisified version of your function and you understand how promises work, you can then write the actual async code much simpler. And, if you also had error handling or had a sequence of async operations that had to be run one after the other (something you don't have here), the advantages of using promises is even greater.
P.S. I think Bluebird's Promise.map() can be used to combine the hosts.map() and Promise.all() into a single call, but I've not used that function myself so I didn't offer it here.
In nodejs, a lot of function calls are implemented as non-blocking function. Sometimes there could be dependencies between these function calls. For example, when user login, I want first get the user type of the user and then according to the user type I can get the tasks which are just assigned this user type. In the non-blocking function, I use nested function call:
$http.post(url_login, login_request)
.success(function(req) {
$http.post(url_get_tasks, req.type)
.success(function(req) {
//update tasks
})
})
A lot of libraries provides non-blocking functions only, for example, the node_redis library. Is there any good way to handle the nested non-blocking function calls to solve the dependency problem.
There is a page dedicated to this exact problem: http://callbackhell.com/
To sum up, you have the option
to use named functions, as codebox suggests
use the async library
or use promises
If I am not mistaken,
you are using the Angular $http service, which should support proper
promise chaining, so you could reduce your code to
$http.post(url_login, login_request)
.then(function(req){
return $http.pos(url_get_tasks, req.type);
})
.then(function(req){
// update tasks
})
.catch(function(err){
// errors from the first AND second call land here
})
You can write a simple wrapper for async functions to turn them into promises:
// this assumes standard node.js callbacks with the
// signature (error, data)
function toPromise(fn){
return function() {
var args = Array.prototype.slice.call(arguments);
var scope = this;
return new Promise(function(resolve, reject){
var cb = function(err, data) {
return err ? reject(err) : resolve(data);
};
args.push(cb);
fn.apply(scope, args);
});
};
}
// turn fs.readdir into a promise
var readdir = toPromise(require('fs').readdir);
readdir('.')
.then(function(files){
console.log(files);
})
.catch(function(err){
console.log('reading files failed');
})
There was another question regarding Angular $http promises which might help you understand them better: here
I'm not 100% sure what you are asking, but if you want to reduce nesting in your code you can move the inner functions out like this:
function success1(req) {
//update tasks
}
function success2(req) {
$http.post(url_get_tasks, req.type).success(success1)
}
$http.post(url_login, login_request).success(success2)
obviously you should pick better names than success1 though