I have a function which creates a database object out of three arrays. The arrays are filled in an each loop, one of the arrays relies on the value in the same iteration of the loop.
The dependent array uses the requests library and the cheerio library to grab a string to populate the array with.
Currently the dependent array fills with nulls which I think is because the loop is not waiting for the request to be returned.
I am still learning and would like to get this to work without direct blocking to keep things asynchronous so I'm looking into promises/callbacks.
This is being done server-side but from what I've seen in cheerios docs there is no promises capability.
Here's what I have so far. (getFile() is the function that isn't filling the 'c' array, it also depends on the current value being put into 'b'). I do know that the getFile function gets the correct value with a console log test, so the issue must be in the implementation of filling 'c'.
addToDB() is a function which saves a value into mongoDB, from testing I know that the objects are correctly being put into the db, just the c array is not correct.
function getInfo(path) {
$(path).each(function(i,e) {
a.push(...)
b.push(value)
c.push(getFile(value))
})
var entry = new DB...//(a,b,c)
addToDB(entry);
}
function getFile(...) {
request(fullUrl, function (err, resp, page) {
if (!err && resp.statusCode == 200) {
var $ = cheerio.load(page); // load the page
srcEp = $(this).attr("src");
return srcEp;
} // end error and status code
}); // end request
}
I've been reading about promises/callbacks and then() but I've yet to find anything which works.
First, you have to get your mind around the fact that any process that relies, at least in part, on an asynchronous sub-process, is itself inherently asynchronous.
At the lowest level of this question's code, request() is asynchronous, therefore its caller, getFile() is asynchronous, and its caller, getInfo() is also asynchronous.
Promises are an abstraction of the outcome of asynchronous processes and help enormously in coding the actions to be taken when such processes complete - successfully or under failure.
Typically, low-level asynchronous functions should return a promise to be acted on by their callers, which will, in turn, return a promise to their callers, and so on up the call stack. Inside each function, returned promise(s) may be acted on using promise methods, chiefly .then(), and may be aggregated using Promise.all() for example (syntax varies).
In this question, there is no evidence that request() currently returns a promise. You have three options :
discover that request() does, in fact, return a promise.
rewrite request() to return a promise.
write an adapter function (a "promisifier") that calls request(), and generates/returns the promise, which is later fulfilled or rejected depending on the outcome of request().
The first or second options would be ideal but the safe assumption for me (Roamer) is to assume that an adapter is required. Fortunately, I know enough from the question to be able to write one. Cheerio appears not to include jQuery's promise implementation, so a dedicated promise lib will be required.
Here is an adapter function, using syntax that will work with the Bluebird lib or native js promises:
//Promisifier for the low level function request()
function requestAsync(url) {
return new Promise(function(resolve, reject) {
request(url, function(err, resp, page) {
if (err) {
reject(err);
} else {
if (resp.statusCode !== 200) {
reject(new Error('request error: ' + resp.statusCode));
}
} else {
resolve(page);
}
});
});
}
Now getFile(...) and getInfo() can be written to make use of the promises returned from the lowest level's adapter.
//intermediate level function
function getFile(DOMelement) {
var fullUrl = ...;//something derived from DOMelement. Presumably either .val() or .text()
return requestAsync(fullUrl).then(function (page) {
var $ = cheerio.load(page);
srcEp = $(???).attr('src');//Not too sure what the selector should be. `$(this)` definitely won't work.
return srcEp;
});
}
//high level function
function getInfo(path) {
var a = [], b = [], c = [];//presumably
// Now map the $(path) to an array of promises by calling getFile() inside a .map() callback.
// By chaining .then() a/b/c are populated when the async data arrives.
var promises = $(path).map(function(i, e) {
return getFile(e).then(function(srcEp) {
a[i] = ...;
b[i] = e;
c[i] = srcEp;
});
});
//return an aggregated promise to getInfo's caller,
//in case it needs to take any action on settlement.
return Promise.all(promises).then(function() {
//What to do when all promises are fulfilled
var entry = new DB...//(a,b,c)
addToDB(entry);
}, function(error) {
//What to do if any of the promises fails
console.log(error);
//... you may want to do more.
});
}
Related
I'm trying to use Promise.all() with an array of Promises that is being populated inside of a foreach looop right before but it seems like the Promise.all() is not waiting the promises to be all completed before executing its callback.
What's wrong with the following code? (I tried to simplify it before posting, so parts of it might not make complete sense, but that promised and loops are all there).
class test {
constructor(sql) {
Promise.all([this.sync(sql, 0), this.sync(sql, 1)]).then((data) => {
console.log(data);
});
}
sync(sql, id = 0) {
return new Promise((resolve, reject) => {
request.get('http://localhost/test/' + id, {
json: true
}, (req, res) => {
var promises = [];
res.body['items'].forEach(item => {
promises.push(new Promise((resolve, reject) => {
this.existingRecord(sql, item['id']).then(() => {
resolve(false);
}).catch(() => {
this.add(sql, item).then(resolve(id));
})
}))
});
Promise.all(promises).then((data) => resolve(data));
});
});
}
add(sql, data) {
return new Promise((resolve, reject) => {
console.log('Inserting ' + data['id']);
var request = new sql.Request();
var query = `INSERT INTO test (col1, col2) VALUES (${utils.prepareInsertdata(data)})`;
request.query(query, (err, result) => {
if (err) {
console.log('ERROR INSERTING: ' + data['id']);
console.log(err);
}
resolve();
});
});
}
}
First off, you're making it much harder to write good, clean, error-free code when you have a mix of promises and regular callbacks in your control flow. I find that the best way to write asynchronous code using promises is to first take any asynchronous operations that are not promise based and create promise-based wrappers for them and then write my logic and control flow using only promises. This makes a consistent path for flow of control and for error handling and it removes the mess of promisifying things from the actual main logic.
Then, I see several significant issues in your code.
Asynchronous operations in the constructor
It's almost never a good idea to put asynchronous operations in the constructor. This is because the constructor HAS to return the object itself so that leaves no simple way to communicate back to the code that created your object when the asynchronous operations are actually done and if they succeeded of failed. It is not entirely clear to me what you're trying to accomplish with those async operations, but this is likely a bad design pattern. I favor a factory function that returns a promise that resolves to the new object for combining the creation of an object with asynchronous operations. This gives you everything you need, a fully formed object, knowledge of when the async operations are done and an ability to have error handling for the async operations. You can see more about this factory function option and some other design options here:
Asynchronous operations in constructor
Improve .then() handler construction
When you do this:
this.add(sql, item).then(resolve(id));
You are calling resolve(id) immediately and passing that to .then() rather than waiting for the .then() handler to be called before calling resolve(id). All of this is complicated because you're mixing regular callbacks and promises.
Creating new wrapped promises rather than just returning existing promises
This is related to your mix of regular callbacks and regular promises, but you'd much rather just return an existing promise than wrap it in a new promise that you have to manually resolve and reject. More than half the time, you will miss proper error handling when manually wrapping things in a new promise and it just results in more code than is needed.
Race Conditions
In any sort of multi-user database environment, you can't write database code such as:
if (record exists) {
do one thing
} else {
create new record
}
This is a race condition. If some other database request comes in during the processing of this, it could change the database in the middle of this and you'd be trying to create a record that just got created by another piece of code.
The usual solution varies by database (and you don't say exactly which database library you're using). Usually, you want to let the database manage the creation of unique records making it so that a duplicate record (by whatever key you're managing uniqueness in this table by) isn't allowed in the database and the concurrency of that is managed by the database itself. Some databases have an atomic operation such as findOrCreate() that will find an existing record or create a new one in an atomic fashion. Other databases have other approaches. But, it's important to make sure that adding unique records to the database is an atomic operation that can't ever create unwanted duplicates.
I'd suggest this implementation:
// use promise version of request library (already promisified for us)
const rp = require('request-promise');
class test {
constructor() {
}
init(sql) {
return Promise.all([this.sync(sql, 0), this.sync(sql, 1)]).then((data) => {
console.log(data);
// do something with the data here - probably store it in instance data
});
}
sync(sql, id = 0) {
return rp.get('http://localhost/test/' + id, {json: true}).then(res => {
// process all items
return Promise.all(res.body.items.map(item => {
return this.existingRecord(sql, item.id).then(() => {
return false;
}).catch(() => {
// it's probably bad form here to do this for all possible database errors
// probably this should be looking for a specific error of id not found
// or something like that.
// This is also likely a race condition. You would typically avoid the race
// condition by making the item key unique in the database and just doing an add and letting
// the database tell you the add failed because the item already exists
// This will allow the database to control the concurrency and avoid race conditions
return this.add(sql, item);
});
}));
});
}
}
// factory function that returns promise that resolves to a new object
// don't use new test() elsewhere
function createTestObj(sql) {
let t = new test();
return t.init(sql).then(() => {
// resolve to our new object
return t;
});
}
For your add() method, I'd switch to using the promise interface in your sql database. There should either be one built-in or a 3rd party package that will add one on top of your database interface. This will prevent the manual creation of promises and the incomplete error handling in your add() method.
I've got a complicated (at least for me) set up of nested loops, ajax calls, and deferreds. The code is calling an API, parsing out relevant data, then using it to make further calls to other APIs.
It works almost as intended. I used the answer to this question (Using $.Deferred() with nested ajax calls in a loop) to build it. Here's my code:
function a() {
var def = $.Deferred();
var req = [];
for (var i = 0 /*...*/) {
for (var j = 0 /*...*/) {
(function(i, j) {
req.push($.ajax({
//params
}).done(function(resp) {
var def2 = $.Deferred();
var req2 = [];
for (var k = 0 /*...*/) {
for (var l = 0 /*...*/) {
req2.push(b(l));
}
}
$.when.apply($, req2).done(function() {
console.log("Got all data pieces");
def2.resolve();
})
}));
})(i, j);
}
}
$.when.apply($, req).done(function() {
console.log("Got all data");
def.resolve();
});
return def.promise();
}
function b(j) {
var def = $.Deferred();
$.when.apply(
$.ajax({
//params
})
).then(function() {
console.log("Got data piece #" + l);
def.resolve();
});
return def.promise();
}
function main() {
//...
$.when.apply($, a()).then(function() {
console.log("All done");
displayPage();
})
//...
}
Here's what I'm expecting to see when the calls complete
(In no specific order)
Got data piece #1
Got data piece #0
Got data piece #2
Got all data pieces
Got data piece #2
Got data piece #1
Got data piece #0
Got all data pieces
Got data piece #0
Got data piece #1
Got data piece #2
Got all data pieces
Got all data <-- These two must be last, and in this order
All done
Here's what I'm seeing
All done
Got data piece #0
Got data piece #1
Got data piece #2
Got all data pieces
Got data piece #0
Got data piece #1
Got data piece #2
Got all data pieces
Got data piece #0
Got data piece #1
Got data piece #2
Got all data pieces
I stepped through it in the debugger, and the 'Got all data' line in function a() gets printed in the correct sequence after everything else completes, after which def.resolve() should get called and resolve the returned promise.
However, in main(), a() is seen as resolved right away and the code jumps right into printing 'All done' and displaying the page. Any ideas as to why it doesn't wait as it's supposed to?
You have illustrated a set of code and said it isn't doing what you expected, but you haven't really described the overall problem. So, I don't actually know exactly what code to recommend. We do a lot better here with real problems rather than pseudo code problems. So, instead, what I can do is to outline a bunch of things that are wrong with your code:
Expecting serial order of parallel async operations
Based on what you say you are expecting, the basic logic for how you control your async operations seems to be missing. When you use $.when() on a series of promises that have already been started, you are running a whole bunch of async operations in parallel. Their completion order is completely unpredictable.
Yes, you seem to expect to be able to run a whole bunch of b(i) in parallel and have them all complete in order. That seems to be the case because you say you are expecting this type of output:
Got data piece #0
Got data piece #1
Got data piece #2
where each of those statements is generated by the completion of some b(i) operation.
That simply will not happen (or it would be blind luck if it did in the real world because there is no code that guarantees the order). Now, you can run them in parallel and use $.when() to track them and $.when() will let you know when they are all done and will collect all the results in order. But when each individual async operation in that group finishes is up to chance.
So, if you really wanted each of your b(i) operations to run and complete in order, then you would have to purposely sequence them (run one, wait for it to complete, then run the next, etc...). In general, if one operation does not depend upon the other, it is better to run them in parallel and let $.when() track them all and order the results for you (because you usually get your end result faster by running them all in parallel rather than sequencing them).
Creation of unnecessary deferreds in lots of places - promse anti-pattern
In this code, there is no need to create a deferred at all. $.ajax() already returns a promise. You can just use that promise. So, instead of this:
function b(j) {
var def = $.Deferred();
$.when.apply(
$.ajax({
//params
})
).then(function() {
console.log("Got data piece #" + l);
def.resolve();
});
return def.promise();
}
You can do this:
function b(j) {
return $.ajax({
//params
}).then(function(data) {
console.log("Got data piece #" + l);
return data;
});
}
Note, that you just directly return the promise that is already produced by $.ajax() and no deferred needs to be created at all. This is also a lot more bulletproof for error handling. One of the reason your method is called an anti-pattern is you don't handle errors at all (a common mistake when using this anti-pattern). But, the improved code, propagates errors right back to the caller just like they should be. In your version, if the $.ajax() call rejects its promise (due to an error), your deferred is NEVER resolved and the caller never sees the error either. Now, you could write extra code to handle the error, but there is no reason to. Just return the promise you already have. When coding with async operations that return promises, you should pretty much never need to create your own deferred.
$.when() is only needed when you have more than one promise
In your b() function, there is no need to use $.when() in this piece of code:
$.when(
$.ajax({
//params
})).then(...);
When you have a single promise, you just use .then() directly on it.
$.ajax({
//params
}).then(...);
Only use $.when() when you have more than one promise and you want to know when all of them are done. If you only have one promise, just use its own .then() handler.
More anti-pattern - just return promises from .then() handlers
In your inner loop, you have this:
$.when.apply($, req2).done(function() {
console.log("Got all data pieces");
def2.resolve();
})
There are several things wrong here. It's not clear what you're trying to do because def2 is a deferred that nothing else uses. So, it appears you're trying to tell someone when this req2 group of promises is done, but nobody is using it. In addition it's another version of the anti-pattern. $.when() already returns a promise. You don't need to create a deferred to resolve when $.when() completes. You can just use the promise that $.when() already returns.
Though I don't fully know your intent here, it appears that what you should probably do is to get rid of the def2 deferred entirely and do just this:
return $.when.apply($, req2).done(function() {
console.log("Got all data pieces");
});
Returning this promise from the .then() handler that it is within will chain this sequence of actions to the parent promise and make the parent promise wait for this new promise to be resolved (which is tied to when all the req2 promises are done) before the parent promise will resolve. This is how you make parent promises dependent upon other promise within a .then() handler. You return a promise from the .then() handler.
And, the exact same issue is true for your outer $.when.apply($, req) also. You don't need a deferred there at all. Just use the promise that $.when() already returns.
Putting it together
Here's a cleaned up version of your code that gets rid of the anti-patterns in multiple places. This does not change the sequencing of the b(i) calls among themselves. If you care about that, it is a bigger change and we need to see more of the real/actual problem to know what best to recommend.
function a() {
var req = [];
for (var i = 0 /*...*/) {
for (var j = 0 /*...*/) {
(function(i, j) {
req.push($.ajax({
//params
}).then(function(resp) {
var req2 = [];
for (var k = 0 /*...*/) {
for (var l = 0 /*...*/) {
req2.push(b(l));
}
}
return $.when.apply($, req2).done(function() {
console.log("Got all data pieces");
});
}));
})(i, j);
}
}
return $.when.apply($, req).done(function() {
console.log("Got all data");
});
}
function b(j) {
return $.ajax({
//params
}).then(function(data) {
console.log("Got data piece #" + l);
return data;
});
}
function main() {
//...
a().then(function() {
console.log("All done");
displayPage();
});
//...
}
P.S. If you want to process the b(i) results from within the same group in order, then don't use a .then() handler on the individual promise because those will execute in arbitrary order. Instead, use the results that come with $.when().then(result1, result2, ...) and process them all there. Though the individual promises complete in an arbitrary order, $.when() will collect the results into the original order so if you process the results in the $.when() handler, then you can process them all in order.
What is the best way to create parallel asynchronous HTTP requests and take the first result that comes back positive? I am familiar with the async library for JavaScript and would happy to use that but am not sure if it has exactly what I want.
Background - I have a Redis store that serves as state for a server. There is an API we can call to get some data that takes much longer than reaching the Redis store.
In most cases the data will already be in the Redis store, but in some cases it won't be there yet and we need to retrieve it from the API.
The simple thing to do would be to query Redis, and if the value is not in Redis then go to the API afterwards. However, we'll needlessly lose 20-50ms if the data is not yet in our Redis cache and we have to go to the API after failing to find the data with Redis. Since this particular API server is not under great load, it won't really hurt to go to the API simultaneously/in parallel, even if we don't absolutely need the returned value.
//pseudocode below
async.minimum([
function apiRequest(cb){
request(opts,function(err,response,body){
cb(err,body.result.hit);
}
},
function redisRequest(cb){
client.get("some_key", function(err, reply) {
cb(err,reply.result.hit);
});
}],
function minimumCompleted(err,result){
// this mimimumCompleted final callback function will be only fired once,
// and would be fired by one of the above functions -
// whichever one *first* returned a defined value for result.hit
});
is there a way to get what I am looking for with the async library or perhaps promises, or should I implement something myself?
Use Promise.any([ap, bp]).
The following is a possible way to do it without promises. It is untested but should meet the requirements.
To meet requirement of returning the first success and not just the first completion, I keep a count of the number of completions expected so that if an error occurs it can be ignored it unless it is the last error.
function asyncMinimum(a, cb) {
var triggered = false;
var completions = a.length;
function callback(err, data) {
completions--;
if (err && completions !== 0) return;
if (triggered) return;
triggered = true;
return cb(err, data);
}
a.map(function (f) { return f(callback); });
}
asyncMinimum([
function apiRequest(cb){
request(opts,function(err,response,body){
cb(err,body.result.hit);
}
},
function redisRequest(cb){
client.get("some_key", function(err, reply) {
cb(err,reply.result.hit);
});
}],
function minimumCompleted(err,result){
// this mimimumCompleted final callback function will be only fired once,
// and would be fired by one of the above functions -
// whichever one had a value for body.result.hit that was defined
});
The async.js library (and even promises) keep track of the number of asynchronous operations pending by using a counter. You can see a simple implementation of the idea in an answer to this related question: Coordinating parallel execution in node.js
We can use the same concept to implement the minimum function you want. Only, instead of waiting for the counter to count all responses before triggering a final callback, we deliberately trigger the final callback on the first response and ignore all other responses:
// IMHO, "first" is a better name than "minimum":
function first (async_functions, callback) {
var called_back = false;
var cb = function () {
if (!called_back) {
called_back = true; // block all other responses
callback.apply(null,arguments)
}
}
for (var i=0;i<async_functions.length;i++) {
async_functions[i](cb);
}
}
Using it would be as simple as:
first([apiRequest,redisRequest],function(err,result){
// ...
});
Here's an approach using promises. It takes a little extra custom code because of the non-standard result you're looking for. You aren't just looking for the first one to not return an error, but you're looking for the first one that has a specific type of result so that takes a custom result checker function. And, if none get a result, then we need to communicate that back to the caller by rejecting the promise too. Here's the code:
function firstHit() {
return new Promise(function(resolve, reject) {
var missCntr = 0, missQty = 2;
function checkResult(err, val) {
if (err || !val) {
// see if all requests failed
++missCntr;
if (missCntr === missQty) {
reject();
}
} else {
resolve(val);
}
}
request(opts,function(err, response, body){
checkResult(err, body.result.hit);
}
client.get("some_key", function(err, reply) {
checkResult(err, reply.result.hit);
});
});
}
firstHit().then(function(hit) {
// one of them succeeded here
}, function() {
// neither succeeded here
});
The first promise to call resolve() will trigger the .then() handler. If both fail to get a hit, then it will reject the promise.
I'm working on a script that pings websites and returns the results in a web UI. However, I've run into a problem which I am trying to figure out the best solution for.
This block of code needs to return a array of statuses but due to the asynchronous behaviour of Node.js, it returns an empty array because the code takes time to execute.
Here is what I have:
var ping = require('ping');
function checkConnection(hosts) {
var results = [];
hosts.forEach(function (host) {
ping.sys.probe(host, function (isAlive) {
results.push({"host": host, "status": isAlive});
});
});
return {results: results, timestamp: new Date().getTime()};
}
module.exports.checkConnection = checkConnection;
I know that you could solve this problem with the use of timers but what would be the simples and most ideal solution here?
How to get around the asynchronous Node.js behaviour?
Don't. Instead, embrace it, by having your checkConection accept a callback or return a promise.
Callback example:
function checkConnection(hosts, callback) {
var results = [];
hosts = hosts.slice(0); // Copy
hosts.forEach(function (host) {
ping.sys.probe(host, function (isAlive) {
results.push({"host": host, "status": isAlive});
if (results.length === hosts.length) {
callback({results: results, timestamp: new Date().getTime()});
}
});
});
}
Note the defensive shallow copy of hosts. If you don't do that, then since this code runs asynchronously, the calling code could add to or remove from the hosts array while you were processing responses, and the lengths would never match.
An alternate way to handle that without copying is to simply count how many requests you've initiated:
function checkConnection(hosts, callback) {
var results = [];
var requests = hosts.length;
hosts.forEach(function (host) {
ping.sys.probe(host, function (isAlive) {
results.push({"host": host, "status": isAlive});
if (results.length === requests) {
callback({results: results, timestamp: new Date().getTime()});
}
});
});
}
That looks like it sets up a race condition (what if something modifies hosts after you set requests but before you're done initiating your probe queries?) but it doesn't, because Node runs your JavaScript on a single thread, so no other code can reach in and modify hosts between the requests = hosts.length and hosts.forEach lines.
Like T.J. said, you will need to embrace asynchronous behavior if you are going to program in node.js as that is a fundamental tenet of how it works and how you code a responsive, scalable server using node.js.
T.J.'s answer is a straightforward way of solving this particular problem. But, since async issues will arise over and over again in node.js, promises can be a very useful tool for managing asynchronous behavior and they quickly become indispensable for more complicated multi-operation sequences with robust error handling.
So, here's a solution to your coding issue using Promises:
var ping = require('ping');
var Promise = require('bluebird');
// make a version of ping.sys.probe that returns a promise when done
ping.sys.probeAsync = function(host) {
return new Promise(function(resolve, reject) {
ping.sys.probe(host, function(isAlive) {
resolve({"host": host, "status": isAlive});
});
}
}
function checkConnection(hosts) {
var promises = hosts.map(function(host) {
return ping.sys.probeAsync(host);
});
return Promise.all(promises).then(function(results) {
return {results: results, timestamp: new Date().getTime()};
});
}
module.exports.checkConnection = checkConnection;
Sample Usage:
myModule.checkConnection(myArrayOfHosts).then(function(results) {
// results is the {results: results, timestamp: time} object
});
Step-by-step, here's how this works:
Load the Bluebird promise library.
Create a promisified version of ping.sys.probe called ping.sys.probeAsync that returns a promise that will be resolved when the underlying call is done.
Using .map() on your array, create an array of promises from calling ping.sys.probeAsync on each item in the array.
Using Promise.all(), create a new promise that is the aggregation of all the promises in the array. It will call it's .then() handler only when all the promises in the array have been resolved (e.g. have finished).
Add a .then() handler to Promise.all() so the timestamp can be added to the results.
Return the Promise.all() promise so the caller of checkConnection() gets a promise back they can use.
When calling checkConnection() use a .then() handler to know when all the operations are done and to obtain the results.
Hopefully you can see that once you have a promisified version of your function and you understand how promises work, you can then write the actual async code much simpler. And, if you also had error handling or had a sequence of async operations that had to be run one after the other (something you don't have here), the advantages of using promises is even greater.
P.S. I think Bluebird's Promise.map() can be used to combine the hosts.map() and Promise.all() into a single call, but I've not used that function myself so I didn't offer it here.
I'm using jQuery to make various ajax POST requests. I need to keep track of the success or failure of each one of them, along with the overall progress of the complete batch, so that I can update the UI with a progress bar and info about how many requests have succeeded, and how many have failed, out of the total.
Before attempting to implement the feature in my app, I've been playing with some code in jsfiddle as a proof of concept, with no luck so far. This is what I've got:
// an alternative to console.log to see the log in the web page
var fnLog = function(message) {
$('#console').append($("<p>" + message + "</p>"));
};
// keeping track of how many ajax calls have been finished (successfully or not)
var count = 0;
// a dummy ajax call that succeeds by default
var fn = function(shouldFail) {
return $.get(shouldFail ? '/echo/fail/' : '/echo/json/')
.done(function() { fnLog("done") })
.fail(function() { fnLog("FAIL") });
};
// a set of different asynchronous ajax calls
var calls = [fn(),fn(),fn(),fn(true),fn(),fn()];
// an attempt to make a collective promise out of all the calls above
$.when.apply($, calls)
.done(function() { fnLog("all done") })
.fail(function() { fnLog("ALL FAIL") })
.always(function() { fnLog("always") })
.progress(function(arg) { fnLog("progress" + arg) })
.then(function() { fnLog("finished") });
It's all in this fiddle: http://jsfiddle.net/mmtbo7v6/1/
What I need is the ability to provide a callback that ought to be called after all promises are resolved (either successfully or not).
When all calls above are set to succeed (by removing the true argument to the fourth fn call in the array) it works fine. The output prints the following:
done
done
done
done
done
done
all done
always
finished
But when even a single call is set to fail (as it is by default in the jsfiddle), the output is the following:
done
FAIL
ALL FAIL
always
done
done
done
done
So none of the collective promise callbacks (the one generated by the $.when call) is called after all promises are resolved. The final .then is not called at all if a single ajax call fails.
Additionally, I would appreciate some insight on how to keep track of the progress of this batch of ajax calls, to update a progress bar in the UI.
Well... I'm going to be unfair. jQuery actually comes bundled with progression events but I myself hate them because I don't think they compose or aggregate well - so I'll show a simpler alternative approach for that progress bar that I believe is superior instead.
First thing's first:
The 'all promises resolved but some possibly rejected' issue is called a 'settle' typically. I've provided an answer to a similar question here with just giving the results and here providing an implementation that gives you access to all results even rejected ones.
function settle(promises){
var d = $.Deferred();
var counter = 0;
var results = Array(promises.length);
promises.forEach(function(p,i){
p.then(function(v){ // add as fulfilled
results[i] = {state:"fulfilled", promise : p, value: v};
}).catch(function(r){ // add as rejected
results[i] = {state:"rejected", promise : p, reason: r};
}).always(function(){ // when any promises resolved or failed
counter++; // notify the counter
if (counter === promises.length) {
d.resolve(results); // resolve the deferred.
}
});
});
return d.promise();
}
You'd use settle in place of $.when to get your desired results.
As for progression - I personally recommend passing a progression callback to the method itself. The pattern goes something like this:
function settle(promises, progress){
progress = progress || function(){}; // in case omitted
var d = $.Deferred();
var counter = 0;
var results = Array(promises.length);
promises.forEach(function(p,i){
p.then(function(v){ // add as fulfilled
results[i] = {state:"fulfilled", promise : p, value: v};
}).catch(function(r){ // add as rejected
results[i] = {state:"rejected", promise : p, reason: r};
}).always(function(){ // when any promises resolved or failed
counter++; // notify the counter
progress((promises.length - counter) / promises.length);
if (counter === promises.length) {
d.resolve(results); // resolve the deferred.
}
});
});
return d.promise();
}
Which would let you do something like:
settle([url1, url2, ... url100].map($.get), function(soFar){
$("#myProgressBar").css("width", (soFar * 100)+"%");
}).then(function(results){
console.log("All settled", results);
]);
It turns out there's a much better alternative to this problem, one which shadows the promises approach. Behold the combination of two patterns: Observables + Iterables = Reactive programming.
Reactive Programming is programming with asynchronous data streams, that is, treating asynchronous data streams as collections that can be traversed and transformed as traditional collection data types. This article is a great introduction.
I won't convert this answer post into a tutorial, so let's go straight to the solution, which is shown below. I'm gonna be using the RxJS library, but there are other libraries for Reactive Programming in JS (bacon.js seems to be really popular too).
function settle(promises) {
return Rx.Observable.from(promises).concatMap(function(promise, index) {
return Rx.Observable.fromPromise(promise).
map(function(response) {
return { count: index+1, total: promises.length, state: "fulfilled", promise: promise, value: response };
}).
catch(function(reason) {
return Rx.Observable.of({ count: index+1, total: promises.length, state: "rejected", promise: promise, reason: reason });
});
});
}
The function itself returns an observable, which is a stream of events. Namely, the events of each promise finished, either successfully or not. We can use that returned observable to listen to this stream (or to subscribe to it, if we're to adhere to RxJS terminology).
var results = settle(promises);
results.subscribeOnNext(function(results) {
// process each result as it arrives
// progress info can be extracted from results.count and results.total
});
results.subscribeOnCompleted(function() {
// completion callback
});
And that's it. Much cleaner code, a more functional-programming approach. No need to keep state, everything expressed in a more declarative way. Just what we want to be done, not how it should be done.