I'm using the Q node library for Promises, question I think can apply to the Bluebird lib as well.
Context
I have a few function calls to make to both my own custom functions and node.js fs style async functions.
if I'm making a call to a function like this:
sync function
do_something = function (input) {
// assign variables, do other sync stuff
}
and need the above to take place before this function:
sync function
do_something_else = function (input) {
// assign variable, do other sync stuff
}
and then need to call a native node function similar to:
async function
writeData = function (obj, callback) {
var deferred = Q.defer();
fs.writeFile(obj.file, obj.datas, function (err, result) {
if (err) deferred.reject(err);
else deferred.resolve('write data completed');
});
return deferred.promise.nodeify(callback);
}
and finally need the above to take place before this function:
sync function
do_something_last = function (input) {
// assign variable, do other sync stuff
}
Question
Is the 'right' thing to do here, to make all my functions 'deferred' or promise aware so I can make sure that they are called in sequence or in the correct order?
like so:
do_something(variable)
.then(do_something_else)
.then(writeData)
.then(do_something_last)
.done();
or should I just do this instead and keep the ordering (sequencing)? Like so:
var variable1 = 'test1.txt'
var variable2 = 'test2.txt'
var return_value = do_something(variable1);
var return_another_value = do_something_else(return_value); <--sync
writeData(return_another_value); <-- async function
var final_value = do_something_last(variable2); <-- sync function
// could potentially have async calls again after a sync call
writeDataAgain(return_another_value); <-- async function
Clarifications
What I thought was since some of these sync functions are going to need to be fired after the async, I needed to make them Promise aware in order to keep the sequence straight, like so:
sync functions made promise aware
do_something = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve(_output_result_);
return deferred.promise;
}
do_something_else = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve(_output_result_);
return deferred.promise;
}
do_something_last = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve('completed workflow');
return deferred.promise;
}
this would allow me to do this:
do_something(variable)
.then(do_something_else) <-- these need to execute before writeData
.then(writeData) <-- a async node fs call to writeFile
.then(do_something_last) <-- I need this to happen after the writeDate
.done();
After the feedback i've read, i guess what it seems like i'm really asking is:
How do I create a function workflow, mixing non-promise sync and
promise-aware async function calls, all the while keeping the ordering
(or sequencing) of execution?
just do this instead and keep the ordering like so:
writeData(return_another_value);
var final_value = do_something_last(variable2);
Well, that simply won't work, as do_something_last is not called after the writeData(…) promise is resolved. It'll just start right after the promise is created and returned. So if you care about that particular order and want to wait until the data is written, then you need to use then with a callback:
var final_promise = writeData(return_another_value).then(function(writeResult) {
return do_something_last(variable2);
});
The general rules are:
make synchronous functions synchronous - no need for promises
make all asynchronous functions always return a promise
use deferreds only at the lowest possible level for promisification
You can just place synchronous functions in a then chain, non-promise return values (or even thrown exceptions) work fine in them.
So while you can write your sequence like
Q('test1.txt')
.then(do_something)
.then(do_something_else)
.then(writeData)
.then(do_something_last.bind(null, 'test2.txt'))
.done();
it looks rather odd. If you don't plan to make the do_somethings asynchronous in the near future for some reason, it's often simpler to write and read
writeData(do_something_else(do_something('test1.txt'))).then(function() {
return do_something_last('test2.txt');
}).done();
Admittedly, it's sometimes more appealing to write
somePromise()
.then(doSomethingSynchronous)
.then(doSomethingAsynchronous)
than
somePromise
.then(function(res) { return doSomethingAsynchronous(doSomethingSynchronous(res)); })
even though they are functionally identical. Choose the style that you like better and that is more consistent.
If all you care about is whether your functions go in sequence or not, then do this:
do_something(variable)
.then(do_something_else)
.then(writeData)
.then(do_something_last)
.done();
You'd only assign promises to variables when you're going to be passing those variables around (e.g. to other services), or using them to create different promise chains.
e.g
var promise = do_something('123')
// two different promise chains
var task1 = promise.then(function(res){
// logic
})
var task2 = promise.then(function(res){
// other logic, independent from task1
})
Related
I have a function which creates a database object out of three arrays. The arrays are filled in an each loop, one of the arrays relies on the value in the same iteration of the loop.
The dependent array uses the requests library and the cheerio library to grab a string to populate the array with.
Currently the dependent array fills with nulls which I think is because the loop is not waiting for the request to be returned.
I am still learning and would like to get this to work without direct blocking to keep things asynchronous so I'm looking into promises/callbacks.
This is being done server-side but from what I've seen in cheerios docs there is no promises capability.
Here's what I have so far. (getFile() is the function that isn't filling the 'c' array, it also depends on the current value being put into 'b'). I do know that the getFile function gets the correct value with a console log test, so the issue must be in the implementation of filling 'c'.
addToDB() is a function which saves a value into mongoDB, from testing I know that the objects are correctly being put into the db, just the c array is not correct.
function getInfo(path) {
$(path).each(function(i,e) {
a.push(...)
b.push(value)
c.push(getFile(value))
})
var entry = new DB...//(a,b,c)
addToDB(entry);
}
function getFile(...) {
request(fullUrl, function (err, resp, page) {
if (!err && resp.statusCode == 200) {
var $ = cheerio.load(page); // load the page
srcEp = $(this).attr("src");
return srcEp;
} // end error and status code
}); // end request
}
I've been reading about promises/callbacks and then() but I've yet to find anything which works.
First, you have to get your mind around the fact that any process that relies, at least in part, on an asynchronous sub-process, is itself inherently asynchronous.
At the lowest level of this question's code, request() is asynchronous, therefore its caller, getFile() is asynchronous, and its caller, getInfo() is also asynchronous.
Promises are an abstraction of the outcome of asynchronous processes and help enormously in coding the actions to be taken when such processes complete - successfully or under failure.
Typically, low-level asynchronous functions should return a promise to be acted on by their callers, which will, in turn, return a promise to their callers, and so on up the call stack. Inside each function, returned promise(s) may be acted on using promise methods, chiefly .then(), and may be aggregated using Promise.all() for example (syntax varies).
In this question, there is no evidence that request() currently returns a promise. You have three options :
discover that request() does, in fact, return a promise.
rewrite request() to return a promise.
write an adapter function (a "promisifier") that calls request(), and generates/returns the promise, which is later fulfilled or rejected depending on the outcome of request().
The first or second options would be ideal but the safe assumption for me (Roamer) is to assume that an adapter is required. Fortunately, I know enough from the question to be able to write one. Cheerio appears not to include jQuery's promise implementation, so a dedicated promise lib will be required.
Here is an adapter function, using syntax that will work with the Bluebird lib or native js promises:
//Promisifier for the low level function request()
function requestAsync(url) {
return new Promise(function(resolve, reject) {
request(url, function(err, resp, page) {
if (err) {
reject(err);
} else {
if (resp.statusCode !== 200) {
reject(new Error('request error: ' + resp.statusCode));
}
} else {
resolve(page);
}
});
});
}
Now getFile(...) and getInfo() can be written to make use of the promises returned from the lowest level's adapter.
//intermediate level function
function getFile(DOMelement) {
var fullUrl = ...;//something derived from DOMelement. Presumably either .val() or .text()
return requestAsync(fullUrl).then(function (page) {
var $ = cheerio.load(page);
srcEp = $(???).attr('src');//Not too sure what the selector should be. `$(this)` definitely won't work.
return srcEp;
});
}
//high level function
function getInfo(path) {
var a = [], b = [], c = [];//presumably
// Now map the $(path) to an array of promises by calling getFile() inside a .map() callback.
// By chaining .then() a/b/c are populated when the async data arrives.
var promises = $(path).map(function(i, e) {
return getFile(e).then(function(srcEp) {
a[i] = ...;
b[i] = e;
c[i] = srcEp;
});
});
//return an aggregated promise to getInfo's caller,
//in case it needs to take any action on settlement.
return Promise.all(promises).then(function() {
//What to do when all promises are fulfilled
var entry = new DB...//(a,b,c)
addToDB(entry);
}, function(error) {
//What to do if any of the promises fails
console.log(error);
//... you may want to do more.
});
}
I'm defining a class which instantiates several modules which depend on previous ones. The modules themselves may require an async operation before they are ready (i.e. establishing a mysql connection) so I've provided each constructor with a callback to be called once the module is ready. However I've run into a problem when instantiating classes which are ready immediately:
var async = require('async');
var child = function(parent, cb) {
var self = this;
this.ready = false;
this.isReady = function() {
return self.ready;
}
/* This does not work, throws error below stating c1.isReady is undefined*/
cb(null, true);
/* This works */
setTimeout(function() {
self.ready = true;
cb(null, true);
}, 0);
}
var Parent = function(cb) {
var self = this;
async.series([
function(callback){
self.c1 = new child(self, callback);
},
function(callback){
self.c2 = new child(self, callback);
}
],
function(err, results){
console.log(self.c1.isReady(), self.c2.isReady);
console.log(err, results);
});
}
var P = new Parent();
I'm guessing the issue is calling cb within the constructor means async proceeds to the next function before the constructor finishes. Is there a better approach to this? I considered using promises, but I find this approach easier to understand/follow.
You will have to delay the call to the callback when everything is synchronous because any code in the callback won't be able to reference the object yet (as you've seen). Because the constructor hasn't finished executing, the assignment of it's return value also hasn't finished yet.
You could pass the object to the callback and force the callback to use that reference (which will exist and be fully formed), but you're requiring people to know that they can't use a normally accepted practice so it's much better to make sure the callback is only called asynchronously and people can then write their code in the normal ways they would expect to.
In node.js, it will be more efficient to use process.nextTick() instead of setTimeout() to accomplish your goal. See this article for more details.
var child = function(parent, cb) {
var self = this;
this.ready = false;
this.isReady = function() {
return self.ready;
}
/* This works */
process.nextTick(function() {
self.ready = true;
cb(null, true);
}, 0);
}
Here's a general observation. Many people (myself included) think that it overcomplicates things to put any async operation in a constructor for reasons related to this. Instead, most objects that need to do async operations in order to get themselves set up will offer a .init() or .connect() method or something like that. Then, you construct the object like you normally would in a synchronous fashion and then separately initiate the async part of the initialization and pass it the callback. That gets you away from this issue entirely.
If/when you want to use promises for tracking your async operation (which is great feature direction to go), it's way, way easier to let the constructor return the object and the .init() operation to return a promise. It gets messy to try to return both the object and a promise from the constructor and even messier to code with that.
Then, using promises you can do this:
var o = new child(p);
o.init().then(function() {
// object o is fully initialized now
// put code in here to use the object
}, function(err) {
// error initializing object o
});
If you can't put the object into a Promise, put a promise into the object.
Give it a Ready property that is a promise.
If you have a class hierarchy (I'm talking about Typescript now) of (say) widgets, the base class can define the Ready property and assign it an already resolved Promise object. Doing this is a base class means it's always safe to write code like this
var foo = Foo();
foo.Ready.then(() => {
// do stuff that needs foo to be ready
});
Derived classes can take control of promise resolution by replacing the value of Ready with a new promise object, and resolving it when the async code completes.
Here's a comprehensive workup of Asynchronous Constructor design pattern
I have several async functions with varying numbers of parameters, in each the last param is a callback. I wish to call these in order. For instance.
function getData(url, callback){
}
function parseData(data, callback){
}
By using this:
Function.prototype.then = function(f){
var ff = this;
return function(){ ff.apply(null, [].slice.call(arguments).concat(f)) }
}
it is possible to call these functions like this, and have the output print to console.log.
getData.then(parseData.then(console.log.bind(console)))('/mydata.json');
I've been trying to use this syntax instead, and cannot get the Then function correct. Any ideas?
getData.then(parseData).then(console.log.bind(console))('/mydata.json');
Implementing a function or library that allows you to chain methods like above is a non-trivial task and requires substantial effort. The main problem with the example above is the constant context changing - it is very difficult to manage the state of the call chain without memory leaks (i.e. saving a reference to all chained functions into a module-level variable -> GC will never free the functions from memory).
If you are interested in this kind of programming strategy I highly encourage you to use an existing, established and well-tested library, like Promise or q. I personally recommend the former as it attempts to behave as close as possible to ECMAScript 6's Promise specification.
For educational purposes, I recommend you take a look at how the Promise library works internally - I am quite sure you will learn a lot by inspecting its source code and playing around with it.
Robert Rossmann is right. But I'm willing to answer purely for academic purposes.
Let's simplify your code to:
Function.prototype.then = function (callback){
var inner = this;
return function (arg) { return inner(arg, callback); }
}
and:
function getData(url, callback) {
...
}
Let's analyze the types of each function:
getData is (string, function(argument, ...)) → null.
function(argument, function).then is (function(argument, ...)) → function(argument).
That's the core of the problem. When you do:
getData.then(function (argument) {}) it actually returns a function with the type function(argument). That's why .then can't be called onto it, because .then expects to be called onto a function(argument, function) type.
What you want to do, is wrap the callback function. (In the case of getData.then(parseData).then(f), you want to wrap parseData with f, not the result of getData.then(parseData).
Here's my solution:
Function.prototype.setCallback = function (c) { this.callback = c; }
Function.prototype.getCallback = function () { return this.callback; }
Function.prototype.then = function (f) {
var ff = this;
var outer = function () {
var callback = outer.getCallback();
return ff.apply(null, [].slice.call(arguments).concat(callback));
};
if (this.getCallback() === undefined) {
outer.setCallback(f);
} else {
outer.setCallback(ff.getCallback().then(f));
}
return outer;
}
This looks like an excellent use for the Promise object. Promises improve reusability of callback functions by providing a common interface to asynchronous computation. Instead of having each function accept a callback parameter, Promises allow you to encapsulate the asynchronous part of your function in a Promise object. Then you can use the Promise methods (Promise.all, Promise.prototype.then) to chain your asynchronous operations together. Here's how your example translates:
// Instead of accepting both a url and a callback, you accept just a url. Rather than
// thinking about a Promise as a function that returns data, you can think of it as
// data that hasn't loaded or doesn't exist yet (i.e., promised data).
function getData(url) {
return new Promise(function (resolve, reject) {
// Use resolve as the callback parameter.
});
}
function parseData(data) {
// Does parseData really need to be asynchronous? If not leave out the
// Promise and write this function synchronously.
return new Promise(function (resolve, reject) {
});
}
getData("someurl").then(parseData).then(function (data) {
console.log(data);
});
// or with a synchronous parseData
getData("someurl").then(function (data) {
console.log(parseData(data));
});
Also, I should note that Promises currently don't have excellent browser support. Luckily you're covered since there are plenty of polyfills such as this one that provide much of the same functionality as native Promises.
Edit:
Alternatively, instead of changing the Function.prototype, how about implementing a chain method that takes as input a list of asynchronous functions and a seed value and pipes that seed value through each async function:
function chainAsync(seed, functions, callback) {
if (functions.length === 0) callback(seed);
functions[0](seed, function (value) {
chainAsync(value, functions.slice(1), callback);
});
}
chainAsync("someurl", [getData, parseData], function (data) {
console.log(data);
});
Edit Again:
The solutions presented above are far from robust, if you want a more extensive solution check out something like https://github.com/caolan/async.
I had some thoughts about that problem and created the following code which kinda meets your requirements. Still - I know that this concept is far away from perfect. The reasons are commented in the code and below.
Function.prototype._thenify = {
queue:[],
then:function(nextOne){
// Push the item to the queue
this._thenify.queue.push(nextOne);
return this;
},
handOver:function(){
// hand over the data to the next function, calling it in the same context (so we dont loose the queue)
this._thenify.queue.shift().apply(this, arguments);
return this;
}
}
Function.prototype.then = function(){ return this._thenify.then.apply(this, arguments) };
Function.prototype.handOver = function(){ return this._thenify.handOver.apply(this, arguments) };
function getData(json){
// simulate asyncronous call
setTimeout(function(){ getData.handOver(json, 'params from getData'); }, 10);
// we cant call this.handOver() because a new context is created for every function-call
// That means you have to do it like this or bind the context of from getData to the function itself
// which means every time the function is called you have the same context
}
function parseData(){
// simulate asyncronous call
setTimeout(function(){ parseData.handOver('params from parseData'); }, 10);
// Here we can use this.handOver cause parseData is called in the context of getData
// for clarity-reasons I let it like that
}
getData
.then(function(){ console.log(arguments); this.handOver(); }) // see how we can use this here
.then(parseData)
.then(console.log)('/mydata.json'); // Here we actually starting the chain with the call of the function
// To call the chain in the getData-context (so you can always do this.handOver()) do it like that:
// getData
// .then(function(){ console.log(arguments); this.handOver(); })
// .then(parseData)
// .then(console.log).bind(getData)('/mydata.json');
Problems and Facts:
the complete chain is executed in the context of the first function
you have to use the function itself to call handOver at least with the first Element of the chain
if you create a new chain using the function you already used, it will conflict when it runs to the same time
it is possible to use a function twice in the chain (e.g. getData)
because of the shared conext you can set a property in one function and read it in one of the following functions
At least for the first Problem you could solve it with not calling the next function in the chain in the same context and instead give the queue as parameter to the next function. I will try this approach later. This maybe would solve the conflicts mentioned at point 3, too.
For the other problem you could use the sample Code in the comments
PS: When you run the snipped make sure your console is open to see the output
PPS: Every comment on this approach is welcome!
The problem is that then returns a wrapper for the current function and successive chained calls will wrap it again, instead of wrapping the previous callback. One way to achieve that is to use closures and overwrite then on each call:
Function.prototype.then = function(f){
var ff = this;
function wrapCallback(previousCallback, callback) {
var wrapper = function(){
previousCallback.apply(null, [].slice.call(arguments).concat(callback));
};
ff.then = wrapper.then = function(f) {
callback = wrapCallback(callback, f); //a new chained call, so wrap the callback
return ff;
}
return wrapper;
}
return ff = wrapCallback(this, f); //"replace" the original function with the wrapper and return that
}
/*
* Example
*/
function getData(json, callback){
setTimeout( function() { callback(json) }, 100);
}
function parseData(data, callback){
callback(data, 'Hello');
}
function doSomething(data, text, callback) {
callback(text);
}
function printData(data) {
console.log(data); //should print 'Hello'
}
getData
.then(parseData)
.then(doSomething)
.then(printData)('/mydata.json');
In nodejs, a lot of function calls are implemented as non-blocking function. Sometimes there could be dependencies between these function calls. For example, when user login, I want first get the user type of the user and then according to the user type I can get the tasks which are just assigned this user type. In the non-blocking function, I use nested function call:
$http.post(url_login, login_request)
.success(function(req) {
$http.post(url_get_tasks, req.type)
.success(function(req) {
//update tasks
})
})
A lot of libraries provides non-blocking functions only, for example, the node_redis library. Is there any good way to handle the nested non-blocking function calls to solve the dependency problem.
There is a page dedicated to this exact problem: http://callbackhell.com/
To sum up, you have the option
to use named functions, as codebox suggests
use the async library
or use promises
If I am not mistaken,
you are using the Angular $http service, which should support proper
promise chaining, so you could reduce your code to
$http.post(url_login, login_request)
.then(function(req){
return $http.pos(url_get_tasks, req.type);
})
.then(function(req){
// update tasks
})
.catch(function(err){
// errors from the first AND second call land here
})
You can write a simple wrapper for async functions to turn them into promises:
// this assumes standard node.js callbacks with the
// signature (error, data)
function toPromise(fn){
return function() {
var args = Array.prototype.slice.call(arguments);
var scope = this;
return new Promise(function(resolve, reject){
var cb = function(err, data) {
return err ? reject(err) : resolve(data);
};
args.push(cb);
fn.apply(scope, args);
});
};
}
// turn fs.readdir into a promise
var readdir = toPromise(require('fs').readdir);
readdir('.')
.then(function(files){
console.log(files);
})
.catch(function(err){
console.log('reading files failed');
})
There was another question regarding Angular $http promises which might help you understand them better: here
I'm not 100% sure what you are asking, but if you want to reduce nesting in your code you can move the inner functions out like this:
function success1(req) {
//update tasks
}
function success2(req) {
$http.post(url_get_tasks, req.type).success(success1)
}
$http.post(url_login, login_request).success(success2)
obviously you should pick better names than success1 though
I'm still trying to get my head around all this and I'm obviously missing some fundamental concepts.
In my code I have a scenario whereby I want to call several functions in a row and when they've all completed, wrap it up with a finishing routine where subtotals are calculated and a pie chart is updated.
Although the functions I call themselves aren't asynchronous, they do contain ajax calls which are, hence why I want to wait for all of them to finish before I calculate totals and update the chart.
So I tried doing:
var fnArray= [];
fnArray.push(genericCalc("use_", true, false, false));
fnArray.push(doArticleImpacts(false));
fnArray.push(genericProjectCalc("tpD2C_", false, false));
fnArray.push(genericCalc("eol_", true, false, false));
fnArray.push(calcPackaging(false));
var calcPromise = Q.all(fnArray);
return calcPromise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone() {
calcTotals();
setChart(selectedRow());
}
function calcsFailed() {
logger.logError("Failure in calculations", "", null, true);
}
...but using the above code and using the script debugger with a stop on the "return calcPromise" line, the fnArray is set to "0:undefined, 1:undefined, 2:Object, 3:undefined, 4:Promise" even before the promise is activated.
I understand that this is obviously something to do with my functions, but I don't really understand what I need to do differently. The functions all vary slightly, but are fundamentally something like:
var genericCalc = function (calcPrefix) {
var res_Array = ko.observable(); //holds returned results
prjArticleArray.forEach(function (thisArticle) {
var calcPromise = calcEOL(res_Array, thisArticle); //another function containing async ajax call
return calcPromise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone() {
//do calculation subtotals here and set a knockout observable value
}
function calcsFailed() {
logger.logError("Failure in " + calcPrefix + "calculation", "", null, true);
}
});
};
What is it that makes some of the functions "undefined", some "object" and some "promise" in my array that I want to use for Q.all? Do I have to have "Q.resolve" in the "calcsDone" part of the functions I'm calling?
I've seen other questions/answers on stackoverflow along similar lines, but they seem to always be calls directly to async calls and my fist level functions I'm stacking up in the promise aren't... should I not be using this structure for non-async calls or just add "setTimeout" to my function calls to make them async?
I understand that this is obviously something to do with my functions, but I don't really understand what I need to do differently
You need to return promises from them. Your current genericCalc function does not even contain a return statement (only the forEach callback function inside it does), so it does return undefined.
If you need to wait for the result of each article, use Q.all like you have already done with fnArray (which is a promiseArray actually). In genericCalc it should look like this:
var genericCalc = function (calcPrefix) {
return Q.all(prjArticleArray.map(function (thisArticle) {
var calcPromise = calcEOL(res_Array, thisArticle); //another function containing async ajax call
var result = calcPromise
.then(calcsDone)
.fail(calcsFailed);
return result;
…
});
};
var res_Array = ko.observable(); //holds returned results
…
//do calculation subtotals here and set a knockout observable value
That is a bad idea. You should not use global variables that are set somewhen and use promises just for propagating changes, but the promises should represent these values. This leads to a better, functional programming style.
So instead of setting a global variable, do return the result from your calcsDone function, and it will resolve the result promise with that value.