I'm defining a class which instantiates several modules which depend on previous ones. The modules themselves may require an async operation before they are ready (i.e. establishing a mysql connection) so I've provided each constructor with a callback to be called once the module is ready. However I've run into a problem when instantiating classes which are ready immediately:
var async = require('async');
var child = function(parent, cb) {
var self = this;
this.ready = false;
this.isReady = function() {
return self.ready;
}
/* This does not work, throws error below stating c1.isReady is undefined*/
cb(null, true);
/* This works */
setTimeout(function() {
self.ready = true;
cb(null, true);
}, 0);
}
var Parent = function(cb) {
var self = this;
async.series([
function(callback){
self.c1 = new child(self, callback);
},
function(callback){
self.c2 = new child(self, callback);
}
],
function(err, results){
console.log(self.c1.isReady(), self.c2.isReady);
console.log(err, results);
});
}
var P = new Parent();
I'm guessing the issue is calling cb within the constructor means async proceeds to the next function before the constructor finishes. Is there a better approach to this? I considered using promises, but I find this approach easier to understand/follow.
You will have to delay the call to the callback when everything is synchronous because any code in the callback won't be able to reference the object yet (as you've seen). Because the constructor hasn't finished executing, the assignment of it's return value also hasn't finished yet.
You could pass the object to the callback and force the callback to use that reference (which will exist and be fully formed), but you're requiring people to know that they can't use a normally accepted practice so it's much better to make sure the callback is only called asynchronously and people can then write their code in the normal ways they would expect to.
In node.js, it will be more efficient to use process.nextTick() instead of setTimeout() to accomplish your goal. See this article for more details.
var child = function(parent, cb) {
var self = this;
this.ready = false;
this.isReady = function() {
return self.ready;
}
/* This works */
process.nextTick(function() {
self.ready = true;
cb(null, true);
}, 0);
}
Here's a general observation. Many people (myself included) think that it overcomplicates things to put any async operation in a constructor for reasons related to this. Instead, most objects that need to do async operations in order to get themselves set up will offer a .init() or .connect() method or something like that. Then, you construct the object like you normally would in a synchronous fashion and then separately initiate the async part of the initialization and pass it the callback. That gets you away from this issue entirely.
If/when you want to use promises for tracking your async operation (which is great feature direction to go), it's way, way easier to let the constructor return the object and the .init() operation to return a promise. It gets messy to try to return both the object and a promise from the constructor and even messier to code with that.
Then, using promises you can do this:
var o = new child(p);
o.init().then(function() {
// object o is fully initialized now
// put code in here to use the object
}, function(err) {
// error initializing object o
});
If you can't put the object into a Promise, put a promise into the object.
Give it a Ready property that is a promise.
If you have a class hierarchy (I'm talking about Typescript now) of (say) widgets, the base class can define the Ready property and assign it an already resolved Promise object. Doing this is a base class means it's always safe to write code like this
var foo = Foo();
foo.Ready.then(() => {
// do stuff that needs foo to be ready
});
Derived classes can take control of promise resolution by replacing the value of Ready with a new promise object, and resolving it when the async code completes.
Here's a comprehensive workup of Asynchronous Constructor design pattern
Related
Normally if you aren't using promises you can easily do
var a = function(cb){
setTimeout(function(){
var boundCb = cb.bind({hello: 'world'});
boundCb();
}, 100);
};
a(function(){
alert(this.hello);
});
which isn't particularly useful most of the times (as you can pass the same things as normal arguments), but it does allow for some extremely readable code and/or simple to use interfaces.
When switching to promises however it suddenly becomes a lot harder to do. Unsurprisingly the following doesn't work (as resolve isn't a direct reference)
var a = function(){
return new Promise(function(resolve, reject){
var boundResolve = resolve.bind({hello: 'word'});
boundResolve();
});
};
a().then(function(){
alert(this.hello)
});
so is there any way at all to achieve this?
Non essential postscriptum: 'Luckily' in my situation the this scope I want to set is already the same as what the this scope is at a(). So I am currently simply writing .bind(this) after every single Promise callback function and in the future I will be able to use arrow functions for this, but I am looking for a cleaner solution in the interim.
Native Promises do not offer this functionality. However, the Bluebird Promise library offers Promise.prototype.bind
Here's an example usage
MyClass.prototype.method = function() {
return fs.readFileAsync(this.file).bind(this)
.then(function(contents) {
var url = urlParse(contents);
return this.httpGetAsync(url); // this is bound to the MyClass instance
}).then(function(result) {
// this is still bound, further down the chain.
var refined = this.refine(result);
return this.writeRefinedAsync(refined);
}).catch(function(e) {
this.error(e.stack); // Even in event handlers
});
};
I'm looking to create a class which retrieves information from a database and sets user variables appropriately.
My issue is that, the database API I am using is asynchronous and this becomes an issue because the class becomes accessible before it has been 'fully constructed' (that is to say the constructor function has returned).
Desired synchronous approach
class User{
constructor(sql){
this.name = Database.selectQuery(sql);
}
}
var x = new User(statement);
console.log(x.name); // returns name
Current asynchronous approach
class User{
constructor(sql){
Database.selectQuery(sql, this, function(data, obj){
obj.name = data;
});
}
}
var x = new User(statement);
console.log(x.name); // returns undefined
I'm aware I can add a callback to my constructor in order to make it work asynchronously but is this a good idea?
You will quickly find that all async operations must be able to communicate to the caller when they are complete because they complete some indeterminate time in the future. This means they either need to accept a callback or they need to return a promise.
While you can pass a callback to a constructor, you can't return a promise from a constructor (it has to return the object) so it is generally not recommended to put an async operation in the constructor. Instead, the code flow by the caller is more straightforward if you construct the object and THEN call an async method on the object.
As your example clearly shows, you have created no way for the caller of your User object to know when the async operation in the constructor is complete and thus, there is no reliable way for the caller to use the results of that operation. That's not just a bad idea, but a complete non-starter. You would be forcing the caller to set some sort of timeout and either guess when the async operation is done or poll looking for completion - both bad things to do.
The more common design approach is to separate out the async operation into a method:
class User{
constructor(){ }
select(sql, callback) {
Database.selectQuery(sql, this, function(data, obj){
obj.name = data;
callback();
});
}
}
var x = new User();
x.select(statement, function() {
console.log(x.name);
});
The modern way of designing async interfaces is really moving to promises now and they offer significant advantages in error handling and in coordinating multiple async operations so that would really be my recommendation for an interface.
Your interface above probably also needs error handling.
I'm using the Q node library for Promises, question I think can apply to the Bluebird lib as well.
Context
I have a few function calls to make to both my own custom functions and node.js fs style async functions.
if I'm making a call to a function like this:
sync function
do_something = function (input) {
// assign variables, do other sync stuff
}
and need the above to take place before this function:
sync function
do_something_else = function (input) {
// assign variable, do other sync stuff
}
and then need to call a native node function similar to:
async function
writeData = function (obj, callback) {
var deferred = Q.defer();
fs.writeFile(obj.file, obj.datas, function (err, result) {
if (err) deferred.reject(err);
else deferred.resolve('write data completed');
});
return deferred.promise.nodeify(callback);
}
and finally need the above to take place before this function:
sync function
do_something_last = function (input) {
// assign variable, do other sync stuff
}
Question
Is the 'right' thing to do here, to make all my functions 'deferred' or promise aware so I can make sure that they are called in sequence or in the correct order?
like so:
do_something(variable)
.then(do_something_else)
.then(writeData)
.then(do_something_last)
.done();
or should I just do this instead and keep the ordering (sequencing)? Like so:
var variable1 = 'test1.txt'
var variable2 = 'test2.txt'
var return_value = do_something(variable1);
var return_another_value = do_something_else(return_value); <--sync
writeData(return_another_value); <-- async function
var final_value = do_something_last(variable2); <-- sync function
// could potentially have async calls again after a sync call
writeDataAgain(return_another_value); <-- async function
Clarifications
What I thought was since some of these sync functions are going to need to be fired after the async, I needed to make them Promise aware in order to keep the sequence straight, like so:
sync functions made promise aware
do_something = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve(_output_result_);
return deferred.promise;
}
do_something_else = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve(_output_result_);
return deferred.promise;
}
do_something_last = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve('completed workflow');
return deferred.promise;
}
this would allow me to do this:
do_something(variable)
.then(do_something_else) <-- these need to execute before writeData
.then(writeData) <-- a async node fs call to writeFile
.then(do_something_last) <-- I need this to happen after the writeDate
.done();
After the feedback i've read, i guess what it seems like i'm really asking is:
How do I create a function workflow, mixing non-promise sync and
promise-aware async function calls, all the while keeping the ordering
(or sequencing) of execution?
just do this instead and keep the ordering like so:
writeData(return_another_value);
var final_value = do_something_last(variable2);
Well, that simply won't work, as do_something_last is not called after the writeData(…) promise is resolved. It'll just start right after the promise is created and returned. So if you care about that particular order and want to wait until the data is written, then you need to use then with a callback:
var final_promise = writeData(return_another_value).then(function(writeResult) {
return do_something_last(variable2);
});
The general rules are:
make synchronous functions synchronous - no need for promises
make all asynchronous functions always return a promise
use deferreds only at the lowest possible level for promisification
You can just place synchronous functions in a then chain, non-promise return values (or even thrown exceptions) work fine in them.
So while you can write your sequence like
Q('test1.txt')
.then(do_something)
.then(do_something_else)
.then(writeData)
.then(do_something_last.bind(null, 'test2.txt'))
.done();
it looks rather odd. If you don't plan to make the do_somethings asynchronous in the near future for some reason, it's often simpler to write and read
writeData(do_something_else(do_something('test1.txt'))).then(function() {
return do_something_last('test2.txt');
}).done();
Admittedly, it's sometimes more appealing to write
somePromise()
.then(doSomethingSynchronous)
.then(doSomethingAsynchronous)
than
somePromise
.then(function(res) { return doSomethingAsynchronous(doSomethingSynchronous(res)); })
even though they are functionally identical. Choose the style that you like better and that is more consistent.
If all you care about is whether your functions go in sequence or not, then do this:
do_something(variable)
.then(do_something_else)
.then(writeData)
.then(do_something_last)
.done();
You'd only assign promises to variables when you're going to be passing those variables around (e.g. to other services), or using them to create different promise chains.
e.g
var promise = do_something('123')
// two different promise chains
var task1 = promise.then(function(res){
// logic
})
var task2 = promise.then(function(res){
// other logic, independent from task1
})
I have several async functions with varying numbers of parameters, in each the last param is a callback. I wish to call these in order. For instance.
function getData(url, callback){
}
function parseData(data, callback){
}
By using this:
Function.prototype.then = function(f){
var ff = this;
return function(){ ff.apply(null, [].slice.call(arguments).concat(f)) }
}
it is possible to call these functions like this, and have the output print to console.log.
getData.then(parseData.then(console.log.bind(console)))('/mydata.json');
I've been trying to use this syntax instead, and cannot get the Then function correct. Any ideas?
getData.then(parseData).then(console.log.bind(console))('/mydata.json');
Implementing a function or library that allows you to chain methods like above is a non-trivial task and requires substantial effort. The main problem with the example above is the constant context changing - it is very difficult to manage the state of the call chain without memory leaks (i.e. saving a reference to all chained functions into a module-level variable -> GC will never free the functions from memory).
If you are interested in this kind of programming strategy I highly encourage you to use an existing, established and well-tested library, like Promise or q. I personally recommend the former as it attempts to behave as close as possible to ECMAScript 6's Promise specification.
For educational purposes, I recommend you take a look at how the Promise library works internally - I am quite sure you will learn a lot by inspecting its source code and playing around with it.
Robert Rossmann is right. But I'm willing to answer purely for academic purposes.
Let's simplify your code to:
Function.prototype.then = function (callback){
var inner = this;
return function (arg) { return inner(arg, callback); }
}
and:
function getData(url, callback) {
...
}
Let's analyze the types of each function:
getData is (string, function(argument, ...)) → null.
function(argument, function).then is (function(argument, ...)) → function(argument).
That's the core of the problem. When you do:
getData.then(function (argument) {}) it actually returns a function with the type function(argument). That's why .then can't be called onto it, because .then expects to be called onto a function(argument, function) type.
What you want to do, is wrap the callback function. (In the case of getData.then(parseData).then(f), you want to wrap parseData with f, not the result of getData.then(parseData).
Here's my solution:
Function.prototype.setCallback = function (c) { this.callback = c; }
Function.prototype.getCallback = function () { return this.callback; }
Function.prototype.then = function (f) {
var ff = this;
var outer = function () {
var callback = outer.getCallback();
return ff.apply(null, [].slice.call(arguments).concat(callback));
};
if (this.getCallback() === undefined) {
outer.setCallback(f);
} else {
outer.setCallback(ff.getCallback().then(f));
}
return outer;
}
This looks like an excellent use for the Promise object. Promises improve reusability of callback functions by providing a common interface to asynchronous computation. Instead of having each function accept a callback parameter, Promises allow you to encapsulate the asynchronous part of your function in a Promise object. Then you can use the Promise methods (Promise.all, Promise.prototype.then) to chain your asynchronous operations together. Here's how your example translates:
// Instead of accepting both a url and a callback, you accept just a url. Rather than
// thinking about a Promise as a function that returns data, you can think of it as
// data that hasn't loaded or doesn't exist yet (i.e., promised data).
function getData(url) {
return new Promise(function (resolve, reject) {
// Use resolve as the callback parameter.
});
}
function parseData(data) {
// Does parseData really need to be asynchronous? If not leave out the
// Promise and write this function synchronously.
return new Promise(function (resolve, reject) {
});
}
getData("someurl").then(parseData).then(function (data) {
console.log(data);
});
// or with a synchronous parseData
getData("someurl").then(function (data) {
console.log(parseData(data));
});
Also, I should note that Promises currently don't have excellent browser support. Luckily you're covered since there are plenty of polyfills such as this one that provide much of the same functionality as native Promises.
Edit:
Alternatively, instead of changing the Function.prototype, how about implementing a chain method that takes as input a list of asynchronous functions and a seed value and pipes that seed value through each async function:
function chainAsync(seed, functions, callback) {
if (functions.length === 0) callback(seed);
functions[0](seed, function (value) {
chainAsync(value, functions.slice(1), callback);
});
}
chainAsync("someurl", [getData, parseData], function (data) {
console.log(data);
});
Edit Again:
The solutions presented above are far from robust, if you want a more extensive solution check out something like https://github.com/caolan/async.
I had some thoughts about that problem and created the following code which kinda meets your requirements. Still - I know that this concept is far away from perfect. The reasons are commented in the code and below.
Function.prototype._thenify = {
queue:[],
then:function(nextOne){
// Push the item to the queue
this._thenify.queue.push(nextOne);
return this;
},
handOver:function(){
// hand over the data to the next function, calling it in the same context (so we dont loose the queue)
this._thenify.queue.shift().apply(this, arguments);
return this;
}
}
Function.prototype.then = function(){ return this._thenify.then.apply(this, arguments) };
Function.prototype.handOver = function(){ return this._thenify.handOver.apply(this, arguments) };
function getData(json){
// simulate asyncronous call
setTimeout(function(){ getData.handOver(json, 'params from getData'); }, 10);
// we cant call this.handOver() because a new context is created for every function-call
// That means you have to do it like this or bind the context of from getData to the function itself
// which means every time the function is called you have the same context
}
function parseData(){
// simulate asyncronous call
setTimeout(function(){ parseData.handOver('params from parseData'); }, 10);
// Here we can use this.handOver cause parseData is called in the context of getData
// for clarity-reasons I let it like that
}
getData
.then(function(){ console.log(arguments); this.handOver(); }) // see how we can use this here
.then(parseData)
.then(console.log)('/mydata.json'); // Here we actually starting the chain with the call of the function
// To call the chain in the getData-context (so you can always do this.handOver()) do it like that:
// getData
// .then(function(){ console.log(arguments); this.handOver(); })
// .then(parseData)
// .then(console.log).bind(getData)('/mydata.json');
Problems and Facts:
the complete chain is executed in the context of the first function
you have to use the function itself to call handOver at least with the first Element of the chain
if you create a new chain using the function you already used, it will conflict when it runs to the same time
it is possible to use a function twice in the chain (e.g. getData)
because of the shared conext you can set a property in one function and read it in one of the following functions
At least for the first Problem you could solve it with not calling the next function in the chain in the same context and instead give the queue as parameter to the next function. I will try this approach later. This maybe would solve the conflicts mentioned at point 3, too.
For the other problem you could use the sample Code in the comments
PS: When you run the snipped make sure your console is open to see the output
PPS: Every comment on this approach is welcome!
The problem is that then returns a wrapper for the current function and successive chained calls will wrap it again, instead of wrapping the previous callback. One way to achieve that is to use closures and overwrite then on each call:
Function.prototype.then = function(f){
var ff = this;
function wrapCallback(previousCallback, callback) {
var wrapper = function(){
previousCallback.apply(null, [].slice.call(arguments).concat(callback));
};
ff.then = wrapper.then = function(f) {
callback = wrapCallback(callback, f); //a new chained call, so wrap the callback
return ff;
}
return wrapper;
}
return ff = wrapCallback(this, f); //"replace" the original function with the wrapper and return that
}
/*
* Example
*/
function getData(json, callback){
setTimeout( function() { callback(json) }, 100);
}
function parseData(data, callback){
callback(data, 'Hello');
}
function doSomething(data, text, callback) {
callback(text);
}
function printData(data) {
console.log(data); //should print 'Hello'
}
getData
.then(parseData)
.then(doSomething)
.then(printData)('/mydata.json');
I have a javascript app saving all data on server, then use REST API communicate server and client.
They works fine, until we start have more and more nested async call or nested sync call which hiding async call. For example:
function asyncFoo(callback) {
callback();
}
function syncCallHidingAsyncCall(){
syncStuff();
asyncFoo(function(){
syncFoo()
});
}
function nestedAsyncCall(callback){
asyncFoo(function(){
anotherAsyncCall(callback);
})
}
// this make refactor code become so hard.
// if we want add step2() after nestedAsyncCall();
// instead of add one line of code
// we need first add callback param in every asyncCall, then pass step2 as a callback
And some unnecessary async call:
// we actually only verify this once.
function isLogin(callback){
if (!App._user) {
ServerApi.getCurUser(function(data){
App._user = data.user;
callback(App._user)
});
}
callback(App._user)
}
function syncCallNeedVerfifyLogin(callback){
// user only need login once, so in most case this is really only a sync call.
// but now I have to involve a unnecessary callback to get the return value
isLogin(function(){
callback(syncStuff())
})
}
So after the project become bigger and bigger, we start forgot their relationship, which one need wait, which one will do magic. And more and more function become async only because some very small thing need be verify on server.
So I start feel their must be some design problem in this project. I am looking for the best practice or design patter, or some rules need follow in this kind heavy communicate app.
Thanks for help.
They exist in several patterns to manage asynchronous data exchange and routine execution. They are called in different names as well:
Promises
EventEmitters
Deferred Objects/Deferreds
Control Flow Libraries
Futures
Callback aggregators
Observer / Publisher-Subscriber
A common implementation is jQuery's Deferred Objects which is also used in managing it's AJAX methods. In NodeJS, there is also AsyncJS and the native EventEmitter. There's even a 20-liner library made by some guy that implements EventEmitter which you could use.
As Bergi says in the comments, the pattern you're looking for is called deferred / promises. There's an implementation built into jQuery. From the docs:
a chainable utility object created by calling the jQuery.Deferred()
method. It can register multiple callbacks into callback queues,
invoke callback queues, and relay the success or failure state of any
synchronous or asynchronous function.
There are a variety of other implementations some of which are outlined in this stackoverflow question.
Make yourself a queue system, something like:
function Queue() {
this.queue = [];
}
Queue.prototype.i = -1;
Queue.prototype.add = function(fn) {
if (typeof fn !== "function")
throw new TypeError("Invalid argument");
this.queue.push(fn);
}
Queue.prototype.next = function() {
this.i++;
if (this.i < this.queue.length) {
this.queue[this.i].appy(this, arguments);
}
}
Queue.prototype.start = function() {
if (this.i !== -1)
throw new Error("Already running")
this.next.apply(this, arguments);
}
And use it like this:
var q = new Queue();
q.add(function() {
// do something async
// In the callback, call `this.next()`, passing
// any relevant arguments
})
q.add(function() {
// do something async
// In the callback, call `this.next()`, passing
// any relevant arguments
})
q.add(function() {
// do something async
// In the callback, call `this.next()`, passing
// any relevant arguments
})
q.start();
DEMO: http://jsfiddle.net/4n3kH/