I have several async functions with varying numbers of parameters, in each the last param is a callback. I wish to call these in order. For instance.
function getData(url, callback){
}
function parseData(data, callback){
}
By using this:
Function.prototype.then = function(f){
var ff = this;
return function(){ ff.apply(null, [].slice.call(arguments).concat(f)) }
}
it is possible to call these functions like this, and have the output print to console.log.
getData.then(parseData.then(console.log.bind(console)))('/mydata.json');
I've been trying to use this syntax instead, and cannot get the Then function correct. Any ideas?
getData.then(parseData).then(console.log.bind(console))('/mydata.json');
Implementing a function or library that allows you to chain methods like above is a non-trivial task and requires substantial effort. The main problem with the example above is the constant context changing - it is very difficult to manage the state of the call chain without memory leaks (i.e. saving a reference to all chained functions into a module-level variable -> GC will never free the functions from memory).
If you are interested in this kind of programming strategy I highly encourage you to use an existing, established and well-tested library, like Promise or q. I personally recommend the former as it attempts to behave as close as possible to ECMAScript 6's Promise specification.
For educational purposes, I recommend you take a look at how the Promise library works internally - I am quite sure you will learn a lot by inspecting its source code and playing around with it.
Robert Rossmann is right. But I'm willing to answer purely for academic purposes.
Let's simplify your code to:
Function.prototype.then = function (callback){
var inner = this;
return function (arg) { return inner(arg, callback); }
}
and:
function getData(url, callback) {
...
}
Let's analyze the types of each function:
getData is (string, function(argument, ...)) → null.
function(argument, function).then is (function(argument, ...)) → function(argument).
That's the core of the problem. When you do:
getData.then(function (argument) {}) it actually returns a function with the type function(argument). That's why .then can't be called onto it, because .then expects to be called onto a function(argument, function) type.
What you want to do, is wrap the callback function. (In the case of getData.then(parseData).then(f), you want to wrap parseData with f, not the result of getData.then(parseData).
Here's my solution:
Function.prototype.setCallback = function (c) { this.callback = c; }
Function.prototype.getCallback = function () { return this.callback; }
Function.prototype.then = function (f) {
var ff = this;
var outer = function () {
var callback = outer.getCallback();
return ff.apply(null, [].slice.call(arguments).concat(callback));
};
if (this.getCallback() === undefined) {
outer.setCallback(f);
} else {
outer.setCallback(ff.getCallback().then(f));
}
return outer;
}
This looks like an excellent use for the Promise object. Promises improve reusability of callback functions by providing a common interface to asynchronous computation. Instead of having each function accept a callback parameter, Promises allow you to encapsulate the asynchronous part of your function in a Promise object. Then you can use the Promise methods (Promise.all, Promise.prototype.then) to chain your asynchronous operations together. Here's how your example translates:
// Instead of accepting both a url and a callback, you accept just a url. Rather than
// thinking about a Promise as a function that returns data, you can think of it as
// data that hasn't loaded or doesn't exist yet (i.e., promised data).
function getData(url) {
return new Promise(function (resolve, reject) {
// Use resolve as the callback parameter.
});
}
function parseData(data) {
// Does parseData really need to be asynchronous? If not leave out the
// Promise and write this function synchronously.
return new Promise(function (resolve, reject) {
});
}
getData("someurl").then(parseData).then(function (data) {
console.log(data);
});
// or with a synchronous parseData
getData("someurl").then(function (data) {
console.log(parseData(data));
});
Also, I should note that Promises currently don't have excellent browser support. Luckily you're covered since there are plenty of polyfills such as this one that provide much of the same functionality as native Promises.
Edit:
Alternatively, instead of changing the Function.prototype, how about implementing a chain method that takes as input a list of asynchronous functions and a seed value and pipes that seed value through each async function:
function chainAsync(seed, functions, callback) {
if (functions.length === 0) callback(seed);
functions[0](seed, function (value) {
chainAsync(value, functions.slice(1), callback);
});
}
chainAsync("someurl", [getData, parseData], function (data) {
console.log(data);
});
Edit Again:
The solutions presented above are far from robust, if you want a more extensive solution check out something like https://github.com/caolan/async.
I had some thoughts about that problem and created the following code which kinda meets your requirements. Still - I know that this concept is far away from perfect. The reasons are commented in the code and below.
Function.prototype._thenify = {
queue:[],
then:function(nextOne){
// Push the item to the queue
this._thenify.queue.push(nextOne);
return this;
},
handOver:function(){
// hand over the data to the next function, calling it in the same context (so we dont loose the queue)
this._thenify.queue.shift().apply(this, arguments);
return this;
}
}
Function.prototype.then = function(){ return this._thenify.then.apply(this, arguments) };
Function.prototype.handOver = function(){ return this._thenify.handOver.apply(this, arguments) };
function getData(json){
// simulate asyncronous call
setTimeout(function(){ getData.handOver(json, 'params from getData'); }, 10);
// we cant call this.handOver() because a new context is created for every function-call
// That means you have to do it like this or bind the context of from getData to the function itself
// which means every time the function is called you have the same context
}
function parseData(){
// simulate asyncronous call
setTimeout(function(){ parseData.handOver('params from parseData'); }, 10);
// Here we can use this.handOver cause parseData is called in the context of getData
// for clarity-reasons I let it like that
}
getData
.then(function(){ console.log(arguments); this.handOver(); }) // see how we can use this here
.then(parseData)
.then(console.log)('/mydata.json'); // Here we actually starting the chain with the call of the function
// To call the chain in the getData-context (so you can always do this.handOver()) do it like that:
// getData
// .then(function(){ console.log(arguments); this.handOver(); })
// .then(parseData)
// .then(console.log).bind(getData)('/mydata.json');
Problems and Facts:
the complete chain is executed in the context of the first function
you have to use the function itself to call handOver at least with the first Element of the chain
if you create a new chain using the function you already used, it will conflict when it runs to the same time
it is possible to use a function twice in the chain (e.g. getData)
because of the shared conext you can set a property in one function and read it in one of the following functions
At least for the first Problem you could solve it with not calling the next function in the chain in the same context and instead give the queue as parameter to the next function. I will try this approach later. This maybe would solve the conflicts mentioned at point 3, too.
For the other problem you could use the sample Code in the comments
PS: When you run the snipped make sure your console is open to see the output
PPS: Every comment on this approach is welcome!
The problem is that then returns a wrapper for the current function and successive chained calls will wrap it again, instead of wrapping the previous callback. One way to achieve that is to use closures and overwrite then on each call:
Function.prototype.then = function(f){
var ff = this;
function wrapCallback(previousCallback, callback) {
var wrapper = function(){
previousCallback.apply(null, [].slice.call(arguments).concat(callback));
};
ff.then = wrapper.then = function(f) {
callback = wrapCallback(callback, f); //a new chained call, so wrap the callback
return ff;
}
return wrapper;
}
return ff = wrapCallback(this, f); //"replace" the original function with the wrapper and return that
}
/*
* Example
*/
function getData(json, callback){
setTimeout( function() { callback(json) }, 100);
}
function parseData(data, callback){
callback(data, 'Hello');
}
function doSomething(data, text, callback) {
callback(text);
}
function printData(data) {
console.log(data); //should print 'Hello'
}
getData
.then(parseData)
.then(doSomething)
.then(printData)('/mydata.json');
Related
I have the following code:
function doSomething() {
//xhr here
setTimeout(function() {
var value = 42;
}, 10);
return {
then: function(callback) {
callback(value);
}
};
}
doSomething().then(function(result) {
log("got a result", result);
});
And can't figure out how to access the value.
I need this to be promise-based solution in order to use in multiple places
JSFidle link
Update:
We are not using any libraries in that projects
There are a couple of problems there:
value is local to the function you're passing into setTimeout, because that's where you've declared it. You could fix this issue by declaring it in doSomething instead.
The bigger issue is that what you have there isn't a promise, it's just a function that returns an object when you call it that has a then method. Here's the order in which things happen:
You call doSomething
It sets a timer to set a value.
It creates an object with a then function.
It returns the object.
You call the then function immediately.
then tries to access value (which it can't because of the declaration issue, but would be a problem anyway).
Some time later, value is set by the callback when the timer fires.
To be a promise, the then function on object you return would have to store a reference to the callback passed into it, and call the callback later, when value has been set (e.g., the promise has been fulfilled).
Rather than implementing your own promises library, I'd suggest using one of the several that have already been written and debugged.
I'm just pointing out, that in order to "fix" your issue you need to return the then this way:
function doSomething() {
//xhr here
return {
then: function(callback) {
setTimeout(function() {
callback(42); // callback from within the async action
}, 10);
}
};
}
doSomething().then(function(result) {
log("got a result", result);
});
Please read TJ's answer and consider using a promise library - also consider reading this post that explains how promise resolution looks like.
Namely: Your then needs to in turn return a promise when called (rather than just set a timeout) for chaining, and the callback should be assimilated before waiting for it.
I'm using the Q node library for Promises, question I think can apply to the Bluebird lib as well.
Context
I have a few function calls to make to both my own custom functions and node.js fs style async functions.
if I'm making a call to a function like this:
sync function
do_something = function (input) {
// assign variables, do other sync stuff
}
and need the above to take place before this function:
sync function
do_something_else = function (input) {
// assign variable, do other sync stuff
}
and then need to call a native node function similar to:
async function
writeData = function (obj, callback) {
var deferred = Q.defer();
fs.writeFile(obj.file, obj.datas, function (err, result) {
if (err) deferred.reject(err);
else deferred.resolve('write data completed');
});
return deferred.promise.nodeify(callback);
}
and finally need the above to take place before this function:
sync function
do_something_last = function (input) {
// assign variable, do other sync stuff
}
Question
Is the 'right' thing to do here, to make all my functions 'deferred' or promise aware so I can make sure that they are called in sequence or in the correct order?
like so:
do_something(variable)
.then(do_something_else)
.then(writeData)
.then(do_something_last)
.done();
or should I just do this instead and keep the ordering (sequencing)? Like so:
var variable1 = 'test1.txt'
var variable2 = 'test2.txt'
var return_value = do_something(variable1);
var return_another_value = do_something_else(return_value); <--sync
writeData(return_another_value); <-- async function
var final_value = do_something_last(variable2); <-- sync function
// could potentially have async calls again after a sync call
writeDataAgain(return_another_value); <-- async function
Clarifications
What I thought was since some of these sync functions are going to need to be fired after the async, I needed to make them Promise aware in order to keep the sequence straight, like so:
sync functions made promise aware
do_something = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve(_output_result_);
return deferred.promise;
}
do_something_else = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve(_output_result_);
return deferred.promise;
}
do_something_last = function (input) {
var deferred = Q.defer();
// assign variables, do other sync stuff
deferred.resolve('completed workflow');
return deferred.promise;
}
this would allow me to do this:
do_something(variable)
.then(do_something_else) <-- these need to execute before writeData
.then(writeData) <-- a async node fs call to writeFile
.then(do_something_last) <-- I need this to happen after the writeDate
.done();
After the feedback i've read, i guess what it seems like i'm really asking is:
How do I create a function workflow, mixing non-promise sync and
promise-aware async function calls, all the while keeping the ordering
(or sequencing) of execution?
just do this instead and keep the ordering like so:
writeData(return_another_value);
var final_value = do_something_last(variable2);
Well, that simply won't work, as do_something_last is not called after the writeData(…) promise is resolved. It'll just start right after the promise is created and returned. So if you care about that particular order and want to wait until the data is written, then you need to use then with a callback:
var final_promise = writeData(return_another_value).then(function(writeResult) {
return do_something_last(variable2);
});
The general rules are:
make synchronous functions synchronous - no need for promises
make all asynchronous functions always return a promise
use deferreds only at the lowest possible level for promisification
You can just place synchronous functions in a then chain, non-promise return values (or even thrown exceptions) work fine in them.
So while you can write your sequence like
Q('test1.txt')
.then(do_something)
.then(do_something_else)
.then(writeData)
.then(do_something_last.bind(null, 'test2.txt'))
.done();
it looks rather odd. If you don't plan to make the do_somethings asynchronous in the near future for some reason, it's often simpler to write and read
writeData(do_something_else(do_something('test1.txt'))).then(function() {
return do_something_last('test2.txt');
}).done();
Admittedly, it's sometimes more appealing to write
somePromise()
.then(doSomethingSynchronous)
.then(doSomethingAsynchronous)
than
somePromise
.then(function(res) { return doSomethingAsynchronous(doSomethingSynchronous(res)); })
even though they are functionally identical. Choose the style that you like better and that is more consistent.
If all you care about is whether your functions go in sequence or not, then do this:
do_something(variable)
.then(do_something_else)
.then(writeData)
.then(do_something_last)
.done();
You'd only assign promises to variables when you're going to be passing those variables around (e.g. to other services), or using them to create different promise chains.
e.g
var promise = do_something('123')
// two different promise chains
var task1 = promise.then(function(res){
// logic
})
var task2 = promise.then(function(res){
// other logic, independent from task1
})
I'm defining a class which instantiates several modules which depend on previous ones. The modules themselves may require an async operation before they are ready (i.e. establishing a mysql connection) so I've provided each constructor with a callback to be called once the module is ready. However I've run into a problem when instantiating classes which are ready immediately:
var async = require('async');
var child = function(parent, cb) {
var self = this;
this.ready = false;
this.isReady = function() {
return self.ready;
}
/* This does not work, throws error below stating c1.isReady is undefined*/
cb(null, true);
/* This works */
setTimeout(function() {
self.ready = true;
cb(null, true);
}, 0);
}
var Parent = function(cb) {
var self = this;
async.series([
function(callback){
self.c1 = new child(self, callback);
},
function(callback){
self.c2 = new child(self, callback);
}
],
function(err, results){
console.log(self.c1.isReady(), self.c2.isReady);
console.log(err, results);
});
}
var P = new Parent();
I'm guessing the issue is calling cb within the constructor means async proceeds to the next function before the constructor finishes. Is there a better approach to this? I considered using promises, but I find this approach easier to understand/follow.
You will have to delay the call to the callback when everything is synchronous because any code in the callback won't be able to reference the object yet (as you've seen). Because the constructor hasn't finished executing, the assignment of it's return value also hasn't finished yet.
You could pass the object to the callback and force the callback to use that reference (which will exist and be fully formed), but you're requiring people to know that they can't use a normally accepted practice so it's much better to make sure the callback is only called asynchronously and people can then write their code in the normal ways they would expect to.
In node.js, it will be more efficient to use process.nextTick() instead of setTimeout() to accomplish your goal. See this article for more details.
var child = function(parent, cb) {
var self = this;
this.ready = false;
this.isReady = function() {
return self.ready;
}
/* This works */
process.nextTick(function() {
self.ready = true;
cb(null, true);
}, 0);
}
Here's a general observation. Many people (myself included) think that it overcomplicates things to put any async operation in a constructor for reasons related to this. Instead, most objects that need to do async operations in order to get themselves set up will offer a .init() or .connect() method or something like that. Then, you construct the object like you normally would in a synchronous fashion and then separately initiate the async part of the initialization and pass it the callback. That gets you away from this issue entirely.
If/when you want to use promises for tracking your async operation (which is great feature direction to go), it's way, way easier to let the constructor return the object and the .init() operation to return a promise. It gets messy to try to return both the object and a promise from the constructor and even messier to code with that.
Then, using promises you can do this:
var o = new child(p);
o.init().then(function() {
// object o is fully initialized now
// put code in here to use the object
}, function(err) {
// error initializing object o
});
If you can't put the object into a Promise, put a promise into the object.
Give it a Ready property that is a promise.
If you have a class hierarchy (I'm talking about Typescript now) of (say) widgets, the base class can define the Ready property and assign it an already resolved Promise object. Doing this is a base class means it's always safe to write code like this
var foo = Foo();
foo.Ready.then(() => {
// do stuff that needs foo to be ready
});
Derived classes can take control of promise resolution by replacing the value of Ready with a new promise object, and resolving it when the async code completes.
Here's a comprehensive workup of Asynchronous Constructor design pattern
I am not very familiar with callback() functions. I am looking for an explanation and an example of what a proper use case might be. In my example below, how can I utilize a callback and also, should I?
Here I have two functions:
addShape = function () {
ExampleService.createShape(function () { //ajax
shapeMade = true;
//anything else etc.......
});
}
deleteShape = function () {
ExampleService.removeShape(function () { //ajax
shapeMade = false;
//anything else etc.......
});
}
The third function (focus of my question)
resetShape = function () {
deleteShape();
addShape();
console.log('example');
}
When I call the resetShape() function, the example gets logged to the console before both deleteShape() and addShape() have finished.
Would this be a situation to use a callback()? If so, how? If not, why?
Callbacks are used for asynchronous functions. In this case, since you have multiple async function to wait for, you probably want to use a Promise, and Promise.all to execute the callback when all async operations are complete:
resetShape = function () {
Promise.all([ deleteShape(), addShape() ]).then( function() {
console.log('example');
});
}
Note that, for this to work, your functions "deleteShape" and "addShape" have to return the promise object:
addShape = function () {
return new Promise(function(resolve, reject) {
ExampleService.createShape(function () {
shapeMade = true;
resolve();
});
});
A callback is used mainly for two reasons, to give another object or function a way to tell us something we were waiting for has completed (or changed) or a way to get something from us when they require it.
First case: imagine a function that makes an asynchronous request to
a server. We don't know how much time it'll take to complete, so we
give the requester a way to tell us it's done:
getDataFromServer('foo.php', callback); //callback will be called when the request
//is done. We can do something else meanwhile.
Second case. I could create and object that, when needed, will ask
for more data (ie, getting more rows to append to a table when we
scroll down)
var tableBuilder = new TableBuilder(container, dataGetterCallback);
dataGetterCallback will be called whenever TableBuilder needs more rows. It'll have the logic to give it to them (maybe by receiving a parameter that tells it in which index to start).
Hope this gives you an idea.
I've been struggling for awhile with callbacks in general. I'm trying to go back to the basics to try to understand conceptually and this is what I understand so far (yeah, basic).
function RandHash (callback) {
ds_hash = Math.floor((Math.random()*10000)+1);
callback();
}
function CompiledHash (){
console.log(ds_hash);
}
var ds_hash;
RandHash(Compiled Hash);
Will yield the random number.
However, I'm lost as to how to get the "ds_hash" variable returned from the callback.
Seems this would work:
var ds_hash;
ds_hash = RandHash(Compiled Hash);
It doesn't. If I try to return the value something like:
function CompiledHash (){
return ds_hash;
}
It doesn't do anything.
Please help me with this. Seems I spend 90% of my time with node in callback debugging. I've built some decent applications but everything has been handled through the async library because of this mental block I have.
Thanks.
The first error you've made is that RandHash hasn't returned anything. There's no return statement present in the function, so var anything = RandHash(alsoAnything) will always result in anything being undefined.
Functions are first class variables
Functions can be used just as variables can, and passed round as arguments to functions. This is a powerful feature of javascript and something that you'll need to grok to use callbacks. Think of a function as a defined action, and you're just passing that action around.
Also, callbacks should be used to deal with the completion of a process. So the idea is, you know what is meant to happen after process A, and what it is meant to generate, so you can give process A a callback which will be called once A has terminated. Now the scenario here isn't appropriate for a callback situation. It would be much easier for you to do...
function RandHash () {
return Math.floor((Math.random()*10000)+1);
}
And then get your hash by just calling this function, like so...
var hash = RandHash();
You also want to be aware of javascripts variable scoping, as you're missing the var keyword when referencing ds_hash in the RandHash function which means the assignment defaults to global scope. This is probably what is responsible for confusing you, as your assignment of ds_hash in RandHash will be global and therefore available in CompiledHash, meaning some functions will still be able to access the ds_hash value despite this not being the correct, or proper way to do this.
Assuming you will eventually require asynchronous processing
function randHash (callback) {
var hash = /* do hash calculation */
callback(hash);
}
function compileHash (hash) {
/* do something using the hash variable passed to this function */
}
randHash(compileHash); // pass the compileHash function as the callback
You should pass your variables as arguments to the callback. That will deal with your scoping issues hopefully.
Also, small note, functions in javascript should typically be lowercase if they aren't going to be used with the new statement (ie, a javascript class).
To truly understand asynchronous behaviour, you should try it with something that is actually asynchronous, and passing values around, not just a global variable, as that will never work when you move on to asynchronous functions :
function RandHash (callback) {
setTimeout(function() {
var ds_hash = Math.floor((Math.random()*10000)+1);
callback(ds_hash); // execute callback when async operation is done
}, 300);
}
function CompiledHash(hash){ // pass hash
console.log(hash);
}
RandHash(function(returned_hash) { // this is the callback function
CompiledHash(returned_hash); // pass along the returned data
});
The callback function is executed once the asynchronous function, in this case setTimeout, has completed, and passes the data back as an argument.
The callback argument is just the function that is passes as an argument.
// regular function
function doStuff(argument) {
// where argument can be anything, also a function
}
// you can pass a string
doStuff('string');
// or a function
doStuff(function(argument_returned) {
});
// and passing a function you can execute that function with arguments in the original function
function doStuff(argument) {
var argument_returned = 'string';
argument(argument_returned); // executes the passsed function
});
Try this:
function RandHash (callback) {
ds_hash = Math.floor((Math.random()*10000)+1);
callback(ds_hash);
}
function CompiledHash(ds_hash){
console.log(ds_hash);
}
var ds_hash;
RandHash(CompiledHash);
For callbacks and functions, it's a bit difficult to understand variable scopes, and what you are doing is not really recommended. You can pass parameters into callbacks, and should get passed, but I would suggest building it like this:
function RandHash (callback) {
var ds_hash = Math.floor((Math.random()*10000)+1);
callback(ds_hash);
}
function CompiledHash(ds_hash){
console.log(ds_hash);
}
RandHash(CompiledHash);
A couple notes:
If you want to propagate the result of the callback then you need to return its output in RandHash
You don't need ds_hash to be a global variable if you're planning on returning it, anyway
The line RandHash(Compiled Hash); is a syntax error (notice the space in the variable name)
Try this:
function RandHash(callback) {
var ds_hash = Math.floor((Math.random() * 10000) + 1);
return callback(ds_hash);
}
function CompiledHash(ds_hash) {
console.log(ds_hash);
return ds_hash;
}
RandHash(CompiledHash);
Notice that there are no global variables. The callback returns a value and the function that executes the callback passes it along.
On a style note, you should only capitalize the names of functions that you intend to use as a constructor.
This will do
function RandHash (callback) {
var ds_hash = Math.floor((Math.random()*10000)+1);
callback(ds_hash);
}
var CompiledHash = function (ds_hash){
console.log(ds_hash);
}
RandHash(CompiledHash);