Develop Reference

Async Frustrations: Should I be using callbacks and how do I pass them through multiple modules?

I am attempting to make a simple text game that operates in a socket.io chat room on a node server. The program works as follows:
Currently I have three main modules
Rogue : basic home of rogue game functions
rogueParser : module responsible for extracting workable commands from command strings
Verb_library: module containing a list of commands that can be invoked from the client terminal.
The Client types a command like 'say hello world'. This triggers the following socket.io listener
socket.on('rg_command', function(command){
// execute the verb
let verb = rogueParser(command);
rogue.executeVerb(verb, command, function(result){
console.log(result);
});
});
Which then in turn invokes the executeVerb function from rogue..
executeVerb: function(verb, data, callback){
verb_library[verb](data, callback);
},
Each verb in verb_library should be responsible for manipulating the database -if required- and then returning an echo string sent to the appropriate targets representing the completion of the action.
EDIT: I chose 'say' when I posted this but it was pointed out afterward that it was a poor example. 'say' is not currently async but eventually will be as will be the vast majority of 'verbs' as they will need to make calls to the database.
...
say: function(data, callback){
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
message = ('you say '+'"'+message.trim()+'"');
response.target = data.user;
response.type = 'echo';
response.message = message;
callback(response);
},
...
My problem is that
1 ) I am having issues passing callbacks through so many modules. Should I be able to pass a callback through multiple layers of modules? Im worried that I'm blind so some scope magic that is making me lose track of what should happen when I pass a callback function into a module which then passes the same callback to another module which then calls the callback. Currently it seems I either end up without access to the callback on the end, or the first function tries to execute without waiting on the final callback returning a null value.
2 ) Im not sure if Im making this harder than it needs to be by not using promises or if this is totally achievable with callbacks, in which case I want to learn how to do it that way before I summon extra code.
Sorry if this is a vague question, I'm in a position of design pattern doubt and looking for advice on this general setup as well as specific information regarding how these callbacks should be passed around. Thanks!

1) Passing callback trough multiple layers doesn't sound like a good idea. Usually I'm thinking what will happen, If I will continiue doing this for a year? Will it be flexible enough so that when I need to change to architecture (let's say customer have new idea), my code will allow me to without rewriting whole app? What you're experiencing is called callback hell. http://callbackhell.com/
What we are trying to do, is to keep our code as shallow as possible.
2) Promise is just syntax sugar for callback. But it's much easier to think in Promise then in callback. So personally, I would advice you to take your time and grasp as much as you can of programming language features during your project. Latest way we're doing asynchronus code is by using async/await syntax which allows us to totally get rid of callback and Promise calls. But during your path, you will have to work with both for sure.
You can try to finish your code this way and when you're done, find what was the biggest pain and how could you write it again to avoid it in future. I promise you that it will be much more educative then getting explicit answear here :)

Asynchronousness and JavaScript go back a long way. How we deal with it has evolved over time and there are numerous applied patterns that attempt to make async easier. I would say that there are 3 concrete and popular patterns. However, each one is very related to the other:
Callbacks
Promises
async/await
Callbacks are probably the most backwards compatible and just involve providing a function to some asynchronous task in order to have your provided function be called whenever the task is complete.
For example:
/**
* Some dummy asynchronous task that waits 2 seconds to complete
*/
function asynchronousTask(cb) {
setTimeout(() => {
console.log("Async task is done");
cb();
}, 2000);
}
asynchronousTask(() => {
console.log("My function to be called after async task");
});
Promises are a primitive that encapsulates the callback pattern so that instead of providing a function to the task function, you call the then method on the Promise that the task returns:
/**
* Some dummy asynchronous task that waits 2 seconds to complete
* BUT the difference is that it returns a Promise
*/
function asynchronousTask() {
return new Promise(resolve => {
setTimeout(() => {
console.log("Async task is done");
resolve();
}, 2000);
});
}
asynchronousTask()
.then(() => {
console.log("My function to be called after async task");
});
The last pattern is the async/await pattern which also deals in Promises which are an encapsulation of callbacks. They are unique because they provide lexical support for using Promises so that you don't have to use .then() directly and also don't have to explicitly return a Promise from your task:
/*
* We still need some Promise oriented bootstrap
* function to demonstrate the async/await
* this will just wait a duration and resolve
*/
function $timeout(duration) {
return new Promise(resolve => setTimeout(resolve, duration));
}
/**
* Task runner that waits 2 seconds and then prints a message
*/
(async function() {
await $timeout(2000);
console.log("My function to be called after async task");
}());
Now that our vocabulary is cleared up, we need to consider one other thing: These patterns are all API dependent. The library that you are using uses callbacks. It is alright to mix these patterns, but I would say that the code that you write should be consistent. Pick one of the patterns and wrap or interface with the library that you need to.
If the library deals in callbacks, see if there is a wrapping library or a mechanism to have it deal in Promises instead. async/await consumes Promises, but not callbacks.

Callbacks are fine, but I would only use them if a function is dependent on some asynchronous result. If however the result is immediately available, then the function should be designed to return that value.
In the example you have given, say does not have to wait for any asynchronous API call to come back with a result, so I would change its signature to the following:
say: function(data){ // <--- no callback argument
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
message = ('you say '+'"'+message.trim()+'"');
response.target = data.user;
response.type = 'echo';
response.message = message;
return response; // <--- return it
}
Then going backwards, you would also change the signature of the functions that use say:
executeVerb: function(verb, data){ // <--- no callback argument
return verb_library[verb](data); // <--- no callback argument, and return the returned value
}
And further up the call stack:
socket.on('rg_command', function(command){
// execute the verb
let verb = rogueParser(command);
let result = rogue.executeVerb(verb, command); // <--- no callback, just get the returned value
console.log(result);
});
Of course, this can only work if all verb methods can return the expected result synchronously.
Promises
If say would depend on some asynchronous API, then you could use promises. Let's assume this API provides a callback system, then your say function could return a promise like this:
say: async function(data){ // <--- still no callback argument, but async!
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
response.target = data.user;
response.type = 'echo';
// Convert the API callback system to a promise, and use AWAIT
await respone.message = new Promise(resolve => someAsyncAPIWithCallBackAsLastArg(message, resolve));
return response; // <--- return it
}
Again going backwards, you would also change the signature of the functions that use say:
executeVerb: function(verb, data){ // <--- still no callback argument
return verb_library[verb](data); // <--- no callback argument, and return the returned promise(!)
}
And finally:
socket.on('rg_command', async function(command){ // Add async
// execute the verb
let verb = rogueParser(command);
let result = await rogue.executeVerb(verb, command); // <--- await the fulfillment of the returned promise
console.log(result);
});

Observable determine if subscriber function has finished

What is the best way to determine if the subscriber has finished executing or better yet return something and catch it up-stream? For example:
this._subscriptions.push(this._client
.getCommandStream(this._command) // Returns an IObservable from a Subject stream
.subscribe(msg => {
// Do some processing maybe some promise stuff
http.request(url).then(
// some more stuff
);
});
What's the best know to determine that subscription has finished. I've implemented it as follows:
this._subscriptions.push(this._client
.getCommandStream(this._command)
.subscribe(msg => {
// Do some processing maybe some promise stuff
http.request(url).then(re => {
// some more stuff
msg.done()
}).catch(err => msg.done(err));
});
i.e. added a done method to the object being passed in to determine if this is finished. The issue with that is I'll have to call done in every promise or catch block and find that a little too exhaustive. Is there a cleaner and more automated way of doing this?
I think the examples I've given are not good enough. This implementation is using RX to build an internal messaging bus. The get command stream is actually returning a read-only channel (as an Observable) to get commands and process them. Now the processing could be a http request followed by many other things or just an if statement.
this._client
.getCommandStream(this._command) // Returns an IObservable from a Subject stream
.subscribe(msg => {
// Do some processing maybe some promise stuff
http.request(url).then({
// some more stuff
}).then({
// Here I wanna do some file io
if(x) {
file.read('path', (content) => {
msg.reply(content);
msg.done();
});
} else {
// Or maybe not do a file io or maybe even do some image processing
msg.reply("pong");
msg.done()
}
});
});
I feel like this is a fine usage of the Observable pattern as this is exactly a sequence of commands coming in and this logic would like to act on them. The question is notice msg.done() being called all over the place. I want to know what is the best way to limit that call and know when the entire thing is done. Another option is to wrap it all in a Promise but then again what's the difference between resolve or msg.done()?

Actually, making another asynchronous request inside subscribe() isn't recommended because it just makes things more complicated and using Rx in this way doesn't help you make your code more understandable.
Since you need to make a request to a remote service that returns a PRomise you can merge it into the chain:
this._subscriptions.push(this._client
.getCommandStream(this._command)
.concatMap(msg => http.request(url))
.subscribe(...)
Also the 3rd parameter to subscribe is a callback that is called when the source Observable completes.
You can also add your own teardown logic when the chain is being disposed. This is called after the complete callback in subscribe(...) is called:
const subscription = this._subscriptions.push(this._client
...
.subscribe(...)
subscription.add(() => doWhatever())
Btw, this is equivalent to using the finally() operator.

As per RxJs subscribe method documentation, the last Argument is completed function
var source = Rx.Observable.range(0, 3)
var subscription = source.subscribe(
function (x) {
console.log('Next: %s', x);
},
function (err) {
console.log('Error: %s', err);
},
function () {
console.log('Completed');
});
please refer this documentation
https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/operators/subscribe.md

What is the meaning of using done as a parameter in a javascript function?

i am learning Javascript, and it seems that the done as a parameter in a function is a difficult concept to understand.
I want to know why it does behaves like that (done as a parameter (completed process signal i guess), and if there is some good book or resource Online to study further this concept.
Example, i am following along with a Tutorial and it uses done as a parameter, the thing is that when i run the code on node via gulp (gulpfile.js) the process never stops when using done, if i choose to skip done in the code it runs smoothly. I am trying to track down the problem, and i know that the problem is the done as a parameter, ( it has been checked by me multiple times).
gulp.task('clean-styles', function(done) {
var files = config.temp + '**/*.css';
clean(files, done);
});
function clean(path, done) {
log('Cleaning: ' + $.util.colors.blue(path));
del(path, done).then(function(path) {
console.log("path=",util.inspect(path,false,null))
console.log('Deleted Files\/Folders:\n', path.join('\n'));
console.log('Finishing clean')
});
}
node version: 0.12.4
npm version: 2.10.1
gulp version: 3.9.0
Thanks a lot for any help, it will be really appreciated.
Salutations.

can only explain the concept. what you are trying to achieve is not clear enough.
done is just a non-official standard name for a function (a.k.a callback) that informs the calling function (parent in stacktrace) that a task is completed.
recall that javascript is asynchronous and functions can be passed around as variables.
now, imagine a function startPrinting that has to call printText1, printText2 and printText3 and then output message that process is completed. We have:
function startPrinting() {
printText1();
printText2();
printText3();
console.log("completed");
}
function printText1() {
$.get('http://ps-web.cloudapp.net/proxy.php?url=http://pastebin.com/raw.php?i=3uPCavLN', function(response){
console.log(response)
});
}
function printText2() {
$.get('http://ps-web.cloudapp.net/proxy.php?url=http://pastebin.com/raw.php?i=jZjqKgNN', function(response){
console.log(response)
});
}
function printText3() {
$.get('http://ps-web.cloudapp.net/proxy.php?url=http://pastebin.com/raw.php?i=SreCbunb', function(response){
console.log(response)
});
}
here, there is no assurance that completed will ALWAYS be printed after all three functions have been executed. since they execute asynchronously.
in order to sort this, javascript ninjas will introduce a done function so that startPrinting will only print completed when all three functions have been executed. Notice how a function is passed to printText1 ... 2 below:
function startPrinting() {
/* START OF DONE ROUTINE */
var count = 0;
var printCompleted = function() {
count+=1;
if(count == 3)
console.log("completed");
}
/* END */
printText1(printCompleted);
printText2(printCompleted);
printText3(printCompleted);
}
function printText1(done) {
$.get('http://ps-web.cloudapp.net/proxy.php?url=http://pastebin.com/raw.php?i=3uPCavLN', function(response){
console.log(response)
done();
});
}
function printText2(done) {
$.get('http://ps-web.cloudapp.net/proxy.php?url=http://pastebin.com/raw.php?i=jZjqKgNN', function(response){
console.log(response)
done();
});
}
function printText3(done) {
$.get('http://ps-web.cloudapp.net/proxy.php?url=http://pastebin.com/raw.php?i=SreCbunb', function(response){
console.log(response)
done();
});
}
I hope you are able to apply this principle to better understanding your context.

Functions are first class objects in JavaScript. You can pass them around like any other value. Once they have been passed to another function as an argument, then you can call them using the argument name (or call another function and pass them as an argument to that, or assign them properties, or convert them to strings, or whatever else you'd like to do with them).
function this_sets_the_body() {
document.body.innerHTML = "Hello, world";
}
function this_calls_a_callback(im_a_callback) {
im_a_callback();
}
this_calls_a_callback(this_sets_the_body);
In your code, you've written a function using an anonymous function expression:
function(done) {
// ...
}
… and you've told it to expect to be called with an argument which you are calling done.
Whatever value is passed to it, you are ignoring (your function doesn't mention done after the argument name).
The library you are using (presumably) is passing a function in there and expects you to call it once your function as done whatever it is that it is going to do. This lets it wait until anything asynchronous that you are doing is finished.
So call done() when your code is done.

It looks like your example is fully messed up with respect to callbacks. In some places in your example done is used as a callback -- a function given from outside to be called when everything is finished in an asynchronous process and signal the end of the operation. In other cases it looks to be used as an argument provided by the execution method itself. In yet another you use it in a promise. Anyway, as I am not familiar with gulp I can only guess, but I hope the following example would work for you to explain the concepts of callback and partially promise. I would, however, recommend to avoid situations of missing callbacks and promises in the same code as it leads to confusion.
gulp.task('clean-styles', function(done) {
console.log(1);
/* we are in the callback of gulp.task: we give the
* latter this anonymous function to call when the
* setup is ready and it gives us function done to
* call when we are done and signal the engine any errors
*/
var files = config.temp + '**/*.css';
/* this defines the action to take when files are actually deleted */
var callback = function(err, message) {
console.log(6);
console.log(message); // expect: looks good
// this is provided apparently by gulp and calling it signals the engine that everything is completed
done(err);
};
/* we call this function, but some bits (like deletion
* run asynchronously. The function will return quickly, but
* callback (function) will only be called when files are deleted */
clean(files, callback);
/* the execution of gulp.task callback is finished,
* but files are not yet deleted */
console.log(4);
});
/* done here is not the same done as above, it is actually
* the function we supply into the call above, i.e. `callback` */
function clean(path, done) {
/* the cleanup is starting */
console.log(2);
/* del is scheduled. it returns a `promise` and if
* we call `then`, then the given anonymous function
* will be executed when files are deleted. This is
* where we call the provided function `done` to
* signal that the job is complete and execute some action */
del(path).then(function() {
/* files are deleted and this callback is called */
console.log(5);
/* we let the outer caller know by calling `done` which
* was given to us from outside */
done(null, "looks good"); // null means no error
}).catch(function(err) {
done(err, "looks bad"); // err is given back
});
/* the clean method is through, but files not yet deleted */
console.log(3);
}

How do async tests work in intern?

How do asynchronous tests work in Intern testing framework? I have tried to get them run exactly as in the example, but the async test passes immediately without waiting for the callback to be run.
it('should connect in 5 seconds', function () {
var dfd = this.async(5000);
conn.connect(credentials, dfd.callback(function(result) {
expect(result).to.... something
}));
}
The test passes immediately. What am I doing wrong?

dfd.callback doesn’t execute anything until it itself is executed. Keep in mind that it is designed for promise callbacks (i.e. the function passed to promise.then), not Node.js-style callbacks where the argument might be an error (i.e. function (error, result) {}). It will not check to see if an error is passed as an argument.
Without knowing what conn is, but seeing how you are passing dfd.callback as an argument to something that is not a promise, my suspicion is you are trying to use a Node.js-style callback and the call is erroring immediately. We may provide a convenience wrapper for these types of callbacks in the future to convert them to a promise interface, but until then, you probably just need to do something like this:
it('should connect in 5 seconds', function () {
var dfd = this.async(5000);
conn.connect(credentials, dfd.callback(function(error, result) {
if (error) {
throw error;
}
expect(result).to.... something
}));
});
Otherwise, without knowing what conn is and seeing what your actual assertion is, it’s too hard to say what the issue is here. So long as nothing inside the callback throws an error, the test will be considered successful.
Edit: So based on your comments above it sounds like your callback is an event listener called multiple times with different information. In this case, what you could do is something like this:
it('should connect in 5 seconds', function () {
var dfd = this.async(5000);
conn.connect(credentials, dfd.rejectOnError(function (result) {
if (result !== 'what I want') {
return;
}
expect(result).to.... something
// all other tests…
// nothing threw an error, so it is a successful test
dfd.resolve();
}));
});
dfd.rejectOnError works just like dfd.callback except it does not automatically resolve the promise; you do that yourself at the end.

Your structure is okay. dfd sets a timeout of 5 seconds for the test to succeed, then immediately tries conn.connect(), which is not part of the intern framework. If you are trying a simple XHR request, try the getUrlCallback function instead.
They have a pretty cool list of tests at this url: https://github.com/theintern/intern/wiki/Writing-Tests . Look for the two async examples.

AJAX workflow: How do I order the execution of these functions?

I'm trying to figure the best way to get my functions executing in the correct order.
I have 3 functions
function 1 - squirts OPTIONs into a SELECT via JSON and marks them as selected
function 2 - squirts OPTIONS into a 2nd SELECT and marks them as selected
function 3 - gets the values from the above SELECTs along with some additional INPUT values, does an AJAX GET resulting in JSON data, which is read and populates a table.
With JQuery Onload, I execute:
function1();
function2();
function3();
I'm finding function3 is executing before the SELECTs have been populated with OPTIONS and hence the table has no results, because the values sent in the GET were blank.
I know this is probably a very simple problem and that there are probably a dozen ways to accomplish this, but basically I need the best way to code this so that function3 only runs if function1 and 2 are complete.
I've come into Javascript via the back door having learnt the basics of JQuery first!
Thanks for your assistance.

Javascript executes synchronously, which means that function3 must wait for function2 to complete, which must wait for function1 to complete before executing.
The exception is when you run code that is asynchronous, like a setTimeout, setInterval or an asynchronous AJAX request.
Any subsequent code that relies on the completion of such asynchronous code needs to be called in such a manner that it doesn't execute until the asynchronous code has completed.
In the case of the setTimeout, you could just place the next function call at the end of the function you're passing to the setTimeout.
In the case of an AJAX call, you can place the next function call in a callback that fires upon a completed request.
If you don't want the execution of the subsequent function to occur every time, you can modify your functions to accept a function argument that gets called at the end of the asynchronous code.
Something like:
function function1( fn ) {
setTimeout(function() {
// your code
// Call the function parameter if it exists
if( fn ) {
fn();
}
}, 200);
}
function function2() {
// some code that must wait for function1
}
onload:
// Call function1 and pass function2 as an argument
function1( function2 );
// ...or call function1 without the argument
function1();
// ...or call function2 independently of function1
function2();

I recommend you use a Promises library. You can hack simple solutions like other answers suggest, but as your application grows, you'll find you are doing more and more of these hacks. Promises are intended to solve these kinds of problems when dealing with asynchronous calls.
The CommonJS project has several Promises proposals which you should check out. Here is a question I asked on SO about Promises a while back with links to other solutions. Learn more about Promises in this Douglas Crockford video. The whole video is good, but skip to just past half way for promises.
I'm using the FuturesJS library currently as it suits my needs. But there are advantages to other implementations as well. It allows you to do sequences very easily:
// Initialize Application
Futures.sequence(function (next) {
// First load the UI description document
loadUI(next); // next() is called inside loadUI
})
.then(function(next) {
// Then load all templates specified in the description
loadTemplates(next); // next() is called inside loadTemplates
})
.then(function(next) {
// Then initialize all templates specified in the description
initTemplates();
});
Even more powerful is when you need to join async events together and do another action when all of the other async events have completed. Here's an example (untested) that will load a bunch of HTML files and then perform an action only once ALL of them have completed loading:
var path = "/templates/",
templates = ["one.html","two.html","three.html"],
promises = [];
$.each(templates, function(i,name) {
promises[i] = Futures.promise();
var $container = $("<div>");
$container.load(path+name, function(response,status,xhr) {
promises[i].fullfill();
}
});
Futures.join(promises, {timeout: 10000}) // Fail if promises not completed in 10 seconds
.when(function(p_arr) {
console.log("All templates loaded");
})
.fail(function(p_arr) {
console.log("Error loading templates");
});
This might be overkill for your application. But if the application is growing in complexity, using promises will help you in the long run.
I hope this helps!

invoke function2 inside of function1 and function3 inside of function2.

It's not clear why f1 and f2 are executing before f3.
Also, are you using the preferred $(document).ready() or some variation of onload?
It might be helpful if you provide a reproducible test case.

fun3() will only run after both are ready. It might run twice. You can fix this with a lock inside fun3() you would need a Singleton to guarantee it works correctly.
var select1ready = false, select2ready = false;
fun1()
{
// do stuff
select1ready = true;
fun3();
}
fun2()
{
// do stuff
select2ready = true;
fun3();
}
fun3()
{
if (select1ready && select2ready)
{
}
}
fun1();
fun2();