Develop Reference

Async Frustrations: Should I be using callbacks and how do I pass them through multiple modules?

I am attempting to make a simple text game that operates in a socket.io chat room on a node server. The program works as follows:
Currently I have three main modules
Rogue : basic home of rogue game functions
rogueParser : module responsible for extracting workable commands from command strings
Verb_library: module containing a list of commands that can be invoked from the client terminal.
The Client types a command like 'say hello world'. This triggers the following socket.io listener
socket.on('rg_command', function(command){
// execute the verb
let verb = rogueParser(command);
rogue.executeVerb(verb, command, function(result){
console.log(result);
});
});
Which then in turn invokes the executeVerb function from rogue..
executeVerb: function(verb, data, callback){
verb_library[verb](data, callback);
},
Each verb in verb_library should be responsible for manipulating the database -if required- and then returning an echo string sent to the appropriate targets representing the completion of the action.
EDIT: I chose 'say' when I posted this but it was pointed out afterward that it was a poor example. 'say' is not currently async but eventually will be as will be the vast majority of 'verbs' as they will need to make calls to the database.
...
say: function(data, callback){
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
message = ('you say '+'"'+message.trim()+'"');
response.target = data.user;
response.type = 'echo';
response.message = message;
callback(response);
},
...
My problem is that
1 ) I am having issues passing callbacks through so many modules. Should I be able to pass a callback through multiple layers of modules? Im worried that I'm blind so some scope magic that is making me lose track of what should happen when I pass a callback function into a module which then passes the same callback to another module which then calls the callback. Currently it seems I either end up without access to the callback on the end, or the first function tries to execute without waiting on the final callback returning a null value.
2 ) Im not sure if Im making this harder than it needs to be by not using promises or if this is totally achievable with callbacks, in which case I want to learn how to do it that way before I summon extra code.
Sorry if this is a vague question, I'm in a position of design pattern doubt and looking for advice on this general setup as well as specific information regarding how these callbacks should be passed around. Thanks!

1) Passing callback trough multiple layers doesn't sound like a good idea. Usually I'm thinking what will happen, If I will continiue doing this for a year? Will it be flexible enough so that when I need to change to architecture (let's say customer have new idea), my code will allow me to without rewriting whole app? What you're experiencing is called callback hell. http://callbackhell.com/
What we are trying to do, is to keep our code as shallow as possible.
2) Promise is just syntax sugar for callback. But it's much easier to think in Promise then in callback. So personally, I would advice you to take your time and grasp as much as you can of programming language features during your project. Latest way we're doing asynchronus code is by using async/await syntax which allows us to totally get rid of callback and Promise calls. But during your path, you will have to work with both for sure.
You can try to finish your code this way and when you're done, find what was the biggest pain and how could you write it again to avoid it in future. I promise you that it will be much more educative then getting explicit answear here :)

Asynchronousness and JavaScript go back a long way. How we deal with it has evolved over time and there are numerous applied patterns that attempt to make async easier. I would say that there are 3 concrete and popular patterns. However, each one is very related to the other:
Callbacks
Promises
async/await
Callbacks are probably the most backwards compatible and just involve providing a function to some asynchronous task in order to have your provided function be called whenever the task is complete.
For example:
/**
* Some dummy asynchronous task that waits 2 seconds to complete
*/
function asynchronousTask(cb) {
setTimeout(() => {
console.log("Async task is done");
cb();
}, 2000);
}
asynchronousTask(() => {
console.log("My function to be called after async task");
});
Promises are a primitive that encapsulates the callback pattern so that instead of providing a function to the task function, you call the then method on the Promise that the task returns:
/**
* Some dummy asynchronous task that waits 2 seconds to complete
* BUT the difference is that it returns a Promise
*/
function asynchronousTask() {
return new Promise(resolve => {
setTimeout(() => {
console.log("Async task is done");
resolve();
}, 2000);
});
}
asynchronousTask()
.then(() => {
console.log("My function to be called after async task");
});
The last pattern is the async/await pattern which also deals in Promises which are an encapsulation of callbacks. They are unique because they provide lexical support for using Promises so that you don't have to use .then() directly and also don't have to explicitly return a Promise from your task:
/*
* We still need some Promise oriented bootstrap
* function to demonstrate the async/await
* this will just wait a duration and resolve
*/
function $timeout(duration) {
return new Promise(resolve => setTimeout(resolve, duration));
}
/**
* Task runner that waits 2 seconds and then prints a message
*/
(async function() {
await $timeout(2000);
console.log("My function to be called after async task");
}());
Now that our vocabulary is cleared up, we need to consider one other thing: These patterns are all API dependent. The library that you are using uses callbacks. It is alright to mix these patterns, but I would say that the code that you write should be consistent. Pick one of the patterns and wrap or interface with the library that you need to.
If the library deals in callbacks, see if there is a wrapping library or a mechanism to have it deal in Promises instead. async/await consumes Promises, but not callbacks.

Callbacks are fine, but I would only use them if a function is dependent on some asynchronous result. If however the result is immediately available, then the function should be designed to return that value.
In the example you have given, say does not have to wait for any asynchronous API call to come back with a result, so I would change its signature to the following:
say: function(data){ // <--- no callback argument
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
message = ('you say '+'"'+message.trim()+'"');
response.target = data.user;
response.type = 'echo';
response.message = message;
return response; // <--- return it
}
Then going backwards, you would also change the signature of the functions that use say:
executeVerb: function(verb, data){ // <--- no callback argument
return verb_library[verb](data); // <--- no callback argument, and return the returned value
}
And further up the call stack:
socket.on('rg_command', function(command){
// execute the verb
let verb = rogueParser(command);
let result = rogue.executeVerb(verb, command); // <--- no callback, just get the returned value
console.log(result);
});
Of course, this can only work if all verb methods can return the expected result synchronously.
Promises
If say would depend on some asynchronous API, then you could use promises. Let's assume this API provides a callback system, then your say function could return a promise like this:
say: async function(data){ // <--- still no callback argument, but async!
var response = {};
console.log('USR:'+data.user);
var message = data.message.replace('say','');
response.target = data.user;
response.type = 'echo';
// Convert the API callback system to a promise, and use AWAIT
await respone.message = new Promise(resolve => someAsyncAPIWithCallBackAsLastArg(message, resolve));
return response; // <--- return it
}
Again going backwards, you would also change the signature of the functions that use say:
executeVerb: function(verb, data){ // <--- still no callback argument
return verb_library[verb](data); // <--- no callback argument, and return the returned promise(!)
}
And finally:
socket.on('rg_command', async function(command){ // Add async
// execute the verb
let verb = rogueParser(command);
let result = await rogue.executeVerb(verb, command); // <--- await the fulfillment of the returned promise
console.log(result);
});

Is there a way to return a value with async/await instead of a Promise ? As a synchronous function do

Firstly I am familiar with the concept of asynchronous/synchronous function.
There is also a lot of questions related to mine. But I can't find my answer anywhere.
So the question is:
Is there a way to return a value instead of a Promise using async/await ? As a synchronous function do.
For example:
async doStuff(param) {
return await new Promise((resolve, reject) => {
setTimeout(() => {
console.log('doStuff after a while.');
resolve('mystuffisdone'+param);
}, 2000);
});
}
console.log(doStuff('1'));
The only way to get the value of this function is by using the .then function.
doStuff('1').then(response => {
console.log(response); // output: mystuffisdone1
doOtherStuffWithMyResponse(response);
// ...
});
Now, what I want is:
const one = doStuff('1');
console.log(one) // mystuffisdone1
const two = doStuff('2');
console.log(two) // mystuffisdone2
To explain myself, I have an asynchronous library full of callbacks. I can turn this asynchronous behavior to a synchronous behavior by using Promises and async/await to faking a synchronous behavior.
But there is still a problem, it is still asynchronous in the end; outside of the scope of the async function.
doStuff('1').then((r) => {console.log(r)};
console.log('Hello wolrd');
It will result in: Hello world then mystuffisdone1. This is the expected behavior when using async/await functions. But that's not what I want.
Now my question would be: Is there a way to do the same thing as await do without the keyword async ? To make the code being synchronous ? And if not possible, why ?
Edit:
Thank you for all you answers, I think my question is not obsvious for all. To clear up what I think here is my comment to #Nikita Isaev answer.
"I understand why all I/O operations are asynchronously done; or done in parallel. But my question is more about the fact that why the engine doesn't block the caller of the sync function in an asynchronous manner ? I mean const a = doStuff(...) is a Promise. We need to call .then to get the result of this function. But why JavaScript or Node engine does not block the caller (just the block where the call is made). If this is possible, we could do const a = doStuff(...), wait and get the result in a without blocking the main thread. As async/await does, why there is no place for sync/wait ?"
Hope this is more clear now, feel free to comment or ask anything :)
Edit 2:
All precisions of the why of the answer are in the comments of the accepted answer.

There are some hacky ways to do what is desired, but that would be an anti-pattern. I’ll try to explain. Callbacks is one of the core concepts in javascript. When your code launches, you may set up event listeners, timers, etc. You just tell the engine to schedule some tasks: “when A happens, do B”. This is what asynchrony is. But callbacks are ugly and difficult to debug, that’s why promises and async-await were introduced. It is important to understand that this is just a syntax sugar, your code still is asynchronous when using async-await. As there are no threads in javascript, waiting for some events to fire or some complicated operations to finish in a synchronous way would block your entire application. The UI or the server would just stop responding to any other user interactions and would keep waiting for a single event to fire.
Real world cases:
Example 1.
Let’s say we have a web UI. We have a button that downloads the latest information from the server on click. Imagine we do it synchronously. What happens?
myButton.onclick = function () {
const data = loadSomeDataSync(); // 0
useDataSomehow(data);
}
Everything’s synchronous, the code is flat and we are happy. But the user is not.
A javascript process can only ever execute a single line of code in a particular moment. User will not be able to click other buttons, see any animations etc, the app is stuck waiting for loadSomeDataSync() to finish. Even if this lasts 3 seconds, it’s a terrible user experience, you can neither cancel nor see the progress nor do something else.
Example 2.
We have a node.js http server which has over 1 million users. For each user, we need to execute a heavy operation that lasts 5 seconds and return the result. We can do it in a synchronous or asynchronous manner. What happens if we do it in async?
User 1 connects
We start execution of heavy operation for user 1
User 2 connects
We return data for user 1
We start execution of heavy operation for user 2
…
I.e we do everything in parallel and asap. Now imagine we do the heavy operation in a sync manner.
User 1 connects
We start execution of heavy operation for user 1, everyone else is waiting for it to accomplish
We return data for user 1
User 2 connects
…
Now imagine the heavy operation takes 5 seconds to accomplish, and our server is under high load, it has over 1 million users. The last one will have to wait for nearly 5 million seconds, which is definitely not ok.
That’s why:
In browser and server API, most of the i/o operations are asynchronous
Developers strive to make all heavy calculation asynchronous, even React renders in an asynchronous manner.

No, going from promise to async/await will not get you from async code to sync code. Why? Because both are just different wrapping for the same thing. Async function returns immediately just like a promise does.
You would need to prevent the Event Loop from going to next call. Simple while(!isMyPromiseResolved){} will not work either because it will also block callback from promises so the isMyPromiseResolved flag will never be set.
BUT... There are ways to achieve what you have described without async/await. For example:
OPTION 1: using deasync approach. Example:
function runSync(value) {
let isDone = false;
let result = null;
runAsync(value)
.then(res => {
result = res;
isDone = true;
})
.catch(err => {
result = err;
isDone = true;
})
//magic happens here
require('deasync').loopWhile(function(){return !isDone;});
return result;
}
runAsync = (value) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
// if passed value is 1 then it is a success
if(value == 1){
resolve('**success**');
}else if (value == 2){
reject('**error**');
}
}, 1000);
});
}
console.log('runSync(2): ', runSync(2));
console.log('runSync(1): ', runSync(1));
OR
OPTION 2: calling execFileSync('node yourScript.js') Example:
const {execFileSync} = require('child_process');
execFileSync('node',['yourScript.js']);
Both approaches will block the user thread so they should be used only for automation scripts or similar purposes.

Wrap the outer body in an asynchronous IIFE:
/**/(async()=>{
function doStuff(param) { // no need for this one to be async
return new Promise((resolve, reject) => { // just return the original promise
setTimeout(() => {
console.log('doStuff after a while.');
resolve('mystuffisdone'+param);
}, 2000);
});
}
console.log(await doStuff('1')); // and await instead of .then
/**/})().then(()=>{}).catch(e=>console.log(e))
The extra cleanup of the doStuff function isn't strictly necessary -- it works either way -- but I hope it helps clarify how async functions and Promises are related. The important part is to wrap the outer body into an async function to get the improved semantics throughout your program.
It's also not strictly necessary to have the final .then and .catch, but it's good practice. Otherwise, your errors might get swallowed, and any code ported to Node will whine about uncaught Promise rejections.

Better way to "loop" promises

This is a post that might come across as quite conceptual, since I first start with a lot of pseudo code. - At the end you'll see the use case for this problem, though a solution would be a "tool I can add to my tool-belt of useful programming techniques".
The problem
Sometimes one might need to create multiple promises, and either do something after all promises have ended. Or one might create multiple promises, based on the results of the previous promises. The analogy can be made to creating an array of values instead of a single value.
There are two basic cases to be considered, where the number of promises is indepedented of the result of said promises, and the case where it is depedent. Simple pseudo code of what "could" be done.
for (let i=0; i<10; i++) {
promise(...)
.then(...)
.catch(...);
}.then(new function(result) {
//All promises finished execute this code now.
})
The basically creates n (10) promises, and the final code would be executed after all promises are done. Of course the syntax isn't working in javascript, but it shows the idea. This problem is relativelly easy, and could be called completely asynchronous.
Now the second problem is like:
while (continueFn()) {
promise(...)
.then(.. potentially changing outcome of continueFn ..)
.catch(.. potentially changing outcome of continueFn ..)
}.then(new function(result) {
//All promises finished execute this code now.
})
This is much more complex, as one can't just start all promises and then wait for them to finish: in the end you'll have to go "promise-by-promise". This second case is what I wish to figure out (if one can do the second case you can also do the first).
The (bad) solution
I do have a working "solution". This is not a good solution as can probably quickly be seen, after the code I'll talk about why I dislike this method. Basically instead of looping it uses recursion - so the "promise" (or a wrapper around a promise which is a promise) calls itself when it's fulfilled, in code:
function promiseFunction(state_obj) {
return new Promise((resolve, reject) => {
//initialize fields here
let InnerFn = (stateObj) => {
if (!stateObj.checkContinue()) {
return resolve(state_obj);
}
ActualPromise(...)
.then(new function(result) {
newState = stateObj.cloneMe(); //we'll have to clone to prevent asynchronous write problems
newState.changeStateBasedOnResult(result);
return InnerFn(newState);
})
.catch(new function(err) {
return reject(err); //forward error handling (must be done manually?)
});
}
InnerFn(initialState); //kickstart
});
}
Important to note is that the stateObj should not change during its lifetime, but it can be really easy. In my real problem (which I'll explain at the end) the stateObj was simply a counter (number), and the if (!stateObj.checkContinue()) was simply if (counter < maxNumber).
Now this solution is really bad; It is ugly, complicated, error prone and finally impossible to scale.
Ugly because the actual business logic is buried in a mess of code. It doesn't show "on the can" that is actually simply doing what the while loop above does.
Complicated because the flow of execution is impossible to follow. First of all recursive code is never "easy" to follow, but more importantly you also have to keep in mind thread safety with the state-object. (Which might also have a reference to another object to, say, store a list of results for later processing).
It's error prone since there is more redundancy than strictly necessary; You'll have to explicitly forward the rejection. Debugging tools such as a stack trace also quickly become really hard to look through.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth. Normally one could either optimize by tail recursion or, more common, create a virtual stack (on the heap) and transform the function to a loop using the manual stack. In this case, however, one can't change the recursive calls to a loop-with-manual-stack; simply because of how promise syntax works.
The alternative (bad) solution
A colleague suggested an alternative approach to this problem, something that initially looked much less problematic, but I discarded ultimatelly since it was against everything promises are meant to do.
What he suggested was basically looping over the promises as per above. But instead of letting the loop continue there would be a variable "finished" and an inner loop that constantly checks for this variable; so in code it would be like:
function promiseFunction(state_obj) {
return new Promise((resolve, reject) => {
while (stateObj.checkContinue()) {
let finished = false;
let err = false;
let res = null;
actualPromise(...)
.then(new function(result) {
res = result;
finished = true;
})
.catch(new function(err) {
res = err;
err = true;
finished = true;
});
while(!finished) {
sleep(100); //to not burn our cpu
}
if (err) {
return reject(err);
}
stateObj.changeStateBasedOnResult(result);
}
});
}
While this is less complicated, since it's now easy to follow the flow of execution. This has problems of its own: not for the least that it's unclear when this function will end; and it's really bad for performance.
Conclusion
Well this isn't much to conclude yet, I'd really like something as simple as in the first pseudo code above. Maybe another way of looking at things so that one doesn't have the trouble of deeply recursive functions.
So how would you rewrite a promise that is part of a loop?
The real problem used as motivation
Now this problem has roots in a real thing I had to create. While this problem is now solved (by applying the recursive method above), it might be interesting to know what spawned this; The real question however isn't about this specific case, but rather on how to do this in general with any promise.
In a sails app I had to check a database, which had orders with order-ids. I had to find the first N "non existing order-ids". My solution was to get the "first" M products from the database, find the missing numbers within it. Then if the number of missing numbers was less than N get the next batch of M products.
Now to get an item from a database, one uses a promise (or callback), thus the code won't wait for the database data to return. - So I'm basically at the "second problem:"
function GenerateEmptySpots(maxNum) {
return new Promise((resolve, reject) => {
//initialize fields
let InnerFn = (counter, r) => {
if (r > 0) {
return resolve(true);
}
let query = {
orderNr: {'>=': counter, '<': (counter + maxNum)}
};
Order.find({
where: query,
sort: 'orderNr ASC'})
.then(new function(result) {
n = findNumberOfMissingSpotsAndStoreThemInThis();
return InnerFn(newState, r - n);
}.bind(this))
.catch(new function(err) {
return reject(err);
});
}
InnerFn(maxNum); //kickstart
});
}
EDIT:
Small post scriptus: the sleep function in the alternative is just from another library which provided a non-blocking-sleep. (not that it matters).
Also, should've indicated I'm using es2015.

The alternative (bad) solution
…doesn't actually work, as there is no sleep function in JavaScript. (If you have a runtime library which provides a non-blocking-sleep, you could just have used a while loop and non-blocking-wait for the promise inside it using the same style).
The bad solution is ugly, complicated, error prone and finally impossible to scale.
Nope. The recursive approach is indeed the proper way to do this.
Ugly because the actual business logic is buried in a mess of code. And error-prone as you'll have to explicitly forward the rejection.
This is just caused by the Promise constructor antipattern! Avoid it.
Complicated because the flow of execution is impossible to follow. Recursive code is never "easy" to follow
I'll challenge that statement. You just have to get accustomed to it.
You also have to keep in mind thread safety with the state-object.
No. There is no multi-threading and shared memory access in JavaScript, if you worry about concurrency where other stuff affects your state object while the loop runs that will a problem with any approach.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow
No. It's asynchronous! The callback will run on a new stack, it's not actually called recursively during the function call and does not carry those stack frames around. The asynchronous event loop already provides the trampoline to make this tail-recursive.
The good solution
function promiseFunction(state) {
const initialState = state.cloneMe(); // clone once for this run
// initialize fields here
return (function recurse(localState) {
if (!localState.checkContinue())
return Promise.resolve(localState);
else
return actualPromise(…).then(result =>
recurse(localState.changeStateBasedOnResult(result))
);
}(initialState)); // kickstart
}
The modern solution
You know, async/await is available in every environment that implemented ES6, as all of them also implemented ES8 now!
async function promiseFunction(state) {
const localState = state.cloneMe(); // clone once for this run
// initialize fields here
while (!localState.checkContinue()) {
const result = await actualPromise(…);
localState = localState.changeStateBasedOnResult(result);
}
return localState;
}

Let’s begin with the simple case: You have N promises that all do some work, and you want to do something when all the promises have finished. There’s actually a built-in way to do exactly that: Promise.all. With that, the code will look like this:
let promises = [];
for (let i=0; i<10; i++) {
promises.push(doSomethingAsynchronously());
}
Promise.all(promises).then(arrayOfResults => {
// all promises finished
});
Now, the second call is a situation you encounter all the time when you want to continue doing something asynchronously depending on the previous asynchronous result. A common example (that’s a bit less abstract) would be to simply fetch pages until you hit the end.
With modern JavaScript, there’s luckily a way to write this in a really readable way: Using asynchronous functions and await:
async function readFromAllPages() {
let shouldContinue = true;
let pageId = 0;
let items = [];
while (shouldContinue) {
// fetch the next page
let result = await fetchSinglePage(pageId);
// store items
items.push.apply(items, result.items);
// evaluate whether we want to continue
if (!result.items.length) {
shouldContinue = false;
}
pageId++;
}
return items;
}
readFromAllPages().then(allItems => {
// items have been read from all pages
});
Without async/await, this will look a bit more complicated, since you need to manage all this yourself. But unless you try to make it super generic, it shouldn’t look that bad. For example, the paging one could look like this:
function readFromAllPages() {
let items = [];
function readNextPage(pageId) {
return fetchSinglePage(pageId).then(result => {
items.push.apply(items, result.items);
if (!result.items.length) {
return Promise.resolve(null);
}
return readNextPage(pageId + 1);
});
}
return readNextPage(0).then(() => items);
}
First of all recursive code is never "easy" to follow
I think the code is fine to read. As I’ve said: Unless you try to make it super generic, you can really keep it simple. And naming also helps a lot.
but more importantly you also have to keep in mind thread safety with the state-object
No, JavaScript is single-threaded. You doing things asynchronously but that does not necessarily mean that things are happening at the same time. JavaScript uses an event loop to work off asynchronous processes, where only one code block runs at a single time.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth.
Also no. This is recursive in the sense that the function references itself. But it will not call itself directly. Instead it will register itself as a callback when an asynchronous process finishes. So the current execution of the function will finish first, then at some point the asynchronous process finishes, and then the callback will eventually run. These are (at least) three separate steps from the event loop, which all run independently from another, so you do no have a problem with recursion depth here.

The crux of the matter seems to be that "the actual business logic is buried in a mess of code".
Yes it is ... in both solutions.
Things can be separated out by :
having an asyncRecursor function that simply knows how to (asynchronously) recurse.
allowing the recursor's caller(s) to specify the business logic (the terminal test to apply, and the work to be performed).
It is also better to allow caller(s) to be responsible for cloning the original object rather than resolver() assuming cloning always to be necessary. The caller really needs to be in charge in this regard.
function asyncRecursor(subject, testFn, workFn) {
// asyncRecursor orchestrates the recursion
if(testFn(subject)) {
return Promise.resolve(workFn(subject)).then(result => asyncRecursor(result, testFn, workFn));
// the `Promise.resolve()` wrapper safeguards against workFn() not being thenable.
} else {
return Promise.resolve(subject);
// the `Promise.resolve()` wrapper safeguards against `testFn(subject)` failing at the first call of asyncRecursor().
}
}
Now you can write your caller as follows :
// example caller
function someBusinessOrientedCallerFn(state_obj) {
// ... preamble ...
return asyncRecursor(
state_obj, // or state_obj.cloneMe() if necessary
(obj) => obj.checkContinue(), // testFn
(obj) => somethingAsync(...).then((result) => { // workFn
obj.changeStateBasedOnResult(result);
return obj; // return `obj` or anything you like providing it makes a valid parameter to be passed to `testFn()` and `workFn()` at next recursion.
});
);
}
You could theoretically incorporate your terminal test inside the workFn but keeping them separate will help enforce the discipline, in writers of the business-logic, to remember to include a test. Otherwise they will consider it optional and sure as you like, they will leave it out!

Sorry, this doesn't use Promises, but sometimes abstractions just get in the way.
This example, which builds from #poke's answer, is short and easy to comprehend.
function readFromAllPages(done=function(){}, pageId=0, res=[]) {
fetchSinglePage(pageId, res => {
if (res.items.length) {
readFromAllPages(done, ++pageId, items.concat(res.items));
} else {
done(items);
}
});
}
readFromAllPages(allItems => {
// items have been read from all pages
});
This has only a single depth of nested functions. In general, you can solve the nested callback problem without resorting to a subsystem that manages things for you.
If we drop the parameter defaults and change the arrow functions, we get code that runs in legacy ES3 browsers.

Accomplish Promise Chaining

Background
I have a nodejs server running and I installed the promise package which follows the promise api specs.
Since I succeeded in making denodeify(fn, length) work, I am now in the process of chaining promises, but I am failing to grasp the main concepts.
What I tried
By reading the documentation example on the specification page, I reached the following code:
let Promise = require("promise");
let write = Promise.denodeify(jsonfile.writeFile);
let read = Promise.denodeify(jsonfile.readFile);
read("dataFile.txt").then( () => {
write("./testFile.txt", "hello_World", TABS_FORMATTING).then(console.log("file complete"));
});
Which is quite different from the examples I see, for example, in the Solutions Optimist tutorial:
loadDeparture( user )
.then( loadFlight )
.then( loadForecast );
Objective
My objective is to make my code as beautiful as the example I showed before, but I don't understand how I can make as concise as it is right now.
Question
1 - What changes do I need to perform in my code to achieve that level?

The given example uses named function to make it look as good as it can get, but that can be a bit redundant because then you're creating functions for every little thing in the chain. You must pick and choose when to use named functions over anonymous functions.
One thing you must also realize is that to chain promises you must return them.
So to make it a proper chain you must return the write method so it is passed down to the next step.
Also make sure that the catch() method is used at the bottom of every promise chain so that errors aren't silently swallowed.
Note that in the example here I'm using the ES2015 arrow functions to return the write() method as that makes it looks better(which seemed to be the purpose of your question).
let Promise = require("promise");
let write = Promise.denodeify(jsonfile.writeFile);
let read = Promise.denodeify(jsonfile.readFile);
read("dataFile.txt")
.then(() => write("./testFile.txt", "hello_World", TABS_FORMATTING))
.then(results => console.log("file complete", results))
.catch(error => console.error(err));
I'd recommend reading this article for some best practices.

Nesting promises kind of defeats the purpose because it creates pyramid code (just like callbacks).
The main concept that may be escaping you is that you can return inside a then and the returned value (can be a promise or a value) can then be accessed in a chained then:
read("dataFile.txt").then( () => {
return write("./testFile.txt", "hello_World", TABS_FORMATTING);
}).then( () => {
console.log("file complete");
});
Now, you can extract the functions:
function writeTheFile() {
return write("./testFile.txt", "hello_World", TABS_FORMATTING);
}
function consoleLog() {
console.log("file complete");
}
read("dataFile.txt")
.then(writeTheFile)
.then(consoleLog);

What is "callback hell" and how and why does RX solve it?

Can someone give a clear definition together with a simple example that explains what is a "callback hell" for someone who does not know JavaScript and node.js ?
When (in what kind of settings) does the "callback hell problem" occur?
Why does it occur?
Is "callback hell" always related to asynchronous computations?
Or can "callback hell" occur also in a single threaded application?
I took the Reactive Course at Coursera and Erik Meijer said in one of his lectures that RX solves the problem of "callback hell". I asked what is a "callback hell" on the Coursera forum but I got no clear answer.
After explaining "callback hell" on a simple example, could you also show how RX solves the "callback hell problem" on that simple example?

1) What is a "callback hell" for someone who does not know javascript and node.js ?
This other question has some examples of Javascript callback hell: How to avoid long nesting of asynchronous functions in Node.js
The problem in Javascript is that the only way to "freeze" a computation and have the "rest of it" execute latter (asynchronously) is to put "the rest of it" inside a callback.
For example, say I want to run code that looks like this:
x = getData();
y = getMoreData(x);
z = getMoreData(y);
...
What happens if now I want to make the getData functions asynchronous, meaning that I get a chance to run some other code while I am waiting for them to return their values? In Javascript, the only way would be to rewrite everything that touches an async computation using continuation passing style:
getData(function(x){
getMoreData(x, function(y){
getMoreData(y, function(z){
...
});
});
});
I don't think I need to convince anyone that this version is uglier than the previous one. :-)
2) When (in what kind of settings) does the "callback hell problem" occur?
When you have lots of callback functions in your code! It gets harder to work with them the more of them you have in your code and it gets particularly bad when you need to do loops, try-catch blocks and things like that.
For example, as far as I know, in JavaScript the only way to execute a series of asynchronous functions where one is run after the previous returns is using a recursive function. You can't use a for loop.
// we would like to write the following
for(var i=0; i<10; i++){
doSomething(i);
}
blah();
Instead, we might need to end up writing:
function loop(i, onDone){
if(i >= 10){
onDone()
}else{
doSomething(i, function(){
loop(i+1, onDone);
});
}
}
loop(0, function(){
blah();
});
//ugh!
The number of questions we get here on StackOverflow asking how to do this kind of thing is a testament to how confusing it is :)
3) Why does it occur ?
It occurs because in JavaScript the only way to delay a computation so that it runs after the asynchronous call returns is to put the delayed code inside a callback function. You cannot delay code that was written in traditional synchronous style so you end up with nested callbacks everywhere.
4) Or can "callback hell" occur also in a single threaded application?
Asynchronous programming has to do with concurrency while a single-thread has to do with parallelism. The two concepts are actually not the same thing.
You can still have concurrent code in a single threaded context. In fact, JavaScript, the queen of callback hell, is single threaded.
What is the difference between concurrency and parallelism?
5) could you please also show how RX solves the "callback hell problem" on that simple example.
I don't know anything about RX in particular, but usually this problem gets solved by adding native support for asynchronous computation in the programming language. The implementations can vary and include: async, generators, coroutines, and callcc.
In Python we can implement that previous loop example with something along the lines of:
def myLoop():
for i in range(10):
doSomething(i)
yield
myGen = myLoop()
This is not the full code but the idea is that the "yield" pauses our for loop until someone calls myGen.next(). The important thing is that we could still write the code using a for loop, without needing to turn out logic "inside out" like we had to do in that recursive loop function.

To address the question of how Rx solves callback hell:
First let's describe callback hell again.
Imagine a case were we must do http to get three resources - person, planet and galaxy. Our objective is to find the galaxy the person lives in. First we must get the person, then the planet, then the galaxy. That's three callbacks for three asynchronous operations.
getPerson(person => {
getPlanet(person, (planet) => {
getGalaxy(planet, (galaxy) => {
console.log(galaxy);
});
});
});
Each callback is nested. Each inner callback is dependent on its parent. This leads to the "pyramid of doom" style of callback hell. The code looks like a > sign.
To solve this in RxJs you could do something like so:
getPerson()
.map(person => getPlanet(person))
.map(planet => getGalaxy(planet))
.mergeAll()
.subscribe(galaxy => console.log(galaxy));
With the mergeMap AKA flatMap operator you could make it more succinct:
getPerson()
.mergeMap(person => getPlanet(person))
.mergeMap(planet => getGalaxy(planet))
.subscribe(galaxy => console.log(galaxy));
As you can see, the code is flattened and contains a single chain of method calls. We have no "pyramid of doom".
Hence, callback hell is avoided.
In case you were wondering, promises are another way to avoid callback hell, but promises are eager, not lazy like observables and (generally speaking) you cannot cancel them as easily.

Just answer the question: could you please also show how RX solves the "callback hell problem" on that simple example?
The magic is flatMap. We can write the following code in Rx for #hugomg's example:
def getData() = Observable[X]
getData().flatMap(x -> Observable[Y])
.flatMap(y -> Observable[Z])
.map(z -> ...)...
It's like you are writing some synchronous FP codes, but actually you can make them asynchronous by Scheduler.

Callback hell is any code where the use of function callbacks in async code becomes obscure or difficult to follow. Generally, when there is more than one level of indirection, code using callbacks can become harder to follow, harder to refactor, and harder to test. A code smell is multiple levels of indentation due to passing multiple layers of function literals.
This often happens when behaviour has dependencies, i.e. when A must happen before B must happen before C. Then you get code like this:
a({
parameter : someParameter,
callback : function() {
b({
parameter : someOtherParameter,
callback : function({
c(yetAnotherParameter)
})
}
});
If you have lots of behavioural dependencies in your code like this, it can get troublesome fast. Especially if it branches...
a({
parameter : someParameter,
callback : function(status) {
if (status == states.SUCCESS) {
b(function(status) {
if (status == states.SUCCESS) {
c(function(status){
if (status == states.SUCCESS) {
// Not an exaggeration. I have seen
// code that looks like this regularly.
}
});
}
});
} elseif (status == states.PENDING {
...
}
}
});
This won't do. How can we make asynchronous code execute in a determined order without having to pass all these callbacks around?
RX is short for 'reactive extensions'. I haven't used it, but Googling suggests it's an event-based framework, which makes sense. Events are a common pattern to make code execute in order without creating brittle coupling. You can make C listen to the event 'bFinished' which only happens after B is called listening to 'aFinished'. You can then easily add extra steps or extend this kind of behaviour, and can easily test that your code executes in order by merely broadcasting events in your test case.

Call back hell means you are inside of a callback of inside another callback and it goes to nth call until your needs not fullfiled.
Let's understand through an example of fake ajax call by using set timeout API, lets assume we have a recipe API, we need to download all recipe.
<body>
<script>
function getRecipe(){
setTimeout(()=>{
const recipeId = [83938, 73838, 7638];
console.log(recipeId);
}, 1500);
}
getRecipe();
</script>
</body>
In the above example after 1.5 sec when timer expires inside code of call back will execute, in other words, through our fake ajax call all recipe will downloaded from the server. Now we need to download a particular recipe data.
<body>
<script>
function getRecipe(){
setTimeout(()=>{
const recipeId = [83938, 73838, 7638];
console.log(recipeId);
setTimeout(id=>{
const recipe = {title:'Fresh Apple Juice', publisher:'Suru'};
console.log(`${id}: ${recipe.title}`);
}, 1500, recipeId[2])
}, 1500);
}
getRecipe();
</script>
</body>
To download a particular recipe data we wrote code inside of our first callback and passed recipe Id.
Now let's say we need to download all the recipes of the same publisher of the recipe which id is 7638.
<body>
<script>
function getRecipe(){
setTimeout(()=>{
const recipeId = [83938, 73838, 7638];
console.log(recipeId);
setTimeout(id=>{
const recipe = {title:'Fresh Apple Juice', publisher:'Suru'};
console.log(`${id}: ${recipe.title}`);
setTimeout(publisher=>{
const recipe2 = {title:'Fresh Apple Pie', publisher:'Suru'};
console.log(recipe2);
}, 1500, recipe.publisher);
}, 1500, recipeId[2])
}, 1500);
}
getRecipe();
</script>
</body>
To full-fill our needs which is to download all the recipes of publisher name suru, we wrote code inside of our second call back. It is clear we wrote a callback chain which is called callback hell.
If you want to avoid callback hell, you can use Promise, which is js es6 feature, each promise takes a callback which is called when a promise is full-filled. promise callback has two options either it is resolved or reject. Suppose your API call is successful you can call resolve and pass data through the resolve, you can get this data by using then(). But if your API failed you can use reject, use catch to catch the error. Remember a promise always use then for resolve and catch for reject
Let's solve the previous callback hell problem using a promise.
<body>
<script>
const getIds = new Promise((resolve, reject)=>{
setTimeout(()=>{
const downloadSuccessfull = true;
const recipeId = [83938, 73838, 7638];
if(downloadSuccessfull){
resolve(recipeId);
}else{
reject('download failed 404');
}
}, 1500);
});
getIds.then(IDs=>{
console.log(IDs);
}).catch(error=>{
console.log(error);
});
</script>
</body>
Now download particular recipe:
<body>
<script>
const getIds = new Promise((resolve, reject)=>{
setTimeout(()=>{
const downloadSuccessfull = true;
const recipeId = [83938, 73838, 7638];
if(downloadSuccessfull){
resolve(recipeId);
}else{
reject('download failed 404');
}
}, 1500);
});
const getRecipe = recID => {
return new Promise((resolve, reject)=>{
setTimeout(id => {
const downloadSuccessfull = true;
if (downloadSuccessfull){
const recipe = {title:'Fresh Apple Juice', publisher:'Suru'};
resolve(`${id}: ${recipe.title}`);
}else{
reject(`${id}: recipe download failed 404`);
}
}, 1500, recID)
})
}
getIds.then(IDs=>{
console.log(IDs);
return getRecipe(IDs[2]);
}).
then(recipe =>{
console.log(recipe);
})
.catch(error=>{
console.log(error);
});
</script>
</body>
Now we can write another method call allRecipeOfAPublisher like getRecipe which will also return a promise, and we can write another then() to receive resolve promise for allRecipeOfAPublisher, I hope at this point you can do it by yourself.
So we learned how to construct and consumed promises, now let's make consuming a promise easier by using async/await which is introduced in es8.
<body>
<script>
const getIds = new Promise((resolve, reject)=>{
setTimeout(()=>{
const downloadSuccessfull = true;
const recipeId = [83938, 73838, 7638];
if(downloadSuccessfull){
resolve(recipeId);
}else{
reject('download failed 404');
}
}, 1500);
});
const getRecipe = recID => {
return new Promise((resolve, reject)=>{
setTimeout(id => {
const downloadSuccessfull = true;
if (downloadSuccessfull){
const recipe = {title:'Fresh Apple Juice', publisher:'Suru'};
resolve(`${id}: ${recipe.title}`);
}else{
reject(`${id}: recipe download failed 404`);
}
}, 1500, recID)
})
}
async function getRecipesAw(){
const IDs = await getIds;
console.log(IDs);
const recipe = await getRecipe(IDs[2]);
console.log(recipe);
}
getRecipesAw();
</script>
</body>
In the above example, we used an async function because it will run in the background, inside async function we used await keyword before each method which returns or is a promise because to wait on that position until that promise fulfilled, in other words in the bellow codes until getIds completed resolved or reject program will stop executing codes bellow that line when IDs returned then we again called getRecipe() function with a id and waited by using await keyword until data returned. So this is how finally we recovered from the callback hell.
async function getRecipesAw(){
const IDs = await getIds;
console.log(IDs);
const recipe = await getRecipe(IDs[2]);
console.log(recipe);
}
To use await we will need a async function, we can return a promise so use then for resolve promise and cath for reject promise
from the above example:
async function getRecipesAw(){
const IDs = await getIds;
const recipe = await getRecipe(IDs[2]);
return recipe;
}
getRecipesAw().then(result=>{
console.log(result);
}).catch(error=>{
console.log(error);
});

One way Callback hell can be avoided is to use FRP which is an "enhanced version" of RX.
I started to use FRP recently because I have found a good implementation of it called Sodium ( http://sodium.nz/ ).
A typical code looks like this ( Scala.js ) :
def render: Unit => VdomElement = { _ =>
<.div(
<.hr,
<.h2("Note Selector"),
<.hr,
<.br,
noteSelectorTable.comp(),
NoteCreatorWidget().createNewNoteButton.comp(),
NoteEditorWidget(selectedNote.updates()).comp(),
<.hr,
<.br
)
}
selectedNote.updates() is a Stream which fires if selectedNode (which is a Cell) changes, the NodeEditorWidget then updates correspondingly.
So, depending on the content of the selectedNode Cell, the currently edited Note will change.
This code avoids Callback-s entirely, almost, Cacllback-s are pushed to the "outer layer"/"surface" of the app, where the state handling logic interfaces with the external world. There are no Callbacks needed to propagate data within the internal state handling logic (which implements a state machine).
The full source code is here
The code snippet above corrosponds to the following simple Create / Display / Update example :
This code also sends updates to the server, so changes to the updated Entities are saved to the server automatically.
All the event handling is taken care by using Streams and Cells. These are FRP concepts. Callbacks are only needed where the FRP logic interfaces with the external world, such as user input, editing text, pressing a button, AJAX call returns.
Data flow is explicitly described, in a declarative manner using FRP (implemented by the Sodium library), so no event handling / callback logic is needed to describe data flow.
FRP (which is a more "strict" version of RX) is a way to describe a data flow graph, which can contain nodes that contain state. Events trigger state changes in the state containing nodes (called Cells).
Sodium is a higher order FRP library, meaning that using the flatMap/switch primitive one can rearrange the data flow graph at runtime.
I recommend to have a look into the Sodium book, it explains in detail how FRP gets rid of all Callbacks which are not essential for describing dataflow logic that has to do with updating the applications state in response to some external stimuli.
Using FRP, only those Callbacks need to be kept which describe interaction with the external world. In other words, the dataflow is described in a functional / declarative manner when one uses an FRP framework (such as Sodium), or when one uses an "FRP like" framework (such as RX).
Sodium is also available for Javascript/Typescript.

Use jazz.js
https://github.com/Javanile/Jazz.js
it simplify like this:
// run sequential task chained
jj.script([
// first task
function(next) {
// at end of this process 'next' point to second task and run it
callAsyncProcess1(next);
},
// second task
function(next) {
// at end of this process 'next' point to thirt task and run it
callAsyncProcess2(next);
},
// thirt task
function(next) {
// at end of this process 'next' point to (if have)
callAsyncProcess3(next);
},
]);

If you don't have a knowledge about callback and hell callback there is no problem.Ist thing is that call back and call back hell.For example:hell call back is like a we can store a class inside a class.As you heard about that nested in C, C++ language.Nested Means that a class inside a another class.