Develop Reference

Javascript, Node, Promises, and recursion

I'm having trouble controlling execution flow. This is a follow-on to node.js, bluebird, poor control of execution path and node.js table search fails with promises in use. Judging by console.log print-outs, my recursive routine works great, except that the first call to resolve() (a signal to the nth recursive call) gives the green light to follow-on code that shouldn't get that green light until the first call to the recursive routine calls resolve(). It turns out the first call to the recursive routine delivers the answer I want reported, but by the time it reports it, the follow-on code is no longer listening for it and is running blissfully along with an "undefined" answer. Bad.
My code is much to long to share here. I tried to write a small model of the problem, but haven't found the combination of factors to replicate the behavior.
Sound familiar? How do you keep proper control over Promises releasing follow-on code on time?
I thought maybe the first call to the routine could start an array passed into a Promise.all and later calls would add another entry to that array. I haven't tried it. Crazy?

Without seeing your actual code, we can't answer specifically.
Sound familiar? How do you keep proper control over Promises releasing
follow-on code on time?
The answer is always to not resolve the first promise in the chain until you're ready for things to execute and to structure your promise chain so that dependent things don't get executed until the things they are waiting on have been properly resolved. If something is executing too soon, then you're either calling something too soon or your promise structure is not correct. Without seeing your actual code, we cannot know for sure.
A common mistake is this:
someAsyncOperation().then(someOtherAync()).then(...)
which should be:
someAsyncOperation().then(someOtherAync).then(...)
where you should pass a reference to the next async function rather than calling it immediately and passing its return value.
I thought maybe the first call to the routine could start an array
passed into a Promise.all and later calls would add another entry to
that array. I haven't tried it. Crazy?
You cannot pass an array to Promise.all() and then add things to the array later - that is not a supported capability of Promise.all(). You can chain subsequent things onto the results of Promise.all() or do another Promise.all() that includes the promise from the previous Promise.all() and some more promises.
var someArrayOfPromises = [...];
var pAll = Promise.all(someArrayOfPromises);
var someMorePromises = [...]
someMorePromises.push(pAll);
Promise.all(someMorePromoises).then(...)

Difference between callback hell and nested promises

I recently started working with NodeJS and MongoDB(using Monk). This is when I came across the term "callback hell". In my code I was exactly doing the same thing. For an example-
DBCall(d1, function(e, docs){
if(docs.length!=0)
DBCall(d2, function(e, docs1){
if(docs1.length!=0)
DBCall(d3, function(e, docs2){
//doing something with docs,docs1,docs2
})
})
})
This is when I started reading on "promises", and I came across this article - https://strongloop.com/strongblog/promises-in-node-js-with-q-an-alternative-to-callbacks/
Since I needed both docs and docs1 inside the third callback so I used nested promises.
DBCall(d1)
.then(function(docs){
if(docs.length!=0)
return DBCall(d2)
.then(function(docs1){
if(docs1.length!=0)
return DBCall(d3)
.then(function(docs2){
//doing something with docs,docs1,docs2
})
})
})
From the above code snippet I have the following questions(/doubts) :
Apart from making the code more readable, does promises have performance benefits?
Nesting promises and callback hell looks similar to me. Is there actually any difference?
I am new to this concept of promises. Any help is appreciated.

Apart from making the code more readable, does promises have performance benefits?
Probably not.
Nesting promises and callback hell looks similar to me. Is there actually any difference?
Promises don't automatically prevent callback hell. But because they are more flexible than "simple" callbacks, they can be composed in different ways, which makes it easier to avoid callback hell.
Since I needed both docs and docs1 inside the third callback so I used nested promises.
Sometimes nested promises are unavoidable. However, there is no need to nest them if they don't have to be executed sequentially. You could execute them in parallel:
Promise.all([
DBCall(d1),
DBCall(d2),
DBCall(d3)
]).then(function(docs) {
// docs is an array: [docs, docs1, docs2]
});

Basically the purpose of Promises is to allow for functional composition and error handling, in a way that reads in a synchronous nature. Promises allow you to read code in a linear (maybe wrong term) or synchronous fasion.
This is sort of a broad question but take a look through these suggested links.
https://promise-nuggets.github.io/
https://blog.domenic.me/youre-missing-the-point-of-promises/
also this link
Edit:
After reading your updates, what you're essentially asking for is to join the promises (I think)
This SO Post gives some good info. Some of the better libraries have utility functions to help with this.
For example, if using bluebird take a look at the join function

Why are Q.js promises asynchronous after they have been resolved?

If I have the following :
var deferred = Q.defer();
deferred.resolve();
var a = deferred.promise.then(function() {
console.log(1);
});
console.log(2);
...why do I see 2, then 1 in the console?
I understand this output is correct according to the Promises spec, which says to call the function on the next tick (e.g. setTimeout()), even if it is already resolved, but I don't understand why.
I would like to have code that calls then on a series of promises synchronously, assuming that all the promises have been resolved.
My real use case is that I am trying to use Angular's implementation, $q, and I want all of the then callbacks to execute in the same $digest cycle, so that I don't get unnecessary subsequent $digest cycles.

Answer is consistency.
In real code, you don't have promises that are always immediately resolved when created, they would be pointless. So you have promises that sometimes may be immediately resolved.
In that case, you don't want to have a different flow. You want always the same, predictable flow. So you want the next function to be always called on next tick.
Don't use a promise when you don't need one.

It's a design mistake based on a set of opinions and assumptions. It has become latched because it was rushed out without full technical verification in a design by committee process that also had the back pressure of a lot of vendors implementing their own already with the same mistake making it hard to backtrack.
Once a standard goes out for JS to the web it can be revoked, even if it's broken with the idea being that webpages shouldn't break. If someone wrote a page today, then died, it should be possible to still view it in your browser in five years. It would be very problematic if when browsing the web you kept bumping into pages that didn't work with your browser.
In very simple use cases it doesn't do much harm and removes confusion about whether something is maybe async.
For increasingly non-trivial use cases it causes increasingly more harm and adds confusion. Initially it appears to make it easier to reason about your code but it sacrifices the less trivial usage for the least trivial usage.
It's overall much easier to reason about your code if it doesn't run things in the next tick that don't need to be async at all. This is a case of choosing to damage the language at level two to cater to level one users, at the expense of level two users and above, rather than working to help ascend level one users to level two users. It's a condescending or pessimistic design decision.
There's an intermediate solution where it's possible to run each task as if it runs after the current code runs to completion but things are scheduled in the correct order. This hasn't been implemented and it's also debatable as compromises don't always produce the best solutions. This compromise produces performance issues for direct return callstacks.
The way promises work means that the forward callstack is depth first and run to completion (isolated) but the return callstack is breadth first and run in pieces interleaved with other return callstacks. These are two radically different concept and behaviours. Traditionally with callbacks both run in the same way.
It also means you can't naively replace callbacks with promises or anything based on promises. Callbacks give you more options which promises take away. If you replace callbacks with promises without taking this difference into account you can create code that has stability issues and potential security issues as well since this behaviour of unordering events can cause the current code flow to unexpectedly jump track.
You can't rely on order which means there are cases with promises where if you ask for lots of things when you get them back you have to double check they're the thing you asked for where as you would not need to do this with callbacks. You may also need to buffer and reorder events which you would not need to do with callbacks. It can make benchmarks unreliable as well if you're not careful to isolate the two things being run.
This can also create serious performance bottlenecks that you can't always easily prevent. If you use promises a hundred returned results returned at once from a single event to an iterator, each taking one second per return call and their promise resolve depth is two, they'll all be broken into half with the first halves all run then the second halves. That means it will take 50.5 seconds before anything can finish where as with callbacks after 50 seconds half of them would already be finished. If the result of the task is passed along to another external service for processing then it leaves that service standing idle for 50 seconds when it could have been processing your results. This makes the promises terrible for when you want both low latency and high throughput under services that take load demonstrating the weakness of the design.
Not being able to naively replace callbacks with promises is one of the most devastating consequences of this design mistake which is also carried over to async/await. If you want to convert a callback library you can't simple change the syntax, you must carefully scrutinise the semantics at every point.
There is no plan to fix this. You can create your own promises and generators can be used to provide the same kind of syntax as async/await but with the same predictable and high performance behaviour of callbacks. You may however have problems playing nice with other libraries that still rely on native promises.

How to think when working with promises

I'm working for the first time with async programming, specifically with promises in Javascript and I'm really confused with it. I'm still thinking like we do in C, PHP, C# and so on where for each step the next one can be sure of it's completion. This is easy to work, because if we have a code like that (in C# for example)
IEnumerable<Page> pages = pagesRepository.getAll();
we can be sure in the next line that all pages are available for us to use in the pages object. This makes life easier, because we can already do work with that data properly.
Now, in JS with promises we would have something like
var pages = dataService.getPages();
and this would not return the data, but just a promise which is very different. Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
So how do we think when we deal with those things? Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?

Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
You don't need to.
The "somewhere else code" will use .then() and yield a new promise for its result, relying on the pages promise. You can easily map the results, or chain other asynchronous tasks. Whenever you need to know about the timing of callbacks because of multiple dependencies, you should use your library's methods for composing promises, like all().
So how do we think when we deal with those things?
Think functional!
Every task in your program should explicitly and only depend on its inputs. A function should try to only use its arguments, not any global variables that contain state.
Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
General Promise Resources in the wiki of the Q library is a very good start.

The simple answer here is that promises are similar to Task asynchronous programming in C#.
However instead of being able to use the 'await' operator, the 'then' function on a promise is analogous to using 'ContinueWith' to chain Tasks in C#.
Callbacks can get complicated to use across scopes, especially with extensive use of nested anonymous functions, so please give promises a go

Javascript callback hell with setTimeout();

Aloha. This is my first question here after eons of lurking. I apologize for the verbose but direct form of this question -- as well as the limitations on acceptable answers -- but I am here to contribute. I have a problem that other questions (for example How to avoid long nesting of asynchronous functions in Node.js or Chained callback readability in Javascript?) do not answer to my satisfaction.
I want to perform several Javascript callbacks so that I have a sequence of four actions occurring one after another. Here is the source:
var self = this;
fade(div, fadeIn, -20, function() {
load(unload, dataOut, -10, function() {
load(self, dataIn, 10, function() {
fade(div, fadeOut, 20),0);
unload = self;}
);}
);}
);
The load() and fade() functions are very similar except for internal semantic details, and they look something very similar to this:
function load(menuCategory, data, step, callback) {
menuCategory.parentNode.style.width = data+"px";
data += step;
if(dataIn <= data && data <= dataOut) {
setTimeout(function() { load(menuCategory,data,step,callback) }, 15);
} else if(callback && (typeof(callback) == 'function')) {
callback();}
}
I have several questions associated with this mess, but first let me give you the answers I am NOT looking for. Answers using jQuery are not acceptable. Answers using other timing or "synchronicity" frameworks or modules are not acceptable. Answers questioning the reason for blocking execution are not acceptable. Answers using languages other than Javascript are unacceptable. Non-abstract answers are acceptable, but the final solution will need to be as abstract as possible, meaning the answer must be able to act on a variety of similar but slightly different actions.
Here is my primary question:
Is there a function/object I could use to apply these actions sequentially, to cut down on the callback hell going on here? In other words, what would an object look like if the object/function were able to synchronously execute each action as it was individually iterated through it (i.e. passed to it)?
Secondary question:
Is anyone aware of how much this callback business looks like goto() in other languages? Meaning, this nesting of callbacks feels very awkward and inelegant to me. Is there a Javascript language construct either being developed or already developed that will reduce this crazy business? Callback nesting is some of the hardest logic flow to follow. It just feels awkward.
My goal here is a tight, elegant, attractive, homegrown Javascript object/function -- that I can call in a for() loop or equivalent -- to apply these operations on each action in sequence. If you've read this far I want to thank you for your time and consideration. :)

using setTimeout(func(),0) in order to execute synchronously
No. Either use func() for a synchronous execution, or setTimeout(func, 0) to queue the execution (which makes it asynchronous).
Is setTimeout(func[…],0) the "most elegant" way to put this action onto the execution queue for rendered webpage objects? In other words, is there a "more elegant" way to block execution (i.e. do these things sequentially)?
Yes, this is the standard method. However, this does not block execution, and has nothing to do with sequencing. If the execution is synchronous, just execute it, if not you will have to deal with the callbacks and queuing a new task does not help.
Is anyone aware of how much this callback business looks like goto() in other languages? Meaning, this nesting of callbacks feels very awkward and inelegant to me. Is there a Javascript language construct either being developed or already developed that will reduce this crazy business?
No, I'm not aware of other languages, but afaik goto is a synchronous, control-flow-structuring statement as does not deal with asynchronous actions.
But no, there are no JavaScript language constructs ("syntactic sugar") that help us around the continuation-passing style with its callback hell.
Is there a function/object I could use to apply these actions sequentially, to cut down on the callback hell going on here? In other words, what would an object look like if the object/function were able to synchronously execute each action as it was individually iterated through it (i.e. passed to it)?
My goal here is a tight, elegant, attractive, homegrown Javascript object/function -- that I can call in a for() loop or equivalent -- to apply these operations on each action in sequence.
Again you want "sequencially" instead of "synchronous" :-) Yes, there are [handcoded/homegrown] solutions which make dealing with callbacks easier. As those callback organisation libraries you mentioned do not satisfy you, I'd recommend to look into the very elegant and attractive Promise Api and its Deferred objects (see futures and promises for the concept).

I'm unsure why you're using setTimeout at all here. Its not necessary to queue callbacks. The following structure will work, assuming that fade() and load() handle callback execution correctly.
fade(div, fadeIn, -20, function() {
load(unload, dataOut, -10, function() {
load(self, dataIn, 10, function() {
fade(div, fadeOut, 20),0);
});
)};
)};
Another topic you should look into is Promises, also called deferreds and futures. Its essentially a coding pattern that specifically handles nested callbacks. At root, you create a Promise object that gets returned immediately, then when your method is done executing it fires a method on the promise that can then execute your next function. Its a different way of writing javascript, but easy to understand and very useful.

As it turns out, the practice that fits as the "best" answer to my question is events. In particular the practice I'm referring to is best described by EventEmitter on Node.js, or creating custom events and event listeners on client browsers. While the Promise API and its deference model is intriguing, it uses extensions to the language instead of core Javascript, which is what I was looking for here in particular (and thus why none of the previous answers -- despite their clarity -- support exactly what I was looking for).
This link (http://wekeroad.com/2012/04/05/cleaning-up-deep-callback-nesting-with-nodes-eventemitter) provides a very good example of what I was looking for. As well, I should mention the book "Async Javascript" by Trevor Burnham (http://www.amazon.com/Async-JavaScript-Trevor-Burnham/dp/1475247362), and "Secrets of the JavaScript Ninja" by John Resig and Bear Bibeault (http://www.amazon.com/Secrets-JavaScript-Ninja-John-Resig/dp/193398869X). These books provide extremely clear working examples of custom events and event handlers in client browsers that provide an elegant way around this "callback hell" I was referring to.