Develop Reference

Managing sync and async functions in an algorithm call stack

TLDR: must I use async and await through a complicated call stack if most functions are not actually async? Are there alternative programming patterns?
Context
This question is probably more about design patterns and overall software architecture than a specific syntax issue. I am writing an algorithm in node.js that is fairly involved. The program flow involves an initial async call to fetch some data, and then moves on to a series of synchronous calculation steps based on the data. The calculation steps are iterative, and as they iterate, they generate results. But occasionally, if certain conditions are met by the calculations, more data will need to be fetched. Here is a simplified version in diagram form:
The calcStep loop runs thousands of times sychronously, pushing to the results. But occasionally it kicks back to getData, and the program must wait for more data to come in before continuing on again to the calcStep loop.
In code
A boiled-down version of the above might look like this in JS code:
let data;
async function init() {
data = await getData();
processData();
calcStep1();
}
function calcStep1() {
// do some calculations
calcStep2();
}
function calcStep2() {
// do more calculations
calcStep3();
}
function calcStep3() {
// do more calculations
pushToResults();
if (some_condition) {
getData(); // <------ question is here
}
if (stop_condition) {
finish();
} else {
calcStep1();
}
}
Where pushToResults and finish are also simple synchronous functions. I write the calcStep functions here are separate because in the real code, they are actually class methods from classes defined based on separation of concerns.
The problem
The obvious problem arises that if some_condition is met, and I need to wait to get more data before continuing the calcStep loop, I need to use the await keyword before calling getData in calcStep3, which means calcStep3 must be called async, and we must await that as well in calcStep2, and all the way up the chain, even synchronous functions must be labeled async and awaited.
In this simplified example, it would not be so offensive to do that. But in reality, my algorithm is far more complicated, with a much deeper call stack involving many class methods, iterations, etc. Is there a better way to manage awaiting functions in this type of scenario? Other tools I can use, like generators, or event emitters? I'm open to simple solutions or paradigm shifts.

If you don't want to make the function async and propagate this up the chain, use .then(). You'll need to duplicate the following code inside .then(); you can simplify this by putting it in its own function.
function maybeRepeat() {
if (stop_condition) {
finish();
} else {
calcStep1();
}
}
function calcStep3() {
// do more calculations
pushToResults();
if (some_condition) {
getData().then(maybeRepeat);
} else {
maybeRepeat()
}
}

Why using Promise()?

No matter how many tutorials I read, I still don't understand why I should use Promise()?!
Ok, let's make an example:
let cleanedUpMyRoom = new Promise(function(resolve, reject) {
if (true === true) {
resolve(true);
}
else {
reject(false);
}
});
cleanedUpMyRoom.then(function(result) {
console.log(result);
});
This code returns either true or false.
function cleanedUpMyRoom() {
if (true === true) {
return true;
}
else {
return false;
}
}
console.log(cleanedUpMyRoom());
This code also returns either true or false.
Why would I use the first code, if the second code is shorter and easier to read?

As several people have pointed out, promises are typically used to serialize asynchronous operations. The idea was developed as an alternative to nested callbacks, and what people have often referred to as "callback hell". E.g. something with callbacks like:
asyncFn1(
function(result1){
asyncFn2(result1,
function(reult2){
asyncFn3(result2)}
)}
)
can be rewritten using promises like:
asyncFn1()
.then(function(result1){asyncFn2(result1)})
.then(function(result2){asyncFn3(result2))
For complex cases where there are both success and failure handlers and functions depend on the results of other asynchronous operations, this can simplify the logic and greatly improve readability.
To look at your cleanedUpMyRoom example, a more common implementation would be to write:
cleanUpMyRoom = function(){
cleanPromise = new Promise();
startCleaning(/*on success*/ function(){console.log('Room cleaned!');
cleanPromise.resolve()},
/*on failure*/ function(){console.log('Room cleaned!');
cleanPromise.resolve(result)})
// or *alternatively* if it's just plain synchronous,
// var result = cleanIt()
// if (result = "success")
// cleanPromise.resolve()
// else
// cleanPromise.reject(result)
return cleanPromise
});
then
cleanUpMyRoom.then(goGetIceCream(),contemplateFailure(result))
cleanUpMyRoom() always returns a promise that has a then function. The important piece is that the function passed to then isn't called until the promise is resolved or rejected, and the result of the promise generating function is ready (whether synchronously or asynchronously).
Ultimately, a promise lets you chain together operations whose results may not be available in the synchronous procedural code so that nothing is called until the available, possibly asynchronous (like network requests), preconditions are met, and you code is able to manage on to other events in the meantime.
Take a look at this:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
You also might be interested in the upcoming (ES2017) async/await syntax which is becoming popular and handles many asynchronous situations without explicitly using promises at all:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

Javascript is an event driven language. In your short example, you are correct that using a promise would be pointless. In a more complex situation, say an AJAX request to the server, the code won't execute immediately.
Javascript will not work by simply returned the value as other imperative languages like python or java. The promise is advantageous by not blocking the main thread, and the code will begin running once the promise has been fullfilled. It makes javascript a great language for managing short spurts of activity through the event abstraction.

Better way to "loop" promises

This is a post that might come across as quite conceptual, since I first start with a lot of pseudo code. - At the end you'll see the use case for this problem, though a solution would be a "tool I can add to my tool-belt of useful programming techniques".
The problem
Sometimes one might need to create multiple promises, and either do something after all promises have ended. Or one might create multiple promises, based on the results of the previous promises. The analogy can be made to creating an array of values instead of a single value.
There are two basic cases to be considered, where the number of promises is indepedented of the result of said promises, and the case where it is depedent. Simple pseudo code of what "could" be done.
for (let i=0; i<10; i++) {
promise(...)
.then(...)
.catch(...);
}.then(new function(result) {
//All promises finished execute this code now.
})
The basically creates n (10) promises, and the final code would be executed after all promises are done. Of course the syntax isn't working in javascript, but it shows the idea. This problem is relativelly easy, and could be called completely asynchronous.
Now the second problem is like:
while (continueFn()) {
promise(...)
.then(.. potentially changing outcome of continueFn ..)
.catch(.. potentially changing outcome of continueFn ..)
}.then(new function(result) {
//All promises finished execute this code now.
})
This is much more complex, as one can't just start all promises and then wait for them to finish: in the end you'll have to go "promise-by-promise". This second case is what I wish to figure out (if one can do the second case you can also do the first).
The (bad) solution
I do have a working "solution". This is not a good solution as can probably quickly be seen, after the code I'll talk about why I dislike this method. Basically instead of looping it uses recursion - so the "promise" (or a wrapper around a promise which is a promise) calls itself when it's fulfilled, in code:
function promiseFunction(state_obj) {
return new Promise((resolve, reject) => {
//initialize fields here
let InnerFn = (stateObj) => {
if (!stateObj.checkContinue()) {
return resolve(state_obj);
}
ActualPromise(...)
.then(new function(result) {
newState = stateObj.cloneMe(); //we'll have to clone to prevent asynchronous write problems
newState.changeStateBasedOnResult(result);
return InnerFn(newState);
})
.catch(new function(err) {
return reject(err); //forward error handling (must be done manually?)
});
}
InnerFn(initialState); //kickstart
});
}
Important to note is that the stateObj should not change during its lifetime, but it can be really easy. In my real problem (which I'll explain at the end) the stateObj was simply a counter (number), and the if (!stateObj.checkContinue()) was simply if (counter < maxNumber).
Now this solution is really bad; It is ugly, complicated, error prone and finally impossible to scale.
Ugly because the actual business logic is buried in a mess of code. It doesn't show "on the can" that is actually simply doing what the while loop above does.
Complicated because the flow of execution is impossible to follow. First of all recursive code is never "easy" to follow, but more importantly you also have to keep in mind thread safety with the state-object. (Which might also have a reference to another object to, say, store a list of results for later processing).
It's error prone since there is more redundancy than strictly necessary; You'll have to explicitly forward the rejection. Debugging tools such as a stack trace also quickly become really hard to look through.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth. Normally one could either optimize by tail recursion or, more common, create a virtual stack (on the heap) and transform the function to a loop using the manual stack. In this case, however, one can't change the recursive calls to a loop-with-manual-stack; simply because of how promise syntax works.
The alternative (bad) solution
A colleague suggested an alternative approach to this problem, something that initially looked much less problematic, but I discarded ultimatelly since it was against everything promises are meant to do.
What he suggested was basically looping over the promises as per above. But instead of letting the loop continue there would be a variable "finished" and an inner loop that constantly checks for this variable; so in code it would be like:
function promiseFunction(state_obj) {
return new Promise((resolve, reject) => {
while (stateObj.checkContinue()) {
let finished = false;
let err = false;
let res = null;
actualPromise(...)
.then(new function(result) {
res = result;
finished = true;
})
.catch(new function(err) {
res = err;
err = true;
finished = true;
});
while(!finished) {
sleep(100); //to not burn our cpu
}
if (err) {
return reject(err);
}
stateObj.changeStateBasedOnResult(result);
}
});
}
While this is less complicated, since it's now easy to follow the flow of execution. This has problems of its own: not for the least that it's unclear when this function will end; and it's really bad for performance.
Conclusion
Well this isn't much to conclude yet, I'd really like something as simple as in the first pseudo code above. Maybe another way of looking at things so that one doesn't have the trouble of deeply recursive functions.
So how would you rewrite a promise that is part of a loop?
The real problem used as motivation
Now this problem has roots in a real thing I had to create. While this problem is now solved (by applying the recursive method above), it might be interesting to know what spawned this; The real question however isn't about this specific case, but rather on how to do this in general with any promise.
In a sails app I had to check a database, which had orders with order-ids. I had to find the first N "non existing order-ids". My solution was to get the "first" M products from the database, find the missing numbers within it. Then if the number of missing numbers was less than N get the next batch of M products.
Now to get an item from a database, one uses a promise (or callback), thus the code won't wait for the database data to return. - So I'm basically at the "second problem:"
function GenerateEmptySpots(maxNum) {
return new Promise((resolve, reject) => {
//initialize fields
let InnerFn = (counter, r) => {
if (r > 0) {
return resolve(true);
}
let query = {
orderNr: {'>=': counter, '<': (counter + maxNum)}
};
Order.find({
where: query,
sort: 'orderNr ASC'})
.then(new function(result) {
n = findNumberOfMissingSpotsAndStoreThemInThis();
return InnerFn(newState, r - n);
}.bind(this))
.catch(new function(err) {
return reject(err);
});
}
InnerFn(maxNum); //kickstart
});
}
EDIT:
Small post scriptus: the sleep function in the alternative is just from another library which provided a non-blocking-sleep. (not that it matters).
Also, should've indicated I'm using es2015.

The alternative (bad) solution
…doesn't actually work, as there is no sleep function in JavaScript. (If you have a runtime library which provides a non-blocking-sleep, you could just have used a while loop and non-blocking-wait for the promise inside it using the same style).
The bad solution is ugly, complicated, error prone and finally impossible to scale.
Nope. The recursive approach is indeed the proper way to do this.
Ugly because the actual business logic is buried in a mess of code. And error-prone as you'll have to explicitly forward the rejection.
This is just caused by the Promise constructor antipattern! Avoid it.
Complicated because the flow of execution is impossible to follow. Recursive code is never "easy" to follow
I'll challenge that statement. You just have to get accustomed to it.
You also have to keep in mind thread safety with the state-object.
No. There is no multi-threading and shared memory access in JavaScript, if you worry about concurrency where other stuff affects your state object while the loop runs that will a problem with any approach.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow
No. It's asynchronous! The callback will run on a new stack, it's not actually called recursively during the function call and does not carry those stack frames around. The asynchronous event loop already provides the trampoline to make this tail-recursive.
The good solution
function promiseFunction(state) {
const initialState = state.cloneMe(); // clone once for this run
// initialize fields here
return (function recurse(localState) {
if (!localState.checkContinue())
return Promise.resolve(localState);
else
return actualPromise(…).then(result =>
recurse(localState.changeStateBasedOnResult(result))
);
}(initialState)); // kickstart
}
The modern solution
You know, async/await is available in every environment that implemented ES6, as all of them also implemented ES8 now!
async function promiseFunction(state) {
const localState = state.cloneMe(); // clone once for this run
// initialize fields here
while (!localState.checkContinue()) {
const result = await actualPromise(…);
localState = localState.changeStateBasedOnResult(result);
}
return localState;
}

Let’s begin with the simple case: You have N promises that all do some work, and you want to do something when all the promises have finished. There’s actually a built-in way to do exactly that: Promise.all. With that, the code will look like this:
let promises = [];
for (let i=0; i<10; i++) {
promises.push(doSomethingAsynchronously());
}
Promise.all(promises).then(arrayOfResults => {
// all promises finished
});
Now, the second call is a situation you encounter all the time when you want to continue doing something asynchronously depending on the previous asynchronous result. A common example (that’s a bit less abstract) would be to simply fetch pages until you hit the end.
With modern JavaScript, there’s luckily a way to write this in a really readable way: Using asynchronous functions and await:
async function readFromAllPages() {
let shouldContinue = true;
let pageId = 0;
let items = [];
while (shouldContinue) {
// fetch the next page
let result = await fetchSinglePage(pageId);
// store items
items.push.apply(items, result.items);
// evaluate whether we want to continue
if (!result.items.length) {
shouldContinue = false;
}
pageId++;
}
return items;
}
readFromAllPages().then(allItems => {
// items have been read from all pages
});
Without async/await, this will look a bit more complicated, since you need to manage all this yourself. But unless you try to make it super generic, it shouldn’t look that bad. For example, the paging one could look like this:
function readFromAllPages() {
let items = [];
function readNextPage(pageId) {
return fetchSinglePage(pageId).then(result => {
items.push.apply(items, result.items);
if (!result.items.length) {
return Promise.resolve(null);
}
return readNextPage(pageId + 1);
});
}
return readNextPage(0).then(() => items);
}
First of all recursive code is never "easy" to follow
I think the code is fine to read. As I’ve said: Unless you try to make it super generic, you can really keep it simple. And naming also helps a lot.
but more importantly you also have to keep in mind thread safety with the state-object
No, JavaScript is single-threaded. You doing things asynchronously but that does not necessarily mean that things are happening at the same time. JavaScript uses an event loop to work off asynchronous processes, where only one code block runs at a single time.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth.
Also no. This is recursive in the sense that the function references itself. But it will not call itself directly. Instead it will register itself as a callback when an asynchronous process finishes. So the current execution of the function will finish first, then at some point the asynchronous process finishes, and then the callback will eventually run. These are (at least) three separate steps from the event loop, which all run independently from another, so you do no have a problem with recursion depth here.

The crux of the matter seems to be that "the actual business logic is buried in a mess of code".
Yes it is ... in both solutions.
Things can be separated out by :
having an asyncRecursor function that simply knows how to (asynchronously) recurse.
allowing the recursor's caller(s) to specify the business logic (the terminal test to apply, and the work to be performed).
It is also better to allow caller(s) to be responsible for cloning the original object rather than resolver() assuming cloning always to be necessary. The caller really needs to be in charge in this regard.
function asyncRecursor(subject, testFn, workFn) {
// asyncRecursor orchestrates the recursion
if(testFn(subject)) {
return Promise.resolve(workFn(subject)).then(result => asyncRecursor(result, testFn, workFn));
// the `Promise.resolve()` wrapper safeguards against workFn() not being thenable.
} else {
return Promise.resolve(subject);
// the `Promise.resolve()` wrapper safeguards against `testFn(subject)` failing at the first call of asyncRecursor().
}
}
Now you can write your caller as follows :
// example caller
function someBusinessOrientedCallerFn(state_obj) {
// ... preamble ...
return asyncRecursor(
state_obj, // or state_obj.cloneMe() if necessary
(obj) => obj.checkContinue(), // testFn
(obj) => somethingAsync(...).then((result) => { // workFn
obj.changeStateBasedOnResult(result);
return obj; // return `obj` or anything you like providing it makes a valid parameter to be passed to `testFn()` and `workFn()` at next recursion.
});
);
}
You could theoretically incorporate your terminal test inside the workFn but keeping them separate will help enforce the discipline, in writers of the business-logic, to remember to include a test. Otherwise they will consider it optional and sure as you like, they will leave it out!

Sorry, this doesn't use Promises, but sometimes abstractions just get in the way.
This example, which builds from #poke's answer, is short and easy to comprehend.
function readFromAllPages(done=function(){}, pageId=0, res=[]) {
fetchSinglePage(pageId, res => {
if (res.items.length) {
readFromAllPages(done, ++pageId, items.concat(res.items));
} else {
done(items);
}
});
}
readFromAllPages(allItems => {
// items have been read from all pages
});
This has only a single depth of nested functions. In general, you can solve the nested callback problem without resorting to a subsystem that manages things for you.
If we drop the parameter defaults and change the arrow functions, we get code that runs in legacy ES3 browsers.

AngularJS - "Chaining promises" with other functions in between

So I have this problem. I'm fairly new to angular and I've been told to modify a directive which manages forms to make the submit button disabled then enabled again when all the work is done.
Since the functions being called usually have async calls, simply adding code sequentially doesn't work.
var ngSubmit = function() {
vm.disabled = true;
$scope.ngSubmitFunction();
vm.disabled = false;
}
Button is enabled before async calls under ngSubmitFunction() finish.
So I thought a promise would fix that and made something like:
var promise = function() {
return $q(function (resolve) {$scope.ngSubmitFunction()});
}
var ngSubmit = function() {
vm.disabled = true;
promise().then(function() {
vm.disabled = false;
});
}
This doesn't output any error but never enables the button again (.then is never called).
Tried different kind of promises declaration, all with the same result, except for this one:
$scope.submitPromise = function() {
return $q.when($scope.ngSubmitFunction());
}
This does call .then function, but again, doesn't wait for any child async function to finish. '.then' is called instantly, like the sequential version.
Have in mind that I don't know what's under ngSubmitFunction(). It is used by dozens developers and it may contain from 0 to multiple async calls. But typical scenario is something like:
ngSubmitFunction() calls func()
-- func() decides wether to call create() or update()
-- -- update() calls a elementFactory.update() which is an async call
-- -- -- elementFactory.update().then(function()) is called when finished.
-- -- -- -- At THIS point, I should enable the button again.
How can I achieve this? is there a way to chain promises with non-promises functions in between? or another way to make a code only execute when everything else is done? I thought about creating an event at DataFactory when an async call is over but if the update() function was calling to more than one async call this wouldn't work.

If you are using promises, your async functions should return promises, if they do it should work like this:
var ngSubmit = function() {
vm.disabled = true;
$scope.ngSubmitFunction().then(() => {
vm.disabled = false;
});
}

I don't know what's under ngSubmitFunction()
Then you're out of luck, promises won't help you here. Promises or $q.when cannot magically inspect the call and see what asynchronous things it started or even wait for them - ngSubmitFunction() needs to return a promise for its asynchronous results itself.
You need to make every function in your codebase which (possibly) does something asynchronous that needs to be awaitable return a promise. There's no way around this.

Well, in case someone wonders, we haven't found a solution to that (there probably isn't). So we will go with the adding returns to all the chain of functions to make sure the ngSubmitFunction recieves a promise and therefor can wait for it to finish before calling '.then'. Not only this makes it work for the cases where there's only one promise implied but it is also a good programming practice.
Cases with more than one promise are scarce so we will treat them individually on the controller itself.
Thank you all for your comments.

How to break sequence handlers of Promises/A?

How to break sequence of handlers of Promises/A?
For example
if we have sequence:
method1().then(method2())
.then(method3(), errorHandler3());
and in method2() we decided to break sequence and don't go farther nor to method3() or errorHandler3(). What we should do?

You can return a promise that never is resolved from the function that method2() yields.
However, that doesn't make much sense, as "breaking the sequence" means going to error state actually, and should be possible to handle.
Maybe you rather want
method1().then(function method2(res) {
// do something
if (/* you want to break*/)
throw new Error("reason for breaking");
else
return when(/* what is done */).then(method3(), errorHandler3());
});

In that case you should nest your logic in a callback passed to then, e.g.
method1().then(function (value) {
if (method2(value)) return method3();
}).done();
I simplified my example, so it doesn't take callbacks from other function calls (as it is in your code). It looks a bit weird, and definitely is not common in promises world, what is your reasoning behind it?
Commenting on other solution, you definitely should not use promise that never resolves. In well written logic all promises resolve, and if there are ones that doesn't it means you have a bug in your logic (it's like creating async function that is never meant to call a callback, doesn't make sense).