Synchronicity in js loops is still driving me up the wall.
What I want to do is fairly simple
async doAllTheThings(data, array) {
await array.forEach(entry => {
let val = //some algorithm using entry keys
let subVal = someFunc(/*more entry keys*/)
data[entry.Namekey] = `${val}/${subVal}`;
});
return data; //after data is modified
}
But I can't tell if that's actually safe or not. I simply don't like the simple loop pattern
for (i=0; i<arrayLength; i++) {
//do things
if (i === arrayLength-1) {
return
}
}
I wanted a better way to do it, but I can't tell if what I'm trying is working safely or not, or I simply haven't hit a data pattern that will trigger the race condition.
Or perhaps I'm overthinking it. The algorithm in the array consists solely of some MATH and assignment statements...and a small function call that itself also consists solely of more MATH and assignment statements. Those are supposedly fully synchronous across the board. But loops are weird sometimes.
The Question
Can you use await in that manner, outside the loop itself, to trigger the code to wait for the loop to complete? Or is the only safe way to accomplish this the older manner of simply checking where you are in the loop, and not returning until you hit the end, manually.
One of the best ways to handle async and loops is to put then on a promise and wait for Promise.all remember that await returns a Promise so you can do:
async function doAllTheThings(array) {
const promises = []
array.forEach((entry, index) => {
promises.push(new Promise((resolve) => {
setTimeout(() => resolve(entry + 1), 200 )
}))
});
return Promise.all(promises)
}
async function main () {
const arrayPlus1 = await doAllTheThings([1,2,3,4,5])
console.log(arrayPlus1.join(', '))
}
main().then(() => {
console.log('Done the async')
}).catch((err) => console.log(err))
Another option is to use generators but they are a little bit more complex so if you can just save your promises and wait for then that is an easier approach.
About the question at the end:
Can you use await in that manner, outside the loop itself, to trigger the code to wait for the loop to complete? Or is the only safe way to accomplish this the older manner of simply checking where you are in the loop, and not returning until you hit the end, manually.
All javascript loops are synchronous so the next line will wait for the loop to execute.
If you need to do some async code in loop a good approach is the promise approach above.
Another approach for async loops specially if you have to "pause" or get info from outside the loop is the iterator/generator approach.
Related
I am trying to write something like this but not able to resolve inner promises.
const traceSamplesRequests = transactionTypes.map(async (transactionType) => {
// This part should be synchronous since we need transactionNames
const transactionNames = await api.getTransactionNames();
return transactionNames.map((transactionName) => api.getTraceSamples());
});
And I want to write with lambda functions with Promise.all not imperative loops.
So, .map() is not promise-aware. It calls the callback and just dutifully collects the return value from the callback and advances to the next iteration of the loop. Since your callback is async with an await in it, that return value is an unresolved promise. So, what you get back from .map() is an array of promises.
So, there is no way to use .map() or any of the array iteration methods without then using something to wait for the promises.
It sounds like you already know this, but with the code you have, you would insert two Promise.all() statements:
const traceSamplesRequests = await Promise.all(transactionTypes.map(async (transactionType) => {
// This part should be synchronous since we need transactionNames
const transactionNames = await api.getTransactionNames();
return Promise.all(transactionNames.map((transactionName) => api.getTraceSamples()));
}));
That would get you an array of values (and the parent function would have to be async). This will run all the asynchronous operation in the loop in parallel (all in flight at the same time).
Your other choices are to use a for or while loop which will suspend loop iteration for each await and will run your asynchronous operations in sequence (second iteration of the loop not running until the first is done).
Or, you can write some sort of helper function that would be a promise-aware version of .map(). You'd probably just either use for or .map() with Promise.all() inside the helper function though.
The Async promise library has multiple such helper functions as choices to use. They are designed to work with either promises or plain callbacks.
I actually made it work like this:
const traceSamplesRequests = await Promise.all(
transactionTypes.map(async (transactionType) => {
const transactionNames = await api.getTransactionNames({ transactionType });
return await Promise.all(transactionNames.map((transactionName) => api.getTraceSamples({ transactionType })));
})
);
I am working with this while loop and it is not working. I decided to use the Google Chrome debugger and I saw that the code inside is not being executed.
All the time it checks the condition, starts the first line of the code inside, and goes back again to check the condition.
It is a NodeJS server and I am using the Spotify API.
app.get('/process', ensureAuthenticated, function (req, res) {
let init_array = text.split(" ");
let modtext = init_array;
while (init_array.length != 0) {
spotifyApi.searchTracks(modtext.join(" "))
.then(function (data) {
try {
console.log(data.body.tracks.items[0].name);
for (let i = 0; i < modtext.length; i++) {
init_array.shift();
}
modtext = init_array;
} catch (err) {
console.log("No song");
modtext.pop();
}
});
}
res.redirect('/');
});
This question is best understood by understanding how node.js uses an event loop. At its core, node.js runs your Javascript in a single thread and it uses an event loop in order to manage the completion of things outside that single thread such as timers, network operations, file operations, etc...
Let's first start with a very simple while() loop:
let done = false;
setTimeout(() => {
done = true;
}, 100);
while(!done) {
// do whatever you want here
}
console.log("done with while loop"); // never gets called
At first blush, you would think that the while loop would run for 100ms and then done would be set to true and the while loop would exit. That is not what happens. In fact, this is an infinite while loop. It runs and runs and runs and the variable done is never set to true. The console.log() at the end never runs.
It has this issue because setTimeout() is an asynchronous operation and it communicates its completion through the event loop. But, as we described above, node.js runs its Javascript as single threaded and only gets the next event from the event loop when that single thread finishes what it's doing. But, the while can't finish what it's doing until done gets set to true, but done can't get set to true until the while loop finishes. It's a stand-off and the while loop just runs forever.
So, in a nutshell, while any sort of loop is running, NO asynchronous operation ever gets its result processed (unless it's using await inside the loop which is something different). Asynchronous operations (or anything that uses the event loop) has to wait until the current running Javascript is done and then the interpreter can go back to the event loop.
Your while loop has the exact same issue. spotifyApi.searchTracks() is an asynchronous operation that returns a promise and all promises communicate their results via the event queue. So, you have the same standoff. Your .then() handler can't get called until the while loop finishes, but your while loop can't finish until the .then() handler gets called. Your while loop will just loop infinitely until you exhaust some system resource and your .then() handlers never get a chance to execute.
Since you haven't included code in your request handler that actually produces some result or action (all it appears to do is just modify some local variables), it's not obvious what exactly you're trying to accomplish and thus how to better write this code.
You appear to have N searches to do and you're logging something in each search. You could do them all in parallel and just use Promise.all() to track when they are all done (no while loop at all). Or, you can sequence them so you run one, get its result, then run another. Your question doesn't give us enough info to know what the best option would be.
Here's one possible solution:
Sequence the operations using async/await
Here the request handler is declared async so we can use await inside the while loop. That will suspend the while loop and allow other events to process while waiting for the promise to resolve.
app.get('/process', ensureAuthenticated, async function (req, res) {
let init_array = text.split(" ");
let modtext = init_array;
while (init_array.length != 0) {
try {
let data = await spotifyApi.searchTracks(modtext.join(" "));
console.log(data.body.tracks.items[0].name);
for (let i = 0; i < modtext.length; i++) {
init_array.shift();
}
modtext = init_array;
} catch (err) {
console.log("No song");
modtext.pop();
}
}
res.redirect('/');
});
The reason you're only seeing one line execute is because it's asynchronous. When you call an asynchronous function, it returns immediately and continues to do its work in the background. Once it's done, it calls another function (a "callback") and passes the results of the function to that. That's why your code has to go inside of a call to then() rather than just being on the next line of code.
In this case, spotifyApi.searchTracks() returns a Promise. Once the Spotify API has completed the search, the function in then() will run.
Lets use async/await to solve this problem, I have no clue what data you get in a text but I think it is good example to understand a concept of asynchronous processing.
app.get('/process', ensureAuthenticated, async function (req, res, next) {
try {
const modtext = text.split(" ");
const promises = modtext.map(item => spotify.searchTracks(item));
const response = await Promise.all(promises);
response.forEach(data => {
// if you use lodash, simply use a get function `get(data, 'body.tracks.items[0].name')` => no need to check existence of inner attributes
// or use something like this
if (data && data.body && data.body.track && data.body.tracks.items && data.body.tracks.items[0]) {
console.log(data.body.tracks.items[0].name);
} else {
console.log('No song');
}
});
res.redirect('/');
} catch(err) {
next(err)
}
});
For me much simpler and cleaner code, again, i dont know what is structure of your text attribute and logic behind it, so maybe you will have to make some changes.
I am trying to iterate over a list of keywords, and call mysql.query() for each one of them.
Problem is, I need each query call to end before the next one begins. How do I make that happen?
I have tried making it work with Promise requests but I do not wholly understand how asynchronous calls work.
keywords.forEach(keyword => {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {...}
});
You can do it recursivly
function call(keywords, index){
if(keyworkds[index]){
sql.query("Select Id from Keyword Where Word= ?", [keyworkds[index].keyword],
function (err, ids) {
call(keywords, index + 1)
}
}
}
call(keywords, 0)
forEach will not wait for the asynchronous function to be done. The reason you need to pass a callback is because that's the only real way you have to do something after completion.
For that reason forEach simply will not work. It's possible to rewrite this to have the second iteration happen asynchronously, but it's ugly as hell.
I would recommend (if you can) to switch to async / await and use a mysql client that supports promises. Example:
for (const keyword of keywords) {
const result = await sql.query(
"SELECT Id from Keyword Where Word = ?",
[keyword]
);
}
You can use mysql2/promise from the mysql2 package to use promisified mysql function.
You won't be able to do this with just the forEach. Each call of sql.query will fire as quickly as the loop iterates and will not wait until it finishes. Also, there is no guarantee those queries will resolve in the order you call them in.
At the bare minimum, you can use callback functions, but that would be some really ugly and difficult code to deal with. https://en.wikipedia.org/wiki/Pyramid_of_doom_(programming)
That leaves Promises and async/await. I strongly recommend spending some time on this topic as you will be running into it a lot
https://javascript.info/async
Instead of looking for an async / loop solution, you can solve your original problem with one SQL query:
const args = keywords.map(_ => "?").join();
sql.query("Select Id, Word from Keyword Where Word in (" + args + ")", keywords,
function (err, records) {...}
);
I think async js library (https://caolan.github.io/async) is a good choice for something similar to your problem and with this library you have a cleaner code without nested async calls that produce Pyramid of doom. Whenever you face a problem that has many async calls that will run either in parallel or synchronously you can use it.
as the documentation said you can run only a single async operation at a time with series method like eachSeries.
async.eachSeries(keywords, function(keyword, callback) {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {
if (err){
//if you want to stop reminding queries call callback with err
callback(err)
} else {
callback();
}
}
}, function(err) {
// if any of the queris produced an error, err would equal that error
if( err ) {
// One of the iterations produced an error.
// All processing will now stop.
console.log('A query failed to process');
}else {
console.log('All queries have been processed successfully');
}
});
forEach doesn't do anything special with async functions. It will call an async function, but it won't wait for it to complete before moving on. Therefore, if you need to call async promises sequentially, then you need another approach.
It looks like though you aren't using a library that returns a promise, so you will need to wrap the call in a promise (util.promisefy can help with that, or you can do it manually), or you can use a version of the library that does return promises. It is possible to do this with callbacks, but much harder.
The most straight forward is to use a for … of loop in side of an async function. In order to do this, you need a new enough version of javascript, and async functions are rather new. The basic idea look like this:
async function processCollection(collection) {
for (let item of collection) {
await processItem(item)
}
}
Where processItem would be another async function or a function that returns a promise (which are basically the same thing, just different syntaxes).
Another way, that doesn't require async functions, is to chain together promises. This can be done in a functional way as follows:
collection.reduce((previousPromise, item) => {
return previousPromose.then(() => processItem(item))
}, Promise.resolve(null))
(again, processItem is a function that returns a promise)
Whats goin on here is that by using reduce, you are basically calling promise.then(…).then(…).then(…)… once for each item in the collection. if the callback to then returns a promise, then it will wait for that promise to finish before calling the next then callback. So you get sequential execution. Also, any promise rejections are propagated as well, so errors will stop the next call from happening.
I understand that this is a basic question, but I can't figure it out myself, how to export my variable "X" (which is actually a JSON object) out of "for" cycle. I have tried a various ways, but in my case function return not the JSON.object itself, but a "promise.pending".
I guess that someone more expirienced with this will help me out. My code:
for (let i = 0; i < server.length; i++) {
const fetch = require("node-fetch");
const url = ''+(server[i].name)+'';
const getData = async url => {
try {
const response = await fetch(url);
return await response.json();
} catch (error) {
console.log(error);
}
};
getData(url).then(function(result) { //promise.pending w/o .then
let x = result; //here is real JSON that I want to export
});
}
console.log(x); // -element is not exported :(
Here's some cleaner ES6 code you may wish to try:
const fetch = require("node-fetch");
Promise.all(
server.map((srv) => {
const url = String(srv.name);
return fetch(url)
.then((response) => response.json())
.catch((err) => console.log(err));
})
)
.then((results) => {
console.log(results);
})
.catch((err) => {
console.log('total failure!');
console.log(err);
});
How does it work?
Using Array.map, it transforms the list of servers into a list of promises which are executed in parallel. Each promise does two things:
fetch the URL
extract JSON response
If either step fails, that one promise rejects, which will then cause the whole series to reject immediately.
Why do I think this is better than the accepted answer? In a word, it's cleaner. It doesn't mix explicit promises with async/await, which can make asynchronous logic muddier than necessary. It doesn't import the fetch library on every loop iteration. It converts the server URL to a string explicitly, rather than relying on implicit coercion. It doesn't create unnecessary variables, and it avoids the needless for loop.
Whether you accept it or not, I offer it up as another view on the same problem, solved in what I think is a maximally elegant and clear way.
Why is this so hard? Why is async work so counterintuitive?
Doing async work requires being comfortable with something known as "continuation passing style." An asynchronous task is, by definition, non-blocking -- program execution does not wait for the task to complete before moving to the next statement. But we often do async work because subsequent statements require data that is not yet available. Thus, we have the callback function, then the Promise, and now async/await. The first two solve the problem with a mechanism that allows you to provide "packages" of work to do once an asynchronous task is complete -- "continuations," where execution will resume once some condition obtains. There is absolutely no difference between a boring node-style callback function and the .then of a Promise: both accept functions, and both will execute those functions at specific times and with specific data. The key job of the callback function is to act as a receptacle for data about the asynchronous task.
This pattern complicates not only basic variable scoping, which was your main concern, but also the issue of how best to express complicated workflows, which are often a mix of blocking and non-blocking statements. If doing async work requires providing lots of "continuations" in the form of functions, then we know that doing this work will be a constant battle against the proliferation of a million little functions, a million things needing names that must be unique and clear. This is a problem that cannot be solved with a library. It requires adapting one's style to the changed terrain.
The less your feet touch the ground, the better. :)
Javascript builds on the concept of promises. When you ask getData to to do its work, what is says is that, "OK, this is going to take some time, but I promise that I'll let you know after the work is done. So please have faith on my promise, I'll let you know once the work is complete", and it immediately gives you a promise to you.
That's what you see as promise.pending. It's pending because it is not completed yet. Now you should register a certain task (or function) with that promise for getData to call when he completes the work.
function doSomething(){
var promiseArray = [];
for (let i = 0; i < server.length; i++) {
const fetch = require("node-fetch");
const url = ''+(server[i].name)+'';
const getData = async url => {
try {
const response = await fetch(url);
return await response.json();
} catch (error) {
console.log(error);
}
};
promiseArray.push(getData(url)); // keeping track of all promises
}
return Promise.all(promiseArray); //see, I'm not registering anything to promise, I'm passing it to the consumer
}
function successCallback(result) {
console.log("It succeeded with " + result);
}
function failureCallback(error) {
console.log("It failed with " + error);
}
let promise = doSomething(); // do something is the function that does all the logic in that for loop and getData
promise.then(successCallback, failureCallback);
This is a post that might come across as quite conceptual, since I first start with a lot of pseudo code. - At the end you'll see the use case for this problem, though a solution would be a "tool I can add to my tool-belt of useful programming techniques".
The problem
Sometimes one might need to create multiple promises, and either do something after all promises have ended. Or one might create multiple promises, based on the results of the previous promises. The analogy can be made to creating an array of values instead of a single value.
There are two basic cases to be considered, where the number of promises is indepedented of the result of said promises, and the case where it is depedent. Simple pseudo code of what "could" be done.
for (let i=0; i<10; i++) {
promise(...)
.then(...)
.catch(...);
}.then(new function(result) {
//All promises finished execute this code now.
})
The basically creates n (10) promises, and the final code would be executed after all promises are done. Of course the syntax isn't working in javascript, but it shows the idea. This problem is relativelly easy, and could be called completely asynchronous.
Now the second problem is like:
while (continueFn()) {
promise(...)
.then(.. potentially changing outcome of continueFn ..)
.catch(.. potentially changing outcome of continueFn ..)
}.then(new function(result) {
//All promises finished execute this code now.
})
This is much more complex, as one can't just start all promises and then wait for them to finish: in the end you'll have to go "promise-by-promise". This second case is what I wish to figure out (if one can do the second case you can also do the first).
The (bad) solution
I do have a working "solution". This is not a good solution as can probably quickly be seen, after the code I'll talk about why I dislike this method. Basically instead of looping it uses recursion - so the "promise" (or a wrapper around a promise which is a promise) calls itself when it's fulfilled, in code:
function promiseFunction(state_obj) {
return new Promise((resolve, reject) => {
//initialize fields here
let InnerFn = (stateObj) => {
if (!stateObj.checkContinue()) {
return resolve(state_obj);
}
ActualPromise(...)
.then(new function(result) {
newState = stateObj.cloneMe(); //we'll have to clone to prevent asynchronous write problems
newState.changeStateBasedOnResult(result);
return InnerFn(newState);
})
.catch(new function(err) {
return reject(err); //forward error handling (must be done manually?)
});
}
InnerFn(initialState); //kickstart
});
}
Important to note is that the stateObj should not change during its lifetime, but it can be really easy. In my real problem (which I'll explain at the end) the stateObj was simply a counter (number), and the if (!stateObj.checkContinue()) was simply if (counter < maxNumber).
Now this solution is really bad; It is ugly, complicated, error prone and finally impossible to scale.
Ugly because the actual business logic is buried in a mess of code. It doesn't show "on the can" that is actually simply doing what the while loop above does.
Complicated because the flow of execution is impossible to follow. First of all recursive code is never "easy" to follow, but more importantly you also have to keep in mind thread safety with the state-object. (Which might also have a reference to another object to, say, store a list of results for later processing).
It's error prone since there is more redundancy than strictly necessary; You'll have to explicitly forward the rejection. Debugging tools such as a stack trace also quickly become really hard to look through.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth. Normally one could either optimize by tail recursion or, more common, create a virtual stack (on the heap) and transform the function to a loop using the manual stack. In this case, however, one can't change the recursive calls to a loop-with-manual-stack; simply because of how promise syntax works.
The alternative (bad) solution
A colleague suggested an alternative approach to this problem, something that initially looked much less problematic, but I discarded ultimatelly since it was against everything promises are meant to do.
What he suggested was basically looping over the promises as per above. But instead of letting the loop continue there would be a variable "finished" and an inner loop that constantly checks for this variable; so in code it would be like:
function promiseFunction(state_obj) {
return new Promise((resolve, reject) => {
while (stateObj.checkContinue()) {
let finished = false;
let err = false;
let res = null;
actualPromise(...)
.then(new function(result) {
res = result;
finished = true;
})
.catch(new function(err) {
res = err;
err = true;
finished = true;
});
while(!finished) {
sleep(100); //to not burn our cpu
}
if (err) {
return reject(err);
}
stateObj.changeStateBasedOnResult(result);
}
});
}
While this is less complicated, since it's now easy to follow the flow of execution. This has problems of its own: not for the least that it's unclear when this function will end; and it's really bad for performance.
Conclusion
Well this isn't much to conclude yet, I'd really like something as simple as in the first pseudo code above. Maybe another way of looking at things so that one doesn't have the trouble of deeply recursive functions.
So how would you rewrite a promise that is part of a loop?
The real problem used as motivation
Now this problem has roots in a real thing I had to create. While this problem is now solved (by applying the recursive method above), it might be interesting to know what spawned this; The real question however isn't about this specific case, but rather on how to do this in general with any promise.
In a sails app I had to check a database, which had orders with order-ids. I had to find the first N "non existing order-ids". My solution was to get the "first" M products from the database, find the missing numbers within it. Then if the number of missing numbers was less than N get the next batch of M products.
Now to get an item from a database, one uses a promise (or callback), thus the code won't wait for the database data to return. - So I'm basically at the "second problem:"
function GenerateEmptySpots(maxNum) {
return new Promise((resolve, reject) => {
//initialize fields
let InnerFn = (counter, r) => {
if (r > 0) {
return resolve(true);
}
let query = {
orderNr: {'>=': counter, '<': (counter + maxNum)}
};
Order.find({
where: query,
sort: 'orderNr ASC'})
.then(new function(result) {
n = findNumberOfMissingSpotsAndStoreThemInThis();
return InnerFn(newState, r - n);
}.bind(this))
.catch(new function(err) {
return reject(err);
});
}
InnerFn(maxNum); //kickstart
});
}
EDIT:
Small post scriptus: the sleep function in the alternative is just from another library which provided a non-blocking-sleep. (not that it matters).
Also, should've indicated I'm using es2015.
The alternative (bad) solution
…doesn't actually work, as there is no sleep function in JavaScript. (If you have a runtime library which provides a non-blocking-sleep, you could just have used a while loop and non-blocking-wait for the promise inside it using the same style).
The bad solution is ugly, complicated, error prone and finally impossible to scale.
Nope. The recursive approach is indeed the proper way to do this.
Ugly because the actual business logic is buried in a mess of code. And error-prone as you'll have to explicitly forward the rejection.
This is just caused by the Promise constructor antipattern! Avoid it.
Complicated because the flow of execution is impossible to follow. Recursive code is never "easy" to follow
I'll challenge that statement. You just have to get accustomed to it.
You also have to keep in mind thread safety with the state-object.
No. There is no multi-threading and shared memory access in JavaScript, if you worry about concurrency where other stuff affects your state object while the loop runs that will a problem with any approach.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow
No. It's asynchronous! The callback will run on a new stack, it's not actually called recursively during the function call and does not carry those stack frames around. The asynchronous event loop already provides the trampoline to make this tail-recursive.
The good solution
function promiseFunction(state) {
const initialState = state.cloneMe(); // clone once for this run
// initialize fields here
return (function recurse(localState) {
if (!localState.checkContinue())
return Promise.resolve(localState);
else
return actualPromise(…).then(result =>
recurse(localState.changeStateBasedOnResult(result))
);
}(initialState)); // kickstart
}
The modern solution
You know, async/await is available in every environment that implemented ES6, as all of them also implemented ES8 now!
async function promiseFunction(state) {
const localState = state.cloneMe(); // clone once for this run
// initialize fields here
while (!localState.checkContinue()) {
const result = await actualPromise(…);
localState = localState.changeStateBasedOnResult(result);
}
return localState;
}
Let’s begin with the simple case: You have N promises that all do some work, and you want to do something when all the promises have finished. There’s actually a built-in way to do exactly that: Promise.all. With that, the code will look like this:
let promises = [];
for (let i=0; i<10; i++) {
promises.push(doSomethingAsynchronously());
}
Promise.all(promises).then(arrayOfResults => {
// all promises finished
});
Now, the second call is a situation you encounter all the time when you want to continue doing something asynchronously depending on the previous asynchronous result. A common example (that’s a bit less abstract) would be to simply fetch pages until you hit the end.
With modern JavaScript, there’s luckily a way to write this in a really readable way: Using asynchronous functions and await:
async function readFromAllPages() {
let shouldContinue = true;
let pageId = 0;
let items = [];
while (shouldContinue) {
// fetch the next page
let result = await fetchSinglePage(pageId);
// store items
items.push.apply(items, result.items);
// evaluate whether we want to continue
if (!result.items.length) {
shouldContinue = false;
}
pageId++;
}
return items;
}
readFromAllPages().then(allItems => {
// items have been read from all pages
});
Without async/await, this will look a bit more complicated, since you need to manage all this yourself. But unless you try to make it super generic, it shouldn’t look that bad. For example, the paging one could look like this:
function readFromAllPages() {
let items = [];
function readNextPage(pageId) {
return fetchSinglePage(pageId).then(result => {
items.push.apply(items, result.items);
if (!result.items.length) {
return Promise.resolve(null);
}
return readNextPage(pageId + 1);
});
}
return readNextPage(0).then(() => items);
}
First of all recursive code is never "easy" to follow
I think the code is fine to read. As I’ve said: Unless you try to make it super generic, you can really keep it simple. And naming also helps a lot.
but more importantly you also have to keep in mind thread safety with the state-object
No, JavaScript is single-threaded. You doing things asynchronously but that does not necessarily mean that things are happening at the same time. JavaScript uses an event loop to work off asynchronous processes, where only one code block runs at a single time.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth.
Also no. This is recursive in the sense that the function references itself. But it will not call itself directly. Instead it will register itself as a callback when an asynchronous process finishes. So the current execution of the function will finish first, then at some point the asynchronous process finishes, and then the callback will eventually run. These are (at least) three separate steps from the event loop, which all run independently from another, so you do no have a problem with recursion depth here.
The crux of the matter seems to be that "the actual business logic is buried in a mess of code".
Yes it is ... in both solutions.
Things can be separated out by :
having an asyncRecursor function that simply knows how to (asynchronously) recurse.
allowing the recursor's caller(s) to specify the business logic (the terminal test to apply, and the work to be performed).
It is also better to allow caller(s) to be responsible for cloning the original object rather than resolver() assuming cloning always to be necessary. The caller really needs to be in charge in this regard.
function asyncRecursor(subject, testFn, workFn) {
// asyncRecursor orchestrates the recursion
if(testFn(subject)) {
return Promise.resolve(workFn(subject)).then(result => asyncRecursor(result, testFn, workFn));
// the `Promise.resolve()` wrapper safeguards against workFn() not being thenable.
} else {
return Promise.resolve(subject);
// the `Promise.resolve()` wrapper safeguards against `testFn(subject)` failing at the first call of asyncRecursor().
}
}
Now you can write your caller as follows :
// example caller
function someBusinessOrientedCallerFn(state_obj) {
// ... preamble ...
return asyncRecursor(
state_obj, // or state_obj.cloneMe() if necessary
(obj) => obj.checkContinue(), // testFn
(obj) => somethingAsync(...).then((result) => { // workFn
obj.changeStateBasedOnResult(result);
return obj; // return `obj` or anything you like providing it makes a valid parameter to be passed to `testFn()` and `workFn()` at next recursion.
});
);
}
You could theoretically incorporate your terminal test inside the workFn but keeping them separate will help enforce the discipline, in writers of the business-logic, to remember to include a test. Otherwise they will consider it optional and sure as you like, they will leave it out!
Sorry, this doesn't use Promises, but sometimes abstractions just get in the way.
This example, which builds from #poke's answer, is short and easy to comprehend.
function readFromAllPages(done=function(){}, pageId=0, res=[]) {
fetchSinglePage(pageId, res => {
if (res.items.length) {
readFromAllPages(done, ++pageId, items.concat(res.items));
} else {
done(items);
}
});
}
readFromAllPages(allItems => {
// items have been read from all pages
});
This has only a single depth of nested functions. In general, you can solve the nested callback problem without resorting to a subsystem that manages things for you.
If we drop the parameter defaults and change the arrow functions, we get code that runs in legacy ES3 browsers.