Develop Reference

JavaScript how to determine when AJAX calls are done in a loop

I've spent the last few days trying to tackle this issue and have read all sorts of solutions on StackOverflow and other sites.
I'm building a site that grabs XML data from an outside source and then gets more XML depending on the results to build a network graph. The problem is that I have to essentially wait until this loop of AJAX calls (which may loop into more AJAX calls) is finished before drawing.
I don't know if this just has an especially high cognitive load, but it really has me stumped.
My Code:
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
Note: I understand I may be completely misunderstanding JavaScript synchronicity here, and I very likely am. I have used callbacks and promises successfully in the past, I just can't seem to wrap my head around this one.
If I have not been totally clear, please let me know.
I did try implementing a counter of sorts that is incremented in the process() function, like so:
if (processCount < 15) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
} else {
draw(nodes, edges);
}
However, this ended up with several draw() calls which made my performance abysmal.

There are nice new well-supported APIs and language constructs we can use. The Fetch API, await, and for...of loops.
The Fetch API uses Promises. Promises can be awaited. The for...of loop is aware of await and won't continue the loop until the await has passed.
// Loop through, one-at-a-time
for (const node of outerNodes) {
// Make the HTTP request
const res = await fetch(someUrl);
// Do something with the response here...
}
Don't forget a try/catch (which also works with await), and check res.ok.

Brad's answer changes the code to by synchronious and to me that defeats the purpose. If you are constantly waiting on all request to be finished then it could take a while, while normal browsers can handle multiple requests.
The problem you have in your original questions is with scope. Since each call to cont(outerNodes) will trigger it's own scope, it has no idea what are calls are doing. So basically if you call cont(outerNodes) twice, each call will handle it's own list of outernodes and then call draw.
The solution is to share information between the different scopes, which can be done with one, but preferably two global variables: 1 to track active processes and 1 to track errors.
var inProcess = 0;
var nrErrors = 0;
function cont(outerNodes) {
//make sure you have outerNodes before you call outerNodes.length
if (outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
inProcess++; //add one more in process
getXML(node["label"], node["id"]);
}
}
//only trigger when nothing is in proces.
if (inProcess==0) {
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
inProcess--; //one is done
cont(outerNodes);
},
error: function() {
inProcess--; //one is done
nrErrors++; //one more error
cont(null); //run without new outerNodes, to trigger a possible draw
}
});
}
Please note that I track nrErrors but dont do anything with it. You could use it to stop further processing all together or warn the user that the draw is not complete.
[important] Keep in mind that this works in javascript because at best it mimics multithreading. That means the the call to inProcess--; and then right after cont(outerNodes); is always execute directly after eachother.
If you would port this to a true multithreading environment, it could very well be that another scope/version of cont(null); would cut in between the two lines and there would still be multiple draws.

The best way to solve this question should be using either promise or callback.
If you really want to avoid promise or callback(Although i don't know why...)
You can try with a counter.
let processCount = 0;
// Increasing the processCount in getXML callback method
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
processCount++;
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
while (processCount < outerNodes.length) {
// do nothing, just wait'
}
draw(nodes, edges);

If after testing it many times, you know that it will never take more than say 5 seconds... you can use a setTimeout.
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// Display a 5 second progress bar here
setTimeout(function(){ draw(nodes, edges); },5000);
}

Async/Await Not Waiting as I expect it

Please bear with me I've been dropped into a new project and trying to take it all in. I've been making progress over the last couple of day but can't seem to get over this last hump. Hopefully I can explain it correctly.
I'm loading up a web form and need to make a call out to the API to get some information that may or may not be present based on the data currently loading up. I've simplified my page to basically be this.
...Get some information the user wants and start to do some work to load
up the page and set up the form.
...old stuff working fine...
//Time for my new stuff
var testValue
async function test() {
await http.post(appConfig.serviceRootUrl + '/api/XXX/YYY',
{ mProperty: myObject.collectionInObject.itemInCollection }).then(function (result) {
if (result.length < 1) {
testValue= false;
}
else if (result[0].infoIWant.trim().length > 0) {
testValue= true;
}
});
}
test();
//Originally above in the if I was just seeing if I got a result
//and setting testValue to true/false but changed it for debugging
//and I am surely getting back correct values when the data exists
//or result.length zero when no data for it
...Do a bunch of more stuff that is old and working correctly....
//Test the new stuff up above
alert(testValue);
Most of the time I get back the correct true or false in the alert but once in a while I get back undefined. I'm guessing the undefined is because it is getting to the alert before the async/await finishes. I was under the impression it won't go past the line where I call "test();". I thought it was in effect making it halt anything below test(); until the await finished. Originally it was a bit more complex but I keep stripping it down to make it (hopefully) more basic/simple.
What am I missing in my thoughts or implementation?
Any help greatly appreciated as I'm chasing my tail at this point.

This isn't how async functions work. The function only appears to wait inside the function itself. Outside the function it is called and returns a promise synchronously.
In other words if you write:
let t = test()
t will be a promise that resolves when test() returns. In your current code, if you want to respond outside the function you would need something like:
async function test() {
let result = await http.post(appConfig.serviceRootUrl + '/api/XXX/YYY',
{ mProperty: myObject.collectionInObject.itemInCollection })
if (result.length < 1) return false
else if (result[0].infoIWant.trim().length > 0) return true;
}
// test is an async function. It returns a promise.
test().then(result => alert("result: " + result ))
Edit based on comments
Here's a working version using Axios for the http.post command:
async function test() {
let result = await axios.post('https://jsonplaceholder.typicode.com/posts',
{ mProperty: "some val" })
return result.data
}
// test is an async function. It returns a promise.
test().then(result => console.log(result ))
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>

How about doing this?
...Get some information the user wants and start to do some work to load
up the page and set up the form.
...old stuff working fine...
//Time for my new stuff
var testValue
async function test() {
let promise = new Promise( (resolve, reject) => resolve( http.post(appConfig.serviceRootUrl + '/api/XXX/YYY',
{ mProperty: myObject.collectionInObject.itemInCollection }).then(function (result) {
if (result.length < 1) {
testValue= false;
}
else if (result[0].infoIWant.trim().length > 0) {
testValue= true;
}
})));
await promise;
alert(testValue);
}
test();
//Originally above in the if I was just seeing if I got a result
//and setting testValue to true/false but changed it for debugging
//and I am surely getting back correct values when the data exists
//or result.length zero when no data for it
...Do a bunch of more stuff that is old and working correctly....
//Test the new stuff up above

If you use .then() syntax, don't await, and vice versa. I was incorrect about browser compatibility with async/await, seems I haven't kept up with browser scripting, in favor of Node. But also, since you're using jQuery, $.ajax() might be a good option for you, because you don't need async/await or .then(), and you can do like so:
$.ajax(appConfig.serviceRootUrl + '/api/XXX/YYY', {
method: 'POST',
data: { mProperty: myObject.collectionInObject.itemInCollection }
}).done(function(data) {
//use result here just as you would normally within the `.then()`
})
I hope this is more helpful than my original answer.

Call hierarchy of async functions inside a loop?

There's a async call I'm making that queries a database on a service, but this service has a limit of how many it can output at once, so I need to check if it hit its limit through the result it sends and repeat the query until it doesn't.
Synchronous mockup :
var query_results = [];
var limit_hit = true; #While this is true means that the query hit the record limit
var start_from = 0; #Pagination parameter
while (limit_hit) {
Server.Query(params={start_from : start_from}, callback=function(result){
limit_hit = result.limit_hit;
start_from = result.results.length;
query_result.push(result.results);
}
}
Obviously the above does not work, I've seen some other questions here about the issue, but they don't mention what to do when you need each iteration to wait for the last one to finish and you don't know before hand the number of iterations.
How can I turn the above asynchronous? I'm open to answers using promise/deferred-like logic, but preferably something clean.
I can probably think of a monstruous and horrible way of doing this using waits/timeouts, but there has to be a clean, clever and modern way to solve it.
Another way is to make a "pre-query" to know the number of features before hand so you know the number of loops, I'm not sure if this is the correct way.
Here we use Dojo sometimes, but the examples I found does not explain what to do when you have an unknown amount of loops https://www.sitepen.com/blog/2015/06/10/dojo-faq-how-can-i-sequence-asynchronous-operations/

although many answers already, still I believe async/await is the cleanest way.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
and you might need babel
https://babeljs.io/
JS async logic syntax changed from callback to promise then to async/await, they all do the same thing, when callback nests a lot we need something like a chain, then promise come, when promise goes in loop, we need something make the chain more plain more simple, then async/await come. But not all browsers support the new syntax, so babel come to compile new syntax to old syntax, then you can always code in new syntax.
getData().then((data) => {
//do something with final data
})
async function getData() {
var query_results = [];
var limit_hit = true;
var start_from = 0;
//when you use await, handle error with try/catch
try {
while (limit_hit) {
const result = await loadPage(start_from)
limit_hit = result.limit_hit;
start_from = result.results.length;
query_result.push(result.results);
}
} catch (e) {
//when loadPage rejects
console.log(e)
return null
}
return query_result
}
async function loadPage(start_from) {
//when you use promise, handle error with reject
return new Promise((resolve, reject) => Server.Query({
start_from
}, (result, err) => {
//error reject
if (err) {
reject(err)
return
}
resolve(result)
}))
}

If you want to use a loop then I think there is no (clean) way to do it without Promises.
A different approach would be the following:
var query_results = [];
var start_from = 0;
funciton myCallback(result) {
if(!result) {
//first call
Server.Query({ start_from: start_from}, myCallback);
} else {
//repeated call
start_from = result.results.length
query_result.push(result.results);
if(!result.limit_hit) {
//limit has not been hit yet
//repeat the query with new start value
Server.Query({ start_from: start_from}, myCallback);
} else {
//call some callback function here
}
}
}
myCallback(null);
You could call this recursive, but since the Query is asynchronous you shouldn't have problems with call stack limits etc.
Using promises in an ES6 environment you could make use of async/await. Im not sure if this is possible with dojo.

You don't understand callbacks until you have written a rate limiter or queue ;) The trick is to use a counter: Increment the counter before the async request, and decrement it when you get the response, then you will know how many requests are "in flight".
If the server is choked you want to put the item back in the queue.
There are many things you need to take into account:
What will happen to the queue if the process is killed ?
How long to wait before sending another request ?
Make sure the callback is not called many times !
How many times should you retry ?
How long to wait before giving up ?
Make sure there are no loose ends ! (callback is never called)
When all edge cases are taken into account you will have a rather long and not so elegant solution. But you can abstract it into one function! (that returns a Promise or whatever you fancy).
If you have a user interface you also want to show a loading bar and some statistics!

You must await for the server response every time. Here a encapsulated method
var query = (function(){
var results = [];
var count = 0;
return function check(fun){
Server.Query({ start_from: count}, function(d){
count = d.results.length;
results.push(d.results);
if (d.limit_hit && fun) fun(results);
else check(fun);
});
};
})();
// Call here
var my_query = query(function(d){
// --> retrive all data when limit_hit is true)
});

You can use a generator function Generators to achieve this
For POC:
some basics
- You define a generator with an asterick *
- it exposes a next function which returns the next value
- generators can pause with yield statement internally and can resume externally by calling the next()
- While (true) will ensure that the generator is not done until limit has reached
function *limitQueries() {
let limit_hit = false;
let start_from = 0;
const query_result = [];
while (true) {
if (limit_hit) {break;}
yield Server.Query(params={start_from : start_from},
callback=function* (result) {
limit_hit = result.limit_hit;
start_from = result.results.length;
yield query_result.push(result.results);
}
}
}
So apparently, the generator function maintains its own state. Generator function exposes two properties { value, done } and you can call it like this
const gen = limitQueries();
let results = [];
let next = gen.next();
while(next.done) {
next = gen.next();
}
results = next.value;
You might have to touch your Server.Query method to handle generator callback. Hope this helps! Cheers!

AngularJS $q.all - wait between http calls

So I have a situation where I need to perform a bunch of http calls, then once they are complete, continue on to the next step in the process.
Below is the code which does this and works fine.
However, I now need to wait a few seconds between each of the http calls. Is there a way to pass in a timeout with my current set up, or will it involve a good bit of refactoring?
Can post more code if needs be. I have tried passing in a timeout config varable into the http call, however, they still get fired at the same time.
Any advice would be great.
Code
var allThings = array.map(function(object) {
var singleThingPromise = getFile(object.id);
return singleThingPromise;
});
$q.all(allThings).then(function() {
deferred.resolve('Finished');
}, function(error) {
deferred.reject(error);
});

Instead of using $q.all, you might want to perform sequential calls one on success of previous and probably with use of $timeout. Maybe you could build a recursive function.
Something like this..
function performSequentialCalls (index) {
if(angular.isUndefined(array[index])) {
return;
}
getFile(array[index].id).then(function() {
$timeout(function() {
performSequentialCalls(index + 1)
}, 1000) // waiting 1 sec after each call
})
}
Inject required stuff properly. This assumes array to contain objects with ids using which you perform API calls. Also assumes that you are using $http. If using $resource, add $promise accordingly.
Hope that helps a bit!

function getItemsWithDelay(index) {
getFile(object[index].id).then(()=>{
setTimeout(()=>{
if(index+1 > object.length) { return }
getItemsWithDelay(index+1)
}, 5000)
})
}
You can make sequential calls

This is a awesome trick question to be asked in an interview, anyways I had a similar requirement and did some research on the internet and thanks to reference https://codehandbook.org/understanding-settimeout-inside-for-loop-in-javascript
I was able to delay all promise call in angularjs and the same can be applied in normal JS syntax as well.
I need to send tasks to a TTP API, and they requested to add a delay in each call
_sendTasks: function(taskMeta) {
var defer = $q.defer();
var promiseArray = [];
const delayIncrement = 1000 * 5;
let delay = 0;
for (i = 0; i < taskMeta.length; i++) {
// using 'let' keyword is VERY IMPORTANT else 'var' will send the same task in all http calls
let requestTask = {
"action": "SOME_ACTION",
"userId": '',
"sessionId": '',
};
// new Promise can be replaced with $q - you can try that, I haven't test it although.
promiseArray.push(new Promise(() => setTimeout(() => $http.post(config.API_ROOT_URL + '/' + requestTask.action, requestTask), delay)));
delay += delayIncrement;
}
$q.all(promiseArray).
then(function(results) {
// handle the results and resolve it at the end
defer.resolve(allResponses);
})
.catch(error => {
console.log(error);
defer.reject("failed to execute");
});
return defer.promise;
}
Note:: using 'let' keyword in FOR loop is VERY IMPORTANT else 'var' will send the same task in all http calls - due to closure/context getting switched

while(count!==3) fail to check callback state

I was learning Node with a tutorial called learnyounode from NodeSchool. This is about one of the 13 questions it provided: send 3 http get requests to 3 urls indicated by first 3 command line arguments, print out the responses in the order of urls when all the response chunks are collected.
I came up with this code snippet:
var http = require("http");
var count = 0;
var strArr = ["","",""];
getData(0);
getData(1);
getData(2);
while(count!==3);
console.log(strArr[0]);
console.log(strArr[1]);
console.log(strArr[2]);
function getData(i) {
http.get(process.argv[i+2], function (response) {
response.setEncoding("utf8");
response.on("data", function (data) {
strArr[i] += data;
});
response.on("end", function (data) {
count++;
});
});
}
I was expecting the while loop to hold back the print statements for me until count turns 3, that is, all 3 responses are gathered completely. However, it didn't work as I expected. Also, I put a print statement in the while loop and it showed that count would always be 0.
I then peeked the answer and learned that a way around is to check the value of count in the callback for response.on("end", ...), like below:
var http = require("http");
var count = 0;
var strArr = ["","",""];
getData(0);
getData(1);
getData(2);
function getData(i) {
http.get(process.argv[i+2], function (response) {
response.setEncoding("utf8");
response.on("data", function (data) {
strArr[i] += data;
});
response.on("end", function (data) {
count++;
if(count===3) {
console.log(strArr[0]);
console.log(strArr[1]);
console.log(strArr[2]);
}
});
});
}
This way, I did pass the test, but why the while-loop method didn't work out still puzzles me.
Thanks in advance for anyone who looks at this.

JavaScript is single-threaded. It executes each execution context until it is finished, then it checks with the event loop to see if there are any new execution contexts queued up that it should execute (such as the callback of an asynchronous function).
The three getData calls all return immediately, then the while loop executes in the thread. The callbacks to http.get cannot execute until the current execution context is finished (until the while loop and everything after it have executed), so there is no way for count to increase, and no way for the loop to end.
The solution you have found works well, but to help with understanding you should realize that setTimeout and setInterval are asynchronous, so they do not block the thread. You could have solved this with something like:
getData(0);
getData(1);
getData(2);
setTimeout( function check_count ( ) {
if ( count !== 3 )
return setTimeout( check_count, 100 );
console.log(strArr[0]);
console.log(strArr[1]);
console.log(strArr[2]);
}, 100 );
That's not a nice solution, since it is arbitrarily checking every 100 ms instead of just waiting until the third callback executes and then immediately logging the results. It is just a demonstatrion of how you can "loop" without blocking the thread.