Develop Reference

JavaScript how to determine when AJAX calls are done in a loop

I've spent the last few days trying to tackle this issue and have read all sorts of solutions on StackOverflow and other sites.
I'm building a site that grabs XML data from an outside source and then gets more XML depending on the results to build a network graph. The problem is that I have to essentially wait until this loop of AJAX calls (which may loop into more AJAX calls) is finished before drawing.
I don't know if this just has an especially high cognitive load, but it really has me stumped.
My Code:
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
Note: I understand I may be completely misunderstanding JavaScript synchronicity here, and I very likely am. I have used callbacks and promises successfully in the past, I just can't seem to wrap my head around this one.
If I have not been totally clear, please let me know.
I did try implementing a counter of sorts that is incremented in the process() function, like so:
if (processCount < 15) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
} else {
draw(nodes, edges);
}
However, this ended up with several draw() calls which made my performance abysmal.

There are nice new well-supported APIs and language constructs we can use. The Fetch API, await, and for...of loops.
The Fetch API uses Promises. Promises can be awaited. The for...of loop is aware of await and won't continue the loop until the await has passed.
// Loop through, one-at-a-time
for (const node of outerNodes) {
// Make the HTTP request
const res = await fetch(someUrl);
// Do something with the response here...
}
Don't forget a try/catch (which also works with await), and check res.ok.

Brad's answer changes the code to by synchronious and to me that defeats the purpose. If you are constantly waiting on all request to be finished then it could take a while, while normal browsers can handle multiple requests.
The problem you have in your original questions is with scope. Since each call to cont(outerNodes) will trigger it's own scope, it has no idea what are calls are doing. So basically if you call cont(outerNodes) twice, each call will handle it's own list of outernodes and then call draw.
The solution is to share information between the different scopes, which can be done with one, but preferably two global variables: 1 to track active processes and 1 to track errors.
var inProcess = 0;
var nrErrors = 0;
function cont(outerNodes) {
//make sure you have outerNodes before you call outerNodes.length
if (outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
inProcess++; //add one more in process
getXML(node["label"], node["id"]);
}
}
//only trigger when nothing is in proces.
if (inProcess==0) {
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
inProcess--; //one is done
cont(outerNodes);
},
error: function() {
inProcess--; //one is done
nrErrors++; //one more error
cont(null); //run without new outerNodes, to trigger a possible draw
}
});
}
Please note that I track nrErrors but dont do anything with it. You could use it to stop further processing all together or warn the user that the draw is not complete.
[important] Keep in mind that this works in javascript because at best it mimics multithreading. That means the the call to inProcess--; and then right after cont(outerNodes); is always execute directly after eachother.
If you would port this to a true multithreading environment, it could very well be that another scope/version of cont(null); would cut in between the two lines and there would still be multiple draws.

The best way to solve this question should be using either promise or callback.
If you really want to avoid promise or callback(Although i don't know why...)
You can try with a counter.
let processCount = 0;
// Increasing the processCount in getXML callback method
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
processCount++;
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
while (processCount < outerNodes.length) {
// do nothing, just wait'
}
draw(nodes, edges);

If after testing it many times, you know that it will never take more than say 5 seconds... you can use a setTimeout.
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// Display a 5 second progress bar here
setTimeout(function(){ draw(nodes, edges); },5000);
}

AngularJS $q.all - wait between http calls

So I have a situation where I need to perform a bunch of http calls, then once they are complete, continue on to the next step in the process.
Below is the code which does this and works fine.
However, I now need to wait a few seconds between each of the http calls. Is there a way to pass in a timeout with my current set up, or will it involve a good bit of refactoring?
Can post more code if needs be. I have tried passing in a timeout config varable into the http call, however, they still get fired at the same time.
Any advice would be great.
Code
var allThings = array.map(function(object) {
var singleThingPromise = getFile(object.id);
return singleThingPromise;
});
$q.all(allThings).then(function() {
deferred.resolve('Finished');
}, function(error) {
deferred.reject(error);
});

Instead of using $q.all, you might want to perform sequential calls one on success of previous and probably with use of $timeout. Maybe you could build a recursive function.
Something like this..
function performSequentialCalls (index) {
if(angular.isUndefined(array[index])) {
return;
}
getFile(array[index].id).then(function() {
$timeout(function() {
performSequentialCalls(index + 1)
}, 1000) // waiting 1 sec after each call
})
}
Inject required stuff properly. This assumes array to contain objects with ids using which you perform API calls. Also assumes that you are using $http. If using $resource, add $promise accordingly.
Hope that helps a bit!

function getItemsWithDelay(index) {
getFile(object[index].id).then(()=>{
setTimeout(()=>{
if(index+1 > object.length) { return }
getItemsWithDelay(index+1)
}, 5000)
})
}
You can make sequential calls

This is a awesome trick question to be asked in an interview, anyways I had a similar requirement and did some research on the internet and thanks to reference https://codehandbook.org/understanding-settimeout-inside-for-loop-in-javascript
I was able to delay all promise call in angularjs and the same can be applied in normal JS syntax as well.
I need to send tasks to a TTP API, and they requested to add a delay in each call
_sendTasks: function(taskMeta) {
var defer = $q.defer();
var promiseArray = [];
const delayIncrement = 1000 * 5;
let delay = 0;
for (i = 0; i < taskMeta.length; i++) {
// using 'let' keyword is VERY IMPORTANT else 'var' will send the same task in all http calls
let requestTask = {
"action": "SOME_ACTION",
"userId": '',
"sessionId": '',
};
// new Promise can be replaced with $q - you can try that, I haven't test it although.
promiseArray.push(new Promise(() => setTimeout(() => $http.post(config.API_ROOT_URL + '/' + requestTask.action, requestTask), delay)));
delay += delayIncrement;
}
$q.all(promiseArray).
then(function(results) {
// handle the results and resolve it at the end
defer.resolve(allResponses);
})
.catch(error => {
console.log(error);
defer.reject("failed to execute");
});
return defer.promise;
}
Note:: using 'let' keyword in FOR loop is VERY IMPORTANT else 'var' will send the same task in all http calls - due to closure/context getting switched

Async request into for loop angular.js

I have an array and i need to send values of array to webservice through http post request one by one . For the node.js , i'm using "async" package to do that for ex: async.eachSeries doing it well , how can i do that same thing for angular.js , my normal async code;
//this code sends all queries of array (maybe 5.000 request at same time , it is hard to process for webservice :=) ) at same time and wait for all responses.
//it works but actually for me , responses should wait others at end of loop should work one by one
//like async.eachSeries module!
for (var i = 0; i < myArr.lenght; i++) {
(function (i) {
var data = {
"myQuery": myArr[i].query
};
$http.post("/myServiceUrl", data).success(function (result) {
console.log(result);
});
})(i);
}
Both Matt Way and Chris L answers Correct , you can investigate Chris's answer for understanding about async to sync functions in for loops.

You can use $q to create a similar requirement by chaining promises together. For example:
var chain = $q.when();
angular.forEach(myArr, function(item){
chain = chain.then(function(){
var data = {
myQuery: item.query
};
return $http.post('/myServiceUrl', data).success(function(result){
console.log(result);
});
});
});
// the final chain object will resolve once all the posts have completed.
chain.then(function(){
console.log('all done!');
});
Essentially you are just running the next promise once the previous one has completed. Emphasis here on the fact that each request will wait until the previous one has completed, as per your question.

function logResultFromWebService(value)
{
$http.post("/myServiceUrl", value).success(console.log);
}
angular.forEach(myArray, logResultFromWebService);

If I understand your question correctly. You want to run a for loop in a synchronized manner such that the next iteration only occurs once the previous iteration is completed. For that, you can use a synchronized loop/callbacks. Especially if the order matters.
var syncLoop = function (iterations, process, exit) {
var index = 0,
done = false,
shouldExit = false;
var loop = {
next: function () {
if (done) {
if (shouldExit && exit) {
return exit(); // Exit if we're done
}
}
// If we're not finished
if (index < iterations) {
index++; // Increment our index
process(loop); // Run our process, pass in the loop
// Otherwise we're done
} else {
done = true; // Make sure we say we're done
if (exit) exit(); // Call the callback on exit
}
},
iteration: function () {
return index - 1; // Return the loop number we're on
},
break: function (end) {
done = true; // End the loop
shouldExit = end; // Passing end as true means we still call the exit callback
}
};
console.log('running first time');
loop.next();
return loop;
}
For your particular implementation:
syncLoop(myArray.length, function (loop) {
var index = loop.iteration();
var data = {
"myQuery": myArray[index].query
};
$http.post("/myServiceUrl", data).success(function (result) {
console.log(result);
loop.next();
});
}, function () {
console.log('done');
});
If you intend on doing something with the data once returned (such as perform calculations) you can do so with this method because you will return the data in a specified order.
I implemented something similar in a statistical calculation web app I built.
EDIT:
To illustrate the problem I had when using $q.when I have set up a fiddle. Hopefully this will help illustrate why I did this the way I did.
https://jsfiddle.net/chrislewispac/6atp3w8o/
Using the following code from Matt's answer:
var chain = $q.when(promise.getResult());
angular.forEach(myArr, function (item) {
chain = chain.then(function () {
$rootScope.status = item;
console.log(item);
});
});
// the final chain object will resolve once all the posts have completed.
chain.then(function () {
console.log('all done!');
});
And this fiddle is an example of my solution:
https://jsfiddle.net/chrislewispac/Lgwteone/3/
Compare the $q version to my version. View the console and imagine those being delivered to the user interface for user intervention in the process and/or performing statistical operations on the sequential returns.
You will see that it does not sequentially give the numbers 1,2,3,4 etc. either in the console or in the view in Matt's answer. It 'batches' the responses and then returns them. Therefore, if step 3 is not to be run depending on the response in step 2 there is not, at least in the answer provided, a way to break out or explicitly control the synchronous operation here. This presents a significant problem when attempting to perform sequential calculations and/or allow the user to control break points, etc.
Now, I am digging through both the $q libraries and the Q library to see if there is a more elegant solution for this problem. However, my solution does work as requested and is very explicit which allows me to place the function in a service and manipulate for certain use cases at my will because I completely understand what it is doing. For me, that is more important than using a library (at least at this stage in my development as a programmer and I am sure there are lots of other people at the same stage on StackOverflow as well).

If the order doesn't matter in which they are sent
var items = [/* your array */];
var promises = [];
angular.forEach(items, function(value, key){
var promise = $http.post("/myServiceUrl", { "myQuery": value.query });
promises.push(promise);
});
return $q.all(promises);

Node module: Don't return until all async requests have finished

I'm new to node and am having trouble understanding node's async behavior. I know this is a very frequently addressed question on SO, but I simply can't understand how to get any of the solutions I've read to work in my context.
I'm writing this module which I want to return an object containing various data.
var myModule = (function () {
var file,
fileArray,
items = [],
getBlock = function (fileArray) {
//get the data from the file that I want, return object
return block;
},
parseBlock = function (block) {
//[find various items in the block, put them into an "myItems" object, then
//take the items and do a look up against a web api as below]...
for (var i = 0, l = myItems.length; i < l; i ++) {
(function (i) {
needle.post(MY_URL, qstring, function(err, resp, body){
if (!err && resp.statusCode === 200){
myItems[i].info = body;
if (i === (myItems.length -1)) {
return myItems;
}
}
});
})(i);
}
},
getSomeOtherData = function (fileArray) {
//parse some other data from the file
}
return {
setFile: function (file) {
fileArray = fs.readFileSync(file).toString().split('\n');
},
render: function () {
var results = [];
results.someOtherData = getsomeOtherData();
var d = getBlock();
results.items = parseBlock(d);
return results;
}
}
})();
When I call this module using:
myModule.setFile('myFile.txt');
var res = myModule.render();
the variable res has the values from the someOtherData property, but not the items property. I understand that my long-running http request has not completed and that node just zooms ahead and finishes executing, but that's not what I want. I looked at a bunch of SO questions on this, and looked at using Q or queue-async, but with no success.
How do I get this module to return no data until all requests have completed? Or is that even possible in node? Is there a better way to design this to achieve my goal?

The problem in your example is your calling getBlock() but you have declared your function as getBlockData(). So you will not get a result. Try changing it to both the same.
Presuming that you have them both the same, your next problem is that your processing data from a file, so I presume that your reading the contents of the file and then parsing it.
If this is the case then there are sync reads that you can use to force sync, however I wouldn't recommend this.
You really want to structure your program based on events. Your thinking in the paradigm of 'call a function, when it returns continue'. You need to be thinking more along the lines of 'call a process and add a listener, the listener then does reply handling'.
This works very well for comms. You receive a request. You need to reply based on contents of file. So you start the read process with two possible results. It calls the completed function or the error function. Both would then call the reply function to process how to handle a reply for the request.
It's important not to block as you will be blocking the thread via which all processes are handled.
Hope that helps, if not add some comments and I will try and elaborate.
Have a look at this answer to another question to see a good example of processing a file using the standard listeners. All async calls have a listener concept for what can happen. All you need to do is pass a function name (or anon if you prefer) to them when you call them.
A quick example (based on node.js stream.Readable API:
fs.createReadStream(filename, {
'flags': 'r'
}).addListener( "data", function(chunk) {
// do your processing logic
}).addListener( "end", function(chunk) {
// do your end logic
response(...);
}).addListener( "error", function(chunk) {
// do your error logic
response(...);
}).addListener( "close",function() {
// do your close logic
});
function response(info) {
}

Sequencing ajax requests

I find I sometimes need to iterate some collection and make an ajax call for each element. I want each call to return before moving to the next element so that I don't blast the server with requests - which often leads to other issues. And I don't want to set async to false and freeze the browser.
Usually this involves setting up some kind of iterator context that i step thru upon each success callback. I think there must be a cleaner simpler way?
Does anyone have a clever design pattern for how to neatly work thru a collection making ajax calls for each item?

jQuery 1.5+
I developed an $.ajaxQueue() plugin that uses the $.Deferred, .queue(), and $.ajax() to also pass back a promise that is resolved when the request completes.
/*
* jQuery.ajaxQueue - A queue for ajax requests
*
* (c) 2011 Corey Frang
* Dual licensed under the MIT and GPL licenses.
*
* Requires jQuery 1.5+
*/
(function($) {
// jQuery on an empty object, we are going to use this as our Queue
var ajaxQueue = $({});
$.ajaxQueue = function( ajaxOpts ) {
var jqXHR,
dfd = $.Deferred(),
promise = dfd.promise();
// queue our ajax request
ajaxQueue.queue( doRequest );
// add the abort method
promise.abort = function( statusText ) {
// proxy abort to the jqXHR if it is active
if ( jqXHR ) {
return jqXHR.abort( statusText );
}
// if there wasn't already a jqXHR we need to remove from queue
var queue = ajaxQueue.queue(),
index = $.inArray( doRequest, queue );
if ( index > -1 ) {
queue.splice( index, 1 );
}
// and then reject the deferred
dfd.rejectWith( ajaxOpts.context || ajaxOpts,
[ promise, statusText, "" ] );
return promise;
};
// run the actual query
function doRequest( next ) {
jqXHR = $.ajax( ajaxOpts )
.done( dfd.resolve )
.fail( dfd.reject )
.then( next, next );
}
return promise;
};
})(jQuery);
jQuery 1.4
If you're using jQuery 1.4, you can utilize the animation queue on an empty object to create your own "queue" for your ajax requests for the elements.
You can even factor this into your own $.ajax() replacement. This plugin $.ajaxQueue() uses the standard 'fx' queue for jQuery, which will auto-start the first added element if the queue isn't already running.
(function($) {
// jQuery on an empty object, we are going to use this as our Queue
var ajaxQueue = $({});
$.ajaxQueue = function(ajaxOpts) {
// hold the original complete function
var oldComplete = ajaxOpts.complete;
// queue our ajax request
ajaxQueue.queue(function(next) {
// create a complete callback to fire the next event in the queue
ajaxOpts.complete = function() {
// fire the original complete if it was there
if (oldComplete) oldComplete.apply(this, arguments);
next(); // run the next query in the queue
};
// run the query
$.ajax(ajaxOpts);
});
};
})(jQuery);
Example Usage
So, we have a <ul id="items"> which has some <li> that we want to copy (using ajax!) to the <ul id="output">
// get each item we want to copy
$("#items li").each(function(idx) {
// queue up an ajax request
$.ajaxQueue({
url: '/echo/html/',
data: {html : "["+idx+"] "+$(this).html()},
type: 'POST',
success: function(data) {
// Write to #output
$("#output").append($("<li>", { html: data }));
}
});
});
jsfiddle demonstration - 1.4 version

A quick and small solution using deferred promises. Although this uses jQuery's $.Deferred, any other should do.
var Queue = function () {
var previous = new $.Deferred().resolve();
return function (fn, fail) {
return previous = previous.then(fn, fail || fn);
};
};
Usage, call to create new queues:
var queue = Queue();
// Queue empty, will start immediately
queue(function () {
return $.get('/first');
});
// Will begin when the first has finished
queue(function() {
return $.get('/second');
});
See the example with a side-by-side comparison of asynchronous requests.
This works by creating a function that will automatically chain promises together. The synchronous nature comes from the fact that we are wrapping $.get calls in function and pushing them into a queue. The execution of these functions are deferred and will only be called when it gets to the front of the queue.
A requirement for the code is that each of the functions you give must return a promise. This returned promise is then chained onto the latest promise in the queue. When you call the queue(...) function it chains onto the last promise, hence the previous = previous.then(...).

You can wrap all that complexity into a function to make a simple call that looks like this:
loadSequantially(['/a', '/a/b', 'a/b/c'], function() {alert('all loaded')});
Below is a rough sketch (working example, except the ajax call). This can be modified to use a queue-like structure instead of an array
// load sequentially the given array of URLs and call 'funCallback' when all's done
function loadSequantially(arrUrls, funCallback) {
var idx = 0;
// callback function that is called when individual ajax call is done
// internally calls next ajax URL in the sequence, or if there aren't any left,
// calls the final user specified callback function
var individualLoadCallback = function() {
if(++idx >= arrUrls.length) {
doCallback(arrUrls, funCallback);
}else {
loadInternal();
}
};
// makes the ajax call
var loadInternal = function() {
if(arrUrls.length > 0) {
ajaxCall(arrUrls[idx], individualLoadCallback);
}else {
doCallback(arrUrls, funCallback);
}
};
loadInternal();
};
// dummy function replace with actual ajax call
function ajaxCall(url, funCallBack) {
alert(url)
funCallBack();
};
// final callback when everything's loaded
function doCallback(arrUrls, func) {
try {
func();
}catch(err) {
// handle errors
}
};

Ideally, a coroutine with multiple entry points so every callback from server can call the same coroutine will be neat. Damn, this is about to be implemented in Javascript 1.7.
Let me try using closure...
function BlockingAjaxCall (URL,arr,AjaxCall,OriginalCallBack)
{
var nextindex = function()
{
var i =0;
return function()
{
return i++;
}
};
var AjaxCallRecursive = function(){
var currentindex = nextindex();
AjaxCall
(
URL,
arr[currentindex],
function()
{
OriginalCallBack();
if (currentindex < arr.length)
{
AjaxCallRecursive();
}
}
);
};
AjaxCallRecursive();
}
// suppose you always call Ajax like AjaxCall(URL,element,callback) you will do it this way
BlockingAjaxCall(URL,myArray,AjaxCall,CallBack);

Yeah, while the other answers will work, they are lots of code and messy looking. Frame.js was designed to elegantly address this situation. https://github.com/bishopZ/Frame.js
For instance, this will cause most browsers to hang:
for(var i=0; i<1000; i++){
$.ajax('myserver.api', { data:i, type:'post' });
}
While this will not:
for(var i=0; i<1000; i++){
Frame(function(callback){
$.ajax('myserver.api', { data:i, type:'post', complete:callback });
});
}
Frame.start();
Also, using Frame allows you to waterfall the response objects and deal with them all after the entire series of AJAX request have completed (if you want to):
var listOfAjaxObjects = [ {}, {}, ... ]; // an array of objects for $.ajax
$.each(listOfAjaxObjects, function(i, item){
Frame(function(nextFrame){
item.complete = function(response){
// do stuff with this response or wait until end
nextFrame(response); // ajax response objects will waterfall to the next Frame()
$.ajax(item);
});
});
Frame(function(callback){ // runs after all the AJAX requests have returned
var ajaxResponses = [];
$.each(arguments, function(i, arg){
if(i!==0){ // the first argument is always the callback function
ajaxResponses.push(arg);
}
});
// do stuff with the responses from your AJAX requests
// if an AJAX request returned an error, the error object will be present in place of the response object
callback();
});
Frame.start()

I am posting this answer thinking that it might help other persons in future, looking for some simple solutions in the same scenario.
This is now possible also using the native promise support introduced in ES6. You can wrap the ajax call in a promise and return it to the handler of the element.
function ajaxPromise(elInfo) {
return new Promise(function (resolve, reject) {
//Do anything as desired with the elInfo passed as parameter
$.ajax({
type: "POST",
url: '/someurl/',
data: {data: "somedata" + elInfo},
success: function (data) {
//Do anything as desired with the data received from the server,
//and then resolve the promise
resolve();
},
error: function (err) {
reject(err);
},
async: true
});
});
}
Now call the function recursively, from where you have the collection of the elements.
function callAjaxSynchronous(elCollection) {
if (elCollection.length > 0) {
var el = elCollection.shift();
ajaxPromise(el)
.then(function () {
callAjaxSynchronous(elCollection);
})
.catch(function (err) {
//Abort further ajax calls/continue with the rest
//callAjaxSynchronous(elCollection);
});
}
else {
return false;
}
}

I use http://developer.yahoo.com/yui/3/io/#queue to get that functionality.
The only solutions I can come up with is, as you say, maintaining a list of pending calls / callbacks. Or nesting the next call in the previous callback, but that feels a bit messy.

You can achieve the same thing using then.
var files = [
'example.txt',
'example2.txt',
'example.txt',
'example2.txt',
'example.txt',
'example2.txt',
'example2.txt',
'example.txt'
];
nextFile().done(function(){
console.log("done",arguments)
});
function nextFile(text){
var file = files.shift();
if(text)
$('body').append(text + '<br/>');
if(file)
return $.get(file).then(nextFile);
}
http://plnkr.co/edit/meHQHU48zLTZZHMCtIHm?p=preview

I would suggest a bit more sophisticated approach which is reusable for different cases.
I am using it for example when I need to slow down a call sequence when the user is typing in text editor.
But I am sure it should also work when iterating through the collection. In this case it can queue requests and can send a single AJAX call instead of 12.
queueing = {
callTimeout: undefined,
callTimeoutDelayTime: 1000,
callTimeoutMaxQueueSize: 12,
callTimeoutCurrentQueueSize: 0,
queueCall: function (theCall) {
clearTimeout(this.callTimeout);
if (this.callTimeoutCurrentQueueSize >= this.callTimeoutMaxQueueSize) {
theCall();
this.callTimeoutCurrentQueueSize = 0;
} else {
var _self = this;
this.callTimeout = setTimeout(function () {
theCall();
_self.callTimeoutCurrentQueueSize = 0;
}, this.callTimeoutDelayTime);
}
this.callTimeoutCurrentQueueSize++;
}
}

There's a very simple way to achieve this by adding async: false as a property to the ajax call. This will make sure the ajax call is complete before parsing the rest of the code. I have used this successfully in loops many times.
Eg.
$.ajax({
url: "",
type: "GET",
async: false
...