Develop Reference

How to run a function in a loop multiple times asynchronously in node.js using the async module

I am reading multiple google docs using the google-spreadsheet module. This involves looping through all of the credentials for each module I have saved in a json file and then pulling the spreadsheet. So if there are 10 spreadsheet credentials in my json file, I will need to within a loop call the function with these details, 10 times.
I will then save the information from these spreadsheets. I need to do this asynchronously. How can I achieve this, maybe using the async module? I tried to understand the queue aspect of this module but couldn't make much sense of it.
for (var i = 0; i < sheets.length; i++) {
// All of the below could be put in a separate function that needs to be run asynchronously
var sheetID=sheets[i]
// spreadsheet key is the long id in the sheets URL
var doc = new GoogleSpreadsheet(sheetID);
doc.useServiceAccountAuth(creds, function(err) {
if (err) {
console.log(err);
return
}
doc.getInfo(function(err, info) {
if (err) {
return console.log(err);
}
sheet = info.worksheets[0];
sheet.getRows({
orderby: "marketname",
start: 0
}, function(err, row_data) {
for (var i = 0; i < row_data.length; i++) {
console.log(row_data[i].marketname);
}
})
})
})
}

So if I understand correctly what you want to do is an asynchronous for loop. This mean that you have an async treatment (here everything under your comment), that you need to chain together, one after another.
Without using the async lib, you can do it as in this example
// Async task
function async(arg, callback) {
//You put in here what needs to be done with each of your data
}
// Final task: display or save your data in your files
function final() { console.log('Done', results); }
// A simple async series:
var items = [ 1, 2, 3, 4, 5, 6 ];
var results = [];
//Recursive function that will go through your items.
function series(item) {
//if there is still some item to work with
if(item) {
//call your async function with our current item
async( item, function(result) {
//push in the results
results.push(result);
//Go to the next item.
return series(items.shift());
});
} else {
//Otherwise this means we're done, and results[] is populated!
return final();
}
}
//Start the treatment
series(items.shift());
Hope this helps!
You can find more information about some control flow in this great paper.

node.js nested callback, get final results array

I am doing a for loop to find the result from mongodb, and concat the array. But I am not getting the final results array when the loop is finished. I am new to node.js, and I think it's not working like objective-c callback.
app.get('/users/self/feed', function(req, res){
var query = Bill.find({user: req.param('userId')});
query.sort('-createdAt');
query.exec(function(err, bill){
if (bill) {
var arr = bill;
Following.findOne({user: req.param('userId')}, function(err, follow){
if (follow) {
var follows = follow.following; //this is a array of user ids
for (var i = 0; i < follows.length; i++) {
var followId = follows[i];
Bill.find({user: followId}, function(err, result){
arr = arr.concat(result);
// res.send(200, arr);// this is working.
});
}
} else {
res.send(400, err);
}
});
res.send(200, arr); //if put here, i am not getting the final results
} else {
res.send(400, err);
}
})
});

While I'm not entirely familiar with MongoDB, a quick reading of their documentation shows that they provide an asynchronous Node.js interface.
That said, both the findOne and find operations start, but don't necessarily complete by the time you reach res.send(200, arr) meaning arr will still be empty.
Instead, you should send your response back once all asynchronous calls complete meaning you could do something like:
var billsToFind = follows.length;
for (var i = 0; i < follows.length; i++) {
var followId = follows[i];
Bill.find({user: followId}, function(err, result){
arr = arr.concat(result);
billsToFind -= 1;
if(billsToFind === 0){
res.send(200, arr);
}
});
}
The approach uses a counter for all of the inner async calls (I'm ignoring the findOne because we're currently inside its callback anyway). As each Bill.find call completes it decrements the counter and once it reaches 0 it means that all callbacks have fired (this works since Bill.find is called for every item in the array follows) and it sends back the response with the full array.

That's true. Your codes inside for will be executed in parallel at the same time (and with the same value of i I think). If you added console.log inside and after your for loop you will found the outside one will be printed before inside one.
You can wrap the code that inside your for into array of functions and execute them by using async module (https://www.npmjs.org/package/async) in parallel or series, and retrieve the final result from async.parallel or async.series's last parameter.

How to capture results from end of FOR loop with Nested/Dependent APIs calls in Node JS

This is my first JavaScript & Node project and I am stuck….
I am trying to call a REST API that returns a set of Post IDs... and based on the set of retrieved IDs I am trying to call another API that returns details for each ID from the first API. The code uses Facebook API provided by Facebook-NodeSDK.
The problem I am having is that the second API fires of in a FOR Loop…. As I understand the for loop executes each request asynchronously…. I can see both the queries executing however I can’t figure out how to capture the end of the second for loop to return the final result to the user…
Following is the code…
exports.getFeeds = function(req, res) {
var posts = [];
FB.setAccessToken(’SOME TOKEN');
var resultLength = 0;
FB.api(
//ARG #1 FQL Statement
'fql', { q: 'SELECT post_id FROM stream WHERE filter_key = "others"' },
//ARG #2 passing argument as a anonymous function with parameter result
function (result)
{
if(!result || result.error) {
console.log(!result ? 'error occurred' : result.error);
return;
} //closing if handling error in this block
var feedObj
console.log(result.data);
console.log(result.data.length);
for (var i = 0; i<resultLengthj ; i++) {
(function(i) {
feedObj = {};
FB.api( result.data[ i].post_id, { fields: ['name', 'description', 'full_picture' ] },
// fbPost is data returned by query
function (fbPost) {
if(!fbPost || fbPost.error) {
console.log(!fbPost ? 'error occurred' : result.error);
return;
}
// else
feedObj=fbPost;
posts.push(feedObj);
});
})(i);
}// end for
}//CLOSE ARG#2 Function
);// close FB.api Function
NOTE I need to call…... res.Send(post)…. and have tried to call it at several places but just can’t get all the posts… I have removed the console statements from the above code…which have shown that the data is being retrieved...
Thanks a lot for your help and attention....

If you just stick res.send almost anywhere in your code, it will be certain to get called before your posts have returned.
What you want to do in a case like this is to declare a counter variable outside your for loop and set it to zero. Then increment it inside the for loop. Then inside your inner callback (in your case, the one that is getting called once for each post), you would decrement the counter and test for when it hits zero. Below I apply the technique to an edited version of your code (I didn't see what feedObj was actually doing, nor did I understand why you were using the immediately-invoked function, so eliminated both - please let me know if i missed something there).
var posts = [];
FB.setAccessToken(’SOME TOKEN');
var resultLength = 0;
FB.api(
//ARG #1 FQL Statement
'fql', { q: 'SELECT post_id FROM stream WHERE filter_key = "others"' },
//ARG #2 passing argument as a anonymous function with parameter result
function (result)
{
if(!result || result.error) {
return;
} //closing if handling error in this block
var counter = 0;
for (var i = 0; i<resultLengthj ; i++) {
counter++;
FB.api( result.data[ i].post_id, { fields: ['name', 'description', 'full_picture' ] },
function (fbPost) { // fbPost is data returned by query
if(!fbPost || fbPost.error) {
return;
}
posts.push(fbPost);
counter--;
if (counter === 0){
// Let's render that page/send that result, etc
}
});
}// end for
}//CLOSE ARG#2 Function
);// close FB.api Function
Hope this helps.

Essentially you are wanting to do an async map operation for each id.
There is a really handy library for doing async operations on collections called async that has a async.map method.
var async = require('async');
FB.api(
'fql', { q: 'SELECT post_id FROM stream WHERE filter_key = "others"' },
//ARG #2 passing argument as a anonymous function with parameter result
function (result) {
async.map(
result,
function (item, callback) {
FB.api(
item.post_id,
{ fields: ['name', 'description', 'full_picture' ] },
callback
);
},
function (err, allPosts) {
if (err) return console.log(err);
// Array of all posts
console.log(allPosts);
}
);
}
);
You definitely don't need to use this, but it simplifies your code a bit. Just run npm install --save async in your project directory and you should be good to go.

Handling multiple call asynchronous callbacks

I am learning node.js with learnyounode.
I am having a problem with JUGGLING ASYNC.
The problem is described as follows:
You are given three urls as command line arguments. You are supposed to make http.get() calls to get data from these urls and then print them in the same order as their order in the list of arguments.
Here is my code:
var http = require('http')
var truecount = 0;
var printlist = []
for(var i = 2; i < process.argv.length; i++) {
http.get(process.argv[i], function(response) {
var printdata = "";
response.setEncoding('utf8');
response.on('data', function(data) {
printdata += data;
})
response.on('end', function() {
truecount += 1
printlist.push(printdata)
if(truecount == 3) {
printlist.forEach(function(item) {
console.log(item)
})
}
})
})
}
Here is the questions I do not understand:
I am trying to store the completed data in response.on('end', function(){})for each url using a dictionary. However, I do not know how to get the url for that http.get(). If I can do a local variable inside http.get(), that would be great but I think whenever I declare a variable as var url, it will always point to the last url. Since it is global and it keeps updating through the loop. What is the best way for me to store those completed data as the value with the key equal to the url?

This is how I would go about solving the problem.
#!/usr/bin/env node
var http = require('http');
var argv = process.argv.splice(2),
truecount = argv.length,
pages = [];
function printUrls() {
if (--truecount > 0)
return;
for (i = 0; i < pages.length; i++) {
console.log(pages[i].data + '\n\n');
}
}
function HTMLPage(url) {
var _page = this;
_page.data = '### [URL](' + url + ')\n';
http.get(url, function(res) {
res.setEncoding('utf8');
res.on('data', function(data) {
_page.data += data;
});
res.on('end', printUrls);
});
}
for (var i = 0; i < argv.length; i++)
pages.push(new HTMLPage(argv[i]));
It adds the requests to an array on the start of each request, that way once done I can iterate nicely through the responses knowing that they are in the correct order.
When dealing with asynchronous processing, I find it much easier to think about each process as something with a concrete beginning and end. If you require the order of the requests to be preserved then the entry must be made on creation of each process, and then you refer back to that record on completion. Only then can you guarantee that you have things in the right order.
If you were desperate to use your above method, then you could define a variable inside your get callback closure and use that to store the urls, that way you wouldn't end up with the last url overwriting your variables. If you do go this way though, you'll dramatically increase your overhead when you have to use your urls from process.argv to access each response in that order. I wouldn't advise it.

I went about this challenge a little differently. I'm creating an array of functions that call http.get, and immediately invoking them with their specifc context. The streams write to an object where the key is the port of the server which that stream is relevant to. When the end event is triggered, it adds to that server to the completed array - when that array is full it iterates through and echos in the original order the servers were given.
There's no right way but there are probably a dozen or more ways. Wanted to share mine.
var http = require('http'),
request = [],
dataStrings = {},
orderOfServerInputs = [];
var completeResponses = [];
for(server in process.argv){
if(server >= 2){
orderOfServerInputs[orderOfServerInputs.length] = process.argv[server].substr(-4);
request[request.length] = function(thisServer){
http.get(process.argv[server], function(response){
response.on("data", function(data){
dataStrings[thisServer.substr(-4)] = dataStrings[thisServer.substr(-4)] ? dataStrings[thisServer.substr(-4)] : ''; //if not set set to ''
dataStrings[thisServer.substr(-4)] += data.toString();
});
response.on("end", function(data){
completeResponses[completeResponses.length] = true;
if(completeResponses.length > 2){
for(item in orderOfServerInputs){
serverNo = orderOfServerInputs[item].substr(-4)
console.log(dataStrings[serverNo]);
}
}
});
});
}(process.argv[server]);
}
}

Immediately-Invoked Function Expression (IIFE) could be a solution to your problem. It allows us to bind to function a specific value, in your case, the url which gets the response. In the code below, I bind variable i to index and so, whichever url gets the response, that index of print list will be updated. For more information, refer to this website
var http = require('http')
var truecount = 0;
var printlist = [];
for(var i = 2; i < process.argv.length; i++) {
(function(index){
http.get(process.argv[index], function(response) {
response.setEncoding('utf8');
response.on('data', function(data) {
if (printlist[index] == undefined)
printlist[index] = data;
else
printlist[index]+= data;
})
response.on('end', function() {
truecount += 1
if(truecount == 3) {
printlist.forEach(function(item) {
console.log(item)
})
}
})
})
})(i)
}

node.js and express : Sequential execution flow one mongodb query request after another

I have a webserver running in node.js and Express which retrieves data from mongodb . In mongodb collections are getting created dynamically and the name of newly created collection will be stored in one metadata collection “project” . My requirement is to firstly iterate to metadata collection to get the collection name and then get inside the each collection to do multiple query based on some condition . Because my collection metadata is dynamic I have tried to do using for loop .
But it is giving wrong data . It is not executing sequent . Before finishing the loop execution it is returning the value .How to perform sequential execution in node.js using node core modules only (Not other library like async..);
exports.projectCount = function (req, res) {
var mongo = require("mongodb"),
Server = mongo.Server,
Db = mongo.Db;
var server = new Server("localhost", 27017, {
auto_reconnect: true
});
var db = new Db("test", server);
// global JSON object to store manipulated data
var projectDetail = {
projectCount: 0,
projectPercent: 0
};
var totalProject = 0;
db.open(function (err, collection) {
//metadata collection
collection = db.collection("project");
collection.find().toArray(function (err, result) {
// Length of metadata collection
projectDetail.projectCount = result.length;
var count = 0;
//iterate through each of the array which is the name of collection
result.forEach(function (item) {
//change collection object to new collection
collection = db.collection(item.keyParameter.wbsName);
// Perform first query based on some condition
collection.find({
$where: "this.status == 'Created'"
}).toArray(function (err, result) {
// based on result of query one increment the value of count
count += result.lenght;
// Perform second query based on some condition
collection.find({
$where: "this.status=='Completed'"
}).toArray(function (err, result) {
count += result.length;
});
});
});
// it is returning the value without finishing the above manipulation
// not waiting for above callback and value of count is coming zero .
res.render('index', {
projectDetail: projectDetail.projectCount,
count: count
});
});
});
};

When you want to call multiple asynchronous functions in order, you should call the first one, call the next one in it's callback and so on. The code would look like:
asyncFunction1(args, function () {
asyncFunction2(args, function () {
asyncFunction3(args, function () {
// ...
})
})
});
Using this approach, you may end up with an ugly hard-to-maintain piece of code.
There are various ways to achieve the same functionality without nesting callbacks, like using async.js or node-fibers.
Here is how you can do it using node.js EventEmitter:
var events = require('events');
var EventEmitter = events.EventEmitter;
var flowController = new EventEmitter();
flowController.on('start', function (start_args) {
asyncFunction1(args, function () {
flowController.emit('2', next_function_args);
});
});
flowController.on('2', function (args_coming_from_1) {
asyncFunction2(args, function () {
flowController.emit('3', next_function_args);
});
});
flowController.on('3', function (args_coming_from_2) {
asyncFunction3(args, function () {
// ...
});
});
flowController.emit('start', start_args);
For loop simulation example:
var events = require('events');
var EventEmitter = events.EventEmitter;
var flowController = new EventEmitter();
var items = ['1', '2', '3'];
flowController.on('doWork', function (i) {
if (i >= items.length) {
flowController.emit('finished');
return;
}
asyncFunction(item[i], function () {
flowController.emit('doWork', i + 1);
});
});
flowController.on('finished', function () {
console.log('finished');
});
flowController.emit('doWork', 0);

Use callbacks or promises or a flow control library. You cannot program servers in node without understanding at the very least one of these approaches, and honestly all halfway decent node programmers thoroughly understand all three of them (including a handful of different flow control libraries).
This is not a something you are going to just get an answer coded for you by someone else on stackoverflow and then move on. This is a fundamental thing you have to go and study and learn generically as it is only going to come up over and over again on a daily basis.
http://howtonode.org/control-flow
http://callbackhell.com/

Per the resources in the answer above me, nesting the callback when you iterate and only calling it if you are on the last iteration will solve you problem.