Passing data into node.js callbacks - the asynchronous issue [duplicate] - javascript

This question already has answers here:
JavaScript closure inside loops – simple practical example
(44 answers)
Closed 8 years ago.
Take the code below, in which an object data is initialized with some values that are then processed by some time-intensive function, like database access. If the function succeeds, the name of the successful data item is printed to the console. Otherwise, a failure notice is printed:
data = {first: 'someinfo', second: 'somemoreinfo', third: 'evenmoreinfo'};
for (var item in data) {
timeIntensiveFunction(item, data[item], function(err) {
if (!err) {
console.log(item + ' processed successfully');
} else {
console.log(item + ' failed');
}
});
}
You would expect the console to show this, assuming the function succeeds for all three data items:
first processed successfully
second processed successfully
third processed successfully
Instead it will show this, assuming the first database access takes longer than the for loop:
third processed successfully
third processed successfully
third processed successfully
This is because the console logging is done in a callback, which would reasonably only be called after the for loop is done, because timeIntensiveFunction() takes so long. By the time the first callback is called, item already has its last value, third.
How do you pass the 'current' value of item into the callback?

Problem is because it's calling the callback with the last item only.
You can bind each of your item with a function like bellow.
var printStatus = function(item){
return function(err) {
if (!err) {
console.log(item + ' processed successfully');
} else {
console.log(item + ' failed');
}
}
}
for (var item in data) {
timeIntensiveFunction(item, data[item], printStatus(item));
}

This is a common "gotcha" with closures in javascript. One way around it is to wrap your function call in an anonymous function and rescope item. Like so:
for (var item in data) {
(function(item){
timeIntensiveFunction(item, data[item], function(err) {
if (!err) {
console.log(item + ' processed successfully');
} else {
console.log(item + ' failed');
}
});
})(item);
}

If you're looking for a library to make working with async tasks more easy, check out caolan/async.
var async = require("async");
var data = [{id: "first"}, {id: "second"}, {id: "third"}];
function timeIntensiveFunction(item, done) {
// do something
console.log("time intensive task started:", item.id);
// err?
// if (err) return done(err);
done();
}
function processItem(item, done) {
timeIntensiveFunction(item, function(err) {
if (err) return done(err);
console.log("task complete:", item.id);
done();
});
}
async.map(data, processItem);
Output
time intensive task started: first
task complete: first
time intensive task started: second
task complete: second
time intensive task started: third
task complete: third
For users looking to learn how to do this without a library, you can see the revision history of this answer.

Related

Trying to wait for list to be populated before iterating through it NodeJS

I have some code that takes an uploaded file, executes it and gets the output. It then compares this output against expected output to check if the script did as expected.
I am now trying to improve this functionality so that an uploaded file will be run several times, each time being checked against a different expected output, or "test case". I then want to push "correct" or "incorrect" onto a results array, so that I can go through that array at the end and check whether there are any "incorrect" (whether the file failed any test case).
I have tried just callbacks within each function.
I have tried using await and async on the getArray as seen below
Using both callbacks and async together.
This is the parent function code that calls for the array to be created, and wants to iterate through it after it has been created.
var resultsArr = await getResults(file.name, files.length, markerDir);
//file.name is the name from the uploaded file object
//files.length is the number of subdirectories (number of testcases to run against)
//markerDir is the str path to where these testcases are stored
if (resultsArr){
for(var i=0;i<resultsArr.length;i++) {
if (resultsArr[i] == "incorrect"){
checkForMarkerCb('incorrect'); //Calls back to frontend to
break; //display result on web app
}
else if (i+1 == resultsArr.length) {
checkForMarkerCb('correct');
}
}
}
The following is inside the getResults function that is called above
for(var i=1; i<=fileLength; i++) {
var sampleOut = markerDir + '/test' + i + '/stdout.txt';
//Grab expected stdout.txt
var markerOut = fs.readFileSync(sampleOut, 'utf-8', function(err){
if (err){
throw err;
};
});
//Run the file and grab the output
executeFile(filename, function(fileOut){
//Compare output with sample stdout
if (markerOut == fileOut){
resultsArr.push('correct');
}
else {
resultsArr.push('incorrect');
}
});
}
//If results array has a response for each testcase
if (resultsArr.length == fileLength) {
return resultsArr;
}
Implementation of executeFile() as requested:
function executeFile(filename, execFileCb){
//pathToUpload is a str path to where the upload is stored
const child = execFile('python', [pathToUpload], (err,stdout,stderr) => {
if (err) {
throw err;
}
execFileCb(stdout); //Callback with output of file
});
}
function executeFileAsync(filename) {
return new Promise(function(resolve,reject){
executeFile(filename, function(err, data){
if (err !== null) reject(err);
else resolve(data);
});
});
}
which was called inside getResults() using
var fileOut = await executeFileAsync(filename)
The initial function that calls getResults().
getResults(): which gets the path to each directory and calls pushes the results of comparing outputs onto a results array.
executeFile(): uses 'child_process' to run a file and calls back with the output.
I expect the code to wait for getResults to return with the resultsArr so that the for loop can iterate through and check for any "incorrect". Instead, getResults returns before resultsArr is populated.
Using some logging, I see that the code for checking markerOut == fileOut is executed at the end after the getResults() for loop has already completed. I tried setting up the call to executeFile() to also be an async/await similar to how getResults() is called but still no change.
I may not be using async/callbacks correctly, any help is greatly appreciated.
Your executeFileAsync function currently calls executeFile with a callback that is expecting two arguments, but executeFile then does call this execFileCb always with only one argument which is interpreted as an error. It also should not use throw in an asynchronous callback.
Instead, merge them into one function:
function executeFile(filename) {
return new Promise(function(resolve,reject){
//pathToUpload is a str path to where the upload is stored
const child = execFile('python', [pathToUpload], (err,stdout,stderr) => {
if (err) reject(err);
else resolve(stdout); //Callback with output of file
});
});
}

Function Return not Working

I am working in Node and trying to load the next sequence from my db. I am able to access the db, load and return the sequence within my function, but I am not able to access it outside of the function.
function getRunId() {
counters.findOne({_id: 'Run_ID'}, function(err, resp) {
if(err) {
console.log(err);
}
console.log('Seq: ' + resp.sequence); // Console Output = Seq: 1234
return resp.sequence;
});
};
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId); // Console Output = CRID: undefined
I've checked several pages worth of Stack Overflow issues relating to using callback's, async (node module), how to properly return values in the function, etc... but none of them get me closer to accessing currentRunId outside of the function.
Is this issue further complicated by the use of Mongo queries inside my function?
For anyone stumbling on this later, start by reading this answer.
I've dealt with this a few times so I understand the frustration. You are trying to mix sync and async code by doing this:
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId);
The trouble is that console.log('Run_ID: ' + currentRunId) is called immediately after you invoke getRunID() by assigning it to current RunID, and getRunID() resolves after console.log('Run_ID: ' + currentRunId), causing the currentRunId variable to be undefined.
But, you have some options to deal with this. Option one is to return a callback, and log the results of the callback instead. Option 2 is to use an ES6 promise. To use option 2, you need node version 7, and you need to use 'use strict' in your code.
Here are 3 examples built around a function stub that spoofs the results of findOne(). The getRunIdA() is your function, and getRunIdB, and getRunIdC are two example solutions to your current problem.
'use strict'
// A function stub which represents a simplified version of findOne.
// Accepts callback and returns a callback with the results of data
function findOne (callback) {
var data = {
sequence: 6
}
return callback(null, data)
}
// This is a simplified version of your function, which reproduces the undefined result
function getRunIdA () {
findOne(function (err, resp) {
if (err) {
console.log(err)
}
console.log('Seq: ' + resp.sequence)
return resp.sequence
})
}
// This is your function with a callback
function getRunIdB (callback) {
findOne(function (err, resp) {
if (err) {
console.log(err)
}
console.log('Seq: ' + resp.sequence)
return callback(resp.sequence)
})
}
// This is your function with a promise
var getRunIdC = new Promise(function (resolve, reject) {
resolve(findOne(function (err, resp) {
if (err) {
console.log(err)
}
return resp.sequence
}))
})
// Invoke your funciton; get undefined
var currentRunID = getRunIdA()
console.log('Run_ID: ' + currentRunID) // Run_ID: undefined
// Invoke getRunIdB using callback, get 6
getRunIdB(function (id) {
console.log('Run_ID: ' + id) // Run_ID: 6
})
// Invoke getRunIdC with a promise; get 6
getRunIdC.then(function (currentRunID) {
console.log('Run_ID: ' + currentRunID) // Run_ID: 6
})
/*
results for all 3:
Seq: 6
Run_ID: undefined
Seq: 6
Run_ID: 6
Run_ID: 6
*/
Give this a try by saving to your machine and running:
node test.js
Is this issue further complicated by the use of Mongo queries inside my function?
Nope, you just need to pass the results of your query to a promise or a callback so that you can work with the results somewhere else.
I hope this helps!
Edit: OP added the following code in a comment, which I will try to break down and address.
Unfortunately, using getRunIdB results in callback is not defined and using getRunIdC results in currentRunId is not defined
var currentRunID = '';
var getRunId = new Promise(function (resolve, reject) { resolve(counters.findOne({_id: 'Run_ID'}, function (err, resp) {
if (err) {
console.log(err)
}
return resp.sequence;
}))
});
getRunId.then(function (res) {
console.log('Run_ID: ' + res.sequence) // Run_ID: 1234
currentRunID = res.sequence;
})
console.log(currentRunID); // currentRunID is not defined
Check out an answer I gave to a similar question for more details on the JS concurrency model. Simply put, the getRunID() function is executing asynchronous code. What that means is that getRunID() doesn't get inserted into the message queue that determines what order javascript will execute until it's callbacks are completed. Thus, when you log currentRunID outside of the .then() function, the results is undefined because currentRunID is undefined.
I think that ultimately what OP is trying to do is to export the result of the function so that the something can be done with those results, this needs to be done within a callback like so:
getRunId.then(function (res) {
// Do stuff with the run ID here.
})
You are only returning on a callback function but not on the actual function.. Change your code to this:
function getRunId() {
var result = counters.findOne({_id: 'Run_ID'}, function(err, resp) {
if(err) {
console.log(err);
}
console.log('Seq: ' + resp.sequence); // Console Output = Seq: 1234
return resp.sequence;
});
return result; //<-- return result of your function is here
};
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId);

Node & MySQL - connection.query inside connection.query - Object property not accessible

I got a pretty awkward problem.
I create a pool, connect to the database, create a connection and query, get the results, do a bunch of stuff, then I have to create another connection and query, but actually it has to be dynamically so I loop over my Array teacherHours containing the data.
Then more Code is happening, and I have to create an extra loop, because certain elements of my teacherHours Array have to try multiple times to get the correct response from the upcoming query.
So another loop follows, which is supposed to loop as long as availableHours > 0. Now, here is where it all goes left.
A buch of code happens inside the second loop, I prepare my second query, call connection.query() and inside of the callback function I prepare my third query (after doing some other stuff) and this is actually where Node kicks me out.
It gives me TypeError: Cannot read property 'tid' of undefined. tid needs to be accessed for my third query, so I try to access it just like I did before but Node doesn't allow it.
I know that the queries return useful data (rows) so it can't be a problem of querying but receiving no data. Actually I console.log("the rowRIDS"+rowRIDS); the result of the second query and I see that it returns 2 rows and just after that it gives me the error.
What is also strange to me, all the console.logs inside my my two loops are being logged, and my console.log of the second query (containing the 2 rows) are being logged after the loops ran through, since the queries are nested shouldn't the returned 2 rows and the error appear within the first iteration of the loop, since the code should access the second query at that point.
BTW, I've tried to set a hardcoded number instead of the tid just to get the next property datum to be an error. I kind of got a feeling as if the variable teacherHours is out of scope, but it is supposed to be a global variable.
To get a better feeling of what I'm talking about I copied the code and uncommented all the javascript code, where I populate and calculate stuff. Any help would be really great, its been almost 7 hours of try & error without any luck. Thank You!
pool.getConnection(function(err, connection){
if (err) throw err;
connection.query('SELECT * FROM teachers_teaching_tbl WHERE fwdid = 1 ', function(err, rows, fields) {
if (err) {
console.error('error querying: ' + err.stack);
return;
}
rowArray=rows;
console.log(rowArray);
//
// HERE HAPPENS
// A LOOOOT OF STUFF
//
// AND teacherHours IS BEING POPULATED
//
// THEN COMES A FOR LOOP
for(var i=0; i<teacherHours.length;i++){
//
// MORE STUFF
//
//AND ANOTHER LOOP
while(availableHours>0){//do{ ORIGINALLY I TRIED WITH A DO WHILE LOOP
//
// AGAIN A BUNCH OF STUFF
//
// NOW I'M PREPARING MY NEXT QUERY
//
var myQueryGetFreeRoom=" SELECT rms.rid FROM rooms_tbl as rms WHERE NOT EXISTS ( ";
myQueryGetFreeRoom+=" SELECT NULL FROM classes_tbl as cls ";
myQueryGetFreeRoom+=" WHERE ( (cls.bis > '"+bisMinus1+"' AND cls.bis <= '"+realBis+"' ) OR ( cls.von > '"+bisMinus1+"' AND cls.von < '"+realBis+"' ) ) AND (cls.datum = '"+teacherHours[i].datum.getFullYear()+"-"+(teacherHours[i].datum.getMonth()+1)+"-"+teacherHours[i].datum.getDate()+"') AND (cls.rid=rms.rid) ) ";
//
//
connection.query(myQueryGetFreeRoom, function(err, rowRIDS, fields) {
if (err) {
console.error('error querying: ' + err.stack);
return;
}
roomIDs=rowRIDS;
console.log("the rowRIDS"+rowRIDS);
//
// MORE STUFF
// HAPPENING
//
if(roomIDs.length>0){
//
// PREPARING QUERY NO.3 - WHICH IS WHERE MY ERROR POINTS - TO THE USE OF tid PROPERTY
//
var myQueryBookClass = " INSERT INTO classes_tbl ( rid , tid , belegtAnz, datum, von , bis ) ";
myQueryBookClass+=" VALUES ( "+Math.floor(Math.random() * roomIDs.length)+", "+teacherHours[i].tid+" , 0, '"+teacherHours[i].datum.getFullYear()+"-"+(teacherHours[i].datum.getMonth()+1)+"-"+teacherHours[i].datum.getDate()+"' , '"+bisMinus1+"' , '"+realBis+"' ) ";
console.log("myQueryBookClass: "+myQueryBookClass);
availableHours = 0;
//
// HERE WAS SUPPOSED TO FOLLOW QUERY 3 - myQueryBookClass
//
// BUT SINCE I DONT EVEN GET INSIDE HERE IT IS IN COMMENTS
//
/*connection.query(myQueryBookClass, function(err, insertRows, fields){
if(err){
console.error('error querying: '+err.stack);
return;
}
console.log("Inserted Rows: "+ insertRows);
}); */
} else {
availableHours= availableHours - 1;
//
// STUFF HAPPENING
//
}
});
availableHours= availableHours - 1;
}//while(availableHours>0);
//
}
connection.release(function(err){
if (err){
console.error('error disconnecting: ' + err.stack);
return;
}
});
});
});
I think you are coming from a non-async language like Python, Java, etc. which is why Node, i.e. JavaScript, seems to screw things up for you, but actually it isn't.
The problem you have in your code is that you execute async functions like query synchronously all at the same time in the same while loop. You need to use a module like async which helps to run and collect results asynchronously.
Here is the updated code.
var async = require('async'),
connection;
async.waterfall([
function (cb) {
pool.getConnection(cb);
},
function (conn, cb) {
connection = conn;
connection.query('SELECT * FROM teachers_teaching_tbl WHERE fwdid = 1', cb);
},
function (rows, fields, cb) {
rowArray = rows;
console.log(rowArray);
// HERE HAPPENS
// A LOOOOT OF STUFF
//
// AND teacherHours IS BEING POPULATED
//
// THEN COMES A FOR LOOP
async.eachSeries(teacherHours, function (teacherHour, done) {
// MORE STUFF
//
//AND ANOTHER LOOP
async.whilst(function () {
return availableHours > 0;
}, function (cb) {
// AGAIN A BUNCH OF STUFF
//
// NOW I'M PREPARING MY NEXT QUERY
//
var myQueryGetFreeRoom =
"SELECT rms.rid FROM rooms_tbl AS rms WHERE NOT EXISTS ("
+ "SELECT NULL FROM classes_tbl AS cls"
+ " WHERE ("
+ "(cls.bis > '" + bisMinus1 + "' AND cls.bis <= '" + realBis + "')"
+ " OR (cls.von > '" + bisMinus1 + "' AND cls.von < '" + realBis + "')"
+ ") AND ("
+ "cls.datum = '" + teacherHour.datum.getFullYear() + "-" + (teacherHour.datum.getMonth() + 1) + "-" + teacherHour.datum.getDate() + "'"
+ ") AND cls.rid = rms.rid";
async.waterfall([
function (cb) {
connection.query(myQueryGetFreeRoom, cb);
},
function(rowRIDS, fields, cb) {
roomIDs = rowRIDS;
console.log("the rowRIDS" + rowRIDS);
//
// MORE STUFF
// HAPPENING
//
if (roomIDs.length > 0) {
//
// PREPARING QUERY NO.3 - WHICH IS WHERE MY ERROR POINTS - TO THE USE OF tid PROPERTY
//
var myQueryBookClass = "INSERT INTO classes_tbl (rid, tid, belegtAnz, datum, von, bis) VALUES ("
+ Math.floor(Math.random() * roomIDs.length)
+ ", " + teacherHour.tid
+ ", 0, '" + teacherHour.datum.getFullYear() + "-" + (teacherHour.datum.getMonth() + 1) + "-" + teacherHour.datum.getDate() + "', '" + bisMinus1 + "', '" + realBis + "')";
console.log("myQueryBookClass: " + myQueryBookClass);
availableHours = 0;
//
// HERE WAS SUPPOSED TO FOLLOW QUERY 3 - myQueryBookClass
//
// BUT SINCE I DONT EVEN GET INSIDE HERE IT IS IN COMMENTS
//
connection.query(myQueryBookClass, function (err, insertRows, fields) {
if (err) {
console.error('error querying: '+err.stack);
return;
}
console.log("Inserted Rows: "+ insertRows);
// Here do whatever you need to do, then call the callback;
cb();
});
} else {
--availableHours;
//
// STUFF HAPPENING
//
cb();
}
}
], function (err) {
if (!err) {
// Notice that you are decrementing the `availableHours` twice here and above.
// Make sure this is what you want.
--availableHours;
}
cb(err);
});
}, done);
}, function (err) {
connection.release(function (err) {
if (err) {
console.error('error disconnecting: ' + err.stack);
return;
}
});
});
}
], function (err) {
conn && pool.release(conn);
err && throw err;
});
Next time please format your code properly for better readability which will help yourself to get answers faster, and split your question text into paragraphs for the same purpose.
Explanation
There are four nested async flows:
async.waterfall
-> async.eachSeries
-> async.whilst
-> async.waterfall
Basically, the async.waterfall library allows you to execute a list of functions in series.
Every next function will be executed only after the previous function has returned the response.
To indicate that a function is completed and the results are available, it has to call the callback, in our case it is cb (you can call it whatever you like, eg. callback). The rule is to call it, otherwise, the next function will never be executed because the previous one doesn't seem to have finished its work.
Once the previous function has completed, it calls the provided cb with the following signature:
cb(err, connection);
If there was an error while requesting for a connection, the entire async.waterfall will interrupt and the final callback function will be executed.
If there was no error, the connection will be provided as a second argument. async module passes all arguments of the previous function to the next function as the first, second, etc. arguments which is why the second function receives the conn as the first argument.
Every next function will receive the callback cb as the last argument, which you must eventually call when the job is done.
Thus, in the first async.waterfall flow:
It requests a new database connection.
Once the connection is available, the next function is executed which sends a query to the database.
Waits for the query results, then once the results are available, it is ready to run the next function which iterates over each row.
async.eachSeries allows to iterate over a gives array of values sequentially.
In the second async.eachSeries flow:
It iterates over each element in the teacherHours array sequentially.
Once each element is processed (however you want), you must call the done callback. Again, you could have called this as cb like in the previous async.waterfall or callback. done is just for clarity that the process is done.
Then we have the async.whilst which provides the same logic as the normal while () {} statement but handles the loop asynchronously.
In this third async.whilst flow:
Calls the first function. Its return value indicates whether it has to continue the loop, i.e. call the second asynchronous function.
If the return value is truthful (availableHours > 0), then the second function is called.
When the async function is done, it must call the provided callback cb to indicate that it is over. Then async module will call the first function to check if it has to continue the loop.
In this asynchronous function inside async.whilst we have another async.waterfall because you need to send queries to the database for each teacherHour.
In this last fourth async.watercall flow:
It sends the SELECT query to the database.
Waits for the response. Once the rowRIDS are available, it calls the second function in the waterfall.
If there are rowRIDS (roomIDs.length > 0), it sends the INSERT query to the database.
Once done, it calls the callback cb.
If there were no rowRIDs, it calls the callback cb, too, to indicate that the job is done.
It is a great thing that JavaScript is asynchronous. It might be difficult at the beginning when you convert from other synchronous languages, but once you get the idea, it will be hard to thing synchronously. It becomes so intuitive that you will start thinking why other languages don't work asynchronous.
I hope I could explain the above code thoroughly. Enjoy JavaScript! It's awesome!

async.waterfall inside a for loop escapes the for loop

On a Form Action of type POST, we fetch all the values in Node.JS/Express and try saving it into MongoDB.
A hidden field determines the length of a property from the frontend javascript and it's value is updated as the hidden field's value.
This length is used in the backend (Node) to iterate over a list of items.
I have a async.waterfall function and a for loop running inside it like this.
async.waterfall([
function(callback){
var itemLength = req.body.itemLength;
var itemProp,itemComponent;
var destination;
var destinationsArray =[];
for(var k=1; k<=itemLength; k++){
destination = new Destination({
name: req.body['destinationName'+k],
});
itemComponent = {
"itemCompProp" : req.body['itemCompProp'+k]
};
itemProp = new ItemProp({
itemComponent: itemComponent
});
itemProp.save(function(err,itemPropSaved){
destination.newProperty = itemPropSaved._id
destination.save(function(err,destinationSaved){
if(err){
console.log("Error== " + err);
}
else{
destinationsArray.push(destinationSaved._id);
}
});
});
}// End of For
callback(null,destinationsArray);
},
function(destinationsArray,callback){
var brand = new Brand({
name : req.body.brandName,
});
brand.save(function(err,brandSaved){
if(err){
console.log("Error== " + err);
}else{
console.log('Brand Saved');
}
});
callback(null);
}
], function (err, status) {
if(err){
req.flash('error', {
msg: 'Error Saving Brands'
});
console.log("Error : " + err);
}
else{
console.log("Brand Saved.");
req.flash('success', {
msg: 'Brand Successfully Added!'
});
}
});
res.redirect('/redirectSomewhere');
When we run this, The destinationsArray is returned first as null, as opposed to going through the for loop and then returning the proper value of destinationsArray over a length (itemLength) of destinations.
We want the process to be synchronous. We also tried using a closure wrapping the for Loop but to no avail.
We can't use a async.eachSeries instead of the for Loop as I am just iterating over a numeric property and we don't have any documents to iterate over
Any feasible solution to run a for Loop inside a async.waterfall?
Cheers and Thanks in Advance.
There are few problems with the code you have there:
where the callbacks got called.
where res.redirect() got call.
the for loop.
save() is asynchronous. Regular for loop will just continue without waiting for all save() calls to finish. That's why destinationsArray is empty. As you said, you cannot use async.eachSeries() since you're iterating through numeric property. However, you're on the right track there. Async.whilst() does just that. Here is the revised code with Async.whilst() and proper calling locations of the callbacks:
async.waterfall([
function(callback){
var itemLength = req.body.itemLength;
var itemProp,itemComponent;
var destination;
var destinationsArray =[];
var k = 1; // 1st part of for loop: for(k=1; k<=itemLength; k++)
async.whilst(
function() {
return k <= itemLength; // 2nd part of for loop: for(k=1; k<=itemLength; k++)
},
function(whilstCb) {
destination = new Destination({
name: req.body['destinationName'+k]
});
itemComponent = {
"itemCompProp" : req.body['itemCompProp'+k]
};
itemProp = new ItemProp({
itemComponent: itemComponent
});
itemProp.save(function(err,itemPropSaved){
destination.newProperty = itemPropSaved._id
destination.save(function(err,destinationSaved){
if(err){
console.log("Error== " + err);
} else {
destinationsArray.push(destinationSaved._id);
}
k++; // 3rd part of for loop: for(k=1; k<=itemLength; k++)
whilstCb(null);
});
});
},
function(err) {
// It gets here once the loop is done
console.log(destinationsArray); // This array should have all the values pushed
callback(null, destinationsArray);
}
);
},
function(destinationsArray,callback){
var brand = new Brand({
name : req.body.brandName
});
brand.save(function(err,brandSaved){
if(err){
console.log("Error== " + err);
} else {
console.log('Brand Saved');
}
callback(null);
});
}
], function (err, status) {
if(err){
req.flash('error', {
msg: 'Error Saving Brands'
});
console.log("Error : " + err);
} else {
console.log("Brand Saved.");
req.flash('success', {
msg: 'Brand Successfully Added!'
});
}
res.redirect('/redirectSomewhere');
});
Its not so much the for loop that is causing you problems but that save is an asynchronous operation. The for loop completes and the callback is executed before any of the save callbacks have had chance to complete.
What you want to do is call the async.waterfall callback after all the destination save callbacks have been executed. Something like:
destination.save(function(err,destinationSaved){
if(err){
console.log("Error== " + err);
} else {
destinationsArray.push(destinationSaved._id);
if (k === itemLength) {
// all destination callbacks have completed successfully
callback(null, destinationsArray);
}
}
});
The issue has to do with callback(null, destinationsArray); getting called outside the for loop without checking first to see the loop has been finished.
Try replacing callback(null, destinationsArray); with something like this:
if (itemLength > 0 && destinationsArray.length === k - 1) {
callback(null, destinationsArray);
} else {
callback(true);
}
The above checks to make sure the destination.save() gets completed the proper number of times successfully.
I actually prefer the method proposed by djskinner. However, because of the console.log() that occurs when there is a save() error, the callbacked destinationsArray could possibly hold the incorrect number of items. To fix this, you could make sure to replace the console.log("Error== " + err); with something like callback(err) to end the waterfall with the error returned. In addition, the k === itemLength check doesn't properly account for the correct number of items that should be saved. This should be replaced with k === destinationsArray.length.
I made modifications to fix this and posted an updated version below.
destination.save(function(err, destinationSaved){
if (err) {
callback(err);
}
else {
destinationsArray.push(destinationSaved._id);
if (k === destinationsArray.length) {
callback(null, destinationsArray);
}
}
});
--EDIT-- I really like the solution that Ben posted using whilst(). This allows the creation of a loop where the iterations runs serially. For more info, view the npm page here.

Using Async waterfall in node.js

I have 2 functions that I'm running asynchronously. I'd like to write them using waterfall model. The thing is, I don't know how..
Here is my code :
var fs = require('fs');
function updateJson(ticker, value) {
//var stocksJson = JSON.parse(fs.readFileSync("stocktest.json"));
fs.readFile('stocktest.json', function(error, file) {
var stocksJson = JSON.parse(file);
if (stocksJson[ticker]!=null) {
console.log(ticker+" price : " + stocksJson[ticker].price);
console.log("changing the value...")
stocksJson[ticker].price = value;
console.log("Price after the change has been made -- " + stocksJson[ticker].price);
console.log("printing the the Json.stringify")
console.log(JSON.stringify(stocksJson, null, 4));
fs.writeFile('stocktest.json', JSON.stringify(stocksJson, null, 4), function(err) {
if(!err) {
console.log("File successfully written");
}
if (err) {
console.error(err);
}
}); //end of writeFile
} else {
console.log(ticker + " doesn't exist on the json");
}
});
} // end of updateJson
Any idea how can I write it using waterfall, so i'll be able to control this? Please write me some examples because I'm new to node.js
First identify the steps and write them as asynchronous functions (taking a callback argument)
read the file
function readFile(readFileCallback) {
fs.readFile('stocktest.json', function (error, file) {
if (error) {
readFileCallback(error);
} else {
readFileCallback(null, file);
}
});
}
process the file (I removed most of the console.log in the examples)
function processFile(file, processFileCallback) {
var stocksJson = JSON.parse(file);
if (stocksJson[ticker] != null) {
stocksJson[ticker].price = value;
fs.writeFile('stocktest.json', JSON.stringify(stocksJson, null, 4), function (error) {
if (err) {
processFileCallback(error);
} else {
console.log("File successfully written");
processFileCallback(null);
}
});
}
else {
console.log(ticker + " doesn't exist on the json");
processFileCallback(null); //callback should always be called once (and only one time)
}
}
Note that I did no specific error handling here, I'll take benefit of async.waterfall to centralize error handling at the same place.
Also be careful that if you have (if/else/switch/...) branches in an asynchronous function, it always call the callback one (and only one) time.
Plug everything with async.waterfall
async.waterfall([
readFile,
processFile
], function (error) {
if (error) {
//handle readFile error or processFile error here
}
});
Clean example
The previous code was excessively verbose to make the explanations clearer. Here is a full cleaned example:
async.waterfall([
function readFile(readFileCallback) {
fs.readFile('stocktest.json', readFileCallback);
},
function processFile(file, processFileCallback) {
var stocksJson = JSON.parse(file);
if (stocksJson[ticker] != null) {
stocksJson[ticker].price = value;
fs.writeFile('stocktest.json', JSON.stringify(stocksJson, null, 4), function (error) {
if (!err) {
console.log("File successfully written");
}
processFileCallback(err);
});
}
else {
console.log(ticker + " doesn't exist on the json");
processFileCallback(null);
}
}
], function (error) {
if (error) {
//handle readFile error or processFile error here
}
});
I left the function names because it helps readability and helps debugging with tools like chrome debugger.
If you use underscore (on npm), you can also replace the first function with _.partial(fs.readFile, 'stocktest.json')
First and foremost, make sure you read the documentation regarding async.waterfall.
Now, there are couple key parts about the waterfall control flow:
The control flow is specified by an array of functions for invocation as the first argument, and a "complete" callback when the flow is finished as the second argument.
The array of functions are invoked in series (as opposed to parallel).
If an error (usually named err) is encountered at any operation in the flow array, it will short-circuit and immediately invoke the "complete"/"finish"/"done" callback.
Arguments from the previously executed function are applied to the next function in the control flow, in order, and an "intermediate" callback is supplied as the last argument. Note: The first function only has this "intermediate" callback, and the "complete" callback will have the arguments of the last invoked function in the control flow (with consideration to any errors) but with an err argument prepended instead of an "intermediate" callback that is appended.
The callbacks for each individual operation (I call this cbAsync in my examples) should be invoked when you're ready to move on: The first parameter will be an error, if any, and the second (third, fourth... etc.) parameter will be any data you want to pass to the subsequent operation.
The first goal is to get your code working almost verbatim alongside the introduction of async.waterfall. I decided to remove all your console.log statements and simplified your error handling. Here is the first iteration (untested code):
var fs = require('fs'),
async = require('async');
function updateJson(ticker,value) {
async.waterfall([ // the series operation list of `async.waterfall`
// waterfall operation 1, invoke cbAsync when done
function getTicker(cbAsync) {
fs.readFile('stocktest.json',function(err,file) {
if ( err ) {
// if there was an error, let async know and bail
cbAsync(err);
return; // bail
}
var stocksJson = JSON.parse(file);
if ( stocksJson[ticker] === null ) {
// if we don't have the ticker, let "complete" know and bail
cbAsync(new Error('Missing ticker property in JSON.'));
return; // bail
}
stocksJson[ticker] = value;
// err = null (no error), jsonString = JSON.stringify(...)
cbAsync(null,JSON.stringify(stocksJson,null,4));
});
},
function writeTicker(jsonString,cbAsync) {
fs.writeFile('stocktest.json',jsonString,function(err) {
cbAsync(err); // err will be null if the operation was successful
});
}
],function asyncComplete(err) { // the "complete" callback of `async.waterfall`
if ( err ) { // there was an error with either `getTicker` or `writeTicker`
console.warn('Error updating stock ticker JSON.',err);
} else {
console.info('Successfully completed operation.');
}
});
}
The second iteration divides up the operation flow a bit more. It puts it into smaller single-operation oriented chunks of code. I'm not going to comment it, it speaks for itself (again, untested):
var fs = require('fs'),
async = require('async');
function updateJson(ticker,value,callback) { // introduced a main callback
var stockTestFile = 'stocktest.json';
async.waterfall([
function getTicker(cbAsync) {
fs.readFile(stockTestFile,function(err,file) {
cbAsync(err,file);
});
},
function parseAndPrepareStockTicker(file,cbAsync) {
var stocksJson = JSON.parse(file);
if ( stocksJson[ticker] === null ) {
cbAsync(new Error('Missing ticker property in JSON.'));
return;
}
stocksJson[ticker] = value;
cbAsync(null,JSON.stringify(stocksJson,null,4));
},
function writeTicker(jsonString,cbAsync) {
fs.writeFile('stocktest.json',jsonString,,function(err) {
cbAsync(err);
});
}
],function asyncComplete(err) {
if ( err ) {
console.warn('Error updating stock ticker JSON.',err);
}
callback(err);
});
}
The last iteration short-hands a lot of this with the use of some bind tricks to decrease the call stack and increase readability (IMO), also untested:
var fs = require('fs'),
async = require('async');
function updateJson(ticker,value,callback) {
var stockTestFile = 'stocktest.json';
async.waterfall([
fs.readFile.bind(fs,stockTestFile),
function parseStockTicker(file,cbAsync) {
var stocksJson = JSON.parse(file);
if ( stocksJson[ticker] === null ) {
cbAsync(new Error('Missing ticker property in JSON.'));
return;
}
cbAsync(null,stocksJson);
},
function prepareStockTicker(stocksJson,cbAsync) {
stocksJson[ticker] = value;
cbAsync(null,JSON.stringify(stocksJson,null,4));
},
fs.writeFile.bind(fs,stockTestFile)
],function asyncComplete(err) {
if ( err ) {
console.warn('Error updating stock ticker JSON.',err);
}
callback(err);
});
}
Basically nodejs (and more generally javascript) functions that require some time to execute (be it for I/O or cpu processing) are typically asynchronous, so the event loop (to make it simple is a loop that continuously checks for tasks to be executed) can invoke the function right below the first one, without getting blocked for a response. If you are familiar with other languages like C or Java, you can think an asynchronous function as a function that runs on another thread (it's not necessarily true in javascript, but the programmer shouldn't care about it) and when the execution terminates this thread notifies the main one (the event loop one) that the job is done and it has the results.
As said once the first function has ended its job it must be able to notify that its job is finished and it does so invoking the callback function you pass to it. to make an example:
var callback = function(data,err)
{
if(!err)
{
do something with the received data
}
else
something went wrong
}
asyncFunction1(someparams, callback);
asyncFunction2(someotherparams);
the execution flow would call: asyncFunction1, asyncFunction2 and every function below until asyncFunction1 ends, then the callback function which is passed as the last parameter to asyncFunction1 is called to do something with data if no errors occurred.
So, to make 2 or more asynchronous functions execute one after another only when they ended you have to call them inside their callback functions:
function asyncTask1(data, function(result1, err)
{
if(!err)
asyncTask2(data, function(result2, err2)
{
if(!err2)
//call maybe a third async function
else
console.log(err2);
});
else
console.log(err);
});
result1 is the return value from asyncTask1 and result2 is the return value for asyncTask2. You can this way nest how many asynchronous functions you want.
In your case if you want another function to be called after updateJson() you must call it after this line:
console.log("File successfully written");

Categories

Resources