I am getting a request from client that request needs to select the data from two tables , from one table i need to select 1000 records in table two i need to select only two records, handling it with a call back . I want the response that needs to send to my client after the two callback function executes , How to achieve this in node js?
You have following options:
Follow Node style callback pattern. Also read, Callback-hell
Follow promises.
Callback pattern
var f = function(somedata, cb){
// pick user from db
pickUserFromTable1(somedata, function(err, users){
// callled on completion of pickUserFromDB
if(err){
//handle error.
cb(err, null);
return;
}
pickUserFromTable2(users, function(err, finallySelectedUsers){
// called on completion of pickUserFromTable2
if(err){
// handle error.
cb(err, null);
// returning is important. or use else part of if to separate the case
return;
}
// call initiator's callback function on completion of both pickUserFromTable1 and pickUserFromTable2
cb(null, finallySelectedUsers);
});
})
}
Note: This approach is good for short project but leads to some maintenance issue (callback hell) if dependency chain is lengthy.
Welcome to callback hell! You just need to build pyramide form callbacks!
Example:
function getDataFromDB (query, successCb, errorCb) {
executeFirstQuery(query, function(firstQueryData) {
console.log('First query ok! Now execute second.');
executeSecondQuery(query, function(secondQueryData) {
console.log('Second query ok! Now execute successCb().');
successCb([firstQueryData, secondQueryData]);
}, function(error) {
errorCb("Error in second query");
})
}, function(error) {
errorCb("Error in first query");
});
}
function executeFirstQuery (query, successCb, errorCb) {
console.log("Execute first query!");
successCb("Some data from first query");
// errorCb("Some Error from first query");
}
function executeSecondQuery (query, successCb, errorCb) {
console.log("Execute second query!");
successCb("Some from second query");
// errorCb("Some Error from second query");
}
getDataFromDB("Some sql", function(data) {
console.log(data);
}, function(error) {
console.error(error);
});
Or use a Promises: http://documentup.com/kriskowal/q/
Related
Please take into consideration that similar questions have been asked on SO and I went through most of them.
I am making a RESTful service that needs querying the DB to get the data. I wrote the code that queries the database correctly but does returns undefined all the time. The code is here:
function returnAll(){
ModuleDBService.find({},function(err,data){
if(err){
console.log('Error occured while retrieving the documents!');
}
return data;
});
}
I was exporting the module using:
module.exports = {
getAll:returnAll
};
After digging SO a lot, I discovered that I will need to use callback to get the data. I went through many examples and tried to apply a similar technique to my code, the modified code looked like this:
function getAllFromDatabase(callback){
ModuleDBService.find({},function(err,data){
if(err){
console.log('Error occured while retrieving the documents!');
}
callback(returnAll(data));
});
}
function returnAll(data){ return data;}
and then returning it in the similar fashion as above.
But now I am getting error that ModuleDAO.getAll is not a function (I am using var ModuleDAO = require('path to the database service').
I tried many variations of the code, went through a couple of videos on YouTube, all of them either lead to returning undefined, or return to the above stated error. If anyone could fix the code and throw light on this whole callback thing (Or could provide solid documentation to understand it), it'll be a great help.
Thanks.
EDIT: After all the extremely helpful answers, here is a summary:
Callbacks cannot return data, pass the function (the callback function) that you want your program to call with the data. In my case, it was my router returning the data.
Here is the corrected code:
function returnAll(callback) {
ModuleDBService.find({}, function (err, data) {
if (err) {
console.log("Error while retrieving the document!")
callback(null);
}
callback(data);
});
}
I used this code in my router as:
mainAPIRouter.post('/api/module', function (req, res) {
try {
moduleDAO.getAll(function(data){
res.status(200);
res.json(data);
});
} catch (error) {
res.status(500);
return res.send("Invalid request");
}
});
Thanks to all those who helped! :)
You are close. You don't need the returnAll() function, and you need to export getAllFromDatabase and pass a callback to it:
function getAllFromDatabase(callback){
ModuleDBService.find({},function(err,data){
if(err) {
console.log('Error occured while retrieving the documents!');
}
callback(data);
});
}
module.exports = {
getAllFromDatabase: getAllFromDatabase
};
Then, when you want to use it, you need a callback function:
dataModule.getAllFromDatabase(callbackHandler);
function callbackHandler(dataFromDatabase) {
// this function will be executed when ModuleDBService executes the callback
console.log(dataFromDatabase);
}
A small detail: if err is Truthy, you should not execute the callback:
if(err) {
console.log('Error occured while retrieving the documents!');
} else {
callback(data);
}
You want to simply call the callback() with the data you need as an argument. You are making things much more complicated by passing another function into the callback. Try something like:
function returnAll(callback) {
ModuleDBService.find({}, function(err, data) {
if (err) return callback(err)
callback(null, data);
});
}
returnAll(function(err, data)) {
// it's customary for callbacks to take an error as their first argument
if (err) {
console.log('Error occured while retrieving the documents!');
} else {
// use data here!!
}
}
As previous answers mentioned, you can use a callback. You can also use a promise if you prefer:
function returnAll(){
return new Promise(function(resolve, reject) {
ModuleDBService.find({},function(err,data){
if(err){
console.log('Error occured while retrieving the documents!');
reject(err);
}
resolve(data);
});
});
}
You would then use something like this to access it:
returnAll()
.then(data=> {console.log(data); })
.catch(err=> { console.log(err); });
*Edit: since you want to use a callback, thought I'd add my $0.02 there as well. The most streamlined approach would be to just use the callback you're passing in with the ModuleDBService.find, without the ad-hoc function. But it's good to verify that callback actually is a function, and if not to make it one...makes your code more fault-tolerant.
function returnAll(cb){
if(typeof cb!=='function') cb = function() {};
ModuleDBService.find({},cb);
}
I have an app.get that will return customer data and customer purchases. Inside this app.get I need run two mysql calls and build a an array to pass back.
How can I execute one query after another and process that data?
app.get('/customer', function (req,res) {
var response1 = [];
var response2 = [];
var processedData = [];
connection.query('QUERY HERE', function(err, rows, fields) {
if (!err){
response.push({rows});
} else {
res.status(400).send(err);
}
});
//for loop 'response' results and perform another query
for (var i = 0; i < response1.length; i++) {
var row = response1[i];
connection.query('QUERY HERE FOR row.customerid', function(err, rows, fields) {
if (!err){
processedData.push({'Customer Name:' : row.customername, 'purchases' : rows});
} else {
res.status(400).send(err);
}
});
}
//Send json back
res.setHeader('Content-Type', 'application/json');
res.status(200).send(JSON.stringify(processedData));
});
There is a very convenient module called async.js that provides a bunch of functions for doing complex async operations. Particularly,
async.waterfall() is great when you need to pass down results from one async operation/task to another.
async.mapSeries() is great when you need to create a new array with results from an array of async operation/tasks.
Let's use both.
If I understood your code correctly, the code would look something similar to
app.get('/customer', function (req, res) {
async.waterfall([
// each task is passed a callback 'cb' as last argument;
// you MUST call it at least and at most once within each task;
// if you pass an error into the callback as the first argument, it will stop the async function
function task1 (cb1) {
//connection.query('QUERY HERE', function(err, rows, fields) {
// if (err) return cb1(err); // stop waterfall() if an error occurred
// cb1(null, rows, fields); // pass results down to next task
//});
connection.query('QUERY HERE', cb1); // shorter version
},
function task2 (rows, fields, cb2) {
// iterate and run async operation over each element in array 'rows'
async.mapSeries(rows, function getPurchases (row, cb3) {
connection.query('QUERY HERE FOR row.customerid', function (err, purchases, fields) {
if (err) return cb3(err); // stop mapSeries() if an error occurred
cb3(null, { 'Customer Name': row.customername, 'purchases': purchases })
});
}, function (err, customers) {
// when mapSeries() is done iterating OR if an error occurred, it will come here
if (err) return cb2(err); // stop waterfall() if an error occurred
cb2(null, customers)
});
// }, cb2); // shorter version
}
], function (err, customers) {
// when waterfall() is done all its tasks OR if an error occurred, it will come here
// handle error and send response here
});
});
I understand why I get this error, however I have no idea how to fix this. This is what my first function looks like:
All I am doing is doing a get request for all of my urls which are stored in the database, and then for each URL response, I go through the body and save the values information by calling saveInformation(value) or saveInfoTwo(value)
function getUrlInfo(urls, callback) {
urls.map(function(url) {
request.get(urls, {
timeout: 1500
}, function(error, response, body) {
if (error) {console.log(error);
} else {
parseString(body, function(error, result) {
if (error) {console.log(error);
} else {
result.Values.forEach(function(listValues) {
listValues.forEach(function(value) {
saveInformation(value);
});
saveInfoTwo(value);
});
}
});
}
callback(null);
});
});
}
My async waterfall looks like this:
async.waterfall([
getUrls,
getUrlInfo
], function(err, result) {
mongoose.connection.close();
});
Where getUrls is just a method that looks for all the urls and adds them to the urls array, so that getUrlInfo can use it.
function getUrls(callback) {
var urls = [];
urlSchema.find(function(error, data) {
data.forEach(function(result) {
clusters.push(result.url);
});
callback(null, urls);
});
}
I use mongodb as my database.
This works great when there is only one url. As soon as I add another url, I get this error:
Error: Callback was already called.
Now, after trying to debug, it seems like everything works fine for the first url, and then when the second url finishes going through its foreach loops, the error is thrown.
How can I fix this?
I am pretty sure that the callback in my getUrlInfo() waits for all of the urls and foreach loops to finish, then continues? - which is not the case I guess.
Any help would be appreciated!
To summarise: How can I prevent the error appearing when I add another url?
Your map call the function for the first URL, when it finishes, the callback is called. When the second one is executed, when it finishes, it will call callback again, thus you are having this error.
The solution, wait for all urls to finish and then call the callback.
As #Bergi said on the comments you can use async map
function getUrlInfo(urls, callback) {
async.map(urls, function(url, cb) {
//do what you must
cb();
}, callback);
}
Ive been playing around with mongodb in node.js. I have made a basic collection with some data (i know its there ive checked). When I try to run a find() on the collection it returns undefined. I dont know why this is. The code is below:
function get_accounts(){
var MongoClient = mongodb.MongoClient;
var url = "url";
MongoClient.connect(url, function (err, db) {
if (err) {
console.log('Unable to connect to the mongoDB server. Error:', err);
} else {
//HURRAY!! We are connected. :)
console.log('Connection established to database');
var collection = db.collection('accounts');
collection.find().toArray(function(err, docs) {
console.log("Printing docs from Array")
docs.forEach(function(doc) {
console.log("Doc from Array ");
console.dir(doc);
});
});
console.log("mission complete");
}
db.close();
}
);
}
If you know why this is happening i would like to hear your thoughts. thanks! The database is a mongolab hosted database if that makes any difference.
You are getting an undefined value because of the asynchronous nature of node.js, nowhere in your code exists logic that tells the console.log statement to wait until the find() statement finishes before it prints out the documents. You have to understand the concept of callbacks in Node.js. There are a few problems here, though, that you could fix. A lot of people getting started with node have the tendency to nest lots of anonymous functions, creating the dreaded "pyramid of doom" or callback hell. By breaking out some functions and naming them, you can make it a lot cleaner and easier to follow:
var MongoClient = require("mongodb").MongoClient
// move connecting to mongo logic into a function to avoid the "pyramid of doom"
function getConnection(cb) {
MongoClient.connect("your-mongo-url", function(err, db) {
if (err) return cb(err);
var accounts = db.collection("accounts");
cb(null, accounts);
})
}
// list all of the documents by passing an empty selector.
// This returns a 'cursor' which allows you to walk through the documents
function readAll(collection, cb) {
collection.find({}, cb);
}
function printAccount(account) {
// make sure you found your account!
if (!account) {
console.log("Couldn't find the account you asked for!");
}
console.log("Account from Array "+ account);
}
// the each method allows you to walk through the result set,
// notice the callback, as every time the callback
// is called, there is another chance of an error
function printAccounts(accounts, cb) {
accounts.each(function(err, account) {
if (err) return cb(err);
printAccount(account);
});
}
function get_accounts(cb) {
getConnection(function(err, collection) {
if (err) return cb(err);
// need to make sure to close the database, otherwise the process
// won't stop
function processAccounts(err, accounts) {
if (err) return cb(err);
// the callback to each is called for every result,
// once it returns a null, you know
// the result set is done
accounts.each(function(err, account) {
if (err) return cb(err)
if (hero) {
printAccount(account);
} else {
collection.db.close();
cb();
}
})
}
readAll(collection, processAccounts);
})
}
// Call the get_accounts function
get_accounts(function(err) {
if (err) {
console.log("had an error!", err);
process.exit(1);
}
});
You might have to add an empty JSON object inside the find.
collection.find({})
Documentation can be found here.
You must enter this code in an async function and you will be fine here data is the your desired value and you must use promises to not make your code look messy.
var accountCollection = db.collection('accounts);
let data = await accountCollection.find().toArray.then(data=>data).catch(err=>err);
I am trying to learn node and understand how callbacks are working. I am doing this by trying to use the async lib. I am trying to hit my db with 2 separate calls and use the async lib to let me know when my object is ready to build. Here is my aysnc code:
async.parallel({
one: function(callback) {
getUser(id, callback);
},
two: function(callback) {
getUserServices(id, callback);
}
}, function(err, results) {
if(err) {
console.log(err);
new Error();
}
res.json(result);
});
Here are what my functions look like where I am calling the db. There are both basically the same function:
var getUser = function(id, callback) {
var query = client.query('SELECT * FROM USERS WHERE USER_ID=$1', [id]);
query.on('row', function(row, result) {
result.addRow(row);
});
query.on('end', function(result) {
callback(result);
});
};
My code hits the db and returns the user, but when it goes back up to the async code the user is in the err object. What am I doing wrong? How do I properly set up callbacks?
As pointed by damphat, your code should be
//additionally, handle errors
query.on('error', function(err){
callback(err) // The first argument to callback should be an error object
})
query.on('end', function(result){
callback(null, result) //passing null error object
})