I understand why I get this error, however I have no idea how to fix this. This is what my first function looks like:
All I am doing is doing a get request for all of my urls which are stored in the database, and then for each URL response, I go through the body and save the values information by calling saveInformation(value) or saveInfoTwo(value)
function getUrlInfo(urls, callback) {
urls.map(function(url) {
request.get(urls, {
timeout: 1500
}, function(error, response, body) {
if (error) {console.log(error);
} else {
parseString(body, function(error, result) {
if (error) {console.log(error);
} else {
result.Values.forEach(function(listValues) {
listValues.forEach(function(value) {
saveInformation(value);
});
saveInfoTwo(value);
});
}
});
}
callback(null);
});
});
}
My async waterfall looks like this:
async.waterfall([
getUrls,
getUrlInfo
], function(err, result) {
mongoose.connection.close();
});
Where getUrls is just a method that looks for all the urls and adds them to the urls array, so that getUrlInfo can use it.
function getUrls(callback) {
var urls = [];
urlSchema.find(function(error, data) {
data.forEach(function(result) {
clusters.push(result.url);
});
callback(null, urls);
});
}
I use mongodb as my database.
This works great when there is only one url. As soon as I add another url, I get this error:
Error: Callback was already called.
Now, after trying to debug, it seems like everything works fine for the first url, and then when the second url finishes going through its foreach loops, the error is thrown.
How can I fix this?
I am pretty sure that the callback in my getUrlInfo() waits for all of the urls and foreach loops to finish, then continues? - which is not the case I guess.
Any help would be appreciated!
To summarise: How can I prevent the error appearing when I add another url?
Your map call the function for the first URL, when it finishes, the callback is called. When the second one is executed, when it finishes, it will call callback again, thus you are having this error.
The solution, wait for all urls to finish and then call the callback.
As #Bergi said on the comments you can use async map
function getUrlInfo(urls, callback) {
async.map(urls, function(url, cb) {
//do what you must
cb();
}, callback);
}
Related
Please take into consideration that similar questions have been asked on SO and I went through most of them.
I am making a RESTful service that needs querying the DB to get the data. I wrote the code that queries the database correctly but does returns undefined all the time. The code is here:
function returnAll(){
ModuleDBService.find({},function(err,data){
if(err){
console.log('Error occured while retrieving the documents!');
}
return data;
});
}
I was exporting the module using:
module.exports = {
getAll:returnAll
};
After digging SO a lot, I discovered that I will need to use callback to get the data. I went through many examples and tried to apply a similar technique to my code, the modified code looked like this:
function getAllFromDatabase(callback){
ModuleDBService.find({},function(err,data){
if(err){
console.log('Error occured while retrieving the documents!');
}
callback(returnAll(data));
});
}
function returnAll(data){ return data;}
and then returning it in the similar fashion as above.
But now I am getting error that ModuleDAO.getAll is not a function (I am using var ModuleDAO = require('path to the database service').
I tried many variations of the code, went through a couple of videos on YouTube, all of them either lead to returning undefined, or return to the above stated error. If anyone could fix the code and throw light on this whole callback thing (Or could provide solid documentation to understand it), it'll be a great help.
Thanks.
EDIT: After all the extremely helpful answers, here is a summary:
Callbacks cannot return data, pass the function (the callback function) that you want your program to call with the data. In my case, it was my router returning the data.
Here is the corrected code:
function returnAll(callback) {
ModuleDBService.find({}, function (err, data) {
if (err) {
console.log("Error while retrieving the document!")
callback(null);
}
callback(data);
});
}
I used this code in my router as:
mainAPIRouter.post('/api/module', function (req, res) {
try {
moduleDAO.getAll(function(data){
res.status(200);
res.json(data);
});
} catch (error) {
res.status(500);
return res.send("Invalid request");
}
});
Thanks to all those who helped! :)
You are close. You don't need the returnAll() function, and you need to export getAllFromDatabase and pass a callback to it:
function getAllFromDatabase(callback){
ModuleDBService.find({},function(err,data){
if(err) {
console.log('Error occured while retrieving the documents!');
}
callback(data);
});
}
module.exports = {
getAllFromDatabase: getAllFromDatabase
};
Then, when you want to use it, you need a callback function:
dataModule.getAllFromDatabase(callbackHandler);
function callbackHandler(dataFromDatabase) {
// this function will be executed when ModuleDBService executes the callback
console.log(dataFromDatabase);
}
A small detail: if err is Truthy, you should not execute the callback:
if(err) {
console.log('Error occured while retrieving the documents!');
} else {
callback(data);
}
You want to simply call the callback() with the data you need as an argument. You are making things much more complicated by passing another function into the callback. Try something like:
function returnAll(callback) {
ModuleDBService.find({}, function(err, data) {
if (err) return callback(err)
callback(null, data);
});
}
returnAll(function(err, data)) {
// it's customary for callbacks to take an error as their first argument
if (err) {
console.log('Error occured while retrieving the documents!');
} else {
// use data here!!
}
}
As previous answers mentioned, you can use a callback. You can also use a promise if you prefer:
function returnAll(){
return new Promise(function(resolve, reject) {
ModuleDBService.find({},function(err,data){
if(err){
console.log('Error occured while retrieving the documents!');
reject(err);
}
resolve(data);
});
});
}
You would then use something like this to access it:
returnAll()
.then(data=> {console.log(data); })
.catch(err=> { console.log(err); });
*Edit: since you want to use a callback, thought I'd add my $0.02 there as well. The most streamlined approach would be to just use the callback you're passing in with the ModuleDBService.find, without the ad-hoc function. But it's good to verify that callback actually is a function, and if not to make it one...makes your code more fault-tolerant.
function returnAll(cb){
if(typeof cb!=='function') cb = function() {};
ModuleDBService.find({},cb);
}
In my node.js project, I need to make iterative API request so I'm using async.each method.
The console complains that url is not defined and I understand this because I've only declared urls which is an array of url.
I'm just not sure how I can put together request() from request module inside async.each. To satisfy async.each() I've placed urls as the first argument but request() itself requires a query string argument which is url.
I'm also using _.extend from Underscore to merge two responses but I'm not sure where I have it currently is in the right place.
Can someone please point me in the right direction?
var urls = [];
Object.keys(result.items).forEach(function(item) {
urls.push("https://www.googleapis.com/youtube/v3/videos?part=contentDetails&id=" + result.items[item].contentDetails
.videoId + "&key=xxx");
})
async.each(
urls,
request(url, function(err, response, body) {
if (err) {
console.log(err);
return;
}
body = JSON.parse(body);
}),
function() {
var extended = _.extend(result, body);
getVideoDetailsCallback(null, extended)
});
It seems you're calling request with callbacks and all, and not just referencing it, which means you probably need an anonymous function call.
Also, if you want an array at the end, or whatever you're extending, you could just use async.map instead, and do something like
var urls = Object.keys(result.items).map(function(item) {
return "https://www.googleapis.com/youtube/v3/videos?part=contentDetails&id=" + result.items[item].contentDetails.videoId + "&key=xxx";
});
async.map(urls, function(url, callback) {
request(url, function(err, response, body) {
if (err) {
return callback(err);
}
callback(null, JSON.parse(body));
});
}, function(err, extended) {
if (err) {
// handle error
}
// extended is an array containing the parsed JSON
getVideoDetailsCallback(null, extended);
});
If you want to get the results from each iteration. I suggest using async.map. Map returns an array of results to the final callback.
Here's an example,
var async = require('async');
async.map([ 1, 2, 3, 4 ], function ( num, cb ) {
cb(null, num);
}, function (error, results) {
console.log(results);
});
// output: [ 1, 2, 3, 4 ]
As for your code snippet, your second argument is the wrong type. You're invoking request which doesn't return a function. The second argument is expected to be a function so what you need to do is wrap the request around function definition.
async.map(urls, function(url, callback) {
request(options, callback);
}, function (error, results) {
// do something with results
});
Ive been playing around with mongodb in node.js. I have made a basic collection with some data (i know its there ive checked). When I try to run a find() on the collection it returns undefined. I dont know why this is. The code is below:
function get_accounts(){
var MongoClient = mongodb.MongoClient;
var url = "url";
MongoClient.connect(url, function (err, db) {
if (err) {
console.log('Unable to connect to the mongoDB server. Error:', err);
} else {
//HURRAY!! We are connected. :)
console.log('Connection established to database');
var collection = db.collection('accounts');
collection.find().toArray(function(err, docs) {
console.log("Printing docs from Array")
docs.forEach(function(doc) {
console.log("Doc from Array ");
console.dir(doc);
});
});
console.log("mission complete");
}
db.close();
}
);
}
If you know why this is happening i would like to hear your thoughts. thanks! The database is a mongolab hosted database if that makes any difference.
You are getting an undefined value because of the asynchronous nature of node.js, nowhere in your code exists logic that tells the console.log statement to wait until the find() statement finishes before it prints out the documents. You have to understand the concept of callbacks in Node.js. There are a few problems here, though, that you could fix. A lot of people getting started with node have the tendency to nest lots of anonymous functions, creating the dreaded "pyramid of doom" or callback hell. By breaking out some functions and naming them, you can make it a lot cleaner and easier to follow:
var MongoClient = require("mongodb").MongoClient
// move connecting to mongo logic into a function to avoid the "pyramid of doom"
function getConnection(cb) {
MongoClient.connect("your-mongo-url", function(err, db) {
if (err) return cb(err);
var accounts = db.collection("accounts");
cb(null, accounts);
})
}
// list all of the documents by passing an empty selector.
// This returns a 'cursor' which allows you to walk through the documents
function readAll(collection, cb) {
collection.find({}, cb);
}
function printAccount(account) {
// make sure you found your account!
if (!account) {
console.log("Couldn't find the account you asked for!");
}
console.log("Account from Array "+ account);
}
// the each method allows you to walk through the result set,
// notice the callback, as every time the callback
// is called, there is another chance of an error
function printAccounts(accounts, cb) {
accounts.each(function(err, account) {
if (err) return cb(err);
printAccount(account);
});
}
function get_accounts(cb) {
getConnection(function(err, collection) {
if (err) return cb(err);
// need to make sure to close the database, otherwise the process
// won't stop
function processAccounts(err, accounts) {
if (err) return cb(err);
// the callback to each is called for every result,
// once it returns a null, you know
// the result set is done
accounts.each(function(err, account) {
if (err) return cb(err)
if (hero) {
printAccount(account);
} else {
collection.db.close();
cb();
}
})
}
readAll(collection, processAccounts);
})
}
// Call the get_accounts function
get_accounts(function(err) {
if (err) {
console.log("had an error!", err);
process.exit(1);
}
});
You might have to add an empty JSON object inside the find.
collection.find({})
Documentation can be found here.
You must enter this code in an async function and you will be fine here data is the your desired value and you must use promises to not make your code look messy.
var accountCollection = db.collection('accounts);
let data = await accountCollection.find().toArray.then(data=>data).catch(err=>err);
I am getting a request from client that request needs to select the data from two tables , from one table i need to select 1000 records in table two i need to select only two records, handling it with a call back . I want the response that needs to send to my client after the two callback function executes , How to achieve this in node js?
You have following options:
Follow Node style callback pattern. Also read, Callback-hell
Follow promises.
Callback pattern
var f = function(somedata, cb){
// pick user from db
pickUserFromTable1(somedata, function(err, users){
// callled on completion of pickUserFromDB
if(err){
//handle error.
cb(err, null);
return;
}
pickUserFromTable2(users, function(err, finallySelectedUsers){
// called on completion of pickUserFromTable2
if(err){
// handle error.
cb(err, null);
// returning is important. or use else part of if to separate the case
return;
}
// call initiator's callback function on completion of both pickUserFromTable1 and pickUserFromTable2
cb(null, finallySelectedUsers);
});
})
}
Note: This approach is good for short project but leads to some maintenance issue (callback hell) if dependency chain is lengthy.
Welcome to callback hell! You just need to build pyramide form callbacks!
Example:
function getDataFromDB (query, successCb, errorCb) {
executeFirstQuery(query, function(firstQueryData) {
console.log('First query ok! Now execute second.');
executeSecondQuery(query, function(secondQueryData) {
console.log('Second query ok! Now execute successCb().');
successCb([firstQueryData, secondQueryData]);
}, function(error) {
errorCb("Error in second query");
})
}, function(error) {
errorCb("Error in first query");
});
}
function executeFirstQuery (query, successCb, errorCb) {
console.log("Execute first query!");
successCb("Some data from first query");
// errorCb("Some Error from first query");
}
function executeSecondQuery (query, successCb, errorCb) {
console.log("Execute second query!");
successCb("Some from second query");
// errorCb("Some Error from second query");
}
getDataFromDB("Some sql", function(data) {
console.log(data);
}, function(error) {
console.error(error);
});
Or use a Promises: http://documentup.com/kriskowal/q/
I'm trying to use the async waterfall method but when it gets to one of the functions, it hangs. I suspect it's because the save() operation is too slow for the execution context, but that's why I was starting to use async's waterfall, so I can wait for the value returned until it goes to the next function in the series (passing along the proper data with it which would be the counted in my case below).
// In my user controller:
async.waterfall([
function(callback) {
getSubmission(id, function(submission) {
if (submission) {
callback(null, submission);
}
});
},
function(submission, callback) {
var submissionId = submission._id;
getViews(submissionId, ip, function(count) {
if (count) {
callback(null, count, submissionId);
}
});
},
// Those top two functions work perfectly passing what
// I need to this one which is where I'm having trouble
function(views, submissionId, callback) {
// addView is called because it is actually
// inserting a row in the db, but never returns from the caller
addView(submissionId, ip, function(added) {
// this callback doesn't fire
if (added) {
callback(null, added);
}
});
},
function(added, callback) {
console.log(added);
}
]);
This is what addView() is (also within user controller which is where the previous async.waterfall code also is in) :
var addView = function(submissionId, ip, callback) {
Submission.addView({
submissionId : submissionId,
ip: ip
}, function(err, counted) {
if (err) {
throw err;
}
if (counted) {
callback(counted);
}
});
};
This is what it's calling (inside my Submission model file) when it calls Submission.addView():
exports.addView = function(obj, fn) {
var ip = obj.ip,
submissionId = obj.submissionId,
submissionView = new SubmissionView(obj);
// it gets to this point
submissionView.save({
ip : ip,
submission_id : submissionId
}, function(err, counted) {
fn(err, counted);
});
};
Whenever async "hangs", it's usually because a callback hasn't been called.
You need to make sure that you call the callback in all code paths. I would also recommend that you reserve the first parameter of any async callback to be an error, even if you don't use it as that is the pattern used throughout node.js. Some modules rely on this pattern. e.g. domains.
If you make the below change, then I would expect some error to pop up somewhere:
getSubmission(id, function(submission) {
if (submission) {
callback(null, submission);
}
});
should be something like this:
getSubmission(id, function(err, submission) {
if(err){
return callback(err);
}
if (!submission) {
return callback('no submission found');
}
callback(null, submission);
});