I have frustrating problem with learning to work with callback style of programming in Node.js. I have a query to a MongoDB database. If I pass in a function to execute on the result it works but I'd rather flatten it out and have it return the value. Any help or direction on how to do this correctly is appreciated. Here's my code:
var getLots = function(response){
db.open(function(err, db){
db.collection('lots', function(err, collection){
collection.find(function(err, cursor){
cursor.toArray(function(err, items){
response(items);
})
})
})
})
}
I want something more like this:
lots = function(){
console.log("Getting lots")
return db.open(openCollection(err, db));
}
openCollection = function(err, db){
console.log("Connected to lots");
return (db.collection('lots',findLots(err, collection))
);
}
findLots = function(err, collection){
console.log("querying 2");
return collection.find(getLots(err, cursor));
}
getLots = function(err, cursor) {
console.log("Getting lots");
return cursor.toArray();
}
Where the final set of data would bubble back up through the function calls.
The problem is that I get an error from Node.js saying that err is not defined or that the collection is not defined. For some reason when I nest the callbacks the correct object is getting passed down. When I try going to this flattened style it complains that things are not defined. I don't know how to get it to pass the necessary objects.
What you need is one of the many control flow libraries available for node via npm and catalogued on the Node.js wiki. My specific recommendation is caolan/async, and you would use the async.waterfall function to accomplish this type of flow where each async operation must be executed in order and each requires the results from the previous operation.
Pseudocode example:
function getLots(db, callback) {
db.collection("lots", callback);
}
function findLots(collection, callback) {
collection.find(callback);
}
function toArray(cursor, callback) {
cursor.toArray(callback);
}
async.waterfall([db.open, getLots, find, toArray], function (err, items) {
//items is the array of results
//Do whatever you need here
response(items);
});
async is a good flow control library. Frame.js offers some specific advantages like better debugging, and better arrangement for synchronous function execution. (though it is not currently in npm like async is)
Here is what it would look like in Frame:
Frame(function(next){
db.open(next);
});
Frame(function(next, err, db){
db.collection('lots', next);
});
Frame(function(next, err, collection){
collection.find(next);
});
Frame(function(next, err, cursor){
cursor.toArray(next);
});
Frame(function(next, err, items){
response(items);
next();
});
Frame.init();
Related
I have been learning JavaScript and started exploring Node. I have spent time understanding callback hell and how to fix it, but now I am more lost than ever and I'm starting to think I'm looking at it backwards (this might be a stupid question). I have read a guide on callbackhell.com and number 1 rule is to keep the code shallow.
Here is an example of connecting to Database, then reading a file, then inserting records to MongoDB and then logging:
MongoClient.connect(url, (err, db)=>{
let dbo = db.db('test');
fs.readFile('./data.json', (err, data)=>{
let records = JSON.parse(data);
dbo.collection('people').insertMany(records, (err, result) =>{
console.log("result");
db.close();
})
});
});
In the example, I have 3 anonymous callback functions and I have no idea how to refactor as the db is used throughout the callbacks. From what I understand I am should be aiming for 3 named functions (or am I missing something?) and call it like this:
MongoClient.connect(url, cb1);
and the functions something like this:
function cb1(err, db){
let dbo = db.db('test');
fs.readFile('./data.json', cb2);
}
function cb2(err, data){
let records = JSON.parse(data);
// ??? dbo.collection('people).insertMany(records, cb3)
//now what?
}
function cb3(err, result){
console.log(result);
// ??? db.close?
}
Looking for any insight on Callback hell. Is this callback hell? How would I go about this? Am I missing something conceptually? Should it even be refactored?
PS. Promises and Async/Await can wait till I understand how to go about async programming using callbacks
Thank you!
The nested callbacks are indeed what is commonly called callback hell.
Your attempt at separating the callbacks into named functions is fine, but you have realised one issue: that you need a reference to the db object in all callbacks, and it is not there.
You can solve this by either binding this to db, or else binding a (first) parameter for passing on that db object as argument. The principle is really the same.
Here is how it would look with binding this to the db object:
function cb1(err, db) {
let dbo = db.db('test');
fs.readFile('./data.json', cb2.bind(db));
}
function cb2(err, data) {
let records = JSON.parse(data);
this.collection('people').insertMany(records, cb3.bind(this));
}
function cb3(err, result) {
console.log(result);
this.close();
}
And here is how it would look with an extra parameter:
function cb1(err, db) {
let dbo = db.db('test');
fs.readFile('./data.json', cb2.bind(null, db));
}
function cb2(db, err, data) {
let records = JSON.parse(data);
db.collection('people').insertMany(records, cb3.bind(null, db));
}
function cb3(db, err, result) {
console.log(result);
db.close();
}
The next step would be to embrace promises and async await syntax. See for instance "How to use MongoDB with promises in Node.js?".
I am listing all files from all directories in /home/myComputer/Desktop/Research, and then filtering them with an if statement to only get the .txt files that I would like to read and store into arrays. All works fine, but pushing the data into the arrays is not functioning. When I console log them, they return no value [].
I tried promise as well as call back function, but they didn't work for me because I didn't know how to implement them properly.
app.get('/jsonData', function(req, res) {
/* Define Arrays */
var theFile = [];
var theCategory = [];
var theContent = [];
var walk = function(dir, done) {
var results = [];
fs.readdir(dir, function(err, list) {
if (err) return done(err);
var i = 0;
(function next() {
var file = list[i++];
if (!file) return done(null, results);
file = dir + '/' + file;
fs.stat(file, function(err, stat) {
if (stat && stat.isDirectory()) {
walk(file, function(err, res) {
results = results.concat(res);
next();
});
} else {
results.push(file);
next();
}
});
})();
});
};
//walk(process.env.HOME, function(err, results) {
walk("/home/myComputer/Desktop/Research", function(err, results) {
if (err) throw err;
//console.log(results);
results.map(function(val) {
//Get the filename
var fileName = val.match(/[^\/]+$/).join();
//Get the category
var category = val.substr(48).match(/[^\/]+/);
if (fileName == 'written-speech.txt') {
console.log('FOUND!: ' + fileName + ' Category: ' + category) //this works
fs.readFile(val, 'utf8', function(err, contents) {
console.log(contents); // this works
theFile.push(fileName);
theCategory.push(category);
theContent.push(contents);
});
}
})
});
console.log(theFile); // The problem: This returns an empty Array []
console.log(theCategory); // The problem: This returns an empty Array []
console.log(theContent); // The problem: This returns an empty Array []
});
I expect console.log(theFile); console.log(theCategory); and console.log(theContent); to return the data pushed in them.
The reason for this is that many callbacks in Javascript are asynchronous, which means both fs.readdir and fs.readFile are asynchronous and their callbacks are not called immediately but slightly later (please read about Event Loop in javascript). So at the moment, when you log your arrays they are empty and data to them will be pushed later, e.g. in future. To avoid this you can either use synchronous methods (fs.readdirSync and fs.readFileSync) which is ugly and can cause performance issues if the app has a lot of other asynchronous operations. If in your case it is just a simple script to read some data, it might be fine.
And the other, preferred way is to use promises or some library for managing callbacks, e.g. async. Please read some articles regarding managing async code if these concepts are fully unfamiliar for you, e.g. https://dev.to/mrm8488/from-callbacks-to-fspromises-to-handle-the-file-system-in-nodejs-56p2 to get a basic understanding and see some use case examples.
Regarding your current version, there is no easy way to make it work without a lot of changes. It is better to rewrite it to use the concepts I described earlier.
walk is an asynchronous function because fs.readdir is an asynchronous method and the console.log statements are running (in a synchronous manner) before the callback of fs.readdir getting invoked.
You can console the values of these variables at the end inside the callback of walk.
My goal is to create a JSON object, from a paragraph of text, that I can then insert as a document into MongoDB. I'm using nodejs and wanted to go for the async approach.
My JSON has parameters like so
{
height:height,
weight:weight
}
My logic is this
create a module with async functions that parse the text and extract weight and height using regex.
but then how would I combine all the responses from these functions into one JSON that I can import at once?
I'm thinking something like this
var get_height = require().height;
var get_weight = require().weight;
exports.contr = function(){
var height,
weight;
get_height(text, function(err, res){
if(err)
throw(err)
height=res;
});
get_weight(text, function(err, res){
if(err)
throw(err)
weight=res;
});
//All other async functions
combine_json(height, weight, ... , function(err, res){
if(err)
throw(err);
console.log(res); //the json was successfully inserted into mongoDB
});
}
I find async confusing and in the above example I'm not sure about two things
wouldn't combine_json run without waiting for the data from the previous two functions (weight, height)
what is the best practice to handle such cases? Should i just use sync functions and wait top-to-bottom for each one to do its thing and then run the final one or I can leverage async?
The simplest way to wait for the results of two independent asynchronous functions is to use promises and Promise.all. For this we'll assume get_height and get_weight return a Promise and can be used as such:
get_height().then(function (height) { console.log(height); });
Then it's trivial to combine two of those promises:
Promise.all([get_height(), get_weight()]).then(function (results) {
combine_json(results[0], results[1]);
});
See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise for documentation and details.
If you do not know nothing of Promises, you first should know how callbacks works. If you do not want a elegant solution with Promise.all() and you just want your code working, you need nested functions. When you are inside get_height callback you should to call get_weight and same when you are inside get_weight callback you should call combine_json() . The only problem is that you have to wait for get_height to call get_weight. This is solved with Promise.all() as it's said.
get_height(text, function(err, height){
if(err)
throw(err);
get_weight(text, function(err, weight){
if(err)
throw(err);
//All other async functions
combine_json(height, weight, ... , function(err, res){
if(err)
throw(err);
console.log(res); //the json was successfully inserted into mongoDB
});
});
});
Promises are your best bet, but if you don't want to use them for some reason and prefer the callback style, then
function get_height_and_weight(text, callback) {
var have_height = false;
var have_weight = false;
var result = {};
get_height(text, function(err, height) {
if (err) callback(err);
have_height = true;
result.height = height;
if (have_weight) callback(null, result);
});
get_weight(text, function(err, weight) {
if (err) callback(err);
have_weight = true;
result.weight = weight;
if (have_height) callback(null, result);
});
}
This is a special case of the parallel async call case which could be handled better by async.parallel.
Hi i'm trying to access the elements returned from a find in mongoose and having some trouble with the asynchronous and callback situation.
Here is the code for better understanding.
function retrieveBudgets(email, callback) {
models.User.find({email: email},{budget:true}, function(err,budgets) {
if (err) {
callback(err, null);
} else {
callback(null, budgets);
}
});
};
retrieveBudgets(user.email, function(err, budgets) {
if (err) {
console.log(err);
}
budgets.forEach(function(budget){
console.log(JSON.stringify(budget, null, 4));
});
});
So this line console.log(JSON.stringify(budget, null, 4)); is working correctly and printing the objects to screen in json format but how do I store each to an array of objects from here? if I try to push to an array at this same line I get an error.
I have seen some questions that are similar but i am not getting any headway with them.
EDIT:____________________________________________________________
I did a little hack to get it working, i moved res.render up, so that rendering the page was done at the same time as the callback but I cant see this being the right solution any thoughts
var user=req.session.user;
res.locals.budgets=[];
function retrieveBudgets(email, callback) {
models.User.find({email: email},{budget:true}, function(err, budgets) {
if (err) {
callback(err, null);
} else {
callback(null, budgets);
}
});
};
retrieveBudgets(user.email, function(err, budgets) {
if (err) {
console.log(err);
}
res.locals.budgets = budgets.map((function(b){ return b; });
res.render('budget/budget.jade',{ csrfToken: req.csrfToken() });
});
This works I can access budgets through locals so any feedback on this would be great I doubt its the right way to do it?
the budgets return value that you get from the retrieveBudgets call is already an array.
this is evidenced by your call to budgets.forEach which is a method on arrays.
is there a specific need to create a new array from the items? that can be easily done:
var myNewArray = budgets.map((function(b){ return b; });
this one line of code will map the original budgets array into a new array containing each of the budget items.
there are other methods of creating a new array, depending on what you need to do exactly
update from comments below
what i really want to do to is use the budgets outside of the query so I can pass it to the view
in that case, you need to render the view from within the callback and pass the budgets to the view:
router.get("/foo", function(req, res, next){
retrieveBudgets(user.email, function(err, budgets) {
if (err) { return next(err); }
res.render('budget/budget.jade',{
budgets: budgets,
csrfToken: req.csrfToken()
});
});
});
This is the only, and correct, way to make this work.
If you tried to do it without waiting for the callback to finish, you would not have any data in your budgets array. Therefore, you must wait for the callback to be executed and then render your view with the budgets (or single budget or whatever) passed to the view.
(There are variations of this using promises, but I find callbacks to be the easier way to handle this.)
Turn the line in which you're passing an object to your jade file
res.render('budget/budget.jade',{
csrfToken: req.csrfToken(),
budgets: budgets.map(function(b) {return b;})
});
This will pass budgets to your jade file, and you should be able to access it there.
I'm new to js.
I am using express for node js, and mongoose as a mongo orm.
function direct_tags_search_in_db(tags){
var final_results = [];
for (var i=0; i<tags.length; ++i) {
var tag = tags[i];
Question.find({tags: tag}).exec(function(err, questions) {
final_results.push(questions);
if (i == tags.length -1 ){
return final_results;
}
});
}
};
I get empty results, because of the asynchronously of the find. But I don't know what the best approach for this.
Appriciate a little help, thanks.
You will often find that methods such as Question.find().exec that accept a function as an argument are async. It is especially common for methods that perform network requests or file system operations. These are most commonly referred to as a callback. That being the case, if you would like something to occur when the async task(s) complete, you need to also implement a callback.
Also, it is possible that your reference to tag is being changed in a way that is likely undesired. There are a number of solutions, here is a simple one.
function direct_tags_search_in_db(tags, callback){
var final_results = [];
// Array.forEach is able to retain the appropriate `tag` reference
tags.forEach(function(tag){
Question.find({tags: tag}).exec(function(err, questions) {
// We should be making sure to handle errors
if (err) {
// Return errors to the requester
callback(err);
} else {
final_results.push(questions);
if (i == tags.length -1 ){
// All done, return the results
callback(null, final_results);
}
}
});
});
};
You will notice that when we implement our own callback, that we follow the same common pattern as the callback for Question.find().exec(function(err, result){}); -- first argument a potential error, second argument the result. That is why when we return the results, we provide null as the first argument callback(null, final_results);
Quick example of calling this function:
direct_tags_search_in_db([1, 2, 3], function(err, results){
if (err) {
console.error('Error!');
console.error(err);
} else {
console.log('Final results');
console.log(results);
}
});
Another option for solving various async goals is the async module, promises, or otherwise.