Develop Reference

Multiple Mongoose Calls Within a For Each Loop

I am reading a JSON object and looping through each item. I am first checking to see if the item already exists in the database and if so I want to log a message. If it doesn't already exist I want to add it.
This is working correctly however, I would like to add a callback or finish the process with process.exit();
Because the mongoose calls are asynchronous I can't put it at the end of the for loop because they haven't finished.
Whats the best way I should handle this?
function storeBeer(data) {
data.forEach((beer) => {
let beerModel = new Beer({
beer_id: beer.id,
name: beer.name,
image_url: beer.image_url
});
Beer.findOne({
'name': beer.name
}).then(function (result) {
if (result) {
console.log(`Duplicate ${result.name}`)
} else {
beerModel.save(function (err, result) {
console.log(`Saved: ${result.name}`)
});
}
});
});
}
Is there anything I should read up on to help solve this?

One means of managing asynchronous resources is through Promise objects, which are first-class objects in Javascript. This means that they can be organized in Arrays and operated on like any other object. So, assuming you want to store these beers in parallel like the question implies, you can do something like the following:
function storeBeer(data) {
// This creates an array of Promise objects, which can be
// executed in parallel.
const promises = data.map((beer) => {
let beerModel = new Beer({
beer_id: beer.id,
name: beer.name,
image_url: beer.image_url
});
return Beer.findOne({
'name': beer.name
}).then(function (result) {
if (result) {
console.log(`Duplicate ${result.name}`)
} else {
beerModel.save(function (err, result) {
console.log(`Saved: ${result.name}`)
});
}
});
);
});
return Promise.all(promises);
}
Don't forget that the storeBeer function is now asynchronous, and will need to be utilized either through a Promise chain or through async/await.
For example, to add process exit afterwards:
async function main() {
const beers = [ ... ];
await storeBeer(beer);
process.exit(0);
}
You can also modify the above example to invoke the storeBeer function within a try / catch block to exit with a different error code based on any thrown errors.

Updating multiple documents with different values

I'm using jquery ui's sortable function (source) to re-arrange elements. I've built custom callbacks to create an list of those elements. So when i move an element, all elements is given a new position id. It could look like this:
[{
id_of_element_in_database: 12,
new_position: 0
}, {
id_of_element_in_database: 16,
new_position: 1
}, {
id_of_element_in_database: 14,
new_position: 2
}]
I'm sending this list to my back-end by doing a simple Ajax post
$.post('/position', { data: list });
Route
router.post('/position', (req, res) => {
console.log(req.body.data); // This prints the array of objects above.
});
Schema
mongoose.Schema({
id: Number,
position: Number,
...
});
Now I can't figure out how to change the position of all documents effectively. Creating a crappy loop of the array and doing multiple database-requests can't be the best approach.
I've tried that here and this feels so wrong.
for (let i in req.body.data) {
collection.update({ id: req.body.data[i].id }, { position: req.body.data[i].position });
There must be something else i can do to achieve this. I've tried google without any luck.

You could try the bulkWrite API to carry out the updates in a better way without multiple requests to the server:
var callback = function(err, r){
console.log(r.matchedCount);
console.log(r.modifiedCount);
}
// Initialise the bulk operations array
var ops = req.body.data.map(function (item) {
return {
"updateOne": {
"filter": {
"id": parseInt(item.id),
"position": { "$ne": parseInt(item.position) }
},
"update": { "$set": { "position": parseInt(item.position) } }
}
}
});
// Get the underlying collection via the native node.js driver collection object
Model.collection.bulkWrite(ops, callback);

Here's how I did it with Node.js:
var forEach = require('async-foreach').forEach;
var bulk = Things.collection.initializeOrderedBulkOp();
Things.find({}).lean().execAsync(execSettings).then(function(resp){
forEach(resp, function(template, index, arr) {
var done = this.async();
bulk.find({'_id': template._id}).update({$set: {_sid: generateShortId()}});
bulk.execute(function (error) {
done();
});
}, function(notAborted, arr){
res.json({done: true, arr: arr.length});
});
});

Access Backbone fetch parameters

I need to be able to pass the server some info when calling fetch on a Backbone collection. On my front-end I have this code, passing data:{} to the fetch function along with the success and error callbacks. Looking at the docs for Backbone fetch:
http://backbonejs.org/#Collection-fetch
perhaps I am not passing in the parameters correctly to the fetch function? Perhaps they should be in an array []...
var lineupCollection = new LineupCollection([]);
var loadTeamsOnPageLoad = (function() {
lineupCollection.url = '/api/lineups';
lineupCollection.fetch(
{
processData: true,
data: {
team_id:$("#team_id").attr("value")
}
},
{
success: function (result) {
if(window.teamDashboardTableView === undefined){
window.teamDashboardTableView = new TeamDashboardTableView({el: $("#team-dashboard-table-div")});
}
window.teamDashboardTableView.render();
},
error: function (err) {
setTimeout(function () {
alert('error fetching lineupCollection - ' + err.message);
}, 1);
}
}
);
})();
which will load a Backbone collection into a table.
It's call this method on the back-end:
exports.getBackboneLineups = function(req,res,next){
var user_id = req.mainUser._id;
console.log('req.params',req.params); //empty {}
console.log('req.team',req.team); //undefined
console.log('team_id',req.team_id); //undefined
console.log('req.data',req.data); //undefined
console.log('req.body',req.body); //empty {}
var team_id = req.team._id; //undefined EXCEPTION, can't read property _id of undefined
var system_db = req.system_db;
var Lineup = LineupModel.getNewLineup(system_db,user_id);
Lineup.find({team_id:team_id}, function (err, lineups) {
if (err) {
return next(err);
}
res.json(lineups);
});
};
however, I am totally failing as far as how to access the data that I supposedly past to the server from the client. I can't really find a good example and I am getting tired of guessing. What's the best way to do this?
edit:
also tried passing {data:{}, success: function(), error: function()} all in the same JavaScript object, but that didn't work.

fetch only takes one argument (see http://backbonejs.org/#Collection-fetch). Instead, you should probably alter your backend to specify the team id in the resource's URL. For example:
lineupCollection.url = function(){ return '/api/lineups?team_id=' + this.team_id; };
lineupCollection.team_id = $("#team_id").attr("value");
lineupCollection.fetch({ success: ..., error: ...});

Node.js Twitter API cursors

I'm using npm-twit to get followers of a specific account.
The Twitter API returns up to 5000 results from a single GET request.
If the user I'm querying has over 5000 followers a "next_cursor" value is returned with the data.
To get the next 5000 results, I need to re-run the GET function, passing it the "next_cursor" value as an argument. I just can't seem to work out how to do it.
I was thinking a while loop, but I can't reset the global variable, I think because of scope:
var cursor = -1
while ( cursor != 0 ) {
T.get('followers/ids', { screen_name: 'twitter' }, function (err, data, response) {
// Do stuff here to write data to a file
cursor = data["next_cursor"];
})
}
Obviously I'm not a JS genius, so any help would be much appreciated.

The issue you are having is due to Node.js being asynchronous.
T.get('followers/ids', { screen_name: 'twitter' }, function getData(err, data, response) {
// Do stuff here to write data to a file
if(data['next_cursor'] > 0) T.get('followers/ids', { screen_name: 'twitter', next_cursor: data['next_cursor'] }, getData);
})
}
Please note:
I gave a name to the internal callback function. That is so that we can recursively call it from the inside.
The loop is replaced with a recursive callback.
If there is a next_cursor data, then we call T.get using the same function getData.
Be aware that Do stuff here code will be executed many times (as many as there are next cursors). Since it is recursive callback - the order is guaranteed.
If you do not like the idea of recursive callbacks, you can avoid it by:
Finding out beforehand all the next_cursor's if possible, and generate requests using for loop.
Alternatively, use asynchronous-helper modules like Async (though for learning purposes, I would avoid modules unless you are fluent in the concept already).

Consider testing with some 5K+ account.
const T = new Twit(tokens)
function getFollowers (screenName, followers = [], cur = -1) {
return new Promise((resolve, reject) => {
T.get('followers/ids', { screen_name: screenName, cursor: cur, count: 5000 }, (err, data, response) => {
if (err) {
cur = -1
reject(err)
} else {
cur = data.next_cursor
followers.push(data.ids)
if (cur > 0) {
return resolve(getFollowers(screenName, followers, cur))
} else {
return resolve([].concat(...followers))
}
}
})
})
}
async function getXaqron () {
let result = await getFollowers('xaqron')
return result
}
console.log(getXaqron().catch((err) => {
console.log(err) // Rate limit exceeded
}))

Struggled with this one.. Everything seemed to work, but data['next_cursor'] didn't change, EVER!
Code should be like this:
T.get('followers/ids', { screen_name: 'twitter' }, function getData(err, data, response) {
// Do stuff here to write data to a file
if(data['next_cursor'] > 0) T.get('followers/ids', { screen_name: 'twitter', cursor: data['next_cursor'] }, getData);
})
}
Parameter for Twit isn't "next_cursor", it's just "cursor" ;)

Mongoose - setting virtuals via return on callback function [duplicate]

This question already has an answer here:
Mongoose complex (async) virtuals
(1 answer)
Closed 8 years ago.
I see similar questions posted all over stackoverflow, but can't seem to find a satisfactory answer.
I'm using MongooseJS as my ODM and I'm trying to set up virtual getters than query, analyze and return information from a different collection.
Unfortunately, (because of nodejs asynchronous nature) I can't return information from within a callback function. Is there an easy way of going about this?
Here's my code:
UserSchema.virtual('info').get(function () {
var data = {
a: 0,
b: 0
};
OtherSchema.find({}, function (err, results) {
results.forEach(function (result) {
if (result.open) {
data.a += 1
} else {
data.b += 1
}
});
return data; //return this information
})
});
Any help would be greatly appreciated!

You need to pass a callback function into your virtual method, like so:
UserSchema.virtual('info').get(function (cb) {
var data = {
a: 0,
b: 0
};
OtherSchema.find({}, function(err, results) {
if (err) {
// pass the error back to the calling function if it exists
return cb(err);
}
results.forEach(function(result) {
if(result.open) { data.a+=1 }
else{data.b+=1}
});
// pass null back for the error and data back as the response
cb(null, data);
});
});
Then to call the function you would do (excuse my syntax on calling the virtual method. Not 100% sure how that works in Mongoose):
UserSchema.info(function(err, data) {
// check if there was an error
// if not then do whatever with the data
}