Develop Reference

MondoDB/Mongoose query responce is too slow

I new to MongoDB/Mongoose, and work with a very large database (more than 25000 docs). I need to configure different queries: by fields, first 10 docs, one by id. The problem is with performance - the server responce is too slow (about 10-15 seconds).
Please tell me how to configure this so that the server response is fast?
Does it depend only on the schema settings, or it can also depend on other things, such as database connection parameters, or query parameters?
P.S. Queries should be by 'district' and 'locality'.
Thanks for any help!
Here is the schema:
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const houseSchema = new Schema({
code: {
type: String,
required: false
},
name: {
type: String,
required: true
},
district: {
type: String,
required: true
},
locality: {
type: String,
required: false
},
recountDate: {
type: Date,
default: Date.now
},
eventDate: {
type: Date,
default: Date.now
},
events: {
type: Array,
default: []
}
});
module.exports = mongoose.model('House', houseSchema);
Connection parameters:
mongoose.connect(
`mongodb+srv://${process.env.MONGO_USER}:${process.env.MONGO_PASSWORD}#cluster0-vuauc.mongodb.net/${process.env.MONGO_DB}?retryWrites=true&w=majority`,
{
useNewUrlParser: true,
useUnifiedTopology: true
}
).then(() => {
console.log('Connection to database established...')
app.listen(5555);
}).catch(err => {
console.log(err);
});
Queries are performed using Relay:
query {
viewer {
allPosts (first: 10) {
edges {
node {
id
code
district
locality
recountDate
eventDate
events
}
}
}
}
}

MongoDB is very fast in the execution of queries. But it also depends on how you write your query. For getting the first 10 documents and sort it descending order to the _id from a collection. You need to use limit & sort in your query.
db.collectionName.find({}).limit(10).sort({_id:-1})

Make sure it's not a connection issue. Try to run your query from MongoDB shell
mongo mongodb+srv://${process.env.MONGO_USER}:${process.env.MONGO_PASSWORD}#cluster0-vuauc.mongodb.net/${process.env.MONGO_DB}?retryWrites=true&w=majority
db.collection.find({condition}).limit(10)
If in MongoDB shell it responds faster than Mongoose:
There is an issue for Node.js driver which uses pure Javascript BSON serializer which is very slow to serialize from BSON to JSON.
Try to install bson-ext
The bson-ext module is an alternative BSON parser that is written in C++. It delivers better deserialization performance and similar or somewhat better serialization performance to the pure javascript parser.
https://mongodb.github.io/node-mongodb-native/3.5/installation-guide/installation-guide/#bson-ext-module

Use Projections to Return Only Necessary Data
When you need only a subset of fields from documents, you can achieve better performance by returning only the fields you need:
For example, if in your query to the posts collection, you need only the timestamp, title, author, and abstract fields, you would issue the following command:
db.posts.find( {}, { timestamp : 1 , title : 1 , author : 1 , abstract : 1} ).sort( { timestamp : -1 } ).limit(10)
You can read for Query optimize here

Dynamoose - cannot query and where with 2 keys

I have opened a related issue on GitHub, but maybe someone here will be able to help quicker.
Summary:
ValidationException: Query key condition not supported
I need to find records in last (amount) seconds on a given location.
Pretty simple, but already related to other issues:
One and another one
WORKS:
Activity.query('locationId').eq(locationId).exec();
DOES NOT WORK:
Activity.query('locationId').eq(locationId).where('createdAt').ge(date).exec();
Code sample:
Schema
const Activity = (dynamoose: typeof dynamooseType) => dynamoose.model<ActivityType, {}>('Activity',
new Schema({
id: {
type: String,
default: () => {
return uuid();
},
hashKey: true,
},
userId: {
type: String,
},
locationId: {
type: String,
rangeKey: true,
index: {
global: true,
},
},
createdAt: { type: Number, rangeKey: true, required: true, default: Date.now },
action: {
type: Number,
},
}, {
expires: 60 * 60 * 24 * 30 * 3, // activity logs to expire after 3 months
}));
Code which executes the function
Funny part is that I found this as workaround proposed to be used until they merge PR giving ability to specify timestamps as keys, but unfortunately it does not work.
async getActivitiesInLastSeconds(locationId: string, timeoutInSeconds: number) {
const Activity = schema.Activity(this.dynamoose);
const date = moment().subtract(timeoutInSeconds, 'seconds').valueOf();
return await Activity.query('locationId').eq(locationId)
.where('createdAt').ge(date).exec();
}

I suspect createdAt is not a range key of your table / index. You need to either do .filter('createdAt').ge(date) or modify your table / index schema.

I'm pretty sure the problem is that when you specifying rangeKey: true on the createdAt property you are telling that to be used on the global index (I don't think that is the correct term). That range key will be linked to the id property.
I believe the easiest solution would be to change your locationId index to be something like the following:
index: {
global: true,
rangeKey: 'createdAt',
},
That way you are being very explicit about which index you want to set createdAt as the rangeKey for.
After making that change please remember to sync your changes with either your local DynamoDB server or the actual DynamoDB service, so that the changes get populated both in your code and on the database system.
Hopefully this helps! If it doesn't fix your problem please feel free to comment and I'll help you further.

Mongoose find/update subdocument

I have the following schemas for the document Folder:
var permissionSchema = new Schema({
role: { type: String },
create_folders: { type: Boolean },
create_contents: { type: Boolean }
});
var folderSchema = new Schema({
name: { type: string },
permissions: [ permissionSchema ]
});
So, for each Page I can have many permissions. In my CMS there's a panel where I list all the folders and their permissions. The admin can edit a single permission and save it.
I could easily save the whole Folder document with its permissions array, where only one permission was modified. But I don't want to save all the document (the real schema has much more fields) so I did this:
savePermission: function (folderId, permission, callback) {
Folder.findOne({ _id: folderId }, function (err, data) {
var perm = _.findWhere(data.permissions, { _id: permission._id });
_.extend(perm, permission);
data.markModified("permissions");
data.save(callback);
});
}
but the problem is that perm is always undefined! I tried to "statically" fetch the permission in this way:
var perm = data.permissions[0];
and it works great, so the problem is that Underscore library is not able to query the permissions array. So I guess that there's a better (and workgin) way to get the subdocument of a fetched document.
Any idea?
P.S.: I solved checking each item in the data.permission array using a "for" loop and checking data.permissions[i]._id == permission._id but I'd like a smarter solution, I know there's one!

So as you note, the default in mongoose is that when you "embed" data in an array like this you get an _id value for each array entry as part of it's own sub-document properties. You can actually use this value in order to determine the index of the item which you intend to update. The MongoDB way of doing this is the positional $ operator variable, which holds the "matched" position in the array:
Folder.findOneAndUpdate(
{ "_id": folderId, "permissions._id": permission._id },
{
"$set": {
"permissions.$": permission
}
},
function(err,doc) {
}
);
That .findOneAndUpdate() method will return the modified document or otherwise you can just use .update() as a method if you don't need the document returned. The main parts are "matching" the element of the array to update and "identifying" that match with the positional $ as mentioned earlier.
Then of course you are using the $set operator so that only the elements you specify are actually sent "over the wire" to the server. You can take this further with "dot notation" and just specify the elements you actually want to update. As in:
Folder.findOneAndUpdate(
{ "_id": folderId, "permissions._id": permission._id },
{
"$set": {
"permissions.$.role": permission.role
}
},
function(err,doc) {
}
);
So this is the flexibility that MongoDB provides, where you can be very "targeted" in how you actually update a document.
What this does do however is "bypass" any logic you might have built into your "mongoose" schema, such as "validation" or other "pre-save hooks". That is because the "optimal" way is a MongoDB "feature" and how it is designed. Mongoose itself tries to be a "convenience" wrapper over this logic. But if you are prepared to take some control yourself, then the updates can be made in the most optimal way.
So where possible to do so, keep your data "embedded" and don't use referenced models. It allows the atomic update of both "parent" and "child" items in simple updates where you don't need to worry about concurrency. Probably is one of the reasons you should have selected MongoDB in the first place.

In order to validate subdocuments when updating in Mongoose, you have to 'load' it as a Schema object, and then Mongoose will automatically trigger validation and hooks.
const userSchema = new mongoose.Schema({
// ...
addresses: [addressSchema],
});
If you have an array of subdocuments, you can fetch the desired one with the id() method provided by Mongoose. Then you can update its fields individually, or if you want to update multiple fields at once then use the set() method.
User.findById(userId)
.then((user) => {
const address = user.addresses.id(addressId); // returns a matching subdocument
address.set(req.body); // updates the address while keeping its schema
// address.zipCode = req.body.zipCode; // individual fields can be set directly
return user.save(); // saves document with subdocuments and triggers validation
})
.then((user) => {
res.send({ user });
})
.catch(e => res.status(400).send(e));
Note that you don't really need the userId to find the User document, you can get it by searching for the one that has an address subdocument that matches addressId as follows:
User.findOne({
'addresses._id': addressId,
})
// .then() ... the same as the example above
Remember that in MongoDB the subdocument is saved only when the parent document is saved.
Read more on the topic on the official documentation.

If you don't want separate collection, just embed the permissionSchema into the folderSchema.
var folderSchema = new Schema({
name: { type: string },
permissions: [ {
role: { type: String },
create_folders: { type: Boolean },
create_contents: { type: Boolean }
} ]
});
If you need separate collections, this is the best approach:
You could have a Permission model:
var mongoose = require('mongoose');
var PermissionSchema = new Schema({
role: { type: String },
create_folders: { type: Boolean },
create_contents: { type: Boolean }
});
module.exports = mongoose.model('Permission', PermissionSchema);
And a Folder model with a reference to the permission document.
You can reference another schema like this:
var mongoose = require('mongoose');
var FolderSchema = new Schema({
name: { type: string },
permissions: [ { type: mongoose.Schema.Types.ObjectId, ref: 'Permission' } ]
});
module.exports = mongoose.model('Folder', FolderSchema);
And then call Folder.findOne().populate('permissions') to ask mongoose to populate the field permissions.
Now, the following:
savePermission: function (folderId, permission, callback) {
Folder.findOne({ _id: folderId }).populate('permissions').exec(function (err, data) {
var perm = _.findWhere(data.permissions, { _id: permission._id });
_.extend(perm, permission);
data.markModified("permissions");
data.save(callback);
});
}
The perm field will not be undefined (if the permission._id is actually in the permissions array), since it's been populated by Mongoose.

just try
let doc = await Folder.findOneAndUpdate(
{ "_id": folderId, "permissions._id": permission._id },
{ "permissions.$": permission},
);

Update embedded document mongoose

I'm looking for an easy way of updating an embedded document using mongoose without having to set each specific field manually. Looking at the accepted answer to this question, once you find the embedded document that you want to update you have to actually set each respective property and then save the parent. What I would prefer to do is pass in an update object and let MongoDB set the updates.
e.g. if I was updating a regular (non embedded) document I would do this:
models.User.findOneAndUpdate({_id: req.params.userId}, req.body.user, function(err, user) {
err ? resp.status(500).send(err) : user ? resp.send(user) : resp.status(404).send();
});
Here I don't actually have to go through each property in req.body.user and set the changes. I can't find a way of doing this kind of thing with sub documents as well ?
My Schema is as follows:
var UserSchema = BaseUserSchema.extend({
isActivated: { type: Boolean, required: true },
files: [FileSchema]
});
var FileSchema = new mongoose.Schema(
name: { type: String, required: true },
size: { type: Number, required: true },
type: { type: String, required: true },
});
And I'm trying to update a file based on user and file id.
Do I need to create a helper function to set the values, or is there a MongoDB way of doing this ?
Many thanks.

Well presuming that you have something that has you "filedata" in a variable, and of course the user _id that you are updating, then you wan't the $set operator:
var user = { /* The user information, at least the _id */
var filedata = { /* From somewhere with _id, name, size, type */ };
models.User.findOneAndUpdate(
{ "_id": user._id, "files._id": filedata._id },
{
"$set": {
"name": filedata.name,
"size": filedata.size,
"type": filedata.type
}
},
function(err,user) {
// Whatever in here such a message, but the update is already done.
}
);
Or really, just only $set the fields that you actually mean to "update" as long as you know which ones you mean. So if you only need to change the "size" then just set that for example.

Mongoose embedded document updating

I have a problem with embedded document update.
My defined Schemas:
var Talk = new Schema({
title: {
type: String,
required: true
},
content: {
type: String,
required: true
},
date: {
type: Date,
required: true
},
comments: {
type: [Comments],
required: false
},
vote: {
type: [VoteOptions],
required: false
},
});
var VoteOptions = new Schema({
option: {
type: String,
required: true
},
count: {
type: Number,
required: false
}
});
Now I would like to update vote.count++, with given Talk id and VoteOption id. I have the following function to do the job:
function makeVote(req, res) {
Talk.findOne(req.params.id, function(err, talk) {
for (var i = 0; i < talk.vote.length; i++) {
if (talk.vote[i]._id == req.body.vote) {
talk.vote[i].count++;
}
}
talk.save(function(err) {
if (err) {
req.flash('error', 'Error: ' + err);
res.send('false');
} else {
res.send('true');
}
});
});
}
Everything executes, I get back the res.send('true'), but the value on count does not change.
When I did some console.log I saw that it changed the value, but the talk.save just doesn't save it in db.
Also I'm quite unhappy about the cycle just to find _id of embedded doc. In the mongoose documentation I read about talk.vote.id(my_id) but that gives me error of not having an id function.

When updating a Mixed type (which seems to be anything else than a basic type, so that also includes embedded documents), one has to call .markModified on the document. In this case, it would be:
talk.markModified("vote"); // mention that `talk.vote` has been modified
talk.save(function(err) {
// ...
});
Hope this helps someone in the future since I couldn't find the answer very quickly.
Reference:
... Mongoose loses the ability to auto detect/save those changes. To "tell" Mongoose that the value of a Mixed type has changed, call the .markModified(path) method of the document passing the path to the Mixed type you just changed.

It's because you are trying to save your talk object before the callback which increments count has been fired. Also, did you make sure to instantiate your Talk schema? eg:
var talk = new Talk();
However, if all you want to do is increment your count variable, mongo supports atomic, in-place updates which you may find useful:
talk.find( { _id : req.body.vote }, { $inc: { count : 1 } } );
have a look at:
http://www.mongodb.org/display/DOCS/Updating#Updating-%24inc