Develop Reference

How to find key with type objectId , ref to <some model>

I have created a Notes model having schema as shown below
const notesschema = new Schema({
user :{
type : Schema.Types.ObjectId,
ref : 'User',
required : true
},
problem : {
type : Schema.Types.ObjectId,
ref : 'Problems',
required : true
},
content : {
type : 'string'
}
},{
timestamps : true
})
To show the User his notes for a particular task/problem I am trying to fetch notes and show to him and possibly update if he do some changes and save, The problem is with this schema I dont know how to write <model.findById >API to find notes from my notes model having particular user and specific task/problem.Which I would know the Id of.
With this particular schema , and my current knowledge i would have to write So much code. So if there is any easier way to do this task is welcomed, I was also thinking to change my schema and just placing my user id in my schema instead of whole user and finding notes from my database
edit : as suggested by all the answers we can simply find using user.id which I thought initially would not word as that was just the path, but which stores actually user.id

You create the notes' collection the same way you're doing it,
const notesSchema = new mongoose.Schema({
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User', // # the name of the user model
required: true
},
problem: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Problem', // # the name of the user model
required: true
},
content: String
})
Then you'll create the model of the notesSchema as follows:
const NoteModel = mongoose.model('Note', notesSchema, 'notes')
export them so you can use them in your controllers:
module.exports = {
NoteModel,
notesSchema
}
or if you're using +es6 modules (think of, if you're using TypeScript):
export default {
NoteModel,
notesSchema
}
This will result in creating the following table (collection) in the database:
Let's think of the following challenges:
To get all the notes:
NoteModel.find({})
To get all the users:
UserModel.find({}) // you should have something like this in your code of course
To get all the problems:
ProblemModel.find({}) // you should have something like this in your code of course
To get all the notes of a user:
NotesModel.find({ user: USER_ID })
To search for notes by problems:
NotesModel.find({ problem: PROBLEM_ID })
Now, the above is how you do it in mongoose, now let's create a RESTFUL controller for all of that: (assuming you're using express)
const expressAsyncHandler = require('express-async-handler') // download this from npm if you want
app.route('/notes').get(expressAsyncHandler(async (req, res, next) => {
const data = await NotesModel.find(req.query)
res.status(200).json({
status: 'success',
data,
})
}))
The req.query is what's going to include the search filters, the search filters will be sent by the client (the front-end) as follows:
http://YOUR_HOST:YOUR_PORT/notes?user=TheUserId
http://YOUR_HOST:YOUR_PORT/notes?problem=TheProblemId
http://YOUR_HOST:YOUR_PORT/notes?content=SomeNotes

const notesschemaOfUser = await notesschema.findOne({user: user_id});

MondoDB/Mongoose query responce is too slow

I new to MongoDB/Mongoose, and work with a very large database (more than 25000 docs). I need to configure different queries: by fields, first 10 docs, one by id. The problem is with performance - the server responce is too slow (about 10-15 seconds).
Please tell me how to configure this so that the server response is fast?
Does it depend only on the schema settings, or it can also depend on other things, such as database connection parameters, or query parameters?
P.S. Queries should be by 'district' and 'locality'.
Thanks for any help!
Here is the schema:
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const houseSchema = new Schema({
code: {
type: String,
required: false
},
name: {
type: String,
required: true
},
district: {
type: String,
required: true
},
locality: {
type: String,
required: false
},
recountDate: {
type: Date,
default: Date.now
},
eventDate: {
type: Date,
default: Date.now
},
events: {
type: Array,
default: []
}
});
module.exports = mongoose.model('House', houseSchema);
Connection parameters:
mongoose.connect(
`mongodb+srv://${process.env.MONGO_USER}:${process.env.MONGO_PASSWORD}#cluster0-vuauc.mongodb.net/${process.env.MONGO_DB}?retryWrites=true&w=majority`,
{
useNewUrlParser: true,
useUnifiedTopology: true
}
).then(() => {
console.log('Connection to database established...')
app.listen(5555);
}).catch(err => {
console.log(err);
});
Queries are performed using Relay:
query {
viewer {
allPosts (first: 10) {
edges {
node {
id
code
district
locality
recountDate
eventDate
events
}
}
}
}
}

MongoDB is very fast in the execution of queries. But it also depends on how you write your query. For getting the first 10 documents and sort it descending order to the _id from a collection. You need to use limit & sort in your query.
db.collectionName.find({}).limit(10).sort({_id:-1})

Make sure it's not a connection issue. Try to run your query from MongoDB shell
mongo mongodb+srv://${process.env.MONGO_USER}:${process.env.MONGO_PASSWORD}#cluster0-vuauc.mongodb.net/${process.env.MONGO_DB}?retryWrites=true&w=majority
db.collection.find({condition}).limit(10)
If in MongoDB shell it responds faster than Mongoose:
There is an issue for Node.js driver which uses pure Javascript BSON serializer which is very slow to serialize from BSON to JSON.
Try to install bson-ext
The bson-ext module is an alternative BSON parser that is written in C++. It delivers better deserialization performance and similar or somewhat better serialization performance to the pure javascript parser.
https://mongodb.github.io/node-mongodb-native/3.5/installation-guide/installation-guide/#bson-ext-module

Use Projections to Return Only Necessary Data
When you need only a subset of fields from documents, you can achieve better performance by returning only the fields you need:
For example, if in your query to the posts collection, you need only the timestamp, title, author, and abstract fields, you would issue the following command:
db.posts.find( {}, { timestamp : 1 , title : 1 , author : 1 , abstract : 1} ).sort( { timestamp : -1 } ).limit(10)
You can read for Query optimize here

Testing validation in Mongoose without hitting the database

If I have in my application the following model:
// employee.js
const mongoose = require('mongoose');
const validators = require('./validators');
const EmployeeSchema = new mongoose.Schema({
firstName: { type: String, required: true },
lastName: { type: String, required: true },
referredBy: {
type: Schema.Types.ObjectId,
ref: 'Employee',
validate: {
isAsync: true,
validator: validators.referralValidator
}
}
});
module.exports = mongoose.model('Employee', EmployeeSchema);
Basically, it's a model for an employee in a firm. That employee could have been referred to work at the firm by a different, existing, employee, so there is a self-referencing in the Employees collection.
The validation makes sure that the user doesn't enter an id of an employee who doesn't exist in the database already. It looks like this:
// validators.js
const mongoose = require('mongoose');
function referralValidator(val, callback) {
if (val) {
const Employee = mongoose.model('Employee');
Employee.findById(val.toString(), (err, found) => {
callback(found !== null);
});
} else {
callback(true);
}
}
module.exports = {
referralValidator
};
Now, I would like to test that this validation work (could be one of many other validations that I'll write in the future for other fields that will be added to the model). However, I would like not to hit the database in the test, so I want to stab out the findById to control manually what the value of "found" will be for the test. I couldn't figure out how to go around the mongo driver that mongoose employs behind the scenes.
How could I stab this function? Or, alternatively, is there a better way to implement the validator in order to make it more testable?

This link should be a good starting point on how to test model validations. If you want to mock database, then check out mockgoose

Why are my Mongoose One-To-Many Relationships not associating properly?

Does anyone know why the following one-to-many relationship between "users" and "posts" (users can have many posts) is not working? It appears I have setup my mongoose associations correctly, but when a new post is created, not only is it not assigned a user, but the users themselves are also not associated with any posts. I'm not sure what I might be doing wrong here.
If you see the JSON object below, it should have a user value, denoting the user whom created the post. You'll see in the Post Model below, that a user value should be created, but does not.
What am I doing wrong?
Here's the JSON object after creating a new post
{
__v: 0
_id: "587ee8f5a99b1709b012ce8f"
createdAt: "2017-01-18T04:03:01.446Z"
message: "This is my first test post!"
updatedAt: "2017-01-18T04:03:01.446Z"
}
Question: Why is the user field missing from the JSON above despite being created in the Post Model below?
Here's my Post Model:
// Setup dependencies:
var mongoose = require('mongoose');
// Setup a schema:
var PostSchema = new mongoose.Schema (
{
message: {
type: String,
minlength: 2,
maxlength: 2000,
required: true,
trim: true,
}, // end message field
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
},
},
{
timestamps: true,
}
);
// Instantiate our model and export it:
module.exports = mongoose.model('Post', PostSchema)
Here's my User Model:
// Setup dependencies:
var mongoose = require('mongoose');
// Setup a schema:
var UserSchema = new mongoose.Schema (
{
username: {
type: String,
minlength: 2,
maxlength: 20,
required: true,
trim: true,
unique: true, // username must be unique
dropDups: true,
lowercase: true,
validate: {
validator: function(username) {
var regex = /^[a-z0-9_]+$/i;
return regex.test(username);
},
message: 'Username may contain only letters, numbers or underscores.',
},
}, // end username field
posts: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Post'
}],
},
{
timestamps: true,
});
// Instantiate our model and export it:
module.exports = mongoose.model('User', UserSchema)
Here's the Controller that queries the DB:
Note: This is the method that runs when the post form is submitted.
// Grab our Mongoose Models:
var User = require('mongoose').model('User');
var Post = require('mongoose').model('Post');
module.exports = {
// Creates a new post for logged in user:
newPost: function(req, res) {
Post.create(req.body)
.then(function(newPost) {
return res.json(newPost);
})
.catch(function(err) {
return res.json(err);
})
}
};
Does anyone know if my associations are improperly setup and this is why I'm not getting any actual posts or users to show up in their respective fields?
It seems that my server-side controller is firing properly, as the post is actually created. But the associations themselves are not linking up and I'm not sure what I'm doing wrong.

I'm adding just a simple answer below to follow up with the example above. Essentially, #cdbajorin was correct, I was absently thinking there was some automation going on and was not appropriately following through the proper mongoose commands to achieve my desired results.
The solution to my question is as follows:
In the User Model, update the UserSchema posts attribute to be an empty array, instead of a mongoose.Schema.Types.ObjectID, since an object ID is not stored here anyhow and I misunderstood how this works.
The code:
posts: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Post'
}],
Instead, should be written simply:
posts: [],
The newPost method, in the server Controller, should be modified as follows (see comments inline for clarification):
newPost: function(req, res) {
// creates new post:
Post.create(req.body)
.then(function(newPost) {
// look up current user based on session ID:
// note: session setup not shown in this example.
User.findById(req.session.userID)
.then(function(user) {
// push new post into users.posts array from model setup**:
user.posts.push(newPost);
user.save();
return res.json(newPost);
})
})
.catch(function(err) {
return res.json(err);
})
This does solve the issue of the new post being generated, and then pushed into a user's posts array (from the UsersSchema).
Though the issue from the initial post is solved, one may question if this is the best use of database management. Storing posts inside of a user, as this example does, can take up a lot of space as users and posts start to add up.
This post ends up being duplicated in the database twice: first, as a document itself in the posts collection, and secondly, as an object in the posts array within the UserSchema.
A better solution is to keep the post as a unique document in the posts collection, but add the userID from the session information to it. Then, if all of user's posts are needed for any reason, a query to the Posts collection, based on the userID, would return all posts with that userID assigned to it. Then, only one copy of the post exists in the DB instead of two.
** Additional Note: Another way to modify the existing document would be to use an instance method, where an actual method would be inserted into the User Model (Schema) file, and called when needed:
For example, inserting the following code before the module.exports line in the UserSchema Model above, allows for convenient access this function when needed:
UserSchema.methods.addPost = function(post) {
this.posts.push(post);
this.save();
return true;
};
To call this instance method from our server Controller, we could re-write our Controller as follows:
User.findById(req.session.userID)
.then(function(user) {
// call our instance method above:
user.addPost(newPost);
return res.json(newPost);
});
The post will be pushed and saved by the instance method, which has been built into the instance object itself.

MongoDB query on populated fields

I have models called "Activities" that I am querying for (using Mongoose). Their schema looks like this:
var activitySchema = new mongoose.Schema({
actor: {
type: mongoose.Schema.ObjectId,
ref: 'User',
required: true
},
recipient: {
type: mongoose.Schema.ObjectId,
ref: 'User'
},
timestamp: {
type: Date,
default: Date.now
},
activity: {
type: String,
required: true
},
event: {
type: mongoose.Schema.ObjectId,
ref: 'Event'
},
comment: {
type: mongoose.Schema.ObjectId,
ref: 'Comment'
}
});
When I query for them, I am populating the actor, recipient, event, and comment fields (all the references). After that, I also deep-populate the event field to get event.creator. Here is my code for the query:
var activityPopulateObj = [
{ path: 'event' },
{ path: 'event.creator' },
{ path: 'comment' },
{ path: 'actor' },
{ path: 'recipient' },
{ path: 'event.creator' }
],
eventPopulateObj = {
path: 'event.creator',
model: User
};
Activity.find({ $or: [{recipient: user._id}, {actor: {$in: user.subscriptions}}, {event: {$in: user.attending}}], actor: { $ne: user._id} })
.sort({ _id: -1 })
.populate(activityPopulateObj)
.exec(function(err, retrievedActivities) {
if(err || !retrievedActivities) {
deferred.reject(new Error("No events found."));
}
else {
User.populate(retrievedActivities, eventPopulateObj, function(err, data){
if(err) {
deferred.reject(err.message);
}
else {
deferred.resolve(retrievedActivities);
}
});
}
});
This is already a relatively complex query, but I need to do even more. If it hits the part of the $or statement that says {actor: {$in: user.subscriptions}}, I also need to make sure that the event's privacy field is equal to the string public. I tried using $elemMatch, but since the event has to be populated first, I couldn't query any of its fields. I need to achieve this same goal in multiple other queries, as well.
Is there any way for me to achieve this further filtering like I have described?

The answer is to change your schema.
You've fallen into the trap that many devs have before you when coming into document database development from a history of using relational databases: MongoDB is not a relational database and should not be treated like one.
You need to stop thinking about foreign keys and perfectly normalized data and instead, keep each document as self-contained as possible, thinking about how to best embed relevant associated data within your documents.
This doesn't mean you can't maintain associations as well. It might mean a structure like this, where you embed only necessary details, and query for the full record when needed:
var activitySchema = new mongoose.Schema({
event: {
_id: { type: ObjectId, ref: "Event" },
name: String,
private: String
},
// ... other fields
});
Rethinking your embed strategy will greatly simplify your queries and keep the query count to a minimum. populate will blow your count up quickly, and as your dataset grows this will very likely become a problem.

You can try below aggregation. Look at this answer: https://stackoverflow.com/a/49329687/12729769
And then, you can use fields from $addFields in your query. Like
{score: {$gte: 5}}

but since the event has to be populated first, I couldn't query any of its fields.
No can do. Mongodb cannot do joins. When you make a query, you can work with exactly one collection at a time. And FYI all those mongoose populates are additional, distinct database queries to load those records.
I don't have time to dive into the details of your schema and application, but most likely you will need to denormalize your data and store a copy of whatever event fields you need to join on in the primary collection.