I'm new to MongoDB, and I'm trying to do a very simple task, but however I can't get it right.
What I want is to change the process status but I tried "FindAndUpdate", "UpdateOne" and "FindByIdAndUpdate" but it won't work.
Maybe it has to do with my Schema. Should I create a new Schema for the Process?
My Database entry inside a MongoDB Collection:
_id: 622c98cfc872bcb2578b97a5
username:"foo"
__v:0
process:Array
0: Object
processname:"bar"
process_status:"stopped"
_id: 6230c1a401c66fc025d3cb88
My current Schema:
const User = new mongoose.Schema(
{
username: { type: String, required: true },
process: [
{
processname: {
type: String,
},
process_status: {
type: String,
},
},
],
},
{ collection: "user-data" }
);
My current code:
const startstopprocess = await User.findByIdAndUpdate(
{ _id: "6230c1a401c66fc025d3cb88" },
{ process_status: "started" }
).then(function (error, result) {
console.log(error);
console.log(result);
});
You can use positional operator $ in this way:
db.collection.update({
"process._id": "6230c1a401c66fc025d3cb88"
},
{
"$set": {
"process.$.process_status": "started"
}
})
Note how using positional operator you can say mongo "from the object you have found in find stage, update the process_status variable to started"
Example here
I am building a social media and one of the features I have is the ability to follow other users and have their posts show up on a home page, much like an Instagram feed. I have an endpoint that loops through all the users you are following and gets the post that are rated for your age, sorts them and puts them in an array.
router.get("/following", auth, async (req, res) => {
try {
const user = await User.findOne({ firebaseUID: req.authId });
const dob = dayjs(user.dob.toISOString().split("T")[0]);
const currentDate = dayjs(new Date().toISOString().split("T")[0]);
const age1 = currentDate.diff(dob, "year");
let posts = [];
for (let i = 0; i < user.following.length; i++) {
posts.push(
await Post.find({
userID: user.following[i].user,
age: { $lte: age1 },
})
);
}
posts = [].concat.apply([], posts);
posts = _.sortBy(posts, "date").reverse();
res.json(posts);
} catch (err) {
console.error(err.message);
res.status(500).json({ msg: "Server Error" });
}
});
The user schema looks like this
const UserSchema = new Schema({
firebaseUID: {
type: String,
required: true,
},
name: {
type: String,
required: true,
},
username: {
type: String,
required: true,
unique: true,
},
avatar: {
type: String,
},
dob: {
type: Date,
default: Date.now,
required: true,
},
slug: {
type: String,
},
following: [
//array of user IDs
{
user: {
type: Schema.Types.ObjectId,
ref: "users", //here so we know which lieks came from which users
},
},
],
followers: [
//array of user IDs
{
user: {
type: Schema.Types.ObjectId,
ref: "user", //here so we know which lieks came from which users
},
},
],
});
The problem is that I am trying to add infinite scroll pagination to the front end that will start by getting the first 15 posts and then get the next 15 and so on from endpoints that provide this.
What would the best way to do this be? For other routes, I have used .skip() and .limit(). I could split the array of posts up right before I send it as a response, but is there a way to do without getting all the posts as this would be taxing if you're following 100s of people with 1000s of posts.
I would need to loop through all the users you are following and get all their posts in line with your age rating but only get the first 15 sorted in date order from newest to oldest. Is there a mongoose function that I could use or would the best way be to split the array of posts?
Thank you.
I am new to working on a MongoDB and Docker, I am working on an application and couldn't find a more subtle way to seed my database using an npm run command. First I created a file called seed.js and then associated it to npm run seed command on the package.json file.
On the seed.js file I import Mongoose and the models but two things I will need to do is:
Create roles, if they don’t exist yet
Create capabilities, if they don’t exist yet and associate it to the
roles
The Roles that i want to create are:
admin (description: Administrator)
viewer (description: Viewer)
Capabilities
I need to check each endpoint of the Users service that should require authentication and create an adequate capability. Example: updateUser updates the user data. This could be done by the own user (so there must be an updateUserOwn capability) and by an administrator (that will have an updateUsers capability). I will have to analyse each endpoint and judge what is adequate but I cannot still find a way around getting the initial role and capabilities to the database.
UPDATE:
On the seeding itself, the updated solution works, but it requires lot of code and repetition that could probably be fixed by loops. I’d like to start creating the roles first which means creating an array with objects, with the data from the roles to be created. Each role has the fields role and description
const userRole = [{
role: admin
description: Administrator
},
{
role: viewer
description: Viewer
}]
The idea is that if the role exist it doesn't need to update but I don't know how do I loop through the array and create a role only if it doesn’t exist. Something like using updateOne, with the upsert: true option, but with the data on $setOnInsert as this will add the data only if a document is inserted.
I only need create and not update because in the future I’ll edit roles directly through the API. So, if a change was made on the admin role, for example, the seed will not overwrite it
During the loop, I'll need to create an associative array called rolesIds that will store the ObjectId of the created roles. It should result in something like this:
[
"admin": "iaufh984whrfj203jref",
"viewer": "r9i23jfeow9iefd0ew0",
]
Also each capability must have an array of roles it must be associated to. Example:
{
capability: "updateUsers",
description: "Update the data of all users",
roles: ["admin"]
}
How do I loop through the array on each element, prepare it to be inserted using the array with object IDs. Instead of roles: ["admin"]? something like roles: ["iaufh984whrfj203jref"], otherwise there’ll be a cast error. Remember each capability may be associated to more than one role, so I'll probably need to loop through them but I cannot find a way to create that logic.
Users Model
const userSchema = new mongoose.Schema(
{
.......
role: {
ref: "roles",
type: mongoose.Schema.Types.ObjectId,
},
);
module.exports = mongoose.model("User", userSchema);
Role Model:
const roles = new mongoose.Schema({
role: {
type: String,
required: true,
},
capabilities: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "capabilities",
},
],
});
module.exports = mongoose.model("roles", roles);
Capabilities Model:
const capabilities = new mongoose.Schema({
capability: {
type: String,
required: true,
},
name: {
type: String,
},
});
module.exports = mongoose.model("capabilities", capabilities);
UPDATED: seed file:
const seedDB = async () => {
if (!process.env.DB_URI) {
throw new Error("Error connecting to MongoDB: DB_URI is not defined.");
}
try {
await mongoose.connect(process.env.DB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
});
console.log("Connected to MongoDB");
const tasks = [
Capability.findOneAndUpdate(
{ name: "updateUserOwn" },
{ capability: "updateUser" },
{ upsert: true }
).exec(),
Capability.findOneAndUpdate(
{ name: "updateUsers" },
{ capability: "updateUser" },
{ upsert: true }
).exec(),
// Seed more...
];
const [updateUserOwn, updateUsers] = await Promise.all(tasks);
Role.bulkWrite([
{
updateOne: {
filter: { role: "Admin" },
update: { capabilities: [updateUsers] },
upsert: true,
},
},
{
updateOne: {
filter: { role: "Viewer" },
update: { capabilities: [updateUserOwn] },
upsert: true,
},
},
]);
console.log("seeded data", tasks);
} catch (error) {
console.log(`Error connecting to MongoDB: ${error}`);
}
};
seedDB();
You are on the right path overall.
Because capabilities are used as a reference you'd have to fetch or create them (get a ref) before assigning them to a role.
This could be your seed logic:
const tasks = [
Capability.findOneAndUpdate(
{ name: 'updateUserOwn' }, // matches or creates this capability
{ capability: 'updateUser' }, // adds this to the object
{ upsert: true, new: true } // `new` guarantees an object is always returned
}).exec(),
Capability.findOneAndUpdate(
{ name: 'updateUsers' },
{ capability: 'updateUser' },
{ upsert: true, new: true }
}).exec(),
// Seed more...
];
const [
updateUserOwn,
updateUsers,
] = await Promise.all(tasks);
// We can use bulk write for the second transaction so it runs in one go
await Role.bulkWrite([
{
updateOne: {
filter: { role: 'Admin' },
update: { capabilities: [updateUsers] },
upsert: true,
}
},
{
updateOne: {
filter: { role: 'Viewer' },
update: { capabilities: [updateUserOwn] },
upsert: true,
}
}
]);
We seed capabilities one by one using findOneAndUpdate so we can get a reference to each capability we intend to use on the roles
Then we use bulkWrite to seed the roles
I might have swapped the capabilities and their names but I hope you get the general idea
The seed would have been simpler if there weren't references involved - you could just use bulkWrite everything in one go, but in order to create object with inner references or add references to such object you first need to have the actual reference
You can create static mapping and loop through which would reduce the code a bit, and make things easier. This would also allow you to skip seeding items that already exist
Since capabilities are reused through roles I want to create them first, but it's no problem to alter the logic to first create roles and then capabilities, though it might not be as straight forward
Also each capability must have an array of roles it must be associated to.
This is called a "many to many" relationship (as roles also have an array of references to capabilities) which would only complicate logic. Are you sure you really need it - mongoose/monogo won't manage it automatically for you:
when you add a capability to a role you'd also need to sync and add the role inside capability.roles - manually
and the reverse - adding a role inside capability.roles you'd need to sync this and also manually add the capability to role.capabilities
the same thing for deleting capabilities or roles - manual cleanup
it can fail and would need to recover - e.g. a capability is added to role.capabilities but for some reason execution stopped and the role was not added to capability.roles - so the whole handling might need to be wrapped in a transaction
there are ways to cross reference roles and capabilities without have to have a "many to many" relationship
Here's a simple approach using save middleware to sync many to many relationships for create/update
Role.js
const mongoose = require('mongoose');
const roles = new mongoose.Schema({
role: {
type: String,
required: true,
},
description: String,
capabilities: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'capabilities',
},
],
});
roles.pre('save', async function save() {
// Doesn't need to run if there are no capabilities
if (!this.capabilities || this.capabilities.length === 0) return;
const Capability = mongoose.model('capabilities');
await Capability.updateMany(
{ _id: {$in: this.capabilities} },
// Adds only if it's missing
{ $addToSet: { roles: this._id }},
);
});
// Todo: similar logic to remove from capabilities if role is deleted
module.exports = mongoose.model("roles", roles);
Capability.js
const mongoose = require('mongoose');
const capabilities = new mongoose.Schema({
capability: {
type: String,
required: true,
},
description: {
type: String,
},
roles: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'roles',
}
]
});
capabilities.pre('save', async function save() {
if (!this.roles || this.roles.length === 0) return;
const Role = mongoose.model('roles');
await Role.updateMany(
{_id: {$in: this.roles}},
{$addToSet: {capabilities: this._id}},
);
})
// Todo: similar logic to remove from roles if capability is deleted
module.exports = mongoose.model("capabilities", capabilities);
Here's an update seed routine:
Seed.js
const mongoose = require('mongoose');
const Capability = require('./models/Capability');
const Role = require('./models/Role');
const CAPABILITIES = {
UPDATE_USERS: {
capability: 'updateUsers',
description: 'Update the data of all users',
},
VIEW_USERS: {
capability: 'viewUsers',
description: 'View public data of users',
},
UPDATE_OWN_RECORD: {
capability: 'updateUserOwn',
description: 'Update user own data',
}
}
const ROLES_TO_SEED = [
{
role: 'admin',
description: 'Administrator',
capabilities: [CAPABILITIES.UPDATE_USERS, CAPABILITIES.VIEW_USERS],
},
{
role: 'viewer',
description: 'Viewer',
capabilities: [CAPABILITIES.VIEW_USERS, CAPABILITIES.UPDATE_OWN_RECORD],
}
]
const seedDB = async () => {
await connectToDb();
await seedRoles();
};
const connectToDb = async () => {
if (!process.env.DB_URI) throw new Error('DB_URI is not defined.');
console.info('Connecting to database...');
await mongoose.connect(process.env.DB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
useFindAndModify: false,
});
console.info('Connected \n');
}
const seedRoles = async () => {
console.log('Seeding Roles...');
// runs sequentially to skip creating duplicate capabilities
for (const role of ROLES_TO_SEED) {
await findOrCreateRole(role);
}
console.log('Complete \n');
}
const findOrCreateRole = async ({capabilities, role, ...defaults}) => {
console.info('Looking for role: ', role);
const fromDb = await Role.findOne({role}).exec();
if (fromDb) {
console.info('Role already exists skipping... \n');
return fromDb;
}
console.info('Role does not exist - creating new \n');
const doc = new Role({role, ...defaults});
// All capabilities (per role) can be created/found in parallel
const roleCapabilities = await Promise.all(capabilities.map(findOrCreateCapability));
doc.capabilities = roleCapabilities.map(c => c._id);
await doc.save();
console.info('Role created: ', role);
console.info('');
return doc;
}
const findOrCreateCapability = async ({capability, ...defaults}) => {
console.info('Looking for capability: ', capability);
let doc = await Capability.findOne({capability}).exec();
if (doc) {
console.info(`Capability ${capability} found - using existing...`);
}
else {
console.info(`Capability ${capability} does not exist - creating new`);
doc = new Capability({capability, ...defaults});
await doc.save();
}
return doc;
}
seedDB()
.then(() => {
console.info('Exiting...: ');
process.exit(0);
})
.catch(error => {
console.error('Seed failed');
console.error(error);
process.exit(1);
})
We have a dictionary of capabilities and a list of roles that we can map to db operations.
The idea is that each role should contain the full definition of a capability, it can be used to either find the existing capability or create it if it doesn't exist
For each role in the list we make a query to see if it exists.
When it exists we do nothing and move to the next role
When it doesn't exist we have all the data needed to create it and create/find any capabilities that it might need
When you figure out all the roles and capabilities of the application you just put them in the ROLES_TO_SEED and CAPABILITIES static mappings
The script relies on the above mentioned middleware modifications in models
And a small bonus
You don't need many to many relationship to match capabilities to the roles they are used in. Here's how you can aggregate that information if only the Role model have an array of capabilities (refs). Run this after the database is seeded:
const showCapabilitiesUsages = async () => {
const result = await Capability.aggregate([
{
$lookup: {
from: 'roles',
let: {searched: '$_id'},
pipeline: [
{
$match: {
$expr: {
$in: ['$$searched', '$capabilities']
}
}
}
],
as: 'roles'
}
}, {
$project: {
_id: 0,
capability: 1,
description: 1,
usedInRoles: {
$map: {
input: '$roles',
as: 'role',
in: '$$role.role',
}
}
}
}
]).exec();
console.log('Aggregate result: ', result);
}
You should get a result like:
Aggregate result: [
{
capability: 'updateUsers',
description: 'Update the data of all users',
usedInRoles: [ 'admin' ]
},
{
capability: 'viewUsers',
description: 'View public data of users',
usedInRoles: [ 'admin', 'viewer' ]
},
{
capability: 'updateUserOwn',
description: 'Update user own data',
usedInRoles: [ 'viewer' ]
}
]
Try something like this, it should would work:
const roles = [
{
name: 'admin',
description: 'Administrator',
},
{
name: 'viewer',
description: 'Viewer',
},
];
const capabilities = [
// Capabilities
{
name: 'createCapability',
description: 'Create a new capability',
roles: ['admin'],
},
{
name: 'deleteCapability',
description: 'Delete a capability',
roles: ['admin'],
}
// Roles
{
name: 'createRole',
description: 'Create a new role',
roles: ['admin'],
},
{
name: 'deleteRole',
description: 'Delete a role',
roles: ['admin'],
},
// Users
{
name: 'updateUser',
description: 'Update current user data',
roles: ['viewer'],
},
{
name: 'updateUsers',
description: 'Update the data from any user',
roles: ['admin'],
},
];
const seedRoles = async (roles) => {
if (0 == roles.length || !Array.isArray(roles)) {
return;
}
console.log('');
for (const role of roles) {
const savedRole = await Role.findOneAndUpdate(
{name: role.name},
{$setOnInsert: role},
{upsert: true, new: true, rawResult: true},
);
if (!savedRole) {
console.log(`Role “${savedRole.value.name}” already on database.`);
} else {
console.log(`Role “${savedRole.value.name}” added to database.`);
}
}
};
const seedCapabilities = async (capabilities) => {
if (0 == capabilities.length || !Array.isArray(capabilities)) {
return;
}
console.log('');
for (const capability of capabilities) {
const rolesToPush = capability.roles;
delete capability.roles;
const addedCapability = await Capability.findOneAndUpdate(
{name: capability.name},
{$setOnInsert: capability},
{upsert: true, new: true, rawResult: true},
);
if (!addedCapability) {
console.log(
`Capability “${addedCapability.value.name}” ` +
`already on database.`,
);
} else {
console.log(
`Capability “${addedCapability.value.name}” ` +
`added to database.`,
);
if (rolesToPush && Array.isArray(rolesToPush)) {
rolesToPush.forEach(async (role) => {
const roleToPush = await Role.findOne({name: role});
if (roleToPush) {
roleToPush.capabilities.push(addedCapability.value);
await roleToPush.save();
}
});
}
}
}
};
const seedDb = async (roles, capabilities, users) => {
try {
await seedRoles(roles);
await seedCapabilities(capabilities);
console.log('roles', roles);
} catch (error) {
console.error(error);
}
};
module.exports = seedDb;
I'm new to Aggregation in MongoDB and I'm trying to understand the concepts of it by making examples.
I'm trying to paginate my subdocuments using aggregation but the returned document is always the overall values of all document's specific field.
I want to paginate my following field which contains an array of Object IDs.
I have this User Schema:
const UserSchema = new mongoose.Schema({
username: {
type: String,
unique: true,
required: true
},
firstname: String,
lastname: String,
following: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
}],
...
}, { timestamps: true, toJSON: { virtuals: true }, toObject: { getters: true, virtuals: true } });
Without aggregation, I am able to paginate following,
I have this route which gets the user's post by their username
router.get(
'/v1/:username/following',
isAuthenticated,
async (req, res, next) => {
try {
const { username } = req.params;
const { offset: off } = req.query;
let offset = 0;
if (typeof off !== undefined && !isNaN(off)) offset = parseInt(off);
const limit = 2;
const skip = offset * limit;
const user = await User
.findOne({ username })
.populate({
path: 'following',
select: 'profilePicture username fullname',
options: {
skip,
limit,
}
})
res.status(200).send(user.following);
} catch (e) {
console.log(e);
res.status(500).send(e)
}
}
);
And my pagination version using aggregate:
const following = await User.aggregate([
{
$match: { username }
},
{
$lookup: {
'from': User.collection.name,
'let': { 'following': '$following' },
'pipeline': [
{
$project: {
'fullname': 1,
'username': 1,
'profilePicture': 1
}
}
],
'as': 'following'
},
}, {
$project: {
'_id': 0,
'following': {
$slice: ['$following', skip, limit]
}
}
}
]);
Suppose I have this documents:
[
{
_id: '5fdgffdgfdgdsfsdfsf',
username: 'gagi',
following: []
},
{
_id: '5fgjhkljvlkdsjfsldkf',
username: 'kuku',
following: []
},
{
_id: '76jghkdfhasjhfsdkf',
username: 'john',
following: ['5fdgffdgfdgdsfsdfsf', '5fgjhkljvlkdsjfsldkf']
},
]
And when I test my route for user john: /john/following, everything is fine but when I test for different user which doesn't have any following: /gagi/following, the returned result is the same as john's following which aggregate doesn't seem to match user by username.
/john/following | following: 2
/kuku/following | following: 0
Aggregate result:
[
{
_id: '5fdgffdgfdgdsfsdfsf',
username: 'kuku',
...
},
{
_id: '5fgjhkljvlkdsjfsldkf',
username: 'gagi',
...
}
]
I expect /kuku/following to return an empty array [] but the result is same as john's. Actually, all username I test return the same result.
I'm thinking that there must be wrong with my implementation since I've only started exploring aggregation.
Mongoose uses a DBRef to be able to populate the field after it has been retrieved.
DBRefs are only handled on the client side, MongoDB aggregation does not have any operators for handling those.
The reason that aggregation pipeline is returning all of the users is the lookup's pipeline does not have a match stage, so all of the documents in the collection are selected and included in the lookup.
The sample document there is showing an array of strings instead of DBRefs, which wouldn't work with populate.
Essentially, you must decide whether you want to use aggregation or populate to handle the join.
For populate, use the ref as shown in that sample schema.
For aggregate, store an array of ObjectId so you can use lookup to link with the _id field.
So I have collections of Teacher, Classes. to understand my problem its necessary to understand my database. the structure is
const TeacherSchema = new Schema(
{
name: {
type: String,
required: true
},
email: {
type: String,
required: true
},
role: {
type: String,
default: "teacher"
},
userid: {
type: String,
required: true
},
password: {
type: String,
required: true
},
profileImage: {
type: String,
required: false
},
classes: [{
type:Schema.Types.ObjectId,
ref:'class'
}]//1 teacher will have multiple classes
},
{ timestamps: true }
);
and the class:
const ClassSchema = new Schema(
{
subject: {
type: String,
required: true
},
teacher:[{
type:Schema.Types.ObjectId,
ref: 'teacher'
}],//once class will have only one teacher
},
{ timestamps: true }
);
What i want to do is, I want to show teacher how many classes he/she has. here is my controller.
exports.getAllClass= async (req, res, next) => {
let teacherClasses
try{
teacherClasses= await Teacher.findById(req.params.id)//teacher database
arrayOfClass= teacherClasses.classes //getting the class it has array of objectID
arrayOfClass.forEach(async (classes)=>{
let classDB =await Class.findById(classes)
console.log(classDB.subject)// in the console it shows all the classes.
return res.status(200).json({
'name':classDB.subject //but here it only shows one class,the first class name
})
});
}catch(err){
console.log(err)
}
}
I have tried many thing, but cant reach to a proper solution. Can you tell me what did i do wrong here or is there any better approach? I want to show all the classes in the return. I am new in programming hence assigned in complex task, please help me.
You are using forEach on the arrayOfClass and within that forEach you return and send the response. So as soon as the first promise is resolved, the response is sent.
One simple way to do this is to use a for .. of loop and await each class-promise inside and keep track of each result. Once that's done you can send the response:
...
const classes = [];
for(const classId of arrayofClass) {
const classDB = await Class.findById(classId);
classes.push({ name: classDB.subject });
}
return res.json(classes);
Another way to do this is to use Promise.all() which might be better if you're dealing with a bigger dataset as is processes in parallel whereas the first solution processes in sequence:
...
const classPromises = await Promise.all(arrayofClass.map(classId => Class.findById(classId)));
const result = classPromises.map(classDB => ({
name: classDB.subject
}));
return res.json(result);