In MongoDB's documentation it is suggested to put as much data as possible in a single document. It is also suggested NOT to use ObjectId ref based sub-documents unless the data of those sub-documents must be referenced from more than one document.
In my case I have a one-to-many relationship like this:
Log schema:
const model = (mongoose) => {
const LogSchema = new mongoose.Schema({
result: { type: String, required: true },
operation: { type: Date, required: true },
x: { type: Number, required: true },
y: { type: Number, required: true },
z: { type: Number, required: true }
});
const model = mongoose.model("Log", LogSchema);
return model;
};
Machine schema:
const model = (mongoose) => {
const MachineSchema = new mongoose.Schema({
model: { type: String, required: true },
description: { type: String, required: true },
logs: [ mongoose.model("Log").schema ]
});
const model = mongoose.model("Machine", MachineSchema);
return model;
};
module.exports = model;
Each Machine will have many Production_Log documents (more than one million). Using embedded documents I hitted the 16mb per document limit very quickly during my tests and I couldn't add any more Production_Log documents to the Machine documents.
My doubts
Is this a case where one should be using sub-documents as ObjectId references instead of embedded documents?
Is there any other solution I could evaluate?
I will be accessing Production_Log documents to generate stats for each Machine using the aggregation framework. Should I have any extra consideration on the schema design?
Thank you very much in advance for your advice!
Database normalization is not applicable to MongoDB
MongoDB scales better if you store full information in the single document (Data redundancy). Database normalization obligate split data in different collections, but once growth your data, it will cause bottlenecks issues.
Use only LOG Schema:
const model = (mongoose) => {
const LogSchema = new mongoose.Schema({
model: { type: String, required: true },
description: { type: String, required: true },
result: { type: String, required: true },
operation: { type: Date, required: true },
x: { type: Number, required: true },
y: { type: Number, required: true },
z: { type: Number, required: true }
});
const model = mongoose.model("Log", LogSchema);
return model;
};
Read / Write operation scales fine in this way.
With Aggregation you can process data and compute desired result.
Please see if this approach suits your need
The Log collection would be having more data generated whereas the Machine collection never exceed 16MB. Instead of embedding Log collection into Machine collection try the vice versa.
Your modified schema would be like this
Machine schema:
const model = (mongoose) => {
const MachineSchema = new mongoose.Schema({
model: { type: String, required: true },
description: { type: String, required: true }
});
const model = mongoose.model("Machine", MachineSchema);
return model;
};
module.exports = model;
Log schema:
const model = (mongoose) => {
const LogSchema = new mongoose.Schema({
result: { type: String, required: true },
operation: { type: Date, required: true },
x: { type: Number, required: true },
y: { type: Number, required: true },
z: { type: Number, required: true },
machine: [ mongoose.model("Machine").schema ]
});
const model = mongoose.model("Log", LogSchema);
return model;
};
If still we are overshooting the size of Document(16MB) then in the Log Schema we can create a new document for every Day/Hour/Week depending on the logs we are generating.
Related
I am getting the following error when trying to create a document in a collection.
MongoServerError: E11000 duplicate key error collection: stock-trading-system-db.trades index: user_id_1 dup key: { user_id: ObjectId('6266de71b90b594dab9037f3') }
Here is my schema:
const tradeSchema = new mongoose.Schema({
user_id: {
type: Schema.Types.ObjectId,
ref: 'User',
required: true
},
stock_id: {
type: Schema.Types.ObjectId,
ref: 'Stock',
required: true
},
stock_amount: {
type: Number,
required: true
},
stock_value: {
type: Number,
required: true
},
time_of_trade: {
type: Date,
required: true,
default: new Date()
},
trade_status: {
type: String,
enum: ['Approved', 'Declined', 'Pending'],
required: true,
default: 'Pending'
}
}, { collection: 'trades' })
I don't want user_id and stock_id to be unique, i just want it to check that those ObjectIDs exist in their respective collections before making the trade document. How do i achieve this?
It looks like a collection saves the schema that is defined in mongoose, and even if you change the schema, the unique values will stay the same inside the collection.
So even though i had removed
unique: true;
from my mongoose schema, it hadn't removed this from the collection in my DB.
Therefore the solution to this is to delete the collection and recreate it.
I have two same functions, first saves some stuff and also saves ref to productId and second saves another stuff and saves ref to productId too. The problem is first writes ref to poductId as object and everything is ok but second function saves ref as string and in mongodb i see ObjectId but when im trying to display data on screen i cant access into object i have only string with ID of refer. Any ideas?
first model which is ok
const mongoose = require('mongoose')
const dishPositionsSchema = mongoose.Schema({
productId: {
type: mongoose.Schema.Types.ObjectId,
required: true,
ref: 'Products'
},
dishId: {
type: mongoose.Schema.Types.ObjectId,
required: true,
ref: 'Dishes'
},
weight: {
type: Number,
required: true
}
})
module.exports = mongoose.model('DishPositions', dishPositionsSchema)
and second which stores only strings instead of objects
const mongoose = require('mongoose')
const diaryPositionsSchema = mongoose.Schema({
productId: {
type: mongoose.Schema.Types.ObjectId,
required: true,
ref: 'Products'
},
dishId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Dishes'
},
weight: {
type: Number,
required: true,
required: true
},
timeOfEatingId: {
type: mongoose.Schema.Types.ObjectId,
required: true,
ref: 'TimeOfEating'
},
date: {
type: String,
required: true
}
})
module.exports = mongoose.model('DiaryPositions', diaryPositionsSchema)
I know where i did mistake... i didnt write populate('productId')... everything is fine now thanks!
I have built a simple MERN app where users can rate phone numbers. Users just fill in the phone number, choose rating (1 - 5 star rating), their city & short text. The app has search function with filter & sorting options. It all works good enough ATM but I think it might break when multiple concurrent users use the website because I update the phone number model (mobileSchema) after a rating (messageSchema) has been submitted - using Mongoose middlewares (post hooks).
For example, I need to calculate number of ratings (messagesCount) for phone number. I use Message.countDocuments({ mobile: mobile._id }) for that. However, I also need to update other properties of phone number (mobileSchema - lastMessageDate, globalRating, averageRating) so that operation takes some time. I believe the number of ratings might not be right when 2 users submit rating at the same time - it will increment the number of ratings (messagesCount) by 1 instead of 2.
Is there a better approach? Can a post hook be fired after the previous post hook already finished?
Sample code:
const mobileSchema = new Schema({
number: { type: String, required: true },
plan: { type: String, required: true },
date: { type: Date, default: Date.now, required: true, index: 1 },
messagesCount: { type: Number, default: 0, index: 1 },
lastMessageDate: { type: Date, index: 1 },
// normal mean
globalRating: { type: Number, default: 0, index: 1 },
// weighted mean
averageRating: { type: Number, default: 0, index: 1 }
});
const messageSchema = new Schema({
comment: { type: String, required: true },
city: { type: Schema.Types.ObjectId, ref: 'City', required: true, index: 1 },
rating: { type: Number, required: true, index: 1 },
date: { type: Date, default: Date.now, required: true, index: 1 },
mobile: { type: Schema.Types.ObjectId, ref: 'Mobile', required: true },
user: { type: Schema.Types.ObjectId, ref: 'User', required: true }
});
messageSchema.post('save', function (message, next) {
const messageModel = this.constructor;
return updateMobile(messageModel, message, next, 1);
});
const updateMobile = (messageModel, message, next, addMessage) => {
const { _id } = message.mobile;
const cityId = message.city._id;
const lastMessageDate = message.date;
let mobile;
hooks.get(Mobile, { _id })
.then(mobileRes => {
mobile = mobileRes;
return Message.countDocuments({ mobile: mobile._id })
})
.then(messagesCount => {
if (messagesCount <= 0) {
const deleteMobile = Mobile.findOneAndDelete({ _id: mobile._id })
const deleteSeen = SeenMobile.findOneAndDelete({ mobile: mobile._id, user: message.user._id })
const cityMobile = updateCityMobile(messageModel, mobile, cityId)
Promise.all([deleteMobile, deleteSeen, cityMobile])
.then(() => {
return next();
})
.catch((err) => {
console.log(err);
return next();
})
}
else {
if (addMessage === -1) lastMessageDate = mobile.lastMessageDate;
const ratings = hooks.updateGlobalRating(mobile, messageModel)
.then(() => hooks.updateAverageRating(mobile, messageModel))
.then(() => {
return new Promise((resolve, reject) => {
mobile.set({
messagesCount,
lastMessageDate
});
mobile.save((err, mobile) => {
if (err) return reject(err);
resolve();
});
})
})
const cityMobile = updateCityMobile(messageModel, mobile, cityId)
Promise.all([ratings, cityMobile])
.then(([ratings, cityMobile]) => {
return next();
})
.catch(err => console.log(err))
}
})
.catch(err => {
console.log(err);
})
}
I think you are always going to run into async issues with your approach. I don't believe you can "synchronize" the hooks; seems to go against everything that is true about MongoDB. However, at a high level, you might have more success grabbing the totals/summaries at run-time, rather than trying to keep them always in sync. For instance, if you need the total number of messages for a given mobile device, why not:
Messages.find({mobile: mobile._id})
and then count the results? That will save you storing the summaries and keeping them updated. However, I also think your current approach could work, but you probably need to scrap the "countDocuments". Something a bit more async friendly, like:
Mobile.aggregation([
{ $match: { _id: mobile._id } },
{ $add: [ "$mobile.messagesCount", 1 ] }
]);
Ultimately I think your design would be strengthened if you stored Messages as an array inside of Mobile, so you can just push the message on it. But to answer the question directly, the aggregation should keep everything tidy.
I found this answer: Locking a document in MongoDB
I will calculate all the values I need (messagesCount, globalRating etc.) in post hook and then I will check if the mobile document has the same __v (version) value during final findOneAndUpdate operation (because this operation locks the document and can increment __v). If it has different __v then I will call the post hook again to ensure it will calculate the right values.
First we need to fix some database structure here
Mobile schema
const mobileSchema = new Schema({
number: { type: String, required: true },
plan: { type: String, required: true },
date: { type: Date, default: Date.now, required: true, index: 1 },
//messagesCount: { type: Number, default: 0, index: 1 },
//lastMessageDate: { type: Date, index: 1 },
// normal mean
//globalRating: { type: Number, default: 0, index: 1 },
// weighted mean
//averageRating: { type: Number, default: 0, index: 1 }
});
Message schema
const messageSchema = new Schema({
comment: { type: String, required: true },
city: { type: Schema.Types.ObjectId, ref: 'City', required: true, index: 1 },
//rating: { type: Number, required: true, index: 1 },
date: { type: Date, default: Date.now, required: true, index: 1 },
mobile: { type: Schema.Types.ObjectId, ref: 'Mobile', required: true },
user: { type: Schema.Types.ObjectId, ref: 'User', required: true }
});
Rating system (take all rating or make them a set)
(numerator & denominator after 100 ratings it is difficult to read every single one) can also check for the mobile
const ratingSchema = new Schema({
mobile: { type: String, required: true },
commmentId:{type:String, required: true, index: 1}
rate: { type: Number required: true, },
//rating: { type: Number, required: true, index: 1 },
timestamp: { type: Date, default: Date.now, required: true, index: 1 }
denominator:{ type: Number},
numerator:{type:Number}
});
Thanks
I've my db already created where I have the following schema:
const ProjectSchema = new mongoose.Schema({
name: { type: String },
description: { type: String },
client: {
type: mongoose.Schema.Types.ObjectId,
ref: 'client'
},
group: {
type: mongoose.Schema.Types.ObjectId,
ref: 'project_group'
}
});
I need to change the schema to
const ProjectSchema = new mongoose.Schema({
name: { type: String, unique: true, required: true },
description: { type: String },
client: {
type: mongoose.Schema.Types.ObjectId,
ref: 'client'
},
group: {
type: mongoose.Schema.Types.ObjectId,
ref: 'project_group'
}
});
because we need to force name to be unique and not null. After change this definition, I see that the db still saving documents with the same name value. My question is, how can I apply any change I've done in my schema? and, how can I do that automatically without doing this manually?
Regards
You most likely need to index that field so it can be quickly searched. Then also restart your server, if you haven't already. It's easy to do if you have MongoDB Compass (the official MongoDB GUI), or you can read this doc.
https://docs.mongodb.com/manual/core/index-single/#create-an-index-on-an-embedded-field
Here is my code. I have a review schema in a separate file called "review.js".
const express = require('express');
const mongoose = require('mongoose');
const User = require('../model/user');
require('mongoose-currency').loadType(mongoose);
const Currency = mongoose.Types.Currency;
const Schema = mongoose.Schema;
let reviewSchema = new Schema();
reviewSchema.add({
rating: {
type: Number,
min: 1,
max: 5,
defualt: 0
},
howEasyToMake: {
type: Number,
required: true,
min: 1,
max: 5
},
howGoodTaste: {
type: Number,
required: true,
min: 1,
max: 5,
},
wouldMakeAgain: {
type: Number,
required: true,
min: 1,
max: 5,
},
comment: {
type: String,
default: ""
},
postedBy: {
type: String,
required: true,
index: true
},
reviewOf: {
type: String,
required: true,
index: true
},
postersCreationDate: {
type: Number,
required: true
},
chefsCreationDate: {
type: Number,
required: true
},
chefsId: {
type: String,
required: true
}
});
module.exports.reviewSchema = reviewSchema;
I have another file called recipe.js, where I import reviewSchema and use it as an embedded schema for my Recipe model schema.
const express = require('express');
const mongoose = require('mongoose');
const User = require('../model/user');
require('mongoose-currency').loadType(mongoose);
const Currency = mongoose.Types.Currency;
const Schema = mongoose.Schema;
const reviewSchema = require('../model/review').reviewSchema;
let recipeSchema = new Schema({
name: {
type: String,
required: true
},
description: {
type: String,
},
steps: {
type: String,
required: true,
},
ingredients: {
type: Array,
default: ['1', '2', '3', '4']
},
category: {
type: String,
required: true,
index: true
},
postedBy: {
type: String,
required: true,
},
reviewsOfRecipe: [reviewSchema],
numberOfRatings: {
type: Number,
default: 0
},
totalAddedRatings: {
type: Number,
default: 0
},
reviewAverage: {
type: Number,
default: undefined
},
postersCreationDate: {
type: Number,
index: true
},
likedBy: {
type: Array
},
reviewedBy: {
type: Array
}
});
recipeSchema.methods.updateReviewAverage = function(){
let recipe = this;
this.reviewAverage = this.totalAddedRatings / this.numberOfRatings;
};
let Recipe = mongoose.model('Recipe', recipeSchema);
module.exports = Recipe;
I have another file called recipeRouter.js where I use reviewSchema to construct a review to then later insert it into the embedded reviewsOfRecipes array in my Recipe document. In my recipRouter.js file, every time my code tries to do this....
Recipe.findOne({name: req.params.name}).then((recipe) => {
let review_ = new reviewSchema({
comment: req.body.comment
rating: req.body.score
});
recipe.reviewsOfRecipe.push(review_);
})
...I get this error.
TypeError: reviewSchema is not a constructor
Previously, when I ran into this problem, I had the reviewSchema in the same file as my Recipe model schema. Since then, I split the two into each having their own file. And I made sure to properly export module.exports.reviewSchema = reviewSchema; in my review.js file, and made sure to have const reviewSchema = require('../model/review').reviewSchema; in both my recipe.js and recipeRouter.js files. Yet still this issue still comes up/ I would greatly appreciate it if someone could point out what may be the issue.
You have to export reviewSchema like you did to the other schema (using mongoose.model):
module.exports.reviewSchema = mongoose.model('Review', reviewSchema);