I have two models: Meal and Ingredient. Here are the schemas:
const mealSchema = new Schema({
title: { type: String, required: true },
image: { type: String, required: false },
ingredients: [{
ingredient: { type: mongoose.Types.ObjectId, required: true, ref: 'Ingredient' },
amount: { type: Number, required: true }
}],
})
const ingredientSchema = new Schema({
name: { type: String, required: true },
unit: { type: String, required: true },
category: { type: String, required: false },
is_vege: { type: Boolean, required: true }
});
When I create the meal, I provide the ingredients as an array in POST request body.
const createMeal = async (req, res, next) => {
const { title, ingredients } = req.body;
// ingredientArray consists of objects with two keys:
// <String> ingredient - id of the associated ingredient in database
// <Number> amount - amount of the ingredient
ingredientArray = JSON.parse(ingredients)
const createdMeal = new Meal({
title,
image: req.file.path,
ingredients: ingredientArray
});
try {
await createdMeal.save();
} catch (err) {
const error = new HttpError('Error occurred, try again later', 500);
return next(error);
}
res.status(201).json(createdMeal);
}
This is the object that is created:
{
"_id": "62ea4531bd7e04fa740e2fee",
"title": "Spaghetti",
"image": "uploads\\images\\8da09ec6-3684-4af5-a513-f90155ddafd8.jpeg",
"ingredients": [
{
"ingredient": "62ea37251212c738a0ce9cee",
"amount": 100,
"_id": "62ea4531bd7e04fa740e2fef",
"id": "62ea4531bd7e04fa740e2fef"
},
{
"ingredient": "62ea371ab9392f3e0107c541",
"amount": 10,
"_id": "62ea4531bd7e04fa740e2ff0",
"id": "62ea4531bd7e04fa740e2ff0"
}
],
"__v": 0,
"id": "62ea4531bd7e04fa740e2fee"
}
I need my "ingredients" list to find the actual ingredient in the database and save its complete data so that the result would look something like this:
"ingredients": [
{
"ingredient": "62ea37251212c738a0ce9cee",
"amount": 100,
"name": "Pasta",
"unit": "g",
"category": "pastas",
"is_vege": true,
"_id": "62ea4531bd7e04fa740e2fef",
"id": "62ea4531bd7e04fa740e2fef"
},
{
"ingredient": "62ea371ab9392f3e0107c541",
"amount": 200,
"name": "Tomato",
"unit": "g",
"category": "vegetables",
"is_vege": true,
"_id": "62ea4531bd7e04fa740e2ff0",
"id": "62ea4531bd7e04fa740e2ff0"
}
]
EDIT: Ideally, the ingredient should be found BEFORE saving, because I want to run some calculations before the meal ends up in the database.
I figured it out. It took a single line of code:
let createdMeal = new Meal({
title,
image: req.file.path,
prep_time,
ingredients: ingredientArray
});
createdMeal = await createdMeal.populate({ path: 'ingredients.ingredient' });
Documentation:
https://mongoosejs.com/docs/populate.html
I'm trying to add 2dSphere index for my field startLocation exists in tourSchema. Here how it looks below
startLocation: {
type: {
type: String,
default: 'Point',
enum: ['Point']
},
coordinates: [Number],
address: String,
description: String
}
And you can also see below what and how I've added indexes on this Schema
tourSchema.index({price:1,ratingsAverage:-1});
tourSchema.index({slug:1});
tourSchema.index({ startLocation: '2dsphere' });
Unfortunately Mongodb can't recognize startLocation index. Using Mongo Compass , I'm able to see all indexes that I've created except startLocation:'2dsphere'.
Here is the error that postman gives me below when I send request to getDistances method in controller:
{
"status": "error",
"error": {
"operationTime": "6791492473605586945",
"ok": 0,
"errmsg": "$geoNear requires a 2d or 2dsphere index, but none were found",
"code": 27,
"codeName": "IndexNotFound",
"$clusterTime": {
"clusterTime": "6791492473605586945",
"signature": {
"hash": "4LYCSBslSoLAoqj93bLXmpubBxs=",
"keyId": "6779443539857113090"
}
},
"name": "MongoError",
"statusCode": 500,
"status": "error"
},
"message": "$geoNear requires a 2d or 2dsphere index, but none were found",
"stack": "MongoError: $geoNear requires a 2d or 2dsphere index, but none were found\n at Connection.<anonymous> (C:\\Users\\timuc\\Downloads\\starter\\starter\\node_modules\\mongodb-core\\lib\\connection\\pool.js:443:61)\n at Connection.emit (events.js:223:5)\n at processMessage (C:\\Users\\timuc\\Downloads\\starter\\starter\\node_modules\\mongodb-core\\lib\\connection\\connection.js:364:10)\n at TLSSocket.<anonymous> (C:\\Users\\timuc\\Downloads\\starter\\starter\\node_modules\\mongodb-core\\lib\\connection\\connection.js:533:15)\n at TLSSocket.emit (events.js:223:5)\n at addChunk (_stream_readable.js:309:12)\n at readableAddChunk (_stream_readable.js:290:11)\n at TLSSocket.Readable.push (_stream_readable.js:224:10)\n at TLSWrap.onStreamRead (internal/stream_base_commons.js:181:23)"
}
I tried to add point: '2dsphere' which was recognized by mongodb but I'm not satisfied. Because when I send request to method in controller that returns success but with empty array.
Here is the method which was triggered in controller:
exports.getDistances = catchAsync(async (req, res, next) => {
const { latlng, unit } = req.params;
const [lat, lng] = latlng.split(",");
if (!lat || !lng) {
new AppError( "Please provide latitude and longitude in the format lat,lng", 400);
}
const distances = await Tour.aggregate([
{
$geoNear: {
near: {
type: "Point",
coordinates: [lng * 1, lat * 1]
},
distanceField: "distance"
}
}
]);
res.status(200).json({
status: "success",
data: {
data: distances
}
});
});
also from router you can see how I send the request URL below
tourRouter.route('/distances/:latlng/unit/:unit').get(tourController.getDistances);
I strongly believe that you are not using the proper collection. This is working for MongoDB 4.2.
Creating the index:
db.location.createIndex({
startLocation: "2dsphere"
})
Indexes of that collection:
db.location.getIndexes()
[{
"v": 2,
"key": {
"_id": 1
},
"name": "_id_",
"ns": "stackoverflow.location"
}, {
"v": 2,
"key": {
"startLocation": "2dsphere"
},
"name": "startLocation_2dsphere",
"ns": "stackoverflow.location",
"2dsphereIndexVersion": 3
}
]
Inserting some data:
db.location.insert({
startLocation: {
type: "Point",
coordinates: [40, 5],
address: "Hellostreet 1",
description: "Hello"
}
})
Aggregate the collection:
db.location.aggregate([{
$geoNear: {
near: {
type: 'Point',
coordinates: [41, 6]
},
distanceField: 'distance'
}
}
])
The result:
{
"_id" : ObjectId("5e404cdd13552bde0a0a9dc5"),
"startLocation" : {
"type" : "Point",
"coordinates" : [
40,
5
],
"address" : "Hellostreet 1",
"description" : "Hello"
},
"distance" : 157065.62445348964
}
I'm trying to query from a field from another collection:
const Order = require("../../models/orders");
const User = require("../../models/user");
module.exports = {
async list(req, res) {
const { page = 1, limit = 10 } = req.query;
let filters = req.query;
filters = {
...filters,
page: undefined,
limit: undefined
};
const myAggregate = Order.aggregate([
{
$lookup: {
from: "users",
localField: "user",
foreignField: "_id",
as: "user"
}
},
{
$unwind: "$user"
},
{
$match: { ...filters }
},
{
$project: {
coupon: 1,
status: 1,
"user._id": 1,
"user.firstName": 1,
"user.lastName": 1,
currency: 1,
rate: 1,
amount: 1,
subtotal: 1,
total: 1,
recipient: 1,
createdAt: 1
}
}
]);
const orders = await Order.aggregatePaginate(myAggregate, { page, limit });
return res.send(orders);
}
};
I'm using a plugin called mongoose-aggregate-paginate-v2, but it just add pagination to the mongoose array, so the problem is with the $match.
The collection Orders stores this object:
{
"_id" : ObjectId("5d447d49a910461fd3b5e155"),
"coupon" : [],
"status" : "Processing",
"user" : ObjectId("5d35f67f1cf34e0018e36d69"),
"currency" : "BRL",
"rate" : 10640,
"amount" : 30,
"subtotal" : 30,
"total" : 30,
"recipient" : ObjectId("5d404b39f005f70419d89ae3"),
"createdAt" : ISODate("2019-07-30T13:59:48.532Z")
}
Which, after the aggregation, turns this:
{
"docs": [
{
"_id": "5d447d49a910461fd3b5e155",
"coupon": [],
"status": "Processing",
"user": {
"_id": "5d35f67f1cf34e0018e36d69",
"firstName": "Boo",
"lastName": "Foo"
},
"currency": "BRL",
"rate": 10640,
"amount": 30,
"subtotal": 30,
"total": 30,
"recipient": "5d404b39f005f70419d89ae3",
"createdAt": "2019-07-30T13:59:48.532Z"
}
],
"totalDocs": 1,
"limit": 10,
"page": 1,
"totalPages": 1,
"pagingCounter": 1,
"hasPrevPage": false,
"hasNextPage": false,
"prevPage": null,
"nextPage": null
}
Inside the docs is the array returned by mongoose. If I make a query of any of the Orders fields, the query returns successful, only with data that corresponds to the query. However, if I try to query the firstName, no matter how I try to format the $match, the query is an empty array.
How should I query aggregated fields?
Model:
const projectSchema = new Schema({
name: {
type: String,
maxlength: 50
},
description: {
type: String
},
projectLead: {
type: String
},
start: {
type: String
},
end: {
type: String
},
projectType: {
type: String
},
claimId: {
type: String
},
organization: {
type: String
},
seats: [{
id: String,
employee: String,
role: String,
start: String,
end: String,
workload: Number,
skills: Array,
approved: Boolean
}]
}, {
timestamps: true
})
Model response example:
{
"project": {
"_id": "5cab4b9bc9b29a7ba2363875",
"name": "Project Title",
"description": "Project description",
"projectLead": "email#email",
"start": "2018-06-01T09:45:00.000Z",
"end": "2019-06-31T09:45:00.000Z",
"claimId": "AIIIIII",
"organization": "Company ACE",
"seats": [
{
"skills": [
"Node.js",
"Vue.js"
],
"_id": "5cab548e5cefe27ef82ca313",
"start": "2018-06-01T09:45:00.000Z",
"end": "2019-06-31T09:45:00.000Z",
"role": "Developer",
"approved": false,
"workload": 20,
"employee": ''
}
],
"createdAt": "2019-04-08T13:24:43.253Z",
"updatedAt": "2019-04-08T14:02:54.257Z",
"__v": 0
}
}
Controller:
exports.updateSeatEmployee = async (req, res, next) => {
try {
const project = await Project.findOneAndUpdate({
"seats._id": req.params.id
}, {
"seats.employee": req.body.employee
}, {
new: true
})
console.log("project", project);
return res.json({
project: project
})
} catch (error) {
next(error)
}
}
Debugger:
Mongoose: users.createIndex({ email: 1 }, { unique: true, background:
true }) { sub: '5cab48ebec24577ab3329fcd', iat: 1554729210 }
Mongoose: users.findOne({ _id: ObjectId("5cab48ebec24577ab3329fcd") },
{ projection: {} })
Mongoose: projects.findAndModify({ 'seats._id':
ObjectId("5cab529bcafa027e30ee229c") }, [], { '$setOnInsert': {
createdAt: new Date("Mon, 08 Apr 2019 14:45:56 GMT") }, '$set': {
'seats.$.employee': 'email#email.com', updatedAt: new
Date("Mon, 08 Apr 2019 14:45:56 GMT") } }, { new: true, upsert: false,
remove: false, projection: {} }) (node:33302) DeprecationWarning:
collection.findAndModify is deprecated. Use findOneAndUpdate,
findOneAndReplace or findOneAndDelete instead.
project null
I want to search for a specific object in seats by its id.
This specific seat should be update. In my case I want to update the employee field.
If I do a findOne({"seats._id": req.params.id}) I get the project back but findOneAndUpdate returns null.
You need the $ positional operator since seats is an array and you should use $set operator if you want to update just one field
const project = await Project.findOneAndUpdate({
"seats._id": req.params.id
}, {
$set: { "seats.$.employee": req.body.employee }
}
I have the two models:
Item.js
const mongoose = require('mongoose');
const itemSchema = new mongoose.Schema({
name: String,
stores: [{ type: mongoose.Schema.Types.ObjectId, ref: 'Store' }]
});
const Item = mongoose.model('Item', itemSchema);
module.exports = Item;
Store.js
const mongoose = require('mongoose');
const storeSchema = new mongoose.Schema({
name: String,
items: [{ type: mongoose.Schema.Types.ObjectId, ref: 'Item' }]
});
const Store = mongoose.model('Store', storeSchema);
module.exports = Store;
And a seed.js file:
const faker = require('faker');
const Store = require('./models/Store');
const Item = require('./models/Item');
console.log('Seeding..');
let item = new Item({
name: faker.name.findName() + " Item"
});
item.save((err) => {
if (err) return;
let store = new Store({
name: faker.name.findName() + " Store"
});
store.items.push(item);
store.save((err) => {
if (err) return;
})
});
The store is saved with the items array containing 1 item. The item though, doesn't have stores. What am I missing? How to automatically update the many-to-many relationships in MongoDB/Mongoose? I was used to Rails and everything was done automatically.
The problem you presently have is that you saved the reference in one model but you did not save it in the other. There is no "automatic referential integrity" in MongoDB, and such concept of "relations" are really a "manual" affair, and in fact the case with .populate() is actually a whole bunch of additional queries in order to retrieve the referenced information. No "magic" here.
Correct handling of "many to many" comes down to three options:
Listing 1 - Keep arrays on Both documents
Following your current design, the parts you are missing is storing the referenced on "both" the related items. For a listing to demonstrate:
const { Schema } = mongoose = require('mongoose');
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
mongoose.set('useFindAndModify', false);
mongoose.set('useCreateIndex', true);
const uri = 'mongodb://localhost:27017/manydemo',
options = { useNewUrlParser: true };
const itemSchema = new Schema({
name: String,
stores: [{ type: Schema.Types.ObjectId, ref: 'Store' }]
});
const storeSchema = new Schema({
name: String,
items: [{ type: Schema.Types.ObjectId, ref: 'Item' }]
});
const Item = mongoose.model('Item', itemSchema);
const Store = mongoose.model('Store', storeSchema);
const log = data => console.log(JSON.stringify(data,undefined,2))
(async function() {
try {
const conn = await mongoose.connect(uri,options);
// Clean data
await Promise.all(
Object.entries(conn.models).map(([k,m]) => m.deleteMany() )
);
// Create some instances
let [toothpaste,brush] = ['toothpaste','brush'].map(
name => new Item({ name })
);
let [billsStore,tedsStore] = ['Bills','Teds'].map(
name => new Store({ name })
);
// Add items to stores
[billsStore,tedsStore].forEach( store => {
store.items.push(toothpaste); // add toothpaste to store
toothpaste.stores.push(store); // add store to toothpaste
});
// Brush is only in billsStore
billsStore.items.push(brush);
brush.stores.push(billsStore);
// Save everything
await Promise.all(
[toothpaste,brush,billsStore,tedsStore].map( m => m.save() )
);
// Show stores
let stores = await Store.find().populate('items','-stores');
log(stores);
// Show items
let items = await Item.find().populate('stores','-items');
log(items);
} catch(e) {
console.error(e);
} finally {
mongoose.disconnect();
}
})();
This creates the "items" collection:
{
"_id" : ObjectId("59ab96d9c079220dd8eec428"),
"name" : "toothpaste",
"stores" : [
ObjectId("59ab96d9c079220dd8eec42a"),
ObjectId("59ab96d9c079220dd8eec42b")
],
"__v" : 0
}
{
"_id" : ObjectId("59ab96d9c079220dd8eec429"),
"name" : "brush",
"stores" : [
ObjectId("59ab96d9c079220dd8eec42a")
],
"__v" : 0
}
And the "stores" collection:
{
"_id" : ObjectId("59ab96d9c079220dd8eec42a"),
"name" : "Bills",
"items" : [
ObjectId("59ab96d9c079220dd8eec428"),
ObjectId("59ab96d9c079220dd8eec429")
],
"__v" : 0
}
{
"_id" : ObjectId("59ab96d9c079220dd8eec42b"),
"name" : "Teds",
"items" : [
ObjectId("59ab96d9c079220dd8eec428")
],
"__v" : 0
}
And produces overall output such as:
Mongoose: items.deleteMany({}, {})
Mongoose: stores.deleteMany({}, {})
Mongoose: items.insertOne({ name: 'toothpaste', _id: ObjectId("59ab96d9c079220dd8eec428"), stores: [ ObjectId("59ab96d9c079220dd8eec42a"), ObjectId("59ab96d9c079220dd8eec42b") ], __v: 0 })
Mongoose: items.insertOne({ name: 'brush', _id: ObjectId("59ab96d9c079220dd8eec429"), stores: [ ObjectId("59ab96d9c079220dd8eec42a") ], __v: 0 })
Mongoose: stores.insertOne({ name: 'Bills', _id: ObjectId("59ab96d9c079220dd8eec42a"), items: [ ObjectId("59ab96d9c079220dd8eec428"), ObjectId("59ab96d9c079220dd8eec429") ], __v: 0 })
Mongoose: stores.insertOne({ name: 'Teds', _id: ObjectId("59ab96d9c079220dd8eec42b"), items: [ ObjectId("59ab96d9c079220dd8eec428") ], __v: 0 })
Mongoose: stores.find({}, { fields: {} })
Mongoose: items.find({ _id: { '$in': [ ObjectId("59ab96d9c079220dd8eec428"), ObjectId("59ab96d9c079220dd8eec429") ] } }, { fields: { stores: 0 } })
[
{
"_id": "59ab96d9c079220dd8eec42a",
"name": "Bills",
"__v": 0,
"items": [
{
"_id": "59ab96d9c079220dd8eec428",
"name": "toothpaste",
"__v": 0
},
{
"_id": "59ab96d9c079220dd8eec429",
"name": "brush",
"__v": 0
}
]
},
{
"_id": "59ab96d9c079220dd8eec42b",
"name": "Teds",
"__v": 0,
"items": [
{
"_id": "59ab96d9c079220dd8eec428",
"name": "toothpaste",
"__v": 0
}
]
}
]
Mongoose: items.find({}, { fields: {} })
Mongoose: stores.find({ _id: { '$in': [ ObjectId("59ab96d9c079220dd8eec42a"), ObjectId("59ab96d9c079220dd8eec42b") ] } }, { fields: { items: 0 } })
[
{
"_id": "59ab96d9c079220dd8eec428",
"name": "toothpaste",
"__v": 0,
"stores": [
{
"_id": "59ab96d9c079220dd8eec42a",
"name": "Bills",
"__v": 0
},
{
"_id": "59ab96d9c079220dd8eec42b",
"name": "Teds",
"__v": 0
}
]
},
{
"_id": "59ab96d9c079220dd8eec429",
"name": "brush",
"__v": 0,
"stores": [
{
"_id": "59ab96d9c079220dd8eec42a",
"name": "Bills",
"__v": 0
}
]
}
]
The key points being that you actually add the reference data to each document in each collection where a relationship exists. The "arrays" present are used here to store those references and "lookup" the results from the related collection and replace them with the object data that was stored there.
Pay attention to parts like:
// Add items to stores
[billsStore,tedsStore].forEach( store => {
store.items.push(toothpaste); // add toothpaste to store
toothpaste.stores.push(store); // add store to toothpaste
});
Because that means not only are we adding the toothpaste to the "items" array in each store, but we are also adding each "store" to the "stores" array of the toothpaste item. This is done so the relationships can work being queried from either direction. If you only wanted "items from stores" and never "stores from items", then you would not need to store the relation data on the "item" entries at all.
Listing 2 - Use Virtuals and an Intermediary Collection
This is essentially the classic "many to many" relation. Where instead of directly defining relationships between the two collections, there is another collection ( table ) that stores the details about which item is related to which store.
As a full listing:
const { Schema } = mongoose = require('mongoose');
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
mongoose.set('useFindAndModify', false);
mongoose.set('useCreateIndex', true);
const uri = 'mongodb://localhost:27017/manydemo',
options = { useNewUrlParser: true };
const itemSchema = new Schema({
name: String,
},{
toJSON: { virtuals: true }
});
itemSchema.virtual('stores', {
ref: 'StoreItem',
localField: '_id',
foreignField: 'itemId'
});
const storeSchema = new Schema({
name: String,
},{
toJSON: { virtuals: true }
});
storeSchema.virtual('items', {
ref: 'StoreItem',
localField: '_id',
foreignField: 'storeId'
});
const storeItemSchema = new Schema({
storeId: { type: Schema.Types.ObjectId, ref: 'Store', required: true },
itemId: { type: Schema.Types.ObjectId, ref: 'Item', required: true }
});
const Item = mongoose.model('Item', itemSchema);
const Store = mongoose.model('Store', storeSchema);
const StoreItem = mongoose.model('StoreItem', storeItemSchema);
const log = data => console.log(JSON.stringify(data,undefined,2));
(async function() {
try {
const conn = await mongoose.connect(uri,options);
// Clean data
await Promise.all(
Object.entries(conn.models).map(([k,m]) => m.deleteMany() )
);
// Create some instances
let [toothpaste,brush] = await Item.insertMany(
['toothpaste','brush'].map( name => ({ name }) )
);
let [billsStore,tedsStore] = await Store.insertMany(
['Bills','Teds'].map( name => ({ name }) )
);
// Add toothpaste to both stores
for( let store of [billsStore,tedsStore] ) {
await StoreItem.update(
{ storeId: store._id, itemId: toothpaste._id },
{ },
{ 'upsert': true }
);
}
// Add brush to billsStore
await StoreItem.update(
{ storeId: billsStore._id, itemId: brush._id },
{},
{ 'upsert': true }
);
// Show stores
let stores = await Store.find().populate({
path: 'items',
populate: { path: 'itemId' }
});
log(stores);
// Show Items
let items = await Item.find().populate({
path: 'stores',
populate: { path: 'storeId' }
});
log(items);
} catch(e) {
console.error(e);
} finally {
mongoose.disconnect();
}
})();
The relations are now in their own collection, so the data now appears differently, for "items":
{
"_id" : ObjectId("59ab996166d5cc0e0d164d74"),
"__v" : 0,
"name" : "toothpaste"
}
{
"_id" : ObjectId("59ab996166d5cc0e0d164d75"),
"__v" : 0,
"name" : "brush"
}
And "stores":
{
"_id" : ObjectId("59ab996166d5cc0e0d164d76"),
"__v" : 0,
"name" : "Bills"
}
{
"_id" : ObjectId("59ab996166d5cc0e0d164d77"),
"__v" : 0,
"name" : "Teds"
}
And now for "storeitems" which maps the relations:
{
"_id" : ObjectId("59ab996179e41cc54405b72b"),
"itemId" : ObjectId("59ab996166d5cc0e0d164d74"),
"storeId" : ObjectId("59ab996166d5cc0e0d164d76"),
"__v" : 0
}
{
"_id" : ObjectId("59ab996179e41cc54405b72d"),
"itemId" : ObjectId("59ab996166d5cc0e0d164d74"),
"storeId" : ObjectId("59ab996166d5cc0e0d164d77"),
"__v" : 0
}
{
"_id" : ObjectId("59ab996179e41cc54405b72f"),
"itemId" : ObjectId("59ab996166d5cc0e0d164d75"),
"storeId" : ObjectId("59ab996166d5cc0e0d164d76"),
"__v" : 0
}
With full output like:
Mongoose: items.deleteMany({}, {})
Mongoose: stores.deleteMany({}, {})
Mongoose: storeitems.deleteMany({}, {})
Mongoose: items.insertMany([ { __v: 0, name: 'toothpaste', _id: 59ab996166d5cc0e0d164d74 }, { __v: 0, name: 'brush', _id: 59ab996166d5cc0e0d164d75 } ])
Mongoose: stores.insertMany([ { __v: 0, name: 'Bills', _id: 59ab996166d5cc0e0d164d76 }, { __v: 0, name: 'Teds', _id: 59ab996166d5cc0e0d164d77 } ])
Mongoose: storeitems.update({ itemId: ObjectId("59ab996166d5cc0e0d164d74"), storeId: ObjectId("59ab996166d5cc0e0d164d76") }, { '$setOnInsert': { __v: 0 } }, { upsert: true })
Mongoose: storeitems.update({ itemId: ObjectId("59ab996166d5cc0e0d164d74"), storeId: ObjectId("59ab996166d5cc0e0d164d77") }, { '$setOnInsert': { __v: 0 } }, { upsert: true })
Mongoose: storeitems.update({ itemId: ObjectId("59ab996166d5cc0e0d164d75"), storeId: ObjectId("59ab996166d5cc0e0d164d76") }, { '$setOnInsert': { __v: 0 } }, { upsert: true })
Mongoose: stores.find({}, { fields: {} })
Mongoose: storeitems.find({ storeId: { '$in': [ ObjectId("59ab996166d5cc0e0d164d76"), ObjectId("59ab996166d5cc0e0d164d77") ] } }, { fields: {} })
Mongoose: items.find({ _id: { '$in': [ ObjectId("59ab996166d5cc0e0d164d74"), ObjectId("59ab996166d5cc0e0d164d75") ] } }, { fields: {} })
[
{
"_id": "59ab996166d5cc0e0d164d76",
"__v": 0,
"name": "Bills",
"items": [
{
"_id": "59ab996179e41cc54405b72b",
"itemId": {
"_id": "59ab996166d5cc0e0d164d74",
"__v": 0,
"name": "toothpaste",
"stores": null,
"id": "59ab996166d5cc0e0d164d74"
},
"storeId": "59ab996166d5cc0e0d164d76",
"__v": 0
},
{
"_id": "59ab996179e41cc54405b72f",
"itemId": {
"_id": "59ab996166d5cc0e0d164d75",
"__v": 0,
"name": "brush",
"stores": null,
"id": "59ab996166d5cc0e0d164d75"
},
"storeId": "59ab996166d5cc0e0d164d76",
"__v": 0
}
],
"id": "59ab996166d5cc0e0d164d76"
},
{
"_id": "59ab996166d5cc0e0d164d77",
"__v": 0,
"name": "Teds",
"items": [
{
"_id": "59ab996179e41cc54405b72d",
"itemId": {
"_id": "59ab996166d5cc0e0d164d74",
"__v": 0,
"name": "toothpaste",
"stores": null,
"id": "59ab996166d5cc0e0d164d74"
},
"storeId": "59ab996166d5cc0e0d164d77",
"__v": 0
}
],
"id": "59ab996166d5cc0e0d164d77"
}
]
Mongoose: items.find({}, { fields: {} })
Mongoose: storeitems.find({ itemId: { '$in': [ ObjectId("59ab996166d5cc0e0d164d74"), ObjectId("59ab996166d5cc0e0d164d75") ] } }, { fields: {} })
Mongoose: stores.find({ _id: { '$in': [ ObjectId("59ab996166d5cc0e0d164d76"), ObjectId("59ab996166d5cc0e0d164d77") ] } }, { fields: {} })
[
{
"_id": "59ab996166d5cc0e0d164d74",
"__v": 0,
"name": "toothpaste",
"stores": [
{
"_id": "59ab996179e41cc54405b72b",
"itemId": "59ab996166d5cc0e0d164d74",
"storeId": {
"_id": "59ab996166d5cc0e0d164d76",
"__v": 0,
"name": "Bills",
"items": null,
"id": "59ab996166d5cc0e0d164d76"
},
"__v": 0
},
{
"_id": "59ab996179e41cc54405b72d",
"itemId": "59ab996166d5cc0e0d164d74",
"storeId": {
"_id": "59ab996166d5cc0e0d164d77",
"__v": 0,
"name": "Teds",
"items": null,
"id": "59ab996166d5cc0e0d164d77"
},
"__v": 0
}
],
"id": "59ab996166d5cc0e0d164d74"
},
{
"_id": "59ab996166d5cc0e0d164d75",
"__v": 0,
"name": "brush",
"stores": [
{
"_id": "59ab996179e41cc54405b72f",
"itemId": "59ab996166d5cc0e0d164d75",
"storeId": {
"_id": "59ab996166d5cc0e0d164d76",
"__v": 0,
"name": "Bills",
"items": null,
"id": "59ab996166d5cc0e0d164d76"
},
"__v": 0
}
],
"id": "59ab996166d5cc0e0d164d75"
}
]
Since the relations are now mapped in a separate collection there are a couple of changes here. Notably we want to define a "virtual" field on the collection that no longer has a fixed array of items. So you add one as is shown:
const itemSchema = new Schema({
name: String,
},{
toJSON: { virtuals: true }
});
itemSchema.virtual('stores', {
ref: 'StoreItem',
localField: '_id',
foreignField: 'itemId'
});
You assign the virtual field with it's localField and foreignField mappings so the subsequent .populate() call knows what to use.
The intermediary collection has a fairly standard definition:
const storeItemSchema = new Schema({
storeId: { type: Schema.Types.ObjectId, ref: 'Store', required: true },
itemId: { type: Schema.Types.ObjectId, ref: 'Item', required: true }
});
And instead of "pushing" new items onto arrays, we instead add them to this new collection. A reasonable method for this is using "upserts" to create a new entry only when this combination does not exist:
// Add toothpaste to both stores
for( let store of [billsStore,tedsStore] ) {
await StoreItem.update(
{ storeId: store._id, itemId: toothpaste._id },
{ },
{ 'upsert': true }
);
}
It's a pretty simple method that merely creates a new document with the two keys supplied in the query where one was not found, or essentially tries to update the same document when matched, and with "nothing" in this case. So existing matches just end up as a "no-op", which is the desired thing to do. Alternately you could simply .insertOne() an ignore duplicate key errors. Whatever takes your fancy.
Actually querying this "related" data works a little differently again. Because there is another collection involved, we call .populate() in a way that considers it needs to "lookup" the relation on other retrieved property as well. So you have calls like this:
// Show stores
let stores = await Store.find().populate({
path: 'items',
populate: { path: 'itemId' }
});
log(stores);
Listing 3 - Use Modern Features to do it on the server
So depending on which approach taken, being using arrays or an intermediary collection to store the relation data in as an alternative to "growing arrays" within the documents, then the obvious thing you should be noting is that the .populate() calls used are actually making additional queries to MongoDB and pulling those documents over the network in separate requests.
This might appear all well and fine in small doses, however as things scale up and especially over volumes of requests, this is never a good thing. Additionally there might well be other conditions you want to apply that means you don't need to pull all the documents from the server, and would rather match data from those "relations" before you returned results.
This is why modern MongoDB releases include $lookup which actually "joins" the data on the server itself. By now you should have been looking at all the output those API calls produce as shown by mongoose.set('debug',true).
So instead of producing multiple queries, this time we make it one aggregation statement to "join" on the server, and return the results in one request:
// Show Stores
let stores = await Store.aggregate([
{ '$lookup': {
'from': StoreItem.collection.name,
'let': { 'id': '$_id' },
'pipeline': [
{ '$match': {
'$expr': { '$eq': [ '$$id', '$storeId' ] }
}},
{ '$lookup': {
'from': Item.collection.name,
'let': { 'itemId': '$itemId' },
'pipeline': [
{ '$match': {
'$expr': { '$eq': [ '$_id', '$$itemId' ] }
}}
],
'as': 'items'
}},
{ '$unwind': '$items' },
{ '$replaceRoot': { 'newRoot': '$items' } }
],
'as': 'items'
}}
])
log(stores);
Which whilst longer in coding, is actually far superior in efficiency even for the very trivial action right here. This of course scales considerably.
Following the same "intermediary" model as before ( and just for example, because it could be done either way ) we have a full listing:
const { Schema } = mongoose = require('mongoose');
const uri = 'mongodb://localhost:27017/manydemo',
options = { useNewUrlParser: true };
mongoose.Promise = global.Promise;
mongoose.set('debug', true);
mongoose.set('useFindAndModify', false);
mongoose.set('useCreateIndex', true);
const itemSchema = new Schema({
name: String
}, {
toJSON: { virtuals: true }
});
itemSchema.virtual('stores', {
ref: 'StoreItem',
localField: '_id',
foreignField: 'itemId'
});
const storeSchema = new Schema({
name: String
}, {
toJSON: { virtuals: true }
});
storeSchema.virtual('items', {
ref: 'StoreItem',
localField: '_id',
foreignField: 'storeId'
});
const storeItemSchema = new Schema({
storeId: { type: Schema.Types.ObjectId, ref: 'Store', required: true },
itemId: { type: Schema.Types.ObjectId, ref: 'Item', required: true }
});
const Item = mongoose.model('Item', itemSchema);
const Store = mongoose.model('Store', storeSchema);
const StoreItem = mongoose.model('StoreItem', storeItemSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri, options);
// Clean data
await Promise.all(
Object.entries(conn.models).map(([k,m]) => m.deleteMany())
);
// Create some instances
let [toothpaste, brush] = await Item.insertMany(
['toothpaste', 'brush'].map(name => ({ name }) )
);
let [billsStore, tedsStore] = await Store.insertMany(
['Bills', 'Teds'].map( name => ({ name }) )
);
// Add toothpaste to both stores
for ( let { _id: storeId } of [billsStore, tedsStore] ) {
await StoreItem.updateOne(
{ storeId, itemId: toothpaste._id },
{ },
{ 'upsert': true }
);
}
// Add brush to billsStore
await StoreItem.updateOne(
{ storeId: billsStore._id, itemId: brush._id },
{ },
{ 'upsert': true }
);
// Show Stores
let stores = await Store.aggregate([
{ '$lookup': {
'from': StoreItem.collection.name,
'let': { 'id': '$_id' },
'pipeline': [
{ '$match': {
'$expr': { '$eq': [ '$$id', '$storeId' ] }
}},
{ '$lookup': {
'from': Item.collection.name,
'let': { 'itemId': '$itemId' },
'pipeline': [
{ '$match': {
'$expr': { '$eq': [ '$_id', '$$itemId' ] }
}}
],
'as': 'items'
}},
{ '$unwind': '$items' },
{ '$replaceRoot': { 'newRoot': '$items' } }
],
'as': 'items'
}}
])
log(stores);
// Show Items
let items = await Item.aggregate([
{ '$lookup': {
'from': StoreItem.collection.name,
'let': { 'id': '$_id' },
'pipeline': [
{ '$match': {
'$expr': { '$eq': [ '$$id', '$itemId' ] }
}},
{ '$lookup': {
'from': Store.collection.name,
'let': { 'storeId': '$storeId' },
'pipeline': [
{ '$match': {
'$expr': { '$eq': [ '$_id', '$$storeId' ] }
}}
],
'as': 'stores',
}},
{ '$unwind': '$stores' },
{ '$replaceRoot': { 'newRoot': '$stores' } }
],
'as': 'stores'
}}
]);
log(items);
} catch(e) {
console.error(e);
} finally {
mongoose.disconnect();
}
})()
And the output:
Mongoose: stores.aggregate([ { '$lookup': { from: 'storeitems', let: { id: '$_id' }, pipeline: [ { '$match': { '$expr': { '$eq': [ '$$id', '$storeId' ] } } }, { '$lookup': { from: 'items', let: { itemId: '$itemId' }, pipeline: [ { '$match': { '$expr': { '$eq': [ '$_id', '$$itemId' ] } } } ], as: 'items' } }, { '$unwind': '$items' }, { '$replaceRoot': { newRoot: '$items' } } ], as: 'items' } } ], {})
[
{
"_id": "5ca7210717dadc69652b37da",
"name": "Bills",
"__v": 0,
"items": [
{
"_id": "5ca7210717dadc69652b37d8",
"name": "toothpaste",
"__v": 0
},
{
"_id": "5ca7210717dadc69652b37d9",
"name": "brush",
"__v": 0
}
]
},
{
"_id": "5ca7210717dadc69652b37db",
"name": "Teds",
"__v": 0,
"items": [
{
"_id": "5ca7210717dadc69652b37d8",
"name": "toothpaste",
"__v": 0
}
]
}
]
Mongoose: items.aggregate([ { '$lookup': { from: 'storeitems', let: { id: '$_id' }, pipeline: [ { '$match': { '$expr': { '$eq': [ '$$id', '$itemId' ] } } }, { '$lookup': { from: 'stores', let: { storeId: '$storeId' }, pipeline: [ { '$match': { '$expr': { '$eq': [ '$_id', '$$storeId' ] } } } ], as: 'stores' } }, { '$unwind': '$stores' }, { '$replaceRoot': { newRoot: '$stores' } } ], as: 'stores' } } ], {})
[
{
"_id": "5ca7210717dadc69652b37d8",
"name": "toothpaste",
"__v": 0,
"stores": [
{
"_id": "5ca7210717dadc69652b37da",
"name": "Bills",
"__v": 0
},
{
"_id": "5ca7210717dadc69652b37db",
"name": "Teds",
"__v": 0
}
]
},
{
"_id": "5ca7210717dadc69652b37d9",
"name": "brush",
"__v": 0,
"stores": [
{
"_id": "5ca7210717dadc69652b37da",
"name": "Bills",
"__v": 0
}
]
}
]
What should be obvious is the significant reduction in the queries issued on the end to return the "joined" form of the data. This means lower latency and more responsive applications as a result of removing all the network overhead.
Final Notes
Those a are generally your approaches to dealing with "many to many" relations, which essentially comes down to either:
Keeping arrays in each document on either side holding the references to the related items.
Storing an intermediary collection and using that as a lookup reference to retrieving the other data.
In all cases it is up to you to actually store those references if you expect things to work on "both directions". Of course $lookup and even "virtuals" where that applies means that you don't always need to store on every source since you could then "reference" in just one place and use that information by applying those methods.
The other case is of course "embedding", which is an entirely different game and what document oriented databases such as MongoDB are really all about. Therefore instead of "fetching from another collection" the concept is of course to "embed" the data.
This means not just the ObjectId values that point to the other items, but actually storing the full data within arrays in each document. There is of course an issue of "size" and of course issues with updating data in multiple places. This is generally the trade off for there being a single request and a simple request that does not need to go and find data in other collections because it's "already there".
There is plenty of material around on the subject of referencing vs embedding. Once such summary source is Mongoose populate vs object nesting or even the very general MongoDB relationships: embed or reference? and many many others.
You should spend some time thinking about the concepts and how this applies to your application in general. And note that you are not actually using an RDBMS here, so you might as well use the correct features that you are meant to exploit, rather than simply making one act like the other.
You first should consider the usage of data in your application before modeling the database.
I don't have the detailed requirements of your application. But why do you have to keep 2 references in 2 schemas? Why not just keep 1 reference from Store to Item (which means 1 store has many items), and then if you want execute a query to find which stores does a item belong to, there is also away to do it by querying Store collection.
In addition, there is nothing called "many-to-many" in MongoDB. It depends on how the data is being used that you must figure out the efficient way to form the relationship between collections, as well as to structure your database.
Anyway, if you still want to use your current schemas, you can first create the item, then create the store and push the created item's id in to items array, then execute a update to the item with created store's id.