Develop Reference

Map the nested data from other table using promise and async-await

I need the expert advice for this code. I need to know Is there any better way to solve this.
I am using the mongoose for db. I have a dataset like this:
Below is matchTable:
{
_id: 617bc0113176d717f4ddd6ce,
car: [],
status: true
},
{
_id: 617bc0113176d717f4ddd6cg,
car: [
{
aid: '5c1b4ffd18e2d84b7d6febcg',
}
],
status: true
}
And I have a Car table in which car name is there on behalf of id
like this
{ _id: ObjectId('5c1b4ffd18e2d84b7d6febce'), name: 'ford' },
{ _id: ObjectId('5c1b4ffd18e2d84b7d6febcg'), name: 'mitsubishi' },
So I want to make join the data from car table, so that response get name on behalf of aid.
Desired result will be like
{
_id: 617bc0113176d717f4ddd6ce,
car: [],
status: true
},
{
_id: 617bc0113176d717f4ddd6cg,
car: [
{
aid: '5c1b4ffd18e2d84b7d6febcg',
name: 'mitsubishi'
}
],
status: true
}
For that I have to merge the car table on matchTable. I have done this but I want to give some suggestion that is there any better way to do or is it fine. I need expert advice.
const getData = await matchTable.find(
{ status: true }
).lean().exec();
let dataHolder = [];
await Promise.all (
getData.map(async x => {
await Promise.all(
x.car.map(async y => {
let data = await Car.findOne(
{ _id: ObjectId(y.aid) },
{ name: 1 }
).lean().exec();
y.name = '';
if (data) {
y.name = data.name;
}
})
)
// If I return { ...x }, then on response it will return {}, {} on car column
dataHolder.push(x) //So I have chosen this approach
})
);
Please guide me if any better and efficient solution is there. Thanks in advance

You can make use of aggregation here.
const pipeline = [
{
$match : { status : true }
},
{
$unwind: '$matchtable',
},
{
$lookup: {
from: "cars",
localField: "car.aid",
foreignField: "_id",
as: "matchcars"
}
},
{
$addFields: {
"car.carName": { $arrayElemAt: ["$matchcars.name", 0] }
}
},
{
$group: {
_id: "$_id",
cars: { $push: "$matchcars" }
}
}
]
const result = await matchTable.aggregate(pipeline).exec();
Please make sure, aid field inside car array (in matchTable collection) is an ObjectId because its being matched to _id (which is an ObjectId) inside cars collection.

Mongoose Model find using an attribute from anohter Schema

Basically I have 2 Schemas.
User and Post.
User have an array which contains _ids from posts.
And post have an attribute that tells if he's an active post. -> is_active.
So, i want to filter User that have at least, one active post.
UserSchema
const UserSchema = new Schema(
{
name: {
type: String,
trim: true,
required: true
},
posts: [
{
type: Schema.Types.ObjectId,
ref: 'Post'
}
],
created_at: {
type: Date,
required: true,
default: Date.now()
}
}
)
export default mongoose.model<User>('User', UserSchema)
Post Schema
const postSchema = new Schema(
{
name: String,
is_active: boolean
}
)

As an alternative to #Tunmee's answer
Since the pipeline $lookup is available from v3.6 and as of v4.2 still has some performance issues. You could also use the "regular" $lookup available from v3.2
db.Users.aggregate([
{
$lookup: {
from: "Posts",
localField: "posts",
foreignField: "_id",
as: "posts"
}
},
{
$match: {
"posts.is_active": true
}
}
])

You can try this:
Users.aggregate([
{
$lookup: {
from: "Posts",
let: { postIds: "$posts", },
pipeline: [
{
$match: {
$expr: {
$and: [
{
$in: [ "$_id", "$$postIds" ]
},
{
$eq: [ "$is_active", true ]
},
]
}
},
},
// You can remove the projection below
// if you need the actual posts data in the final result
{
$project: { _id: 1 }
}
],
as: "posts"
}
},
{
$match: {
$expr: {
$gt: [ { $size: "$posts" }, 0 ]
}
}
}
])
You can test it out in a playground here
I'm not sure about your application's query requirement but you can add a compound index on _id and is_active properties in Posts collection to make the query faster.
You can read more about MongoDB data aggregation here.

Populate Only When Conditions Are Met

I have a mongodb database and I use mongoose with nodejs.
I need return data from the next query populating "tabela_tuss" only if I have the field "temtussvinculado=true".
Here is what I am doing:
ConvProced.find({'convenioId':new ObjectId(req.params.id)})
.populate('convenioId')
.populate({
path:'procedId',
populate:{
path:'tabela_tuss',
match: { 'procedId.temtussvinculado': true}
}
})
.exec( (err,data) => {
callback(err,data,res)
})
My problem is that my match with "procedId.temtussvinculado:true" has no effect and "tabela_tuss" is never populated.
What am I doing wrong?
Here is my schemas:
////
var conveniosSchema = new mongoose.Schema({
nome: {type: String, unique:true},
ativo: {type: Boolean}
});
module.exports = mongoose.model('Convenio', conveniosSchema,'convenios' );
////
////
const agProcedimentosSchema = new mongoose.Schema({
ativo:{type:Boolean},
temtussvinculado:{type:Boolean},
tabela_tuss:{type:mongoose.Schema.Types.ObjectId, ref:'Tuss_22'}
});
module.exports = mongoose.model('Ag_procedimento', agProcedimentosSchema,'ag_procedimentos' );
///
////
const tuss_22Schema = new mongoose.Schema({
codigo: {type: String, unique:true},
descricao:{type: String},
tabela:{type: String}
});
module.exports = mongoose.model('Tuss_22', tuss_22Schema,'tuss_22' );
////
//../models/convenioprocedimento
var conveniosProcedsSchema = new mongoose.Schema({
convenioId:{type:mongoose.Schema.Types.ObjectId, ref:'Convenio'},
procedId:{type:mongoose.Schema.Types.ObjectId, ref:'Ag_procedimento'},
valor_particular:{type:Number},
valor_convenio:{type:Number},
});
module.exports = mongoose.model('ConvenioProcedimento', conveniosProcedsSchema,'conveniosprocedimentos' );
//my query:
const ConvProced = require('../models/convenioprocedimento');
ConvProced.find({'convenioId':new ObjectId(req.params.id)})
.populate('convenioId')
.populate({
path:'procedId',
populate:{
path:'tabela_tuss',
match: { 'procedId.temtussvinculado': true}
}
})
.exec( (err,data) => {
callback(err,data,res)
})

What you are actually asking here is to "Only populate where a condition within the data says to do so", which is something that is not actually a "directly" supported action of .populate() or usage of the "nested populate" syntax.
So if you want to impose "conditions" on which items are actually populated or not, then you must handle the populate calls "manually".
The basic premise in your case is that you would need to inspect the value which you need to get from the "initial" top level .populate() call, but then "only" call the "inner" populate when the given condtions actually allow it.
So your code should then probably look like this using "Promises" using Promise.all() where you basically "loop" or .map() each query result and test the proceedid.temtussvinculado to see if it is true/false, and where true we actually issue a Model.populate() call, otherwise just return the data in it's present state:
ConvProced.find({'convenioId':new ObjectId(req.params.id)})
.populate('convenioId procedId')
.exec()
.then(data =>
Promise.all(
data.map( d =>
( d.proceedid.temtussvinculado )
? mongoose.model('Tuss_22').populate(d,{ path: 'proceedId.tabela_tuss' })
: d
)
)
)
)
// Populated conditionally
.then( data =>
// Do something with data
)
.catch(err => console.error(err)); // or something else with error
There are different options available other than 'Promises', but it is the no dependency option. Alternate cases such as async.map to do much the same thing, but is an additional dependency if you do not already have it:
ConvProced.find({'convenioId':new ObjectId(req.params.id)})
.populate('convenioId procedId')
.exec((err,data) => {
if (err) throw err;
async.map(data,(d,callback) =>
( d.proceedid.temtussvinculado )
? mongoose.model('Tuss_22').populate(d,{ path: 'proceedId.tabela_tuss' },callback)
: callback(null,d)
(err,data) => {
if (err) throw err; // or something
// Conditionally populated
}
)
})
Also demonstrated with a full working example, which is actually a little more complicated than what you need to do, since the "condition" is nested within another array in this example:
const async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
mongoose.connect('mongodb://localhost/test');
const subInnerSchema = new Schema({
label: String
});
const innerSchema = new Schema({
name: String,
populate: Boolean,
subs: [{ type: Schema.Types.ObjectId, ref: 'Sub' }]
});
const outerSchema = new Schema({
title: String,
inners: [{ type: Schema.Types.ObjectId, ref: 'Inner' }]
});
const Sub = mongoose.model('Sub', subInnerSchema);
const Inner = mongoose.model('Inner', innerSchema);
const Outer = mongoose.model('Outer', outerSchema);
function log(data) {
console.log(JSON.stringify(data, undefined, 2))
}
async.series(
[
// Clean data
(callback) =>
async.each(mongoose.models,(model,callback) =>
model.remove({},callback),callback),
// Insert some data
(callback) =>
async.waterfall(
[
(callback) =>
Sub.create([1,2,3,4].map( label => ({ label })),callback),
(subs,callback) =>
Inner.create(
[0,2].map(x => subs.slice(x,x+2))
.map((el,i) => ({
name: i+i,
populate: i == 1,
subs: el
})),
callback
),
(inners,callback) =>
Outer.create(
inners.map((inner,i) => ({
title: i+1,
inners: [inner]
})),
callback
),
],
callback
),
// Conditional populate async.map version
(callback) =>
Outer.find().populate('inners').exec((err,outers) => {
if (err) callback(err);
async.map(
outers,
(outer,callback) =>
async.map(
outer.inners,
(inner,callback) =>
(inner.populate)
? Inner.populate(inner,{ path: 'subs' },callback)
: callback(null,inner),
(err,inners) => {
if (err) callback(err);
outer.inners = inners
callback(null,outer);
}
),
(err,outers) => {
if (err) callback(err);
log(outers);
callback();
}
);
}),
// Conditional populate Promise
(callback) =>
Outer.find().populate('inners').exec()
.then(outers =>
Promise.all(
outers.map( outer =>
new Promise((resolve,reject) => {
Promise.all(
outer.inners.map( inner =>
(inner.populate)
? Inner.populate(inner,{ path: 'subs' })
: inner
)
).then(inners => {
outer.inners = inners;
resolve(outer)
})
.catch(reject)
})
)
)
)
.then(outers => {
log(outers);
callback();
})
.catch(err => callback(err))
],
(err) => {
if (err) throw err;
mongoose.disconnect();
}
);
Which produces the output showing the "conditional" selection, from using either approach of course:
Mongoose: subs.remove({}, {})
Mongoose: inners.remove({}, {})
Mongoose: outers.remove({}, {})
Mongoose: subs.insert({ label: '1', _id: ObjectId("5961830256bf9e2d0fcf13b3"), __v: 0 })
Mongoose: subs.insert({ label: '2', _id: ObjectId("5961830256bf9e2d0fcf13b4"), __v: 0 })
Mongoose: subs.insert({ label: '3', _id: ObjectId("5961830256bf9e2d0fcf13b5"), __v: 0 })
Mongoose: subs.insert({ label: '4', _id: ObjectId("5961830256bf9e2d0fcf13b6"), __v: 0 })
Mongoose: inners.insert({ name: '0', populate: false, _id: ObjectId("5961830256bf9e2d0fcf13b7"), subs: [ ObjectId("5961830256bf9e2d0fcf13b3"), ObjectId("5961830256bf9e2d0fcf13b4") ], __v: 0 })
Mongoose: inners.insert({ name: '2', populate: true, _id: ObjectId("5961830256bf9e2d0fcf13b8"), subs: [ ObjectId("5961830256bf9e2d0fcf13b5"), ObjectId("5961830256bf9e2d0fcf13b6") ], __v: 0 })
Mongoose: outers.insert({ title: '1', _id: ObjectId("5961830256bf9e2d0fcf13b9"), inners: [ ObjectId("5961830256bf9e2d0fcf13b7") ], __v: 0 })
Mongoose: outers.insert({ title: '2', _id: ObjectId("5961830256bf9e2d0fcf13ba"), inners: [ ObjectId("5961830256bf9e2d0fcf13b8") ], __v: 0 })
Mongoose: outers.find({}, { fields: {} })
Mongoose: inners.find({ _id: { '$in': [ ObjectId("5961830256bf9e2d0fcf13b7"), ObjectId("5961830256bf9e2d0fcf13b8") ] } }, { fields: {} })
Mongoose: subs.find({ _id: { '$in': [ ObjectId("5961830256bf9e2d0fcf13b5"), ObjectId("5961830256bf9e2d0fcf13b6") ] } }, { fields: {} })
[
{
"_id": "5961830256bf9e2d0fcf13b9",
"title": "1",
"__v": 0,
"inners": [
{
"_id": "5961830256bf9e2d0fcf13b7",
"name": "0",
"populate": false,
"__v": 0,
"subs": [
"5961830256bf9e2d0fcf13b3",
"5961830256bf9e2d0fcf13b4"
]
}
]
},
{
"_id": "5961830256bf9e2d0fcf13ba",
"title": "2",
"__v": 0,
"inners": [
{
"_id": "5961830256bf9e2d0fcf13b8",
"name": "2",
"populate": true,
"__v": 0,
"subs": [
{
"_id": "5961830256bf9e2d0fcf13b5",
"label": "3",
"__v": 0
},
{
"_id": "5961830256bf9e2d0fcf13b6",
"label": "4",
"__v": 0
}
]
}
]
}
]
Mongoose: outers.find({}, { fields: {} })
Mongoose: inners.find({ _id: { '$in': [ ObjectId("5961830256bf9e2d0fcf13b7"), ObjectId("5961830256bf9e2d0fcf13b8") ] } }, { fields: {} })
Mongoose: subs.find({ _id: { '$in': [ ObjectId("5961830256bf9e2d0fcf13b5"), ObjectId("5961830256bf9e2d0fcf13b6") ] } }, { fields: {} })
[
{
"_id": "5961830256bf9e2d0fcf13b9",
"title": "1",
"__v": 0,
"inners": [
{
"_id": "5961830256bf9e2d0fcf13b7",
"name": "0",
"populate": false,
"__v": 0,
"subs": [
"5961830256bf9e2d0fcf13b3",
"5961830256bf9e2d0fcf13b4"
]
}
]
},
{
"_id": "5961830256bf9e2d0fcf13ba",
"title": "2",
"__v": 0,
"inners": [
{
"_id": "5961830256bf9e2d0fcf13b8",
"name": "2",
"populate": true,
"__v": 0,
"subs": [
{
"_id": "5961830256bf9e2d0fcf13b5",
"label": "3",
"__v": 0
},
{
"_id": "5961830256bf9e2d0fcf13b6",
"label": "4",
"__v": 0
}
]
}
]
}
]
So you can see there that in much the same way there is a "boolean" field which is being tested to determine whether to perform a .populate() or just return the plain data instead.

Mongoose, find documents that contain a value in an array of objects [duplicate]

I'm pretty new to Mongoose and MongoDB in general so I'm having a difficult time figuring out if something like this is possible:
Item = new Schema({
id: Schema.ObjectId,
dateCreated: { type: Date, default: Date.now },
title: { type: String, default: 'No Title' },
description: { type: String, default: 'No Description' },
tags: [ { type: Schema.ObjectId, ref: 'ItemTag' }]
});
ItemTag = new Schema({
id: Schema.ObjectId,
tagId: { type: Schema.ObjectId, ref: 'Tag' },
tagName: { type: String }
});
var query = Models.Item.find({});
query
.desc('dateCreated')
.populate('tags')
.where('tags.tagName').in(['funny', 'politics'])
.run(function(err, docs){
// docs is always empty
});
Is there a better way do this?
Edit
Apologies for any confusion. What I'm trying to do is get all Items that contain either the funny tag or politics tag.
Edit
Document without where clause:
[{
_id: 4fe90264e5caa33f04000012,
dislikes: 0,
likes: 0,
source: '/uploads/loldog.jpg',
comments: [],
tags: [{
itemId: 4fe90264e5caa33f04000012,
tagName: 'movies',
tagId: 4fe64219007e20e644000007,
_id: 4fe90270e5caa33f04000015,
dateCreated: Tue, 26 Jun 2012 00:29:36 GMT,
rating: 0,
dislikes: 0,
likes: 0
},
{
itemId: 4fe90264e5caa33f04000012,
tagName: 'funny',
tagId: 4fe64219007e20e644000002,
_id: 4fe90270e5caa33f04000017,
dateCreated: Tue, 26 Jun 2012 00:29:36 GMT,
rating: 0,
dislikes: 0,
likes: 0
}],
viewCount: 0,
rating: 0,
type: 'image',
description: null,
title: 'dogggg',
dateCreated: Tue, 26 Jun 2012 00:29:24 GMT
}, ... ]
With the where clause, I get an empty array.

With a modern MongoDB greater than 3.2 you can use $lookup as an alternate to .populate() in most cases. This also has the advantage of actually doing the join "on the server" as opposed to what .populate() does which is actually "multiple queries" to "emulate" a join.
So .populate() is not really a "join" in the sense of how a relational database does it. The $lookup operator on the other hand, actually does the work on the server, and is more or less analogous to a "LEFT JOIN":
Item.aggregate(
[
{ "$lookup": {
"from": ItemTags.collection.name,
"localField": "tags",
"foreignField": "_id",
"as": "tags"
}},
{ "$unwind": "$tags" },
{ "$match": { "tags.tagName": { "$in": [ "funny", "politics" ] } } },
{ "$group": {
"_id": "$_id",
"dateCreated": { "$first": "$dateCreated" },
"title": { "$first": "$title" },
"description": { "$first": "$description" },
"tags": { "$push": "$tags" }
}}
],
function(err, result) {
// "tags" is now filtered by condition and "joined"
}
)
N.B. The .collection.name here actually evaluates to the "string" that is the actual name of the MongoDB collection as assigned to the model. Since mongoose "pluralizes" collection names by default and $lookup needs the actual MongoDB collection name as an argument ( since it's a server operation ), then this is a handy trick to use in mongoose code, as opposed to "hard coding" the collection name directly.
Whilst we could also use $filter on arrays to remove the unwanted items, this is actually the most efficient form due to Aggregation Pipeline Optimization for the special condition of as $lookup followed by both an $unwind and a $match condition.
This actually results in the three pipeline stages being rolled into one:
{ "$lookup" : {
"from" : "itemtags",
"as" : "tags",
"localField" : "tags",
"foreignField" : "_id",
"unwinding" : {
"preserveNullAndEmptyArrays" : false
},
"matching" : {
"tagName" : {
"$in" : [
"funny",
"politics"
]
}
}
}}
This is highly optimal as the actual operation "filters the collection to join first", then it returns the results and "unwinds" the array. Both methods are employed so the results do not break the BSON limit of 16MB, which is a constraint that the client does not have.
The only problem is that it seems "counter-intuitive" in some ways, particularly when you want the results in an array, but that is what the $group is for here, as it reconstructs to the original document form.
It's also unfortunate that we simply cannot at this time actually write $lookup in the same eventual syntax the server uses. IMHO, this is an oversight to be corrected. But for now, simply using the sequence will work and is the most viable option with the best performance and scalability.
Addendum - MongoDB 3.6 and upwards
Though the pattern shown here is fairly optimized due to how the other stages get rolled into the $lookup, it does have one failing in that the "LEFT JOIN" which is normally inherent to both $lookup and the actions of populate() is negated by the "optimal" usage of $unwind here which does not preserve empty arrays. You can add the preserveNullAndEmptyArrays option, but this negates the "optimized" sequence described above and essentially leaves all three stages intact which would normally be combined in the optimization.
MongoDB 3.6 expands with a "more expressive" form of $lookup allowing a "sub-pipeline" expression. Which not only meets the goal of retaining the "LEFT JOIN" but still allows an optimal query to reduce results returned and with a much simplified syntax:
Item.aggregate([
{ "$lookup": {
"from": ItemTags.collection.name,
"let": { "tags": "$tags" },
"pipeline": [
{ "$match": {
"tags": { "$in": [ "politics", "funny" ] },
"$expr": { "$in": [ "$_id", "$$tags" ] }
}}
]
}}
])
The $expr used in order to match the declared "local" value with the "foreign" value is actually what MongoDB does "internally" now with the original $lookup syntax. By expressing in this form we can tailor the initial $match expression within the "sub-pipeline" ourselves.
In fact, as a true "aggregation pipeline" you can do just about anything you can do with an aggregation pipeline within this "sub-pipeline" expression, including "nesting" the levels of $lookup to other related collections.
Further usage is a bit beyond the scope of what the question here asks, but in relation to even "nested population" then the new usage pattern of $lookup allows this to be much the same, and a "lot" more powerful in it's full usage.
Working Example
The following gives an example using a static method on the model. Once that static method is implemented the call simply becomes:
Item.lookup(
{
path: 'tags',
query: { 'tags.tagName' : { '$in': [ 'funny', 'politics' ] } }
},
callback
)
Or enhancing to be a bit more modern even becomes:
let results = await Item.lookup({
path: 'tags',
query: { 'tagName' : { '$in': [ 'funny', 'politics' ] } }
})
Making it very similar to .populate() in structure, but it's actually doing the join on the server instead. For completeness, the usage here casts the returned data back to mongoose document instances at according to both the parent and child cases.
It's fairly trivial and easy to adapt or just use as is for most common cases.
N.B The use of async here is just for brevity of running the enclosed example. The actual implementation is free of this dependency.
const async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.Promise = global.Promise;
mongoose.set('debug', true);
mongoose.connect('mongodb://localhost/looktest');
const itemTagSchema = new Schema({
tagName: String
});
const itemSchema = new Schema({
dateCreated: { type: Date, default: Date.now },
title: String,
description: String,
tags: [{ type: Schema.Types.ObjectId, ref: 'ItemTag' }]
});
itemSchema.statics.lookup = function(opt,callback) {
let rel =
mongoose.model(this.schema.path(opt.path).caster.options.ref);
let group = { "$group": { } };
this.schema.eachPath(p =>
group.$group[p] = (p === "_id") ? "$_id" :
(p === opt.path) ? { "$push": `$${p}` } : { "$first": `$${p}` });
let pipeline = [
{ "$lookup": {
"from": rel.collection.name,
"as": opt.path,
"localField": opt.path,
"foreignField": "_id"
}},
{ "$unwind": `$${opt.path}` },
{ "$match": opt.query },
group
];
this.aggregate(pipeline,(err,result) => {
if (err) callback(err);
result = result.map(m => {
m[opt.path] = m[opt.path].map(r => rel(r));
return this(m);
});
callback(err,result);
});
}
const Item = mongoose.model('Item', itemSchema);
const ItemTag = mongoose.model('ItemTag', itemTagSchema);
function log(body) {
console.log(JSON.stringify(body, undefined, 2))
}
async.series(
[
// Clean data
(callback) => async.each(mongoose.models,(model,callback) =>
model.remove({},callback),callback),
// Create tags and items
(callback) =>
async.waterfall(
[
(callback) =>
ItemTag.create([{ "tagName": "movies" }, { "tagName": "funny" }],
callback),
(tags, callback) =>
Item.create({ "title": "Something","description": "An item",
"tags": tags },callback)
],
callback
),
// Query with our static
(callback) =>
Item.lookup(
{
path: 'tags',
query: { 'tags.tagName' : { '$in': [ 'funny', 'politics' ] } }
},
callback
)
],
(err,results) => {
if (err) throw err;
let result = results.pop();
log(result);
mongoose.disconnect();
}
)
Or a little more modern for Node 8.x and above with async/await and no additional dependencies:
const { Schema } = mongoose = require('mongoose');
const uri = 'mongodb://localhost/looktest';
mongoose.Promise = global.Promise;
mongoose.set('debug', true);
const itemTagSchema = new Schema({
tagName: String
});
const itemSchema = new Schema({
dateCreated: { type: Date, default: Date.now },
title: String,
description: String,
tags: [{ type: Schema.Types.ObjectId, ref: 'ItemTag' }]
});
itemSchema.statics.lookup = function(opt) {
let rel =
mongoose.model(this.schema.path(opt.path).caster.options.ref);
let group = { "$group": { } };
this.schema.eachPath(p =>
group.$group[p] = (p === "_id") ? "$_id" :
(p === opt.path) ? { "$push": `$${p}` } : { "$first": `$${p}` });
let pipeline = [
{ "$lookup": {
"from": rel.collection.name,
"as": opt.path,
"localField": opt.path,
"foreignField": "_id"
}},
{ "$unwind": `$${opt.path}` },
{ "$match": opt.query },
group
];
return this.aggregate(pipeline).exec().then(r => r.map(m =>
this({ ...m, [opt.path]: m[opt.path].map(r => rel(r)) })
));
}
const Item = mongoose.model('Item', itemSchema);
const ItemTag = mongoose.model('ItemTag', itemTagSchema);
const log = body => console.log(JSON.stringify(body, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri);
// Clean data
await Promise.all(Object.entries(conn.models).map(([k,m]) => m.remove()));
// Create tags and items
const tags = await ItemTag.create(
["movies", "funny"].map(tagName =>({ tagName }))
);
const item = await Item.create({
"title": "Something",
"description": "An item",
tags
});
// Query with our static
const result = (await Item.lookup({
path: 'tags',
query: { 'tags.tagName' : { '$in': [ 'funny', 'politics' ] } }
})).pop();
log(result);
mongoose.disconnect();
} catch (e) {
console.error(e);
} finally {
process.exit()
}
})()
And from MongoDB 3.6 and upward, even without the $unwind and $group building:
const { Schema, Types: { ObjectId } } = mongoose = require('mongoose');
const uri = 'mongodb://localhost/looktest';
mongoose.Promise = global.Promise;
mongoose.set('debug', true);
const itemTagSchema = new Schema({
tagName: String
});
const itemSchema = new Schema({
title: String,
description: String,
tags: [{ type: Schema.Types.ObjectId, ref: 'ItemTag' }]
},{ timestamps: true });
itemSchema.statics.lookup = function({ path, query }) {
let rel =
mongoose.model(this.schema.path(path).caster.options.ref);
// MongoDB 3.6 and up $lookup with sub-pipeline
let pipeline = [
{ "$lookup": {
"from": rel.collection.name,
"as": path,
"let": { [path]: `$${path}` },
"pipeline": [
{ "$match": {
...query,
"$expr": { "$in": [ "$_id", `$$${path}` ] }
}}
]
}}
];
return this.aggregate(pipeline).exec().then(r => r.map(m =>
this({ ...m, [path]: m[path].map(r => rel(r)) })
));
};
const Item = mongoose.model('Item', itemSchema);
const ItemTag = mongoose.model('ItemTag', itemTagSchema);
const log = body => console.log(JSON.stringify(body, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri);
// Clean data
await Promise.all(Object.entries(conn.models).map(([k,m]) => m.remove()));
// Create tags and items
const tags = await ItemTag.insertMany(
["movies", "funny"].map(tagName => ({ tagName }))
);
const item = await Item.create({
"title": "Something",
"description": "An item",
tags
});
// Query with our static
let result = (await Item.lookup({
path: 'tags',
query: { 'tagName': { '$in': [ 'funny', 'politics' ] } }
})).pop();
log(result);
await mongoose.disconnect();
} catch(e) {
console.error(e)
} finally {
process.exit()
}
})()

what you are asking for isn't directly supported but can be achieved by adding another filter step after the query returns.
first, .populate( 'tags', null, { tagName: { $in: ['funny', 'politics'] } } ) is definitely what you need to do to filter the tags documents. then, after the query returns you'll need to manually filter out documents that don't have any tags docs that matched the populate criteria. something like:
query....
.exec(function(err, docs){
docs = docs.filter(function(doc){
return doc.tags.length;
})
// do stuff with docs
});

Try replacing
.populate('tags').where('tags.tagName').in(['funny', 'politics'])
by
.populate( 'tags', null, { tagName: { $in: ['funny', 'politics'] } } )

Update: Please take a look at the comments - this answer does not correctly match to the question, but maybe it answers other questions of users which came across (I think that because of the upvotes) so I will not delete this "answer":
First: I know this question is really outdated, but I searched for exactly this problem and this SO post was the Google entry #1. So I implemented the docs.filter version (accepted answer) but as I read in the mongoose v4.6.0 docs we can now simply use:
Item.find({}).populate({
path: 'tags',
match: { tagName: { $in: ['funny', 'politics'] }}
}).exec((err, items) => {
console.log(items.tags)
// contains only tags where tagName is 'funny' or 'politics'
})
Hope this helps future search machine users.

After having the same problem myself recently, I've come up with the following solution:
First, find all ItemTags where tagName is either 'funny' or 'politics' and return an array of ItemTag _ids.
Then, find Items which contain all ItemTag _ids in the tags array
ItemTag
.find({ tagName : { $in : ['funny','politics'] } })
.lean()
.distinct('_id')
.exec((err, itemTagIds) => {
if (err) { console.error(err); }
Item.find({ tag: { $all: itemTagIds} }, (err, items) => {
console.log(items); // Items filtered by tagName
});
});

#aaronheckmann 's answer worked for me but I had to replace return doc.tags.length; to return doc.tags != null; because that field contain null if it doesn't match with the conditions written inside populate.
So the final code:
query....
.exec(function(err, docs){
docs = docs.filter(function(doc){
return doc.tags != null;
})
// do stuff with docs
});

Is It possible to use query projection on the same collection that has a $elemMatch projection?

I understand that you can limit the items in a subcollection array using $elemMatch as a projection. When using it as such it returns all fields of the subdocuments that match regardless if query projections are also specified.
Is it possible to limit the fields returned in the matching subdocuments? How would you do so?
Using version 3.8.9.
Given the simple schemas:
var CommentSchema = mongoose.Schema({
body: String,
created: {
by: {
type: String,
required: true
},
date: {
type: Date,
default: Date.now
}
}
});
var BlogSchema = mongoose.Schema({
title: String,
blog: String,
created: {
by: {
type: String,
required: true
},
date: {
type: Date,
default: Date.now
}
},
comments: [CommentSchema]
});
var Blog = mongoose.model('Blog',modelSchema);
Example
Blog.findOne({_id: id}, {_id: 1, comments: {$elemMatch: {'created.by': 'Jane'}, body: 1}}, function(err, blog) {
console.log(blog.toJSON());
});
// outputs:
{
_id: 532cb63e25e4ad524ba17102,
comments: [
_id: 53757456d9c99f00009cdb5b,
body: 'My awesome comment',
created: { by: 'Jane', date: Fri Mar 21 2014 20:34:45 GMT-0400 (EDT) }
]
}
// DESIRED OUTPUT
{
_id: 532cb63e25e4ad524ba17102,
comments: [
body: 'My awesome comment'
]
}

Yes there are two ways to do this. So you can either use the $elemMatch on the projection side as you already have, with slight changes:
Model.findById(id,
{ "comments": { "$elemMatch": {"created.by": "Jane" } } },
function(err,doc) {
Or just add to the query portion and use the positional $ operator:
Model.findOne(
{ "_id": id, "comments.created.by": "Jane" },
{ "comments.$": 1 },
function(err,doc) {
Either way is perfectly valid.
If you wanted something a little more involved than that, you can use the .aggregate() method and it's $project operator instead:
Model.aggregate([
// Still match the document
{ "$match": "_id": id, "comments.created.by": "Jane" },
// Unwind the array
{ "$unwind": "$comments" },
// Only match elements, there can be more than 1
{ "$match": "_id": id, "comments.created.by": "Jane" },
// Project only what you want
{ "$project": {
"comments": {
"body": "$comments.body",
"by": "$comments.created.by"
}
}},
// Group back each document with the array if you want to
{ "$group": {
"_id": "$_id",
"comments": { "$push": "$comments" }
}}
],
function(err,result) {
So the aggregation framework can be used for a lot more than simply aggregating results. It's $project operator gives you more flexibility than is available to projection using .find(). It also allows you to filter and return multiple array results, which is also something that cannot be done with projection in .find().