Develop Reference

return what was not found in array in MongoDB

say my database collection has
* user collection*
[
{id:'1'}
{id:'2'}
]
I have an array of object
[
{id:'1'}
{id:'2'}
{id:'3'}
]
I want the object that was not found in the collection.
I want
[
{id:'3'}
]
I'm currently have this
const records = await dbo
.collection('user collection')
.find({
'id': { $in: newArr },
})
.toArray();
I'm a bit stumped on what to do! ... hope someone can help Thanks!

Option 1:
Looks like this is what you need via the not in operation ( $nin ) when you need to check the not exisitng id in collection documents from provided array:
db.collection.aggregate([
{
$match: {
id: {
"$nin": [
1,
2
]
}
}
},
{
$group: {
_id: null,
"idnotIntheArray": {
$push: "$id"
}
}
}
])
Explained:
$match for any documents with id not in provided array.
$group all id's in an array
plaground1
Option 2:
And this is the option where you output only the array elements not existing in the collection:
db.collection.aggregate([
{
$group: {
_id: null,
ids: {
$push: "$id"
}
}
},
{
$project: {
missingFromCollection: {
"$setDifference": [
[
1,
5,
4
],
"$ids"
]
}
}
}
])
Explained:
Push all id elements from collection to array ids ( note this solution will not allow more then 16MB total size of id's )
Use $setDifference to identify the difference between the two arrays.
playground2

You can use this aggregation:
db.entity.aggregate([
{
$match : {
"myObjList.id" : 1
}
},
{
$unwind : "$myObjList"
},
{
$match : {
"myObjList.id" : 1
}
}
])
and my aggregation result:
{
"_id" : ObjectId("6225a0f78d435fd2845f1dd1"),
"myObjList" : {
"id" : 1
}
}

Use mongoDB $lookup to find documents in another collection not present inside an array

I'm using the aggregate framework to query a collection and create an array of active players (up until the last $lookup) after which I'm trying to use $lookup and $pipeline to select all the players from another collection (users) that are not present inside the activeUsers array.
Is there any way of doing this with my current setup?
Game.aggregate[{
$match: {
date: {
$gte: ISODate('2021-04-10T00:00:00.355Z')
},
gameStatus: 'played'
}
}, {
$unwind: {
path: '$players',
preserveNullAndEmptyArrays: false
}
}, {
$group: {
_id: '$players'
}
}, {
$group: {
_id: null,
activeUsers: {
$push: '$_id'
}
}
}, {
$project: {
activeUsers: true,
_id: false
}
}, {
$lookup: {
from: 'users',
'let': {
active: '$activeUsers'
},
pipeline: [{
$match: {
deactivated: false,
// The rest of the query works fine but here I would like to
// select only the elements that *aren't* inside
// the array (instead of the ones that *are* inside)
// but if I use '$nin' here mongoDB throws
// an 'unrecognized' error
$expr: {
$in: [
'$_id',
'$$active'
]
}
}
},
{
$project: {
_id: 1
}
}
],
as: 'users'
}
}]
Thanks

For negative condition use $not before $in operator,
{ $expr: { $not: { $in: ['$_id', '$$active'] } } }

Mongoose aggregate does not work with $or

I am working on a Chat application. My schema looks like this:
{
from: String,
to: String,
message: String,
attachment: {
name: String,
size: Number,
type: String,
},
unread: Boolean,
sent: Date,
seen: Date,
}
The following code works and returns the latest messages:
Query 1:
ChatDB.aggregate([
{ $match: {
$or: [
{ from, to },
{ from: to, to: from },
],
}},
{ $sort: { sent: -1 }},
{ $limit: messageBatchSize },
{ $sort: { sent: 1 }},
]);
But, when I try to paginate by including a timestamp in the query, it does not work anymore:
Query 2:
ChatDB.aggregate([
{ $match: {
sent: { $lt: new Date(beforeTimestamp) },
$or: [
{ from, to },
{ from: to, to: from },
],
}},
{ $sort: { sent: -1 }},
{ $limit: messageBatchSize },
{ $sort: { sent: 1 }},
]);
If I remove the $or portion and keep only the timestamp check on sent, things work, but (of course) it returns results for all users, which is not what I want:
Query 3:
ChatDB.aggregate([
{ $match: {
sent: { $lt: new Date(beforeTimestamp) },
}},
{ $sort: { sent: -1 }},
{ $limit: messageBatchSize },
{ $sort: { sent: 1 }},
]);
At first I thought it has got to do something with not converting the ids from string to ObjectId and changed my code to use Types.ObjectId accordingly. But that did not help even. I mean, Query 1 works correctly without any conversion.
Any idea what is going on? My mongoose version:
"mongoose": "^5.8.2",
Edit:
I tried running the query in mongo console and it returned the results correctly:
> db.chats.aggregate([
... {
... $match: {
... $or: [
... { from: '5f0319f87278d056876952d5', to: 'org' },
... { to: '5f0319f87278d056876952d5', from: 'org' },
... ],
... sent: { $lt: new Date('2020-07-08T17:05:34.288Z') }
... }
... },
... { $sort: { sent: -1 }},
... { $limit: 20 },
... { $sort: { sent: 1 }}
... ]);

I feel kinda stupid for posting this in the first place.
The problem turned out to be that the values in from and to were of type Types.ObjectId because they were being retrieved from a different collection.
The values stored in ChatDB were strings. Because of this, the query from mongo console worked fine (because I was providing string correctly) and the one with mongoose in the code did not work.
However, I still don't know why Query 1 worked.

Mongo find by sum of subdoc array

I'm trying to find stocks in the Stock collection where the sum of all owners' shares is less than 100. Here is my schema.
const stockSchema = new mongoose.Schema({
owners: [
{
owner: {
type: Schema.Types.ObjectId,
ref: "Owner"
},
shares: {
type: Number,
min: 0,
max: 100
}
}
]
}
const Stock = mongoose.model("Stock", stockSchema);
I've tried to use aggregate but it returns a single object computed over all stocks in the collection, as opposed to multiple objects with the sum of each stock's shares.
stockSchema.statics.getUnderfundedStocks = async () => {
const result = await Stock.aggregate([
{ $unwind: "$owners" },
{ $group: { _id: null, shares: { $sum: "$owners.shares" } } },
{ $match: { shares: { $lt: 100 } } }
]);
return result;
};
So, rather than getting:
[ { _id: null, shares: 150 } ] from getUnderfundedStocks, I'm looking to get:
[ { _id: null, shares: 90 }, { _id: null, shares: 60 } ].
I've come across $expr, which looks useful, but documentation is scarce and not sure if that's the appropriate path to take.
Edit: Some document examples:
/* 1 */
{
"_id" : ObjectId("5ea699fb201db57b8e4e2e8a"),
"owners" : [
{
"owner" : ObjectId("5ea62a94ccb1b974d40a2c72"),
"shares" : 85
}
]
}
/* 2 */
{
"_id" : ObjectId("5ea699fb201db57b8e4e2e1e"),
"owners" : [
{
"owner" : ObjectId("5ea62a94ccb1b974d40a2c72"),
"shares" : 20
},
{
"owner" : ObjectId("5ea62a94ccb1b974d40a2c73"),
"shares" : 50
},
{
"owner" : ObjectId("5ea62a94ccb1b974d40a2c74"),
"shares" : 30
}
]
}
I'd like to return an array that just includes document #1.

You do not need to use $group here. Simply use $project with $sum operator.
db.collection.aggregate([
{ "$project": {
"shares": { "$sum": "$owners.shares" }
}},
{ "$match": { "shares": { "$lt": 100 } } }
])
Or even you do not need to use aggregation here
db.collection.find({
"$expr": { "$lt": [{ "$sum": "$owners.shares" }, 100] }
})
MongoPlayground

nodejs how to dynamically manipulate your MongoDB aggregate in your controller?

I have a very loooong aggregate in my nodejs controller:
agentSaleModel.aggregate([
{
$match: {
$and:
[{
_id: { $in: req.body.propertyID },
active : true
}]
}
}, etc....
And it works great when I got elements in my req.body.propertyID
like ["property01","property02"] etc...
My problem is that I also want the aggregate to work when there are nothing in req.body.propertyID. (when its blank) - and then get all records.
This does not work:
agentSaleModel.aggregate([
{
$match: {
$and:
[{
_id: { $in: '' },
active : true
}]
}
}, etc....
So rather than doing an "if" with two huge sets of almost identical code:
if (req.body.propertyID) {
...the entire aggregate...
} else {
...the entire aggregate minus the _id: { $in: req.body.propertyID },...
}
Is there a smarter way to do this?
SOLUTION!! Thanks to FluffyNights :)
if (req.body.propertyID!='') {
var matchStr = {
$match: {
$and:
[{
_id: { $in: req.body.propertyID },
active : true
}]
}
}
} else {
var matchStr = {
$match: {
active : true
}
}
}
agentSaleModel.aggregate([ matchStr, etc..... (rest of pipeline)

you could do it like this:
let query = [
{
$match: {
$and:
[{
active : true
}]
}
}];
if(req.body.propertyID) {
query[0]["$match"]["$and"][0]["_id"] = { $in: req.body.propertyID };
}
agentSaleModel.aggregate(query, ...)
you could also use regex, like:
if(!req.body.properyID){
req.body.propertyID = [ ".*" ];
}
agentSaleModel.aggregate([
{
$match: {
$and:
[{
_id: { $in: req.body.propertyID },
active : true
}]
}
}, etc....
however, this might get slow.
Im not sure if passing null to $in would work the way you want, you could try it though.

What about trying to construct query before running it.
For example.
var query = req.body.propertyID ? { $and: [{_id: { $in: req.body.propertyID }, active : true}]} : {active : true}
agentSaleModel.aggregate([
{
$match: query
}, etc....
Hope this helps.

Here's an inline solution using computed property names:
$match: {
$and: [
{
_id: { [ req.body.propertyID ? '$in' : '$exists' ] : req.body.propertyID || true },
active: true,
},
],
}
When req.body.propertyID exists, the query becomes:
_id : { $in : req.body.propertyID }
If not:
_id : { $exists : true }
EDIT: this will also allow req.body.propertyID to equal "ALL" if you explicitly want to match all documents:
let selectAll = ! req.body.propertyID || req.body.propertyID === 'ALL';
const query = {
$match: {
$and: [
{
_id: { [ selectAll ? '$exists' : '$in' ] : selectAll || req.body.propertyID },
active: true,
},
],
},
};