Develop Reference

Consolidate two mongodb find and aggregate into one

I have two mongoDB queries one is aggregate and another one is a find.
They are coded in a way that if "aggregate" query gives result then "find" query doesn't run, otherwise, if "aggregate" gives no result then find query runs.
In the following way:-
var pipeline1 = [{
$match: { "user_id": "123" } //dynamic value based on request
}, {
$lookup: {
from: "config_rules",
localField: "group_id",
foreignField: "rule_type_value", //this field has group id mapped || or can be null
as: "rule"
}
},
{
$unwind: "$rule"
},{
$match:{ "rule.configtype": "profile" } //dynamic value based on request
}];
db.getCollection("user_group_mapping").aggregate(pipeline);
If the above aggregate gives a result then, the same is returned. or else we run the following find query to get config rule for the general user, and return it
var query = {
$and: [
{ rule_type_value: null }, //null for general user rules
{ configtype: "profile" }
]
}
db.getCollection("config_rules").find(query)
In simple words for a request, we check if the requester is in a group if yes, then we return config rule based on this group,
If the requester is not in any group then we return general config rule.
So my query is as seen above these are two different query running on different collection, and requires two separate mongo calls. Can I somehow combine these queries into 1 query?,
Like- If for a given user he is in a group return group-specific config or return general config rule.
I want to combine these so that in my code I will need to make only one DB call(this db call itself has both query consolidated in one) instead of two.
Sample document in user_group_mapping collection
{ "user_id": "123",
"group_id": "beta_users"
},
{ "user_id": "213",
"group_id": "alpha_testers";
}
Sample data in config_rules :
{ "rule_type_value":"beta_users",
"configType": "help",
"configVersion": "1.1"
},
{ "rule_type_value":null,
"configType": "help",
"configVersion": "1.0"
},
{ "rule_type_value":"alpha_testers",
"configType": "help",
"configVersion": "1.3"
}
Sample Input:
Req 1 user_id: "123"
configType: "help"
Req 2 user_id : "678"
configType: "help"
Sample output: (I have only written rule content for simplicity)
Req 1 config v1.1 will be returned
{ "rule_type_value":"beta_users",
"configType": "help",
"configVersion": "1.1"
}
Req 2 v1.0 will be returned
{ "rule_type_value":null,
"configType": "help",
"configVersion": "1.0"
}

try:
https://mongoplayground.net/p/m3HxBQIuqpS
please set configType at line 23 and set user_id at line 31
db.config_rules.aggregate([
{
$lookup: {
from: "user_group_mapping",
localField: "rule_type_value",
foreignField: "group_id",
as: "rule"
}
},
{
$addFields: {
"ruleCount": {
$size: "$rule",
},
"user_id": {
$first: "$rule.user_id"
}
}
},
{
$match: {
"configType": "help"
}
},
{
$match: {
$or: [
{
user_id: {
$eq: "678"//123 or 678
}
},
{
user_id: {
$exists: false
}
}
]
}
},
{
$sort: {
"ruleCount": -1
}
},
{
$limit: 1
},
{
$project: {
"_id": 0,
"rule_type_value": 1,
"configType": 1,
"configVersion": 1
}
}
])

MongoDB aggregation does not have flow control, and it will not execute subsequent stages if there are no documents output from a stage.
If you want to retrieve 1 of 2 possible values from the linked collection, change the $lookup stage so that all potential documents are selected, and filter the returned list afterward. Perhaps something similar to:
[
{$match: { "user_id": "123" }},
{$lookup: {
from: "config_rules",
let: {targetgroup: "$group_id"},
pipeline: [{$match:{
configtype: "profile",
$or:[
{$expr:{$eq:["$rule_type_value","$$targetgroup"]}},
{ rule_type_value: null, }
]
}}],
as: "rule"
}},
{$set: {
rule: {$cond: {
if: {$in: ["$group_id", "$rule.rule_type_value"]},
then: {$filter: {
input: "$rule",
cond: {$eq: ["$group_id", "$$this.rule_type_value"]}
}},
else: "$rule"
}},

MongoDB aggregation - combine multiple values of a document from collection A to lookup a single value within a document from collection B

Say I have the following two collections, sites and webpages. I'm trying to understand how to create an aggregation that'll allow me to combine values of a document from the sites collection and use that to lookup a value from the webpages collection. In addition, I need to prepend the combined values with a string.
// sites collection
[
{ "_id" : 3, "host" : "www.example-foo.com", "path": "/bar", "hasVisited": false },
]
// webpages collection
[
{ "_id" : 5, "url" : "https://www.example-foo.com/bar" },
{ "_id" : 8, "url" : "https://www.fizz.com/buzz" },
]
Without an aggregation I would do something like the following.
const site = await db.sites.findOne({ hasVisited: { $eq: false } });
const pages = await db.webpages.find({
url: `https://${site.host}${site.path}`, // <--- how to construct this in a lookup aggregation? string + value + value
});
// pages = [{ "_id" : 5, "url" : "https://www.example-foo.com/bar" }]

This is like translation of your code with the 2 find queries in 1 using $lookup
Query
first findOne is the $match and the $limit 1
$set url is to make the string concat
second find is to do the $lookup (with the 1 site from above stages)
*if you want to do it for more than 1 sites remove the limit, and project more fields, to know where this pages belong to(which site)
Test code here
db.sites.aggregate([
{
"$match": {
"hasVisited": {
"$eq": false
}
}
},
{
"$limit": 1
},
{
"$set": {
"url": {
"$concat": [
"https://",
"$host",
"$path"
]
}
}
},
{
"$lookup": {
"from": "webpages",
"localField": "url",
"foreignField": "url",
"as": "pages"
}
},
{
"$project": {
"_id": 0,
"pages": 1
}
}
])

Adding Objects to Array of Objects in a Document using Mongoose.js

I've looked in stackoverflow, however I've not found the answer. I'm trying to add Object to a New(empty) Array in my local mongodb that is not a duplicate. I also want to update this Array with other Objects.
I've looked at $push and $addToSet, the examples are using an "id" (_id) which wouldn't be created until I add my first Object.
I'm using Node.js, Mongoose.js, Mongodb, Express.js.
My Schema is:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var barSchema = new Schema({
location: [{
name: String,
city: String,
total: Number
}]
});
var Bar = mongoose.model('Bar', barSchema);
module.exports = Bar;
I've tried to use this;
var newBar = bar({
location: [{ "name": req.body.bar, "city": req.body.city, "total": 0 }] });
newBar.save(function(err) {
if (err) throw err;
});
I've also used the $push with success but in this case I've not got an "id"
user.findByIdAndUpdate(req.user._id, { $push: {
barlist: { "name": req.body.bar,
"rsvp": true } } },
function(err, user) { });
Which gives back this;
{
"_id" : ObjectId("######"),
"location" : [
{
"name" : "1st Bar",
"city" : "Boston",
"total" : 0,
"_id" : ObjectId("#########")
}
],
"__v" : 0
}
{
"_id" : ObjectId("######"),
"location" : [
{
"name" : "2nd Bar",
"city" : "Boston",
"total" : 0,
"_id" : ObjectId("#########")
}
],
"__v" : 0
}
However I am trying to get this;
{
"_id" : ObjectId("#######"),
"location" : [
{
"name" : "Biddy Early's",
"city" : "Boston",
"total" : 0
},
{
"name" : "Some Bar Name",
"city" : "Boston",
"total" : 0
}
]
}

Please know there may be better ways to do this, but... The first thing you want to do in this case is create a new instance of your schema or 'model'. Based on what your code looks like you might want to consider something like this;
var newBar = bar({
"city": req.body.city,
location: [{
"bar": req.body.bar,
"total": 0
}]
});
newBar.save(function(err) {
if (err) throw err;
});
Since if you are looking for bars or restaurants in the same city you might want to have a common key/value pair to '.find()' or '.update()' depending on what you want to do with it.
From here, you will want to look into what you mentioned before with '$addToSet' so that you wouldn't be adding duplicate bars to your 'location' array. So for instance;
bar.findOneAndUpdate({'city':req.body.city},
{'$addToSet': { location: {
'bar': req.body.bar, 'total': 0 } } },
function(err, b) {
if (err) throw err;
console.log(b);
});
Consider using a strategy like if/else to determine if the city name exists, if it does then you would utilize the '$addToSet'. Else, if it didn't exist you would utilize the new model like the example I used.

Match Documents where all array members do not contain a value

MongoDB selectors become quickly complicated, especially when you come from mySQL using JOIN and other fancy keywords. I did my best to make the title of this question as clear as possible, but failed miserably.
As an example, let a MongoDB collection have the following schema for its documents:
{
_id : int
products : [
{
qte : int
status : string
},
{
qte : int
status : string
},
{
qte : int
status : string
},
...
]
}
I'm trying to run a db.collection.find({ }) query returning documents where all products do not have the string "finished" as status. Please note that the products array has a variable length.
We could also say we want all documents that has at least one product with a status that is not "finished".
If I were to run it as a Javascript loop, we would have something like the following :
// Will contain queried documents
var matches = new Array();
// The documents variable contains all documents of the collection
for (var i = 0, len = documents.length; i < len; i++) {
var match = false;
if (documents[i].products && documents[i].products.length !== 0) {
for (var j = 0; j < documents[i].products; j++) {
if (documents[i].products[j].status !== "finished") {
match = true;
break;
}
}
}
if (match) {
matches.push(documents[i]);
}
}
// The previous snippet was coded directly in the Stack Overflow textarea; I might have done nasty typos.
The matches array would contain the documents I'm looking for. Now, I wish there would be a way of doing something similar to collection.find({"products.$.status" : {"$ne":"finished"}}) but MongoDB hates my face when I do so.
Also, documents that do not have any products need to be ignored, but I already figured this one out with a $and clause. Please note that I need the ENTIRE document to be returned, not just the product array. If a document has products that are not "finished", then the entire document should be present. If a document has all of its products set at "finished", the document is not returned at all.
MongoDB Version: 3.2.4
Example
Let's say we have a collection that contains three documents.
This one would match because one of the status is not "finished".
{
_id : 1,
products : [
{
qte : 10,
status : "finished"
},
{
qte : 21,
status : "ongoing"
},
]
}
This would not match because all statuses are set to "finished"
{
_id : 2,
products : [
{
qte : 35,
status : "finished"
},
{
qte : 210,
status : "finished"
},
{
qte : 2,
status : "finished"
},
]
}
This would also not match because there are no products. It would also not match if the products field was undefined.
{
_id : 3,
products : []
}
Again, if we ran the query in a collection that had the three documents in this example, the output would be:
[
{
_id : 1,
products : [
{
qte : 10,
status : "finished"
},
{
qte : 21,
status : "ongoing"
},
]
}
]
Only the first document gets returned because it has at least one product that doesn't have a status of "finished", but the last two did not make the cut since they either have all their products' statuses set as "finished", or don't have any products at all.

Try following query. It's fetching documents where status is not equals to "finished"
Note: This query will work with MongoDB 3.2+ only
db.collection.aggregate([
{
$project:{
"projectid" : 1,
"campname" : 1,
"campstatus" : 1,
"clientid" : 1,
"paymentreq" : 1,
products:{
$filter:{
input:"$products",
as: "product",
cond:{$ne: ["$$product.status", "finished"]}
}
}
}
},
{
$match:{"products":{$gt: [0, {$size:"products"}]}}
}
])

You need .aggregate() rather than .find() here. That is the only way to determine if ALL elements actually don't contain what you want:
// Sample data
db.products.insertMany([
{ "products": [
{ "qte": 1 },
{ "status": "finished" },
{ "status": "working" }
]},
{ "products": [
{ "qte": 2 },
{ "status": "working" },
{ "status": "other" }
]}
])
Then the aggregate operation with $redact:
db.products.aggregate([
{ "$redact": {
"$cond": {
"if": {
"$anyElementTrue": [
{ "$map": {
"input": "$products",
"as": "product",
"in": {
"$eq": [ "$$product.status", "finshed" ]
}
}}
]
},
"then": "$$PRUNE",
"else": "$$KEEP"
}
}}
])
Or alternately you can use the poorer and slower cousin with $where
db.products.find(function(){
return !this.products.some(function(product){
return product.status == "finished"
})
})
Both return just the one sample document:
{
"_id" : ObjectId("56fb4791ae26432047413455"),
"products" : [
{
"qte" : 2
},
{
"status" : "working"
},
{
"status" : "other"
}
]
}
So the $anyElementTrue with the $map input or the .some() are basically doing the same thing here and evaluating if there was any match at all. You use the "negative" assertion to "exclude" documents that actually find a match.

MongoDB Retrieve a subset of an array in a collection by specifying two fields which should match

I'm using this structure to store conversations & messages:
{ "_id" : ObjectId( "4f2952d7ff4b3c36d700000d" ),
"messages" : [
{ "_id" : ObjectId( "4f2952d7ff4b3c36d700000c" ),
"sender" : "4f02f16f0364c024678c0e5f",
"receiver" : "4f02f16f0364c024678c0e61",
"receiver_deleted" : "true",
"sender_deleted" : "true",
"body" : "MSG 1",
"timestamp" : "2012-02-01T14:57:27Z" },
{ "_id" : ObjectId( "4f2952daff4b3c36d700000e" ),
"sender" : "4f02f16f0364c024678c0e61",
"receiver" : "4f02f16f0364c024678c0e5f",
"body" : "MSG 2",
"timestamp" : "2012-02-01T14:57:30Z" },
{ "_id" : ObjectId( "4f295305ff4b3c36d700000f" ),
"sender" : "4f02f16f0364c024678c0e5f",
"receiver" : "4f02f16f0364c024678c0e61",
"body" : "TEST",
"timestamp" : "2012-02-01T14:58:13Z" } ],
"participants" : [
"4f02f16f0364c024678c0e5f",
"4f02f16f0364c024678c0e61" ],
"type" : "chat" }
When one of the sender or receiver does delete a specific message, receiver_deleted or sender_deleted gets added to the message (as you see in the first message).
Now how can I fetch a conversation with only the messages in it which haven't the sender/receiver deleted flag set?
First I tried like this:
db.conversations.find({
"_id": ObjectId("4f2952d7ff4b3c36d700000d"),
"participants": {"$in": ["4f02f16f0364c024678c0e5f"]},
"$or": [
{
"$and": [{"messages.sender": "4f02f16f0364c024678c0e5f"}, {"messages.sender_deleted": {"$exists": false}}]
},
{
"$and": [{"messages.receiver": "4f02f16f0364c024678c0e5f"}, {"messages.receiver_deleted": {"$exists": false}}]
}
]
})
But this doesn't work. I also tried with $elemMatch like this:
db.conversations.find({
"_id": ObjectId("4f2952d7ff4b3c36d700000d"),
"participants": {"$in": ["4f02f16f0364c024678c0e5f"]},
"$or": [
{
"messages": {
"$elemMatch": {"sender": "4f02f16f0364c024678c0e5f", "sender_deleted": {"$exists": False}}
}
},
{
"messages": {
"$elemMatch": {"receiver": "4f02f16f0364c024678c0e5f", "receiver_deleted": {"$exists": False}}
}
}
]
})
And a couple of other options with trying $and instead of $or etc. but it doesn't work.. Either it returns nothing or the whole conversation regardless of the receiver/sender deleted fields.
Thank you,
Michael

It is not currently possible to retrieve a subset of an array with mongodb. You will always get the whole document back if there is a match, or nothing if there is no match. $slice allows you to return a subset, but that's based on a starting and stopping index (which is not what you want - as you want to return only the matching messages in the array).
The feature you are describing has been logged and requested here: https://jira.mongodb.org/browse/SERVER-828

Since version 2.2 Aggregation Framework is available. You could perform your query like this:
db.expose.aggregate(
//Find the Documents which contains the desired criteria (document level)
{
$match: {
$or: [
{
"messages.sender_deleted": "true"
},
{
"messages.receiver_deleted": "true"
}]}},
//Peels off the elements of messages array individually
{
$unwind: "$messages"
},
//Perform the same find method, now in the embed collection level
{
$match: {
$or: [
{
"messages.sender_deleted": "true"
},
{
"messages.receiver_deleted": "true"
}]}},
//Select what to show as the result: in this case the Document id and the messages array
{
$group: {
_id: "$_id",
messages: {
$push: "$messages"
}}});
The first match is not required, but is better to filter out as much as possible in the beginning.