I have this problem that I want to sort the result of a query based on the field values from another collection,
Problem: I want to first get the user 123 friends and then get their posts and then sort the post with the friends strength value,
I have this :
POST COLLECTON:
{
user_id: 8976,
post_text: 'example working',
}
{
user_id: 673,
post_text: 'something',
}
USER COLLECTON:
{
user_id: 123,
friends: {
{user_id: 673,strength:4}
{user_id: 8976,strength:1}
}
}
Based on the information you have retrieved from your user you essentially want to come out to an aggregation framework query that looks like this:
db.posts.aggregate([
{ "$match": { "user_id": { "$in": [ 673, 8976 ] } } },
{ "$project": {
"user_id": 1,
"post_text": 1,
"weight": {
"$cond": [
{ "$eq": [ "$user_id", 8976 ] },
1,
{ "$cond": [
{ "$eq": [ "$user_id", 673 ] },
4,
0
]}
]
}
}},
{ "$sort": { "weight": -1 } }
])
So why aggregation when this does not aggregate? As you can see, the aggregation framework does more than just aggregate. Here it is being used to "project" a new field into the document an populate it with a "weight" to sort on. This allows you to get the results back ordered by the value you want them to be sorted on.
Of course, you need to get from your initial data to this form in a "generated" way that you do do for any data. This takes a few steps, but here I'll present the JavaScript way to do it, which should be easy to convert to most languages
Also presuming your actual "user" looks more like this, which would be valid:
{
"user_id": 123,
"friends": [
{ "user_id": 673, "strength": 4 },
{ "user_id": 8976, "strength": 1 }
]
}
From an object like this you then construct the aggregation pipeline:
// user is the structure shown above
var stack = [];
args = [];
user.friends.forEach(function(friend) {
args.push( friend.user_id );
var rec = {
"$cond": [
{ "$eq": [ "user_id", friend.user_id ] },
friend.strength
]
};
if ( stack.length == 0 ) {
rec["$cond"].push(0);
} else {
var last = stack.pop();
rec["$cond"].push( last );
}
stack.push( rec );
});
var pipeline = [
{ "$match": { "user_id": { "$in": args } } },
{ "$project": {
"user_id": 1,
"post_text": 1,
"weight": stack[0]
}},
{ "$sort": { "weight": -1 } }
];
db.posts.aggregate(pipeline);
And that is all there is to it. Now you have some code to go through the list of "friends" for a user and construct another query to get all posts from those friends weighted by the "strength" value for each.
Of course you could do much the same things with a query for all posts by just removing or changing the $match, but keeping the "weight" projection you can "float" all of the "friends" posts to the top.
Related
I have a multilevel nested document (its dynamic and some levels can be missing but maximum 3 levels). I want to update all the children and subchildren routes if any. The scenario is same as in any Windows explorer, where all subfolders' route need to change when a parent folder route is changed. For eg. In the below example, If I am at route=="l1/l2a" and it's name needs to be edited to "l2c", then I will update it's route as route="l1/l2c and I will update all childrens' route to say "l1/l2c/l3a".
{
"name":"l1",
"route": "l1",
"children":
[
{
"name": "l2a",
"route": "l1/l2a",
"children":
[
{
"name": "l3a",
"route": "l1/l2a/l3a"
}]
},
{
"name": "l2b",
"route": "l1/l2b",
"children":
[
{
"name": "l3b",
"route": "l1/l2b/l3b"
}]
}
]
}
Currently I am able to go to a point and I am able to change its name and ONLY its route in the following manner:
router.put('/navlist',(req,res,next)=>{
newname=req.body.newName //suppose l2c
oldname=req.body.name //suppose l2a
route=req.body.route // existing route is l1/l2a
id=req.body._id
newroute=route.replace(oldname,newname); // l1/l2a has to be changed to l1/l2c
let segments = route.split('/');
let query = { route: segments[0]};
let update, options = {};
let updatePath = "";
options.arrayFilters = [];
for(let i = 0; i < segments.length -1; i++){
updatePath += `children.$[child${i}].`;
options.arrayFilters.push({ [`child${i}.route`]: segments.slice(0, i + 2).join('/') });
} //this is basically for the nested children
updateName=updatePath+'name'
updateRoute=updatePath+'route';
update = { $setOnInsert: { [updateName]:newDisplayName,[updateRoute]:newroute } };
NavItems.updateOne(query,update, options)
})
The problem is I am not able to edit the routes of it's children if any i.e it's subfolder route as l1/l2c/l3a. Although I tried using the $[] operator as follows.
updateChild = updatePath+'.children.$[].route'
updateChild2 = updatePath+'.children.$[].children.$[].route'
//update = { $set: { [updateChild]:'abc',[updateChild2]:'abc' } };
Its important that levels are customizable and thus I don't know whether there is "l3A" or not. Like there can be "l3A" but there may not be "l3B". But my code simply requires every correct path else it gives an error
code 500 MongoError: The path 'children.1.children' must exist in the document in order to apply array updates.
So the question is how can I apply changes using $set to a path that actually exists and how can I edit the existing route part. If the path exists, it's well and good and if the path does not exist, I am getting the ERROR.
Update
You could simplify updates when you use references.Updates/Inserts are straightforward as you can only the update target level or insert new level without worrying about updating all levels. Let the aggregation takes care of populating all levels and generating route field.
Working example - https://mongoplayground.net/p/TKMsvpkbBMn
Structure
[
{
"_id": 1,
"name": "l1",
"children": [
2,
3
]
},
{
"_id": 2,
"name": "l2a",
"children": [
4
]
},
{
"_id": 3,
"name": "l2b",
"children": [
5
]
},
{
"_id": 4,
"name": "l3a",
"children": []
},
{
"_id": 5,
"name": "l3b",
"children": []
}
]
Insert query
db.collection.insert({"_id": 4, "name": "l3a", "children": []}); // Inserting empty array simplifies aggregation query
Update query
db.collection.update({"_id": 4}, {"$set": "name": "l3c"});
Aggregation
db.collection.aggregate([
{"$match":{"_id":1}},
{"$lookup":{
"from":"collection",
"let":{"name":"$name","children":"$children"},
"pipeline":[
{"$match":{"$expr":{"$in":["$_id","$$children"]}}},
{"$addFields":{"route":{"$concat":["$$name","/","$name"]}}},
{"$lookup":{
"from":"collection",
"let":{"route":"$route","children":"$children"},
"pipeline":[
{"$match":{"$expr":{"$in":["$_id","$$children"]}}},
{"$addFields":{"route":{"$concat":["$$route","/","$name"]}}}
],
"as":"children"
}}
],
"as":"children"
}}
])
Original
You could make route as array type and format before presenting it to user. It will greatly simplify updates for you. You have to break queries into multiple updates when nested levels don’t exist ( ex level 2 update ). May be use transactions to perform multiple updates in atomic way.
Something like
[
{
"_id": 1,
"name": "l1",
"route": "l1",
"children": [
{
"name": "l2a",
"route": [
"l1",
"l2a"
],
"children": [
{
"name": "l3a",
"route": [
"l1",
"l2a",
"l3a"
]
}
]
}
]
}
]
level 1 update
db.collection.update({
"_id": 1
},
{
"$set": {
"name": "m1",
"route": "m1"
},
"$set": {
"children.$[].route.0": "m1",
"children.$[].children.$[].route.0": "m1"
}
})
level 2 update
db.collection.update({
"_id": 1
},
{
"$set": {
"children.$[child].route.1": "m2a",
"children.$[child].name": "m2a"
}
},
{
"arrayFilters":[{"child.name": "l2a" }]
})
db.collection.update({
"_id": 1
},
{
"$set": {
"children.$[child].children.$[].route.1": "m2a"
}
},
{
"arrayFilters":[{"child.name": "l2a"}]
})
level 3 update
db.collection.update({
"_id": 1
},
{
"$set": {
"children.$[].children.$[child].name": "m3a"
"children.$[].children.$[child].route.2": "m3a"
}
},
{
"arrayFilters":[{"child.name": "l3a"}]
})
I don't think its possible with arrayFilted for first level and second level update, but yes its possible only for third level update,
The possible way is you can use update with aggregation pipeline starting from MongoDB 4.2,
I am just suggesting a method, you can simplify more on this and reduce query as per your understanding!
Use $map to iterate the loop of children array and check condition using $cond, and merge objects using $mergeObjects,
let id = req.body._id;
let oldname = req.body.name;
let route = req.body.route;
let newname = req.body.newName;
let segments = route.split('/');
LEVEL 1 UPDATE: Playground
// LEVEL 1: Example Values in variables
// let oldname = "l1";
// let route = "l1";
// let newname = "l4";
if(segments.length === 1) {
let result = await NavItems.updateOne(
{ _id: id },
[{
$set: {
name: newname,
route: newname,
children: {
$map: {
input: "$children",
as: "a2",
in: {
$mergeObjects: [
"$$a2",
{
route: { $concat: [newname, "/", "$$a2.name"] },
children: {
$map: {
input: "$$a2.children",
as: "a3",
in: {
$mergeObjects: [
"$$a3",
{ route: { $concat: [newname, "/", "$$a2.name", "/", "$$a3.name"] } }
]
}
}
}
}
]
}
}
}
}
}]
);
}
LEVEL 2 UPDATE: Playground
// LEVEL 2: Example Values in variables
// let oldname = "l2a";
// let route = "l1/l2a";
// let newname = "l2g";
else if (segments.length === 2) {
let result = await NavItems.updateOne(
{ _id: id },
[{
$set: {
children: {
$map: {
input: "$children",
as: "a2",
in: {
$mergeObjects: [
"$$a2",
{
$cond: [
{ $eq: ["$$a2.name", oldname] },
{
name: newname,
route: { $concat: ["$name", "/", newname] },
children: {
$map: {
input: "$$a2.children",
as: "a3",
in: {
$mergeObjects: [
"$$a3",
{ route: { $concat: ["$name", "/", newname, "/", "$$a3.name"] } }
]
}
}
}
},
{}
]
}
]
}
}
}
}
}]
);
}
LEVEL 3 UPDATE: Playground
// LEVEL 3 Example Values in variables
// let oldname = "l3a";
// let route = "l1/l2a/l3a";
// let newname = "l3g";
else if (segments.length === 3) {
let result = await NavItems.updateOne(
{ _id: id },
[{
$set: {
children: {
$map: {
input: "$children",
as: "a2",
in: {
$mergeObjects: [
"$$a2",
{
$cond: [
{ $eq: ["$$a2.name", segments[1]] },
{
children: {
$map: {
input: "$$a2.children",
as: "a3",
in: {
$mergeObjects: [
"$$a3",
{
$cond: [
{ $eq: ["$$a3.name", oldname] },
{
name: newname,
route: { $concat: ["$name", "/", "$$a2.name", "/", newname] }
},
{}
]
}
]
}
}
}
},
{}
]
}
]
}
}
}
}
}]
);
}
Why separate query for each level?
You could do single query but it will update all level's data whenever you just need to update single level data or particular level's data, I know this is lengthy code and queries but i can say this is optimized version for query operation.
you can't do as you want. Because mongo does not support it. I can offer you to fetch needed item from mongo. Update him with your custom recursive function help. And do db.collection.updateOne(_id, { $set: data })
function updateRouteRecursive(item) {
// case when need to stop our recursive function
if (!item.children) {
// do update item route and return modified item
return item;
}
// case what happen when we have children on each children array
}
I want to create a mongo db view from two collections with a new value that is a sum of values from one of the collection according to an operation from another collection.
Below is the structure:
/* First collection */
{
"product": "test",
"labels" : [
{"code": "label1", "value": 42},
{"code": "label2", "value": 50}
]
}
/* Second collection */
{
"code": "label3",
"calculation" : [
{"label" : "label1", "operation":"+"},
{"label" : "label2", "operation":"-"}
]
}
In my aggregated collection i want a new field that would be label1 - label2.
{
"product" : "test",
"labels" : [
{"code": "label1", "value": 42},
{"code": "label2", "value": 50}
],
"vlabels" : [
{"code": "label3", "value": -8}
]
}
Although it is possible. I doubt it would be optimal, if you don't need further processing on the database, I suggest you do this at the application layer.
However, I have attempted to do this as an exercise. This approach would check only for the "-" operator and assign a negative value, other operator will use the existing value.
/* First collection: "products" */
/* Second collection: "vlabels" */
db.products.aggregate([
{
$lookup: {
from: "vlabels", // lookup calculation from vlabels
let: {
labels: "$labels"
},
pipeline: [
{
$set: {
calculation: {
$map: {
input: "$calculation", // map over calculation in vlabels
as: "calc",
in: {
operation: "$$calc.operation",
product: {
$arrayElemAt: [
{
$filter: {
input: "$$labels", // filter for matching product labels and get the first element using $arrayAlemAt to get the value
as: "label",
cond: {
$eq: ["$$calc.label", "$$label.code"]
}
}
},
0
]
}
}
}
}
}
},
{
$project: {
_id: false,
code: "$code",
value: {
$reduce: { // reducing by adding all values in calculation array, use negative value on "-" operator
input: "$calculation",
initialValue: 0,
in: {
$add: [
"$$value",
{
$cond: [
{
$eq: ["-", "$$this.operation"]
},
{
$multiply: [
-1,
{ $ifNull: ["$$this.product.value", 0] }
]
},
{ $ifNull: ["$$this.product.value", 0] }
]
}
]
}
}
}
}
}
],
as: "vlabels"
}
}
])
Mongo Playground
Is it possible to have facet to return as an object instead of an array? It seems a bit counter intuitive to need to access result[0].total instead of just result.total
code (using mongoose):
Model
.aggregate()
.match({
"name": { "$regex": name },
"user_id": ObjectId(req.session.user.id),
"_id": { "$nin": except }
})
.facet({
"results": [
{ "$skip": start },
{ "$limit": finish },
{
"$project": {
"map_levels": 0,
"template": 0
}
}
],
"total": [
{ "$count": "total" },
]
})
.exec()
Each field you get using $facet represents separate aggregation pipeline and that's why you always get an array. You can use $addFields to overwrite existing total with single element. To get that first item you can use $arrayElemAt
Model
.aggregate()
.match({
"name": { "$regex": name },
"user_id": ObjectId(req.session.user.id),
"_id": { "$nin": except }
})
.facet({
"results": [
{ "$skip": start },
{ "$limit": finish },
{
"$project": {
"map_levels": 0,
"template": 0
}
}
],
"total": [
{ "$count": "total" },
]
})
.addFields({
"total": {
$arrayElemAt: [ "$total", 0 ]
}
})
.exec()
You can try this as well
Model
.aggregate()
.match({
"name": { "$regex": name },
"user_id": ObjectId(req.session.user.id),
"_id": { "$nin": except }
})
.facet({
"results": [
{ "$skip": start },
{ "$limit": finish },
{
"$project": {
"map_levels": 0,
"template": 0
}
}
],
"total": [
{ "$count": "total" },
]
})
.addFields({
"total": {
"$ifNull": [{ "$arrayElemAt": [ "$total.total", 0 ] }, 0]
}
})
.exec()
imagine that you want to pass the result of $facet to the next stage, let's say $match. well $match accepts an array of documents as input and return an array of documents that matched an expression, if the output of $facet was just an element we can't pass its output to $match because the type of output of $facet is not the same as the type of input of $match ($match is just an example). In my opinion it's better to keep the output of $facet as array to avoid handling those types of situations.
PS : nothing official in what i said
I'm trying to put together a MongoDB query that will return results where a field equals a value that is present in a JavaScript array. I've put together the following code so far but it isn't working. I would like the number field in the collection to find a match within the versions array. I'm thinking that my "list" part using the $map under the first $project is wrong.
//example data
var versions = ['v1', 'v2', 'v3'];
function getResults(versions) {
var query =
{
"db": "db-name",
"collection": "collection-name",
"query":
[
{
$match: {
time: getStartTime(),
}
},
{
$project:
{
number: "$number",
session_id: "$session_id",
round_id: "$_id",
"list": {
"$map": {
"input": { "$filter": {
"input": versions,
"as": "versions",
"cond": {
"$eq": [ "$$versions.value", "$number" ]
}
}},
"as": "versions",
"in": "$$versions.value"
}
}
}
},
{
$group:
{
_id: "$session_id",
number: { $first: "$list" },
rounds: { $addToSet: "$round_id" }
}
},
{
$project:
{
number: "$number",
rounds: "$rounds"
}
}
]
}
return query;
}
Can you provide more info about you are triyng to do?
Maybe you can use filter by javascript array.
Here is an example code:
function isBigEnough(value) {
return value >= 10;
}
var filtered = [12, 5, 8, 130, 44].filter(isBigEnough);
// filtered is [12, 130, 44]
Maybe filter could be useful for you.
The structure of the objects stored in mongodb is the following:
obj = {_id: "55c898787c2ab821e23e4661", ingredients: [{name: "ingredient1", value: "70.2"}, {name: "ingredient2", value: "34"}, {name: "ingredient3", value: "15.2"}, ...]}
What I would like to do is retrieve all documents, which value of specific ingredient is greater than arbitrary number.
To be more specific, suppose we want to retrieve all the documents which contain ingredient with name "ingredient1" and its value is greater than 50.
Trying the following I couldn't retrieve desired results:
var collection = db.get('docs');
var queryTest = collection.find({$where: 'this.ingredients.name == "ingredient1" && parseFloat(this.ingredients.value) > 50'}, function(e, docs) {
console.log(docs);
});
Does anyone know what is the correct query to condition upon specific array element names and values?
Thanks!
You really don't need the JavaScript evaluation of $where here, just use basic query operators with an $elemMatch query for the array. While true that the "value" elements here are in fact strings, this is not really the point ( as I explain at the end of this ). The main point is to get it right the first time:
collection.find(
{
"ingredients": {
"$elemMatch": {
"name": "ingredient1",
"value": { "$gt": 50 }
}
}
},
{ "ingredients.$": 1 }
)
The $ in the second part is the postional operator, which projects only the matched element of the array from the query conditions.
This is also considerably faster than the JavaScript evaluation, in both that the evaluation code does not need to be compiled and uses native coded operators, as well as that an "index" can be used on the "name" and even "value" elements of the array to aid in filtering the matches.
If you expect more than one match in the array, then the .aggregate() command is the best option. With modern MongoDB versions this is quite simple:
collection.aggregate([
{ "$match": {
"ingredients": {
"$elemMatch": {
"name": "ingredient1",
"value": { "$gt": 50 }
}
}
}},
{ "$redact": {
"$cond": {
"if": {
"$and": [
{ "$eq": [ { "$ifNull": [ "$name", "ingredient1" ] }, "ingredient1" ] },
{ "$gt": [ { "$ifNull": [ "$value", 60 ] }, 50 ] }
]
},
"then": "$$DESCEND",
"else": "$$PRUNE"
}
}}
])
And even simplier in forthcoming releases which introduce the $filter operator:
collection.aggregate([
{ "$match": {
"ingredients": {
"$elemMatch": {
"name": "ingredient1",
"value": { "$gt": 50 }
}
}
}},
{ "$project": {
"ingredients": {
"$filter": {
"input": "$ingredients",
"as": "ingredient",
"cond": {
"$and": [
{ "$eq": [ "$$ingredient.name", "ingredient1" ] },
{ "$gt": [ "$$ingredient.value", 50 ] }
]
}
}
}
}}
])
Where in both cases you are effectively "filtering" the array elements that do not match the conditions after the initial document match.
Also, since your "values" are actually "strings" right now, you reaally should change this to be numeric. Here is a basic process:
var bulk = collection.initializeOrderedBulkOp(),
count = 0;
collection.find().forEach(function(doc) {
doc.ingredients.forEach(function(ingredient,idx) {
var update = { "$set": {} };
update["$set"]["ingredients." + idx + ".value"] = parseFloat(ingredients.value);
bulk.find({ "_id": doc._id }).updateOne(update);
count++;
if ( count % 1000 != 0 ) {
bulk.execute();
bulk = collection.initializeOrderedBulkOp();
}
})
]);
if ( count % 1000 != 0 )
bulk.execute();
And that will fix the data so the query forms here work.
This is much better than processing with JavaScript $where which needs to evaluate every document in the collection without the benefit of an index to filter. Where the correct form is:
collection.find(function() {
return this.ingredients.some(function(ingredient) {
return (
( ingredient.name === "ingredient1" ) &&
( parseFloat(ingredient.value) > 50 )
);
});
})
And that can also not "project" the matched value(s) in the results as the other forms can.
Try using $elemMatch:
var queryTest = collection.find(
{ ingredients: { $elemMatch: { name: "ingredient1", value: { $gte: 50 } } } }
);