Related
I have this pipeline :
let pipeline = [
{
$match: {
date: { $gte: new Date("2022-10-19"), $lte: new Date("2022-10-26") },
},
},
{
$group: {
_id: "$date",
tasks: { $push: "$$ROOT" },
},
},
{
$sort: { _id: -1 },
},
];
const aggregationData = await ScheduleTaskModel.aggregate(pipeline);
where i group all "tasks" between a date range by date and i get that result :
[
{
"date": "2022-10-21T00:00:00.000Z",
"tasks": [...tasks with this date]
},
{
"date": "2022-10-20T00:00:00.000Z",
"tasks": [...tasks with this date]
}
]
as you see i have "tasks" only for 2 dates in that range,what if i want all dates to appear even the ones with no tasks so it would be like this with empty arrays ?
[
{
"date": "2022-10-26T00:00:00.000Z",
"tasks": []
},
{
"date": "2022-10-25T00:00:00.000Z",
"tasks": []
},
{
"date": "2022-10-24T00:00:00.000Z",
"tasks": []
},
{
"date": "2022-10-23T00:00:00.000Z",
"tasks": []
},
{
"date": "2022-10-22T00:00:00.000Z",
"tasks": []
},
{
"date": "2022-10-21T00:00:00.000Z",
"tasks": [...tasks with this date]
},
{
"date": "2022-10-20T00:00:00.000Z",
"tasks": [...tasks with this date]
},
{
"date": "2022-10-19T00:00:00.000Z",
"tasks": []
},
]
i tried to use $densify but unfortunately it requires upgrading my mongoDb atlas cluster which is not possible..
The answer of #WernfriedDomscheitAnother has a downside of grouping together all the documents in the collection, creating one large document, while a document has a size limit. A variation on it, without this downside, can be:
$match only the relevant document, same as in your current query
Use $facet to handle the case of no relevant documents at all. This will allow you to group all the relevant documents as you did in your query, but to keep a working-document even if there are any.
Add the relevant dates inside an array (since we use $facet this will happen even if the first match is empty)
Concatenate the array of matched data with the array of empty entries, use the real-data first.
$unwind the separate the documents by date, and $group again by date to remove the duplicates.
Format the result
db.collection.aggregate([
{$match: {date: {$gte: new Date("2022-10-19"), $lte: new Date("2022-10-26")}}},
{$facet: {
data: [
{$group: {_id: "$date", tasks: {$push: "$$ROOT"}}},
{$project: {date: "$_id", tasks: 1}}
]
}},
{$addFields: {
dates: {$map: {
input: {$range: [0, 8]},
// maybe more dynamic with $dateDiff -> { $dateDiff: { startDate: new Date("2022-10-19"), endDate: new Date("2022-10-26") }, unit: "day" } }
in: {
date: {$dateAdd: {
startDate: ISODate("2022-10-19T00:00:00.000Z"),
unit: "day",
amount: "$$this"
}},
tasks: []
}
}}
}},
{$project: {data: {$concatArrays: ["$data", "$dates"]}}},
{$unwind: "$data"},
{$group: {_id: "$data.date", "tasks": {$first: "$data.tasks"}}},
{$project: { _id: 0, date: "$_id", tasks: 1 }},
{$sort: { date: -1 }},
])
See how it works on the playground example
New function $densify would be the simplest, of course. The manual way of doing it would be this one:
db.collection.aggregate([
{
$group: {
_id: null,
data: { $push: "$$ROOT" }
}
},
{
$set: {
dates: {
$map: {
input: { $range: [ 0, 8 ] }, // maybe more dynamic with $dateDiff -> { $dateDiff: { startDate: new Date("2022-10-19"), endDate: new Date("2022-10-26") }, unit: "day" } }
in: {
date: {
$dateAdd: {
startDate: ISODate("2022-10-19T00:00:00.000Z"),
unit: "day",
amount: "$$this"
}
}
}
}
}
}
},
{
$set: {
dates: {
$map: {
input: "$dates",
as: "d",
in: {
$mergeObjects: [
"$$d",
{
tasks: {
$filter: {
input: "$data",
cond: { $eq: [ "$$d.date", "$$this.date" ] }
}
}
}
]
}
}
}
}
},
{
$project: {
data: {
$map: {
input: "$dates",
in: {
$cond: {
if: { $eq: [ "$$this.tasks", [] ] },
then: "$$this",
else: { $first: "$$this.tasks" }
}
}
}
}
}
},
{ $unwind: "$data" },
{ $replaceWith: "$data" }
])
Mongo Playground
I have a collection of various documents similar to what is shown below as 3 objects.
{
comment:{
text_sentiment: "positive",
topic: "A"
}
}, // DOC-1
{
comment:{
text_sentiment: "negative",
topic: "A"
}}, // DOC-2
{
comment:{
text_sentiment: "positive",
topic: "B"
}},..//DOC-3 ..
I want to write an aggregation that returns results in the following structure:
{
topic: "A",
topicOccurance: 2,
sentiment: {
positive: 3,
negative: 2,
neutral: 0
}
},...
I have written an aggregation that is able to group for topic and text_sentiment but I do not know, how can I create a structure similar to the one shown above. Here is the aggregation that I created.
db.MyCollection.aggregate({
$match: {
_id: "xyz",
"comment.topic": {$exists: 1},
}
},{
$group: {
_id: {
topic: "$comment.topic",
text_sentiment: "$comment.text_sentiment"
},
total: {$sum: 1},
}
},{
$project: {
topic: {
name: "$_id.topic",
occurence: "$total"
},
sentiment: "$_id.text_sentiment"
}
},{
$sort: {"topic.occurence": -1}
})
It grouped by topic and sentiment, but the structure does not match the one above. How can I get a similar structure?
Answer 1
You need 2 $group stages.
$match
$group - Group by comment.topic and comment.topic and $sum.
$group - Group by _id.topic, $sum; and add text_sentiment and total from previous stage into text_sentiments via $push.
$project - Decorate output documents. Set sentiment with converting from text_sentiments array to key-value pair via $arrayToObject.
$sort
db.collection.aggregate([
{
$match: {
_id: "xyz",
"comment.topic": {
$exists: 1
},
}
},
{
$group: {
_id: {
topic: "$comment.topic",
text_sentiment: "$comment.text_sentiment"
},
total: {
$sum: 1
},
}
},
{
$group: {
_id: "$_id.topic",
total: {
$sum: 1
},
text_sentiments: {
$push: {
k: "$_id.text_sentiment",
v: "$total"
}
}
}
},
{
$project: {
topic: "$_id",
topicOccurance: "$total",
sentiment: {
"$arrayToObject": "$text_sentiments"
}
}
},
{
$sort: {
"topicOccurance": -1
}
}
])
Sample Mongo Playground (Answer 1)
Answer 2
As mentioned text_sentiment values are fixed, you can use the query below:
db.collection.aggregate([
{
$match: {
_id: "xyz",
"comment.topic": {
$exists: 1
},
}
},
{
$group: {
_id: "$comment.topic",
total: {
$sum: 1
},
text_sentiments: {
$push: "$comment.text_sentiment"
}
}
},
{
$project: {
topic: "$_id",
topicOccurance: "$total",
sentiment: {
"positive": {
$reduce: {
input: "$text_sentiments",
initialValue: 0,
in: {
$sum: [
"$$value",
{
"$cond": {
"if": {
$eq: [
"$$this",
"positive"
]
},
"then": 1,
"else": 0
}
}
]
}
}
},
"negative": {
$reduce: {
input: "$text_sentiments",
initialValue: 0,
in: {
$sum: [
"$$value",
{
"$cond": {
"if": {
$eq: [
"$$this",
"negative"
]
},
"then": 1,
"else": 0
}
}
]
}
}
},
"neutral": {
$reduce: {
input: "$text_sentiments",
initialValue: 0,
in: {
$sum: [
"$$value",
{
"$cond": {
"if": {
$eq: [
"$$this",
"neutral"
]
},
"then": 1,
"else": 0
}
}
]
}
}
}
}
}
},
{
$sort: {
"topicOccurance": -1
}
}
])
Disadvantage: When the text_sentiment value is added/removed, then you have to modify the query.
Sample Mongo Playground (Answer 2)
Answer 3
Another approach similar to Answer 2 is using $size and $filter to replace $reduce.
db.collection.aggregate([
{
$match: {
//_id: "xyz",
"comment.topic": {
$exists: 1
},
}
},
{
$group: {
_id: "$comment.topic",
total: {
$sum: 1
},
text_sentiments: {
$push: "$comment.text_sentiment"
}
}
},
{
$project: {
topic: "$_id",
topicOccurance: "$total",
sentiment: {
"positive": {
$size: {
$filter: {
input: "$text_sentiments",
cond: {
$eq: [
"$$this",
"positive"
]
}
}
}
},
"negative": {
$size: {
$filter: {
input: "$text_sentiments",
cond: {
$eq: [
"$$this",
"negative"
]
}
}
}
},
"neutral": {
$size: {
$filter: {
input: "$text_sentiments",
cond: {
$eq: [
"$$this",
"neutral"
]
}
}
}
},
}
}
},
{
$sort: {
"topicOccurance": -1
}
}
])
Sample Mongo Playground (Answer 3)
I’ve defined the following query which fetches me all items with an id which is in a given list of ids, and a status of either active or retracted.
const query = {
$and : [
{
$or: [
{
status: ‘active’,
},
{
status: ‘retracted’,
},
],
},
{
id: { $in: ids },
},
],
};
Each of these items has a parent_id field, which can either be null if the item does not have a parent, or can be the id of the parent.
I want my query to fetch all items with the ids I supply, as well as their parent items, if such a parent exists.
For example, if I supply the following IDs
[1,2,3]
and item 2 has a parent with id 5, while item 1 and 3 have parent_id set to null, I want my query to return the following items:
[1,2,3,5].
To achieve this I wrote the following query:
const collection = db.collection(‘myCollection’);
const data = await collection.aggregate([
{$match : query},
{
$lookup: {
from: ‘myCollection’,
let: { parentID: ‘$parent_id’},
pipeline: [
{
$match: {
$expr: {
$eq: [‘$id’, ‘$$parentID’],
},
},
},
as: ‘parent’,
},
},
]).sort(‘created_date’, ‘desc’).toArray();
return data;
However, this always returns null.
Sample Data:
[
{
id: 1,
parent_id: 3,
data: ‘bla bla’
},
{
id: 2,
parent_id: null,
data: ‘bla bla bla’
},
{
id: 3,
parent_id: null,
data: ‘bla’
}
]
Input: [1]
Output:
[
{
id: 1,
parent_id: 3,
data: ‘bla bla’
},
{
id: 3,
parent_id: null,
data: ‘bla’
}
]
The approach with $lookup being run upon same collection should work however it gives you a nested array so you need few additional stages to flatten such array and get all elements as on result set:
db.collection.aggregate([
{
$match: { id: { $in: [1] } }
},
{
$lookup: {
from: "collection",
localField: "parent_id",
foreignField: "id",
as: "parent"
}
},
{
$project: {
all: {
$concatArrays: [
"$parent",
[ "$$ROOT" ]
]
}
}
},
{
$project: {
"all.parent": 0
}
},
{
$unwind: "$all"
},
{
$replaceRoot: {
newRoot: "$all"
}
}
])
Mongo Playground
Your aggregation was malformed and lack some "]" for example closing the pipeline fied.
If you fix that the query works fine for me. Example
You can try this. The input array is [2,3] where 2 has parent id=1 and that is not in the input array. But the output array has the entry.
Working Playground
db.collection.aggregate([
{
$match: {
_id: {
$in: [
2,
3
]
}
}
},
{
$lookup: {
from: "collection",
localField: "p",
foreignField: "_id",
as: "parent"
}
},
{
$project: {
_id: 0,
id: {
$concatArrays: [
[
"$_id"
],
"$parent._id"
]
}
}
},
{
$unwind: "$id"
},
{
$sort: {
id: 1
}
}
])
For my DB, I wrote the following pipeline:
let orders = await Order.aggregate(
{
$unwind: "$candidate",
},
{
$lookup: {
from: "groups",
localField: "candidate.groupId",
foreignField: "_id",
as: "groupData",
},
},
{
$lookup: {
from: "users",
let: {
id: "$candidate.groupId",
},
pipeline: [
{ $match: { groupId: { $ne: null } } },
{
$match: {
$expr: { $in: ["$$id", "$groupId"] },
},
},
{ $project: { name: 1, email: 1, _id: 1 } },
],
as: "members",
},
},
{ $match: { "members._id": new ObjectId(req.userId) } },
{
$lookup: {
from: "users",
let: { ids: "$candidate.autonomousId" },
pipeline: [
{
$match: {
$expr: { $in: ["$_id", "$$ids"] },
},
},
{ $project: { name: 1, email: 1, _id: 1 } },
],
as: "candidate",
},
},
{
$project: {
groupData: 1,
members: 1,
candidate: 1,
stillAvailable: 1,
_id: 0,
},
}
).toArray();
The output was the expected...
{ candidate:
[ { _id: 601817dc2eeecd17db3a68f6,
name: 'Maria' },
{ _id: 601817ef2eeecd17db3a68f7,
name: 'Jose' } ],
groupData:
[ { _id: 606632403fffb851b8c41d12,
name: 'Giraia' } ],
members:
[ { _id: 601817dc2eeecd17db3a68f6,
name: 'Maria' },
{ _id: 601817ef2eeecd17db3a68f7,
name: 'Jose' },
{ _id: 60182cbb2b654330d2458f89,
name: 'Jonas'} ] }
The last step in the pipeline would be to compare the arrays, filter which members were not candidates and add them to the array stillAvailable. I tried in many ways but I couldn't achieve my goal with aggregation. The only solution I could find was to process the result of the incomplete pipeline on my backend. The code is:
orders.forEach(
(order) =>
(order.stillAvailable = order.members.filter(
(autonomous) =>
!order.candidate.some((el) => {
return el._id.toString() === autonomous._id.toString();
})
))
);
With that, I reach the expected output...
{ candidate:
[ { _id: 601817dc2eeecd17db3a68f6,
name: 'Maria' },
{ _id: 601817ef2eeecd17db3a68f7,
name: 'Jose' } ],
groupData:
[ { _id: 606632403fffb851b8c41d12,
name: 'Giraia' ],
members:
[ { _id: 601817dc2eeecd17db3a68f6,
name: 'Maria' },
{ _id: 601817ef2eeecd17db3a68f7,
name: 'Jose' },
{ _id: 60182cbb2b654330d2458f89,
name: 'Jonas' ],
stillAvailable:
[ { _id: 60182cbb2b654330d2458f89,
name: 'Jonas' ] }
The problem is to better compartmentalize my code, it would be necessary to realize the last step (done with javascript on my backend) as one more step on the pipeline. Does anyone have an idea how to reach that?
After I wrote the question here, somehow the idea was better structured and I achieved the result, using $map and one more level of $lookup! I left the answer documented here in case someone falls into the same issue.
let orders = await Order.aggregate(
{
$unwind: "$candidate",
},
{
$lookup: {
from: "groups",
localField: "candidate.groupId",
foreignField: "_id",
as: "groupData",
},
},
{
$lookup: {
from: "users",
let: {
id: "$candidate.groupId",
},
pipeline: [
{ $match: { groupId: { $ne: null } } },
{
$match: {
$expr: { $in: ["$$id", "$groupId"] },
},
},
{ $project: { name: 1, email: 1, _id: 1 } },
],
as: "members",
},
},
{ $match: { "members._id": new ObjectId(req.userId) } },
{
$lookup: {
from: "users",
let: { ids: "$candidate.autonomousId" },
pipeline: [
{
$match: {
$expr: { $in: ["$_id", "$$ids"] },
},
},
{ $project: { name: 1, email: 1, _id: 1 } },
],
as: "candidate",
},
},
{
$project: {
groupData: 1,
members: 1,
candidate: 1,
_id: 0,
stillAvailable: {
$setDifference: [
{
$map: {
input: "$members",
as: "member",
in: "$$member._id",
},
},
{
$map: {
input: "$candidate",
as: "el",
in: "$$el._id",
},
},
],
},
},
},
{
$lookup: {
from: "users",
let: {
ids: "$stillAvailable",
},
pipeline: [
{
$match: {
$expr: { $in: ["$_id", "$$ids"] },
},
},
{ $project: { name: 1, email: 1, _id: 1 } },
],
as: "stillAvailable",
},
}
).toArray();
I'm working on a feature that recursively looks up a thread of comments using $graphLookUp, and I almost have it. (albeit in a somewhat convoluted way but it is working!)
The last step I need is the following:
instead of having the nested posteriorThread as a property of the root array ($$ROOT), just merge it onto the root itself.
AGGREGATION CODE:
const posteriorThread = await Comment.aggregate([
{
$match: {
_id: post.threadDescendant
}
},
{
$graphLookup: {
from: 'baseposts',
startWith: '$threadDescendant',
connectFromField: 'threadDescendant',
connectToField: '_id',
as: 'posteriorThread'
}
},
{
$unwind: '$posteriorThread'
},
{
$sort: { 'posteriorThread.depth': 1 }
},
{
$group: {
_id: '$_id',
posteriorThread: { $push: '$posteriorThread' },
root: { $first: '$$ROOT' }
}
},
{
$project: {
'root.posteriorThread': 0
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
{
posteriorThread: '$posteriorThread'
},
'$root'
]
}
}
}
]);
CURRENT OUTPUT
OUTPUT: posteriorThread
[
{
_id: '5f7eab40575e6fc56ee07604',
onModel: 'BasePost',
depth: 1,
user: '5f5da45245c07cc06e51b09f',
text: 'thread 0',
isThread: true,
threadDescendant: '5f7eabad575e6fc56ee07607',
posteriorThread: [
{
_id: '5f7eabad575e6fc56ee07607',
onModel: 'Comment',
depth: 2,
user: '5f5da45245c07cc06e51b09f',
text: 'thread 1',
isThread: true,
threadDescendant: '5f7eac82575e6fc56ee07609'
},
{
_id: '5f7eac82575e6fc56ee07609',
onModel: 'Comment',
depth: 3,
user: '5f5da45245c07cc06e51b09f',
text: 'thread 2',
isThread: true
}
]
}
];
DESIRED OUTPUT
OUTPUT: posteriorThread
[
{
_id: '5f7eab40575e6fc56ee07604',
onModel: 'BasePost',
depth: 1,
user: '5f5da45245c07cc06e51b09f',
text: 'thread 0',
isThread: true,
threadDescendant: '5f7eabad575e6fc56ee07607'
},
{
_id: '5f7eabad575e6fc56ee07607',
onModel: 'Comment',
depth: 2,
user: '5f5da45245c07cc06e51b09f',
text: 'thread 1',
isThread: true,
threadDescendant: '5f7eac82575e6fc56ee07609'
},
{
_id: '5f7eac82575e6fc56ee07609',
onModel: 'Comment',
depth: 3,
user: '5f5da45245c07cc06e51b09f',
text: 'thread 2',
isThread: true
}
];
I could accomplish this after the aggregation in regular js, but I would prefer to do it all in the aggregation.
The part that needs to be replaced is the mergeObjects bit and replaced with something else or the group aggregation and taking a different strategy, but I'm not sure what to put in it's place.
Also, if you have any other suggestions to make this cleaner, I'm all ears.
Thanks in advance.
Its really challenging. Atleast for me. And really very interesting case. Lets try my solution. Hope it works..
db.test.aggregate([
// PREVIOSU STEPS YOU ALREADY DID
{
$group: {
_id: "$_id",
items: {$push: "$$ROOT"},
subItems: {$first: "$posteriorThread"}
}
},
{
$project: {
"items.posteriorThread": 0
}
},
{
$addFields: {
allItems: {
$concatArrays: ["$items", "$subItems"]
}
}
},
{
$group: {
_id: null,
mergedItems: {$push: "$allItems"}
}
},
{
$unwind: "$mergedItems"
},
{
$unwind: "$mergedItems"
},
{
$replaceRoot: {
newRoot: "$mergedItems"
}
}
])
Thanks to #Sunil K Samanta for steering me in the right direction. It's not the prettiest solution, but it does give me the right solution.
const posteriorThread = await Comment.aggregate([
{
$match: {
_id: post.threadDescendant
}
},
{
$graphLookup: {
from: 'baseposts',
startWith: '$threadDescendant',
connectFromField: 'threadDescendant',
connectToField: '_id',
as: 'posteriorThread'
}
},
{
$unwind: '$posteriorThread'
},
{
$sort: { 'posteriorThread.depth': 1 }
},
{
$group: {
_id: '$_id',
items: { $push: '$$ROOT.posteriorThread' },
root: { $push: '$$ROOT' },
},
},
{
$project: {
items: 1,
root: { $slice: ['$$ROOT.root', 0, 1] },
},
},
{
$project: {
'root.posteriorThread': 0,
},
},
{
$addFields: {
allItems: {
$concatArrays: ['$root', '$items'],
},
},
},
{
$replaceRoot: {
newRoot: {
full_posterior: '$$ROOT.allItems',
},
},
},
])
)[0].full_posterior;