First of all the function :
static async getInitRegistrationMetric(account) {
// we want only full day so we exclude current date
const exclude = DateTime.now()
.setZone('utc')
.startOf('day')
.toISODate();
// this count the number of client created by day.
const groupByDate = {
$group: {
_id: {
$dateToString: { format: '%Y-%m-%d', date: '$createdAt' },
},
count: { $sum: 1 },
},
};
// this is a way to rename (_id, count) to (date, value)
const renameData = {
$project: {
_id: 0,
date: '$_id',
value: '$count',
},
};
// this is done to filter data, I want to clean the null date and the today result
const excludeTodayAndNull = {
$match: {
$and: [
{
date: {
$ne: exclude,
},
},
{
date: {
$ne: null,
},
},
],
},
};
// account is the mongoose model.
return account.aggregate([groupByDate, renameData, excludeTodayAndNull]);
}
this code will produce data like this:
const data = [
{ date: '2000-10-01', value: 50 },
{ date: '2000-10-03', value: 12 },
{ date: '2000-10-07', value: 112 },
];
the problem is I don't have value for the 2nd, 4th, 5th and 6th of the month. My idea was to force mongo to "create" void valid for the other days, like this:
const data = [
{ date: '2000-10-01', value: 50 },
{ date: '2000-10-02', value: 0 },
{ date: '2000-10-03', value: 12 },
{ date: '2000-10-04', value: 0 },
{ date: '2000-10-05', value: 0 },
{ date: '2000-10-06', value: 0 },
{ date: '2000-10-07', value: 112 },
];
How can I ask "aggregate" to fill the gap between significant dates with data with 0 as a value ?
Thanks
PS: I already did it by code in js but it looks heavy and ugly. I try to do it cleaner.
You could use the $fill or $densify operators to fill with zeros if you're running a recent enough version of MongoDB:
https://www.mongodb.com/docs/manual/reference/operator/aggregation/densify/
https://www.mongodb.com/docs/manual/reference/operator/aggregation/fill/
e.g
$fill:
{
sortBy: { date: 1 },
output:
{
"value": 0
}
}
Related
I am trying to apply a group by operation based on month from field From_Date and then calculate the sum of Gross_Amount, Net_Amount and Tax_Amount. Have a look at below mongoDB document sample:
{
"Partner_ID" : "0682047456",
"EarningsData" : [
{
"From_Date" : ISODate("2022-01-10T18:30:00.000Z"),
"Gross_Amount" : 300,
"Net_Amount" : 285,
"Tax_Amount" : 15
},
{
"From_Date" : ISODate("2022-10-01T18:30:00.000Z"),
"Gross_Amount" : 1958,
"Net_Amount" : 1860,
"Quantity" : 979,
"Tax_Amount" : 98
},
],
"createdAt" : ISODate("2023-01-23T16:23:02.430Z")
}
Below is the aggregation query which I have written :
var projectQry = [
{
$match: {
"Partner_ID": userId
}
},
{
$unwind: "$EarningsData"
},
{
$group: {
_id: {
$month: "$EarningsData.From_Date"
},
Gross: {
$sum: "$EarningsData.Gross_Amount"
},
Tax: {
$sum: "$EarningsData.Tax_Amount"
},
Net: {
$sum: "$EarningsData.Net_Amount"
},
}
},
{
$project: {
_id: 0,
Month: "$_id",
Gross: 1,
Tax: 1,
Net: 1
}
}
];
Everything is working fine and I am getting the output also. But, I need to sort that output based on Month. I tried to apply sort pipeline at the end as follows
{
$sort: {
Month: 1
}
},
But the problem happening here is previous year Dec month is coming after Jan month of current year.
NOTE: The From_Date field contains the date of either current year or last year only. It will never go beyond last year.
If I understand what you are trying to do, you should group by <year, month> and perform sorting on these fields.
Note:
Check the data you reported in the question as there are inconsistencies with your pipeline, however they are understandable.
The aggregation pipeline should look as follows:
db.getCollection("test01").aggregate([
{
$match: {
"Partner_ID": "0682047456"
}
},
{
$unwind: "$EarningsData"
},
{
$group: {
_id: {
year: { $year: "$EarningsData.From_Date", },
month: { $month: "$EarningsData.From_Date" }
},
Gross: {
$sum: "$EarningsData.Gross_Amount"
},
Tax: {
$sum: "$EarningsData.Tax_Amount"
},
Net: {
$sum: "$EarningsData.Net_Amount"
},
}
},
{
$project: {
_id: 0,
Date: "$_id",
Gross: 1,
Tax: 1,
Net: 1
}
},
{
$sort: {
"Date.year": 1,
"Date.month": 1,
}
}
]);
I'm trying to get all orders between two dates, group them by day of week, then average them. Currently the code looks like this:
export const getOrderValuesBetweenTwoDates = async (
from: number,
to: number,
) => {
// from, and to are guaranteed to be Mondays, 00:00
const orders = await OrderModel.find({
createdAt: { $lt: to, $gte: from },
}).exec();
const totalOfDaysOfWeek = [0, 0, 0, 0, 0, 0, 0];
orders.forEach((order) => {
const daysSinceFrom = (order.createdAt - from) / dayInMilliseconds;
const dayOfWeek = Math.floor(daysSinceFrom) % 7;
totalOfDaysOfWeek[dayOfWeek] =
(totalOfDaysOfWeek[dayOfWeek] || 0) + order.value;
});
const numberOfWeeks = Math.floor((to - from) / dayInMilliseconds / 7);
const averageOfDaysOfWeek = totalOfDaysOfWeek.map((v) =>
Number((v / numberOfWeeks).toFixed(2)),
);
return averageOfDaysOfWeek;
};
However, this is not really performant, and I guess if it could be written in aggregation, it would be. Is that possible to convert the above into aggregation?
Sample input (2 weeks):
[
// 1st mon (total 5)
{ createdAt: 345600000, value: 2 },
{ createdAt: 345600000, value: 3 },
// 1st tue
{ createdAt: 432000000, value: 1 },
// 1st wed
{ createdAt: 518400000, value: 1 },
// 1st thu
{ createdAt: 604800000, value: 1 },
// 1st fri
{ createdAt: 691200000, value: 1 },
// 1st sat
{ createdAt: 777600000, value: 1 },
// 1st sun (2 total)
{ createdAt: 864000000, value: 2 },
// 2nd mon (1 total)
{ createdAt: 950400000, value: 1 },
// 2nd tue
{ createdAt: 1036800000, value: 1 },
// 2nd wed
{ createdAt: 1123200000, value: 1 },
// 2nd thu
{ createdAt: 1209600000, value: 1 },
// 2nd fri
{ createdAt: 1296000000, value: 1 },
// 2nd sat
{ createdAt: 1382400000, value: 1 },
// 2nd sun (4 total)
{ createdAt: 1468800000, value: 1 },
{ createdAt: 1468800000, value: 1 },
{ createdAt: 1468800000, value: 2 },
]
In the above example I've made 2 special cases, for Monday, and Sunday. There are multiple orders for those days.
For the first Monday there is an order with value 2, and 3, to 5 in total. For the second Monday there is only one order with value 1. The average should be 3.
For Sunday, the first one, there's an order with value 2, and for the second Sunday, there are 3 orders with total value of 4. I'm expecting the average to be 3.
I'm expecting the result to be [3,1,1,1,1,1,3]
format the date using $dateToString
use $sum to get sum of same day of week
get day of week by $dayOfWeek
group by days of week and get average by $avg
project to get data as desired format
weekDay in output will be number between 1 (Sunday) and 7 (Saturday).
test it at mongoPlayground
db.collection.aggregate([
{
"$addFields": {
createdAt: {
"$dateToString": {
"date": {
"$toDate": "$createdAt"
},
"format": "%Y-%m-%d"
}
}
}
},
{
"$group": {
"_id": "$createdAt",
"value": {
"$sum": "$value"
}
}
},
{
"$addFields": {
"createdAt": {
$dayOfWeek: {
"$toDate": "$_id"
}
}
}
},
{
"$group": {
"_id": "$createdAt",
"average": {
"$avg": "$value"
}
}
},
{
"$project": {
_id: 0,
weekDay: "$_id",
average: 1
}
}
])
How can I retrieve data with a custom sort in Mongoose?
There is a job starting date that needs to be sorted by the month and year, but currently this script is only sorting from December to January.
router.get('/', (req, res) => {
Job.find()
.sort({ from: -1 })
.then(jobs => res.json(jobs))
.catch(err => res.status(404).json(err));
});
The problem is in the sort; values for from is like 12.2018, 06.2019, 03.2020, 11.2009 and so on.
I want to sort these results first from the year (which is after the dot) and then sort from the months. I cannot currently change how the data is set and it's stored as a String in the model Schema.
You have to use aggregation framework to first transform your string to a valid date by
$spliting it,
$convert parts from string to int
and using $dateFromParts,
then you sort and finally remove created field.
Here's the query :
db.collection.aggregate([
{
$addFields: {
date: {
$dateFromParts: {
year: {
$convert: {
input: {
$arrayElemAt: [
{
$split: [
"$from",
"."
]
},
1
]
},
to: "int"
}
},
month: {
$convert: {
input: {
$arrayElemAt: [
{
$split: [
"$from",
"."
]
},
0
]
},
to: "int"
}
},
}
}
}
},
{
$sort: {
date: -1
}
},
{
$project: {
date: 0
}
}
])
You can test it here
I want to implement retweet feature in my app. I use Mongoose and have User and Message models, and I store retweets as array of objects of type {userId, createdAt} where createdAt is time when retweet occurred. Message model has it's own createdAt field.
I need to create feed of original and retweeted messages merged together based on createdAt fields. I am stuck with merging, whether to do it in a single query or separate and do the merge in JavaScript. Can I do it all in Mongoose with a single query? If not how to find merge insertion points and index of the last message?
So far I just have fetching of original messages.
My Message model:
const messageSchema = new mongoose.Schema(
{
fileId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'File',
required: true,
},
userId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
required: true,
},
likesIds: [{ type: mongoose.Schema.Types.ObjectId, ref: 'User' }],
reposts: [
{
reposterId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
},
createdAt: { type: Date, default: Date.now },
},
],
},
{
timestamps: true,
},
);
Edit: Now I have this but pagination is broken. I am trying to use newCreatedAt field for cursor, that doesn't seem to work. It returns empty array in second call when newCreatedAt is passed from the frontend.
messages: async (
parent,
{ cursor, limit = 100, username },
{ models },
) => {
const user = username
? await models.User.findOne({
username,
})
: null;
const options = {
...(cursor && {
newCreatedAt: {
$lt: new Date(fromCursorHash(cursor)),
},
}),
...(username && {
userId: mongoose.Types.ObjectId(user.id),
}),
};
console.log(options);
const aMessages = await models.Message.aggregate([
{
$addFields: {
newReposts: {
$concatArrays: [
[{ createdAt: '$createdAt', original: true }],
'$reposts',
],
},
},
},
{
$unwind: '$newReposts',
},
{
$addFields: {
newCreatedAt: '$newReposts.createdAt',
original: '$newReposts.original',
},
},
{ $match: options },
{
$sort: {
newCreatedAt: -1,
},
},
{
$limit: limit + 1,
},
]);
const messages = aMessages.map(m => {
m.id = m._id.toString();
return m;
});
//console.log(messages);
const hasNextPage = messages.length > limit;
const edges = hasNextPage ? messages.slice(0, -1) : messages;
return {
edges,
pageInfo: {
hasNextPage,
endCursor: toCursorHash(
edges[edges.length - 1].newCreatedAt.toString(),
),
},
};
},
Here are the queries. The working one:
Mongoose: messages.aggregate([{
'$match': {
createdAt: {
'$lt': 2020 - 02 - 02 T19: 48: 54.000 Z
}
}
}, {
'$sort': {
createdAt: -1
}
}, {
'$limit': 3
}], {})
And the non working one:
Mongoose: messages.aggregate([{
'$match': {
newCreatedAt: {
'$lt': 2020 - 02 - 02 T19: 51: 39.000 Z
}
}
}, {
'$addFields': {
newReposts: {
'$concatArrays': [
[{
createdAt: '$createdAt',
original: true
}], '$reposts'
]
}
}
}, {
'$unwind': '$newReposts'
}, {
'$addFields': {
newCreatedAt: '$newReposts.createdAt',
original: '$newReposts.original'
}
}, {
'$sort': {
newCreatedAt: -1
}
}, {
'$limit': 3
}], {})
This can be done in one query, although its a little hack-ish:
db.collection.aggregate([
{
$addFields: {
reposts: {
$concatArrays: [[{createdAt: "$createdAt", original: true}],"$reports"]
}
}
},
{
$unwind: "$reposts"
},
{
$addFields: {
createdAt: "$reposts.createdAt",
original: "$reposts.original"
}
},
{
$sort: {
createdAt: -1
}
}
]);
You can add any other logic you want to the query using the original field, documents with original: true are the original posts while the others are retweets.
I got this data set from collection
{
item: 124001
price: 6
},
{
item: 124001
price: 6
},
{
item: 124121
price: 16
},
{
item: 124121
price: 13
},
{
item:n
price: x
}
from code:
let INDX = [xxx,xxx,xxx,xxx, ..n]
auctions.aggregate([
{
$match: { item: { $in: INDX }}
}
The problem is right after it, in the $group stage. For example I'd like to receive $min, $max or $avg 'price' for every unique/distinct item.
When I'm trying to use:
{
$group: {
min_1: { $min: "$price",}
}
}
I receive just $min from all data,
[ { _id: 0, min_1: 0 } ]
but I need something like:
{ _id: 124119, min_1: 66500 },
{ _id: 124437, min_1: 26398 }
Ok, here is a simple answer:
Just don't forget about _id field and use it, at $group stage, just like:
{
$group: {
_id: "$item",
min_1: {
$min: '$price',
}
}
}