Here is my query:
db.collection('guilds').aggregate([
{
$match: { users: { $elemMatch: {
$and: [
{ 'registeredAt': { $exists: false } },
{ $expr: {
$gt: [
{ $divide: [
{ $subtract: [
new Date(),
'$joinedAt'] },
1000 * 60 * 60 * 24] },
1] } }],
} } },
},
])
and here is my data:
I'm getting this error when I try to run the aggregation:
MongoError: $expr can only be applied to the top-level document
How can I solve it? I've seen this question but I feel like it's not the best answer:
Also if there is another way to achieve the same result I'd appreciate that too
The $expr operator uses aggregation operations instead of query operations. $elemMatch is a query operator, which doesn't work with $expr.
For this use case, calculating the date on the client side will reduce the work required of the database, and permit the query planner to use an index on 'joinedAt' (if one exists):
let oneDayAgo = new Date(new Date() - (1000 * 60 * 60 * 24]))
db.collection('guilds').aggregate([
{
$match: { users: { $elemMatch: {
'registeredAt': { $exists: false },
'joinedAt': {$gte: oneDayAgo}
} } },
},
])
Related
I have a mongodb collection testdata which contains a field called insertTime. We have a requirement to delete data older than 60 days. So, previously to delete older data from the collections for all documents which are older than 60 days -> I would use the following logic of first finding the deletion date and then comparing it against the updateTime:
var date = new Date();
var daysToDeletion = 60;
var deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
deletionDate = deletionDate.toISOString()
printjson(insertDate);
db.testdata.find({"insertTime":{ $lt: deletionDate}})
However now, I would like to delete the data which is older than the alive time of the record. Alive time would be calculated as the insertTime + endTime(60 days). Now the documents older than this alive time - 60 days should be deleted. Can someone help me achieve this?
All i can think of is something like this but i don't think the command is right:
db.testdata.find({"insertTime"+endTime:{ $lt: deletionDate}})
How do i achieve this in mongodb find command query? Please can insights be provided on this.
Thanks a ton.
I have added all the details above and what i would like to achieve.
EDIT: using AWS documentDB 4.0.0
You can use $dateAdd(available from MongoDB v5.0+) to compute the alive date and compare to $$NOW
db.collection.find({
$expr: {
$lt: [
{
"$dateAdd": {
"startDate": "$insertTime",
"unit": "day",
"amount": 60
}
},
"$$NOW"
]
}
})
Mongo Playground
Here is a version for MongoDB / AWS DocumentDB(v4.0) that OP is using. The idea is to compute 60 days late by adding 60 day * 24 hours * 60 min * 60 sec * 1000 ms = 5184000000.
db.collection.aggregate([
{
"$addFields": {
flag: {
$lt: [
{
$add: [
"$insertTime",
5184000000
]
},
"$$NOW"
]
}
}
},
{
"$match": {
flag: true
}
},
{
"$unset": "flag"
}
])
Mongo Playground
I think this $expr can help you:
var date = new Date();
var daysToDeletion = 60;
var deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
db.testdata.deleteMany({
$expr: {
$lt: [{ $add: ["$insertTime", "$endTime"] }, deletionDate]
}
});
Edit:
With compatible solution with documentdb:
var date = new Date();
var daysToDeletion = 60;
var deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
db.testdata.find(
{
$lt: {
$add: [
"$insertTime",
{ $multiply: [daysToDeletion, 24 * 60 * 60 * 1000] }
]
},
deletionDate
}
);
Edit 2: The solution above wasn't working properly.
This one a little bit tricky but it works
const date = new Date();
const daysToDeletion = 60;
const deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
const aliveTime = { $add: ["$insertTime", "$endTime"] };
db.testdata.deleteMany({
$and: [
{ aliveTime: { $lt: deletionDate } },
{ insertTime: { $lt: deletionDate } }
]
});
I'm trying to find Users who logged in the last day.
The userActivity field is an object on User. The userActivity object contains a field called hourly which is an array of dates. I want the find any users who's hourly array contains a date greater than a day ago using aggregation.
User schema
{
userName:"Bob",
userActivity:
{"hourly":
[
"2022-05-09T02:31:12.062Z", // the user logged in
"2022-05-09T19:37:42.870Z" // saved as date object in the db
]
}
}
query that didn't work
const oneDayAgo = new Date();
oneDayAgo.setDate(oneDayAgo.getDate() - 1);
const usersActiveToday = await User.aggregate([
{
$match: {
$expr: { $gt: [oneDayAgo, '$userActivity'] },
},
},
]);
If today is September 13, 11pm, I'd expect the results to the above to show users who had activity between the 12th and 13th.
Instead I am getting all users returned.
If you want to use an aggregation pipeline, then one option is to use $max to find if there are items that are greater than oneDayAgo:
db.collection.aggregate([
{$match: {$expr: {$gt: [{$max: "$userActivity.hourly"}, oneDayAgo]}}}
])
See how it works on the playground example - aggregation
But, you can also do it simply by using find:
db.collection.find({
"userActivity.hourly": {$gte: oneDayAgo}
})
See how it works on the playground example - find
This can be considered as a 3 step process
Find the max value in hourly array and store it in some key(here max)
Check if the max value is greater than or equal to oneDayAgo timestamp
Unset the key that stored max value
Working Code Snippet:
const oneDayAgo = new Date();
oneDayAgo.setDate(oneDayAgo.getDate() - 1);
const usersActiveToday = await User.aggregate([
{
$set: {
max: {
$max: {
$map: {
input: "$userActivity.hourly",
in: {
$max: "$$this",
},
},
},
},
},
},
{
$match: {
max: {
$gte: oneDayAgo.toISOString(),
},
},
},
{
$unset: "max",
},
]);
Here's code in action: Mongo Playground
You can try something along these lines:
db.collection.aggregate([
{
"$addFields": {
"loggedInADayBefore": {
"$anyElementTrue": [
{
"$map": {
"input": "$userActivity.hourly",
"as": "time",
"in": {
"$gte": [
"$$time",
ISODate("2022-05-13T00:00:00.000Z")
]
}
}
}
]
}
}
},
{
"$match": {
loggedInADayBefore: true
}
},
{
"$project": {
loggedInADayBefore: 0
}
}
])
Here, we $anyElementTrue, to find if any element in the array hourly, is greater than the day before, and store it as a boolean value in a new field. Then we filter the docs on the basis of that field using $match.
Here's the playground link.
Here is my query:
ctas.updateMany({
$and: [
{$expr: { $lt: ['$schedule.start', () => Date.now()] }},
{$expr: { $gt: ['$schedule.end', () => Date.now()] }}
]
},
{
$set: {isActive: true}
}).then(res => {
const { matchedCount, modifiedCount } = res;
console.log(`Successfully matched ${matchedCount} and modified ${modifiedCount} items.`)
}).catch(e => console.error(e));
I'm absolutely positive that start is less than Date.now() and end is greater than Date.now(), but I'm not getting any matches. Is my syntax wrong?
a snippet of my document in mongo:
schedule: {
start: 1642564718042,
end: 3285129434744
}
Edit: In case it makes a difference, I'm writing this code as a mongo scheduled trigger.
Update: If I replace the second expression with an obviously truth expression, { isActive: false }, it matches all the documents. Obviously Date.now()*2 (what I used to set schedule.end) is greater than Date.now(), so why is that second expression failing?
Missing $. And make sure your field paths are correct. $schedule.start and $schedule.end.
And another concern is that both schedule.start and schedule.end are with Timespan value. So you need to cast them to date via $toDate.
db.collection.update({
$and: [
{
$expr: {
$lt: [
{
$toDate: "$schedule.start"
},
new Date()
]
}
},
{
$expr: {
$gt: [
{
$toDate: "$schedule.end"
},
new Date()
]
}
}
]
},
{
$set: {
isActive: true
}
})
Sample Mongo Playground
I have a large MongoDB dataset of around 34gb and I am using Fastify and Mongoose for the API. I want to retrieve all list of unique userUuid from the date range. I tried the distinct method from Mongoose:
These are my filters:
let filters = {
applicationUuid: opts.applicationUuid,
impressions: {
$gte: opts.impressions
},
date: {
$gte: moment(opts.startDate).tz('America/Chicago').format(),
$lt: moment(opts.endDate).tz('America/Chicago').format()
}
}
This is my distinct Mongoose function:
return await Model.distinct("userUuid", filters)
This method will return an array with unique userUuid based from the filters.
This works fine for small dataset, but it has a memory cap of 16MB when it comes to huge dataset.
Therefore, I tried the aggregate method to achieve similar results, having read that it is better optimized. Nevertheless, the same filters object above does not work inside the match pipeline because aggregate does not accept string date that comes as the result of moment; but only JavaScript Date is accepted. However, JavaScript date dissregards all the timezones since it is unix based.
This is my aggregate function to get distinct values based on filters.
return await Model.aggregate(
[
{
$match: filters
},
{
$group: {
_id: {userUuid: "$userUuid" }
}
}
]
).allowDiskUse(true);
As I said, $match does not work with moment, but only with new Date(opts.startDate), however, JavaScript's new Date disregards moment's timezone. Nor it has a proper native timezone. Any thought on how to achieve this array of unique ids based on filters with Mongoose?
This is the solution I came up with and it works pretty well regarding the performance. Use this solution for large dataset:
let filters = {
applicationUuid: opts.applicationUuid,
impressions: { $gte: opts.impressions },
$expr: {
$and: [
{
$gte: [
'$date',
{
$dateFromString: {
dateString: opts.startDate,
timezone: 'America/Chicago',
},
},
],
},
{
$lt: [
'$date',
{
$dateFromString: {
dateString: opts.endDate,
timezone: 'America/Chicago',
},
},
],
},
],
},
}
return Model.aggregate([
{ $match: filters },
{
$group: {
_id: '$userUuid',
},
},
{
$project: {
_id: 0,
userUuid: '$_id',
},
},
])
.allowDiskUse(true)
Which will return a list of unique ids i.e.
[
{ userUuid: "someId" },
{ userUuid: "someId" }
]
Use the following method on small dataset which is more convenient:
let filters = {
applicationUuid: opts.applicationUuid,
impressions: {
$gte: opts.impressions
},
date: {
$gte: opts.startDate,
$lte: opts.endDate
}
}
return Model.distinct("userUuid", filters)
Which will return the following result:
[ "someId", "someOtherId" ]
I have kind of a "simple" problem and i have thought of a "complicated" solution in my head, but i'm having problems executing it...
The thing is, i have this schema:
const DensitySchema = mongoose.Schema(
{
map_id: mongoose.Schema.Types.ObjectId,
name: String,
location: {
x: Number,
y: Number
},
density: Number,
time: Number
},
{
timestamps: true
}
);
And I also have this query:
Density.find({ map_id: req.params.mapId, time: { $gte: req.query.from, $lte: req.query.to } }).then(data => {
console.log(data);
res.send(data);
});
The time and the from - to values are timestamps, for example, 1579148100.
In the database, i have entries for timestamps that occur every 15s. But i want to get only results of those that have happened in timestamps that have minutes that are multipliers of 15 and 0, so 0, 15, 30, 45
the idea would be that if i ask for the values between, let's say, 1am and 2am, i get 4 results instead of dozens.
From what i see on mongo's documentation, i should be able to somehow filter the query request by somehow turning the time value to a readable date and then checking if the minutes are one of those values... but i guess im not smart enough to figure out how... the documentation is not so easy for me to understand yet, since i have started using mongo this week...
I would appreciate your help very much.
Thanks in advance and kind regards.
The timestamp is a long and the corresponding time (in hours, minutes, months, etc., format) can be constructed with the Date field. For example, I have three timestamps and the corresponding time (the hour and minutes):
1579243569270 -> 06:46
1579243509270 -> 06:45
1579244415497 -> 07:00
From the Mongo Shell, the new Date(1579243569270) gets the date/time as ISODate("2020-01-17T06:46:09.270Z"). From this we can find the time minutes, which in this case is 06:46. And, we use this in the query as follows using the find or the aggregate:
The input collection:
{ "ts" : 1579243569270 }
{ "ts" : 1579243509270 }
{ "ts" : 1579244415497 }
The queries:
const MINS_ARRAY = [ 0, 15, 30, 45 ]
db.test.find( { $expr: { $in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] } } )
_or_
db.test.aggregate( [
{
$match: {
$expr: {
$in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] }
}
}
}
] )
The result will be the two documents with timestamps, 1579243509270 and 1579244415497.
The code is tested for Mongo Shell. The queries use aggegation date and array operators which are used with the $expr to construct the queries.
[ EDIT ADD ]
db.test.find( { $expr: { $in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] },
map_id: req.params.mapId,
time: { $gte: req.query.from, $lte: req.query.to }
} )