I have a mongodb collection testdata which contains a field called insertTime. We have a requirement to delete data older than 60 days. So, previously to delete older data from the collections for all documents which are older than 60 days -> I would use the following logic of first finding the deletion date and then comparing it against the updateTime:
var date = new Date();
var daysToDeletion = 60;
var deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
deletionDate = deletionDate.toISOString()
printjson(insertDate);
db.testdata.find({"insertTime":{ $lt: deletionDate}})
However now, I would like to delete the data which is older than the alive time of the record. Alive time would be calculated as the insertTime + endTime(60 days). Now the documents older than this alive time - 60 days should be deleted. Can someone help me achieve this?
All i can think of is something like this but i don't think the command is right:
db.testdata.find({"insertTime"+endTime:{ $lt: deletionDate}})
How do i achieve this in mongodb find command query? Please can insights be provided on this.
Thanks a ton.
I have added all the details above and what i would like to achieve.
EDIT: using AWS documentDB 4.0.0
You can use $dateAdd(available from MongoDB v5.0+) to compute the alive date and compare to $$NOW
db.collection.find({
$expr: {
$lt: [
{
"$dateAdd": {
"startDate": "$insertTime",
"unit": "day",
"amount": 60
}
},
"$$NOW"
]
}
})
Mongo Playground
Here is a version for MongoDB / AWS DocumentDB(v4.0) that OP is using. The idea is to compute 60 days late by adding 60 day * 24 hours * 60 min * 60 sec * 1000 ms = 5184000000.
db.collection.aggregate([
{
"$addFields": {
flag: {
$lt: [
{
$add: [
"$insertTime",
5184000000
]
},
"$$NOW"
]
}
}
},
{
"$match": {
flag: true
}
},
{
"$unset": "flag"
}
])
Mongo Playground
I think this $expr can help you:
var date = new Date();
var daysToDeletion = 60;
var deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
db.testdata.deleteMany({
$expr: {
$lt: [{ $add: ["$insertTime", "$endTime"] }, deletionDate]
}
});
Edit:
With compatible solution with documentdb:
var date = new Date();
var daysToDeletion = 60;
var deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
db.testdata.find(
{
$lt: {
$add: [
"$insertTime",
{ $multiply: [daysToDeletion, 24 * 60 * 60 * 1000] }
]
},
deletionDate
}
);
Edit 2: The solution above wasn't working properly.
This one a little bit tricky but it works
const date = new Date();
const daysToDeletion = 60;
const deletionDate = new Date(date.setDate(date.getDate() - daysToDeletion));
const aliveTime = { $add: ["$insertTime", "$endTime"] };
db.testdata.deleteMany({
$and: [
{ aliveTime: { $lt: deletionDate } },
{ insertTime: { $lt: deletionDate } }
]
});
Related
consider this mongo collection with following documents and props.
{sku: '3344', frequency: 30, lastProccessedDate: 2021-01-07 15:18:07.576Z},
{sku: '2233', frequency: 30, lastProccessedDate: 2021-02-16 15:18:07.576Z},
{sku: '1122', frequency: 30, lastProccessedDate: 2021-04-13 15:18:07.576Z}
I want to query and get all the documents with (lastProcessedDate + frequency (days)) <= current date.
Essentially in SQL world this is possible to do it, but I can't figure it out to do it in mongo or even if it is possible to do it.
In SQL this would be something like
SELECT * FROM table WHERE DATE_FORMAT(DATE_ADD(FROM_UNIXTIME(lastProcessedDate), INTERVAL frequency DAY), '%Y-%m-%d') <= CURDATE()
If it is not possible I know I can store the calculated date in the document and just query based on that but you know I want to know if it is possible to do it.
Thank you all!
Unfortunately, I can't give you a solution for mongoose. But here is the Mongo query that returns the result you want:
db.getCollection("yourCollection").aggregate([
{
$project: {
_id: 1,
sku: 1,
frequency: 1,
lastProccessedDate: 1,
shiftedDate: {
$add: [
"$lastProccessedDate", { $multiply: [ 24 * 60 * 60 * 1000, "$frequency" ] }
]
}
}
}, {
$match: {
shiftedDate: { $lte: new Date() }
}
}, {
$project: {
_id: 1,
sku: 1,
frequency: 1,
lastProccessedDate: 1,
}
}
])
First, we transform documents to new form that contains the same fields plus a new temporary field - a "shifted" date that is defined as lastProccessedDate + frequency (days) (pay attention that we actually add milliseconds, so there's 24 * 60 * 60 * 1000 in the query). Then we select only that documents which shiftedDate is less than (or equals) current timestamp (new Date() returns current timestamp). Finally, we transform the filtered documents to original form, without the temporary field we used to filter the documents previously.
Perhaps there's a better solution to get documents you want, but this one can resolve your problem too.
Here is my query:
db.collection('guilds').aggregate([
{
$match: { users: { $elemMatch: {
$and: [
{ 'registeredAt': { $exists: false } },
{ $expr: {
$gt: [
{ $divide: [
{ $subtract: [
new Date(),
'$joinedAt'] },
1000 * 60 * 60 * 24] },
1] } }],
} } },
},
])
and here is my data:
I'm getting this error when I try to run the aggregation:
MongoError: $expr can only be applied to the top-level document
How can I solve it? I've seen this question but I feel like it's not the best answer:
Also if there is another way to achieve the same result I'd appreciate that too
The $expr operator uses aggregation operations instead of query operations. $elemMatch is a query operator, which doesn't work with $expr.
For this use case, calculating the date on the client side will reduce the work required of the database, and permit the query planner to use an index on 'joinedAt' (if one exists):
let oneDayAgo = new Date(new Date() - (1000 * 60 * 60 * 24]))
db.collection('guilds').aggregate([
{
$match: { users: { $elemMatch: {
'registeredAt': { $exists: false },
'joinedAt': {$gte: oneDayAgo}
} } },
},
])
I have kind of a "simple" problem and i have thought of a "complicated" solution in my head, but i'm having problems executing it...
The thing is, i have this schema:
const DensitySchema = mongoose.Schema(
{
map_id: mongoose.Schema.Types.ObjectId,
name: String,
location: {
x: Number,
y: Number
},
density: Number,
time: Number
},
{
timestamps: true
}
);
And I also have this query:
Density.find({ map_id: req.params.mapId, time: { $gte: req.query.from, $lte: req.query.to } }).then(data => {
console.log(data);
res.send(data);
});
The time and the from - to values are timestamps, for example, 1579148100.
In the database, i have entries for timestamps that occur every 15s. But i want to get only results of those that have happened in timestamps that have minutes that are multipliers of 15 and 0, so 0, 15, 30, 45
the idea would be that if i ask for the values between, let's say, 1am and 2am, i get 4 results instead of dozens.
From what i see on mongo's documentation, i should be able to somehow filter the query request by somehow turning the time value to a readable date and then checking if the minutes are one of those values... but i guess im not smart enough to figure out how... the documentation is not so easy for me to understand yet, since i have started using mongo this week...
I would appreciate your help very much.
Thanks in advance and kind regards.
The timestamp is a long and the corresponding time (in hours, minutes, months, etc., format) can be constructed with the Date field. For example, I have three timestamps and the corresponding time (the hour and minutes):
1579243569270 -> 06:46
1579243509270 -> 06:45
1579244415497 -> 07:00
From the Mongo Shell, the new Date(1579243569270) gets the date/time as ISODate("2020-01-17T06:46:09.270Z"). From this we can find the time minutes, which in this case is 06:46. And, we use this in the query as follows using the find or the aggregate:
The input collection:
{ "ts" : 1579243569270 }
{ "ts" : 1579243509270 }
{ "ts" : 1579244415497 }
The queries:
const MINS_ARRAY = [ 0, 15, 30, 45 ]
db.test.find( { $expr: { $in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] } } )
_or_
db.test.aggregate( [
{
$match: {
$expr: {
$in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] }
}
}
}
] )
The result will be the two documents with timestamps, 1579243509270 and 1579244415497.
The code is tested for Mongo Shell. The queries use aggegation date and array operators which are used with the $expr to construct the queries.
[ EDIT ADD ]
db.test.find( { $expr: { $in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] },
map_id: req.params.mapId,
time: { $gte: req.query.from, $lte: req.query.to }
} )
I have a collection that has an endDateTime field:
If endDateTime is null, the operation is not yet finished.
If it has finished, endDateTime contains the timestamp (Javascript Date.now())
How can I query for records that is not yet finished (endDateTime is null) or has ended in the last 24 hours:
let endDateTime = Date.now();
let startDateTime = endDateTime - 24 * 60 * 60 * 1000;
query.endDateTime = {
$or: [
null,
{ $and: [{ $gte: startDateTime }, { $lte: endDateTime }] }
]
};
My query is returning registers before 24 hours ago. How to solve it?
You can combine your criteria with $or:
db.col.find({
$or: [
{ endDateTime: null },
{ endDateTime: { $gt: Date.now() - 24 * 60 * 60 * 1000 } }
]
})
or if you have no endDateTime key in some of your documents then you need $exists operator:
db.col.find({
$or: [
{ endDateTime: { $exists: false } },
{ endDateTime: { $gt: Date.now() - 24 * 60 * 60 * 1000 } }
]
})
I am new in sequelize, and I don't know how to get the records with createdAt column less than 1 hours with the current date and here is my code :
models.Trip.findAll({
where: {
createdAt: {
$gt: -------> createdAt <= 1 hour with current date
}
},
include: [
models.User,
models.Vehicle,
],
})
How to do that, please help me. Thanks!
Use sequelize.literal() to query based on the database server time.
Using the server:
created_at: {
[Op.gt]: sequelize.literal("NOW() - INTERVAL '24 HOURS'")),
}
Use a Date object set to one hour in the past by subtracting milliseconds from Date.now().
Using Sequelize.Op
created_at: {
[Op.gt]: new Date(Date.now() - (60 * 60 * 1000)),
},
Using old style operator aliases:
created_at: {
$gt: new Date(Date.now() - (60 * 60 * 1000)),
},