Develop Reference

Mongo aggregation; match where a field's subfield has at least one array element greater than specified value

I'm trying to find Users who logged in the last day.
The userActivity field is an object on User. The userActivity object contains a field called hourly which is an array of dates. I want the find any users who's hourly array contains a date greater than a day ago using aggregation.
User schema
{
userName:"Bob",
userActivity:
{"hourly":
[
"2022-05-09T02:31:12.062Z", // the user logged in
"2022-05-09T19:37:42.870Z" // saved as date object in the db
]
}
}
query that didn't work
const oneDayAgo = new Date();
oneDayAgo.setDate(oneDayAgo.getDate() - 1);
const usersActiveToday = await User.aggregate([
{
$match: {
$expr: { $gt: [oneDayAgo, '$userActivity'] },
},
},
]);
If today is September 13, 11pm, I'd expect the results to the above to show users who had activity between the 12th and 13th.
Instead I am getting all users returned.

If you want to use an aggregation pipeline, then one option is to use $max to find if there are items that are greater than oneDayAgo:
db.collection.aggregate([
{$match: {$expr: {$gt: [{$max: "$userActivity.hourly"}, oneDayAgo]}}}
])
See how it works on the playground example - aggregation
But, you can also do it simply by using find:
db.collection.find({
"userActivity.hourly": {$gte: oneDayAgo}
})
See how it works on the playground example - find

This can be considered as a 3 step process
Find the max value in hourly array and store it in some key(here max)
Check if the max value is greater than or equal to oneDayAgo timestamp
Unset the key that stored max value
Working Code Snippet:
const oneDayAgo = new Date();
oneDayAgo.setDate(oneDayAgo.getDate() - 1);
const usersActiveToday = await User.aggregate([
{
$set: {
max: {
$max: {
$map: {
input: "$userActivity.hourly",
in: {
$max: "$$this",
},
},
},
},
},
},
{
$match: {
max: {
$gte: oneDayAgo.toISOString(),
},
},
},
{
$unset: "max",
},
]);
Here's code in action: Mongo Playground

You can try something along these lines:
db.collection.aggregate([
{
"$addFields": {
"loggedInADayBefore": {
"$anyElementTrue": [
{
"$map": {
"input": "$userActivity.hourly",
"as": "time",
"in": {
"$gte": [
"$$time",
ISODate("2022-05-13T00:00:00.000Z")
]
}
}
}
]
}
}
},
{
"$match": {
loggedInADayBefore: true
}
},
{
"$project": {
loggedInADayBefore: 0
}
}
])
Here, we $anyElementTrue, to find if any element in the array hourly, is greater than the day before, and store it as a boolean value in a new field. Then we filter the docs on the basis of that field using $match.
Here's the playground link.

Mongoose findOneAndUpdate in a for loop using set if extist or put if doesn't

I am having a problem with concurrency trying to use findOneAndUpdate with the mongoose in a for a loop.
Don' t focus on the example given but only on its method.
Let' s say i have this schema:
var PlayerSchema = new Schema({
playername: String,
playerstats:[{
matchNumber: Number,
goals: {
type: Number,
default: 0
}
}]
})
Assuming that I have a year's array and I would like to update goals with another array created with a for loop iteration, what I am looking to do is to have a for loop for x = 0 to 5 where x is the matchNumber and scores[x] represents the number of goals for that match (for the last year). I would like to look in the Player document if the matchNumber exists and if it exists I would use findOneAndUpdate with $set to set scores[x] but it doesn' t exist I would like to create it with scores[x] using $push. Let' s give code for the easiest understanding ("Maradona" will be our player name) :
function iterateYears(){
years = ["2019", "2020"]
for (var k = 0; k < years.lenght; k++) {
updateScores(k)
}
}
function updateScores(k) {
var scores = []
for (var y = 0; y < 6; y++) {
scores.splice(y, 0, (y * 10) + k)
}
//at this point we would have the first time scores = [0, 10, 20, 30, 40] and second time scores = [1, 11, 21, 31, 41]
for (var x = 0; x < 6; x++) {
Player.findOneAndUpdate({playername: "Maradona", 'playerstats.matchNumber': x}, {$set: {'playerstats.$.goals': scores[x]}}, function(err, matchfound) {
if (err) {
console.log("Err found")
} else {
if (matchfound == null) {
Player.findOneAndUpdate({playername: "Maradona"}, {$push: {playerstats:{matchNumber: x, goals: scores[x]}}}).exec();
}
}
})
}
}
What I would have at the end is:
{
playername: "Maradona",
playerstats:[
{
matchNumber: 0,
goals: 1
},
{
matchnumber: 1,
goals: 11
},
{
matchNumber: 2,
goals: 21
},
{
matchNumber: 3,
goals: 31
},
{
matchNumber: 4,
goals: 41
}
}
but what i get is something like:
{
playername: "Maradona",
playerstats:[
{
matchNumber: 0,
goals: 1
},
{
matchNumber: 0,
goals: 0
},
{
matchnumber: 1,
goals: 11
},
{
matchnumber: 1,
goals: 10
},
{
matchNumber: 2,
goals: 21
},
{
matchNumber: 2,
goals: 20
},
{
matchNumber: 3,
goals: 31
},
{
matchNumber: 3,
goals: 30
},
{
matchNumber: 4,
goals: 41
},
{
matchNumber: 4,
goals: 40
}
}
or a mix of them.
This happens because of the code will attempt to make the findOneAndUpdate with $set and when the matchNumber is not found (matchfound == null) instead of executing the findOneAndUpdate with $push to create it, it will keep on iterating so will try to do the same with the value of k = 1 (the second year to clarify) and will need to move again to findOneAndUpdate with $push because of it will not be found and so the value with scores = (x * 10) + k is created and after that, it will come back and make the findOneAndUpdate with $push and scores = x * 10. I understand that this is its normal working because of mono threading but I really need to have the desired output.
Thanks to all and sorry for the long reading.

The problem there is that findOneAndUpdate is asynchronous. Each iteration through the for loop calls Player.findOneAndUpdate and passes it a callback function. The findOneAndUpdate call returns immediately, and the loop moves on to the next iteration.
At some point in the future, the query to the mongo completes and the callback is called with the return value, since there are multiple asynchronous calls staged, the order they complete is not strictly defined.
You might try using await with each call to findOneAndUpdate to make them synchronous, which will serialize the requests.
If you are using MongoDB 4.2, you can use the pipeline form of update to be able to do both insert and update array elements in an single statement:
.update(
{playername: "Maradona"},
[{$set:{
playerstats:{
$concatArrays:[
{$filter:{
input:"$playerstats",
cond:{$ne:["$$this.matchNumber",x]}
}},
[{matchNumber: x, goals: scores[x] }]
]
}
}}]
)
In order to make sure these are applied in the correct order, you could build and array of all of the operations to apply, like:
[
{updateOne:{
filter: {playername: "Maradona"},
update: [{$set:{
playerstats:{
$concatArrays:[
{$filter:{
input:"$playerstats",
cond:{$ne:["$$this.matchNumber",0]}
}},
[{matchNumber: 0, goals: 1 }]
]
}
}}]
}},
{updateOne:{
filter: {playername: "Maradona"},
update: [{$set:{
playerstats:{
$concatArrays:[
{$filter:{
input:"$playerstats",
cond:{$ne:["$$this.matchNumber",1]}
}},
[{matchNumber: 1, goals: 11 }]
]
}
}}]
}}
and then pass that array to bulkWrite with the ordered=true option.
Also note that if you are able to build an array of all of the scores that need to be updated, you could use a similar statement to update all of the scores for a single player in a one operation, which should perform a bit better.

MongoDB: how to filter by 15 min period in a query with timestamp input?

I have kind of a "simple" problem and i have thought of a "complicated" solution in my head, but i'm having problems executing it...
The thing is, i have this schema:
const DensitySchema = mongoose.Schema(
{
map_id: mongoose.Schema.Types.ObjectId,
name: String,
location: {
x: Number,
y: Number
},
density: Number,
time: Number
},
{
timestamps: true
}
);
And I also have this query:
Density.find({ map_id: req.params.mapId, time: { $gte: req.query.from, $lte: req.query.to } }).then(data => {
console.log(data);
res.send(data);
});
The time and the from - to values are timestamps, for example, 1579148100.
In the database, i have entries for timestamps that occur every 15s. But i want to get only results of those that have happened in timestamps that have minutes that are multipliers of 15 and 0, so 0, 15, 30, 45
the idea would be that if i ask for the values between, let's say, 1am and 2am, i get 4 results instead of dozens.
From what i see on mongo's documentation, i should be able to somehow filter the query request by somehow turning the time value to a readable date and then checking if the minutes are one of those values... but i guess im not smart enough to figure out how... the documentation is not so easy for me to understand yet, since i have started using mongo this week...
I would appreciate your help very much.
Thanks in advance and kind regards.

The timestamp is a long and the corresponding time (in hours, minutes, months, etc., format) can be constructed with the Date field. For example, I have three timestamps and the corresponding time (the hour and minutes):
1579243569270 -> 06:46
1579243509270 -> 06:45
1579244415497 -> 07:00
From the Mongo Shell, the new Date(1579243569270) gets the date/time as ISODate("2020-01-17T06:46:09.270Z"). From this we can find the time minutes, which in this case is 06:46. And, we use this in the query as follows using the find or the aggregate:
The input collection:
{ "ts" : 1579243569270 }
{ "ts" : 1579243509270 }
{ "ts" : 1579244415497 }
The queries:
const MINS_ARRAY = [ 0, 15, 30, 45 ]
db.test.find( { $expr: { $in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] } } )
_or_
db.test.aggregate( [
{
$match: {
$expr: {
$in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] }
}
}
}
] )
The result will be the two documents with timestamps, 1579243509270 and 1579244415497.
The code is tested for Mongo Shell. The queries use aggegation date and array operators which are used with the $expr to construct the queries.
[ EDIT ADD ]
db.test.find( { $expr: { $in: [ { $minute: { $toDate: "$ts" } }, MINS_ARRAY ] },
map_id: req.params.mapId,
time: { $gte: req.query.from, $lte: req.query.to }
} )

Firestore query by dates and times separately

How do I query my documents by specific date and time range? Ie. I want to query documents by specific date range (01-01-2019 - 31-01-2019) and from those dates only the documents which are made in 10am to 12pm.
It would go something like this:
let ref = db.collection('events')
// Query by date range
ref = ref
.where('created', '>=', new Date(2019, 1, 1, 10, 0, 0, 0))
.where('created', '<=', new Date(2019, 1, 1, 12, 0, 0, 0))
// Query by time range in each date
ref = ref
.where('created', '>=', *START TIME = 10pm*)
.where('created', '<=', *END TIME = 12pm*)
So the question is how do I filter hours and minutes every day I queried? Hopefully you understand what I'm asking here! I'll update the question if needed.
Update
Database sturcture is following
/events/{single event document}:
{
cafeId: String,
created: Firestore timestamp // ex. February 2, 2019 at 10:16:00 AM UTC+2
discount: Number,
eventId: Number,
productCount: Number,
keywords: Array,
products: Map,
payment: Map,
returned: Number,
total: Number
}
And when I get this in the client created will transform to:
{ _seconds: Number, _nanoseconds: 0 }

If you want to use "standard" where() methods, you'll probably have to do the second filtering on your front end, from the results received with
ref = ref
.where('created', '>=', new Date(2019, 1, 1, 10, 0, 0, 0))
.where('created', '<=', new Date(2019, 3, 1, 12, 0, 0, 0))
Another approach would be to store, in an extra field (e.g. createdText), your date/time value with a custom format, like for example YYYYMMDDHHMM, with a 24 hours format for the hours (i.e. HH).
This way the document will be correctly sorted and you can query with only one where, as follows:
ref = ref
.where('createdText', '>=', '201902031000')
.where('createdText', '<=', '201902041200')
It is quite common to duplicate the data in an NoSQL database, in particular to allow querying according to your business needs.

Mongo $and selector

How does mongo $and selector work? I have trouble getting correct results back.
Let's say I have a collection something like this:
{ "_id" : "F7mdaZC2eBQDXA5wx", "quantity" : 5 }
{ "_id" : "F7mdaZC2eBQDXA5wx", "quantity" : 9 }
{ "_id" : "F7mdaZC2eBQDXA5wx", "quantity" : 34 }
{ "_id" : "F7mdaZC2eBQDXA5wx", "quantity" : 66 }
and I run query:
var selectorMin = 9;
var selectorMax = 42;
ScrapReport.find({ $and: [ { quantity: { $gte: selectorMin }, quantity: { $lte: selectorMax } } ] })
I would expect mongo to return me only 9 and 34. But for some reason it also returns 5 and 66.
What's wrong with my query?

Your query is returning all the documents in that sample because it is first looking for documents whose quantity >= 9 i.e. 9, 34 and 66 AND combines that query with documents whose quantity <= 42 i.e 34, 9 and 5. It's not looking for documents within a particular range but your query explicitly looks for all documents that satisify two ranges i.e.
Documents which satisfy "quantity >= 9"
+
Documents which satisfy "quantity <= 42"
not
Documents which satisfy "9 <= quantity <= 42"
Just simplify your query to
ScrapReport.find({ "quantity": { "$gte": selectorMin, "$lte": selectorMax } })
That way, you specify a range for MongoDB to filter your documents with i.e.
9 <= quantity <= 42
Specifying a comma separated list of expressions implies an implicit AND operation and use an explicit AND with the $and operator when when the same field or operator has to be specified in multiple expressions.

Using an implicit AND operation like the other answers suggested would work. But I would like to dig deeper into the specifics. Why is your query not working as you expected it to work?
Why? Why is this seemingly correct query of yours returning not so correct results? After all, whether you use implicit or explicit AND operation should be a matter of your choice and you should be able to achieve your goal irrespective of which form you use. How to make your query work with an explicit AND operation?
Let us look at the syntax of the AND operation.
{ $and: [ { <expression1> }, { <expression2> } , ... , { <expressionN> } ] }
The value of your AND operator should be an array containing expressions on which you would like to perform the AND operation.
After a first glance at your query, everything looks fine. But if you take a moment to look deeper, you would see that your query is not matching the AND syntax exactly. It is still syntactically correct. No doubt about that. But it is logically incorrect. I will explain how.
This is your $and operator value
{ $and: [ { quantity: { $gte: selectorMin }, quantity: { $lte: selectorMax } } ] }
You think you have an expression1 quantity: { $gte: selectorMin } and an expression2 quantity: { $lte: selectorMax }. An AND operation with these expressions should return the documents with quantity 9 and 34. But actually, all you have is one expression. Pay close attention to the braces. You have added both these expressions in a single {} block. Do you see it? So effectively, there is no 2nd expression for the AND operator to work with. But AND operator requires two or more expressions to function properly.
So your query is of the form
{ $and: [ { <expression1> } ] }
With an incorrect form, the results will also be incorrect. The correct query using an explicit AND operation would be
ScrapReport.find({ $and: [ { quantity: { $gte: selectorMin } }, { quantity: { $lte: selectorMax } } ] })
Do you see the difference? Try this query and you will get the results that you expected in the first place.
If you are not satisfied by just having the answer and are curious to know how Mongo interpreted your first query, read further.
Consider this query
ScrapReport.find({ quantity: 9 })
What would you expect the result to be? If you expected Mongo to return a single document whose value in the quantity field is 9, you are right. That is exactly what the result is. Now consider the same query with a small twist.
ScrapReport.find({ quantity: 9, quantity: 5 })
What would the result be now? Things are getting interesting now, huh? If you execute this query and have a look at the result, you will still see only a single document. But the value in the quantity field is 5. Now that is interesting!
ScrapReport.find({ quantity: 9, quantity: 5, quantity: 34 })
What about this? The result is still a single document with value in the quantity field being 34. You can try other combinations. What you will find out is this -
Within an expression, if you are referencing a field multiple times, the result will be determined by the last reference to that field in that expression.
Now apply this concept to your original query. It has already been pointed out that you have a single expression with two parts quantity: { $gte: selectorMin } and quantity: { $lte: selectorMax }. Since within an expression, you are referring to the same field twice, only the last one will be relevant. The selection criteria will be quantity: { $lte: selectorMax }. The result will be 3 documents with quantity values 5, 9 and 34.
If you swap the order i.e. write quantity: { $lte: selectorMax } first and then quantity: { $gte: selectorMin }, the selection criteria will now be determined by quantity: { $gte: selectorMin }. The result will be 3 documents with quantity values 9, 34 and 66.
Although it wasn't your intention, your original query is effectively
ScrapReport.find({ quantity: { $gte: selectorMin }, quantity: { $lte: selectorMax } })
When you miss braces or add them at the wrong position, it can completely change the meaning of your query.
Moral - Pay close attention to where you place your braces in complex queries.

Actually you have two problems in there:
Your query is equivalent to the following:
ScrapReport.find( { "$and": [{ "quantity": { "$lte": selectorMax } } ] } )
or even better:
ScrapReport.find( { "quantity": { "$lte": selectorMax } } )
The reason is because duplicate key are allowed in JSON document but the last value for a given key is maintained.
So this will only return all those documents where "quantity" is less than or equal selectorMax.
The second problem is already mentioned in #chridam's answer so the right query is:
ScrapReport.find({ "quantity": { "$gte": selectorMin, "$lte": selectorMax } })