Develop Reference

How to exclude Sundays from date range in mongodb query?

I need to get each user's transactions every three days. I want to know users that don't have up to a certain amount(200) within the three days period, then get the sum of all the transactions for each user. I want to exclude Sunday since transactions are always low on Sundays.
I want to make sure this is done right from the DB because the transactions from each user can run into thousands even millions.
I am using dayjs to manipulate the time but I am not getting it right
I have been able to get the three previous date and the current date. The previous date will be the startDate and the current date will be endDate.
But I need to remove if Sunday is in the range and use that to query the database.
This is what I have done what I am not close to fixing it.
How can I query the transaction table by dateCreated and exclude sundays?
schema sample
export const TransactionSchema = new mongoose.Schema({
description: {
type: String
},
ref: {
type: String,
required: true
},
userID: {
type: mongoose.SchemaTypes.ObjectId,
ref: 'User'
},
amount: {
type: Number,
required: true,
},
commission: {
type: Number,
default: 0
},
responseDescription: {
type: String
},
authCode: {
type: Number
},
isDeleted: {
type: Boolean,
default: false
},
dateCreated: {
type: Date,
default: Date.now
},
dateUpdated: {
type: Date,
required: false
},
openingBalance: {
type: Number
},
closingBalance: {
type: Number
},
})
method
async getUserRequiredTargetTrans() {
let now = dayjs();//endDate
let fromThreeDays = now.subtract('2', 'day')
let sunday = now.day(0)
let withOutSunday = now.diff(fromThreeDays);//startDate
const response = await this.transactionModel.find({ isDeleted: false, dateCreated: { $gte: withOutSunday, $lt: now } })

To exclude sundays from date range, you can use $where operator like this:
async getUserRequiredTargetTrans() {
let from = dayjs();//endDate
let fromThreeDays = now.subtract('2', 'day')
const response = await this.transactionModel.find({ isDeleted: false, dateCreated: { $gte: fromThreeDays, $lt: now },
// exculde sunday
$where: `function() { return this.dateCreated.getDay() !== 0;}`
} )

You pass a dayjs, don't know whether you can use it directly. Perhaps you have to use
find({ isDeleted: false, dateCreated: { $gte: withOutSunday.toDate(), $lt: now.toDate() } })

mongoose add more key:value inside ref field

organs: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'Organ',
healthValue: { // i want to add this field but it is becoming invalid, not generating auto with default val
type: Number,
default: 0,
}
},
],
I have a user schema and also there is organs key in this schema. I'm keeping organs with ref way to get belongs to user organs. I also should keep organ health value but I can not keep it in ref field together as above. How can i do this? Can not I add more key:value to populate (type & ref) fields?

organs: [
{
organId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Organ',
},
healthValue: {
type: Number,
default: 0,
},
isOwned: {
type: Boolean,
default: false,
},
},
],
i think i solve with above code block, i did use wrong syntax

How to change boolean value within an object of a document sub-array in Mongoose?

I have a rooms model. Within it is an array of User's which has its own model.
Each user has a bunch of different attributes, some of them being boolean. Knowing the ID of the specific room and the specific user, I am attempting to change the boolean value within a specific User element within the sub array like this:
Room.findOne({_id: roomId, "users" : user}, { "$set" : { mutedAudio : false}})
.then(doc => {
console.log("Unmuted audio");
res.json(doc)
io.in(roomId).emit('userchange');
})
.catch(err => {
console.log(err);
})
(I'm using a user model instead of a user ID for seeking the user within the sub array. Could not get ID to work but can fetch object by comparing it to itself entirely.)
I get the error:
MongoError: Unsupported projection option: $set: { mutedAudio: true }
Anyone know the answer to this?
Thank you.
EDIT:
const RoomSchema = new Schema({
owner: {
id: {
type: String
},
username: {
type: String
}
},
roomname: {
type: String,
required: true
},
category: {
type: String,
required: true
},
password: {
type: String,
required: false
},
users: [UserSchema],
messages: [{
username: {
type: String
},
message: {
type: String
},
time: {
type: String
}
}],
date: {
type: Date,
default: Date.now
}
});
const UserSchema = new Schema({
id: {
type: String
},
username: {
type: String,
required: true
},
email: {
type: String,
required: true
},
password: {
type: String,
required: true
},
avatar: {
type: String
},
date: {
type: Date,
default: Date.now
},
micEnabled: {
type: Boolean,
default: false
},
mutedAudio: {
type: Boolean,
default: true
}
});

Model.findOne() takes 4 parameters, the second being "optional fields to return", that's why you're getting the error, mongoose is trying to select fields to return according to $set: { mutedAudio: true } which is being passed as a second parameter (therefore considered to be a projection option).
Use Model.findOneAndUpdate() which takes an update object as a second parameter, along with the positional operator $.
Room.findOneAndUpdate(
{ "_id": roomId, "users._id": userID },{ "$set": { "users.$.mutedAudio": false } } )
.then(doc => {
console.log("Unmuted audio");
res.json(doc)
io.in(roomId).emit('userchange');
})
.catch(err => {
console.log(err);
})
Original answer by #Neil Lunn in Mongoose find/update subdocument

Mongoose find and create multidimensional arrays

I'm trying to get the userid, and the array object that contains the specific location within locations.
What I want to accomplish is the following:
The query will return the array location's result.
If not userid exist at all, create it, then return the array of the matching location.
If the location id Number is not there. create a new one, then return the array of the matching location.
How can I accomplish this?
Current query:
easyCrime.findOne({
userid: userid,
"locations.location": location
}, {"locations.$.location": 1}).then(function (err, stats) {
}
});
model:
userid: {
type: String,
default: '57c1c0f3b6b20c011242bf22'
},
locations: [
{
userid: {
type: String,
default: '57c1c0f3b6b20c011242bf22'
},
location: {
type: Number,
default: 1
},
easycrime: [
{
optioname : {
type: String,
default: 'unknown'
},
chance: {
type: Number,
default: 500
}
}
],
heavycrime: [
{
optioname : {
type: String,
default: 'unknown'
},
chance: {
type: Number,
default: 500
}
}
],
}
],
timesteal: {
type:Number,
default: 0
}

I presume that easyCrime is Model, cause there is no such thing as findOne query in a Document. If it is a Model please name it EasyCrime.
I had a really hard time interpreting your question. Base on what I understand, this is your solution
EasyCrime
.findOne({ userid: param.userid})
.exec((err, crime) => {
//userid not exists at all, create new
if (!crime) {
let newCrime = new EasyCrime({...});
newCrime.save(...);
return;
}
//Check if location exists
for (let i = 0; i < crime.locations.length; ++i) {
if (crime.locations[i].location === param.location) {
//crime.location[i] is what you're finding
return;
}
}
//Cannot find any location with provided param.location
crime.locations.push({
userid: ...,
location: param.location,
...
});
crime.save(...);
})

MongoDB queries optimisation

I wish to retrieve several information from my User model that looks like this:
var userSchema = new mongoose.Schema({
email: { type: String, unique: true, lowercase: true },
password: String,
created_at: Date,
updated_at: Date,
genre : { type: String, enum: ['Teacher', 'Student', 'Guest'] },
role : { type: String, enum: ['user', 'admin'], default: 'user' },
active : { type: Boolean, default: false },
profile: {
name : { type: String, default: '' },
headline : { type: String, default: '' },
description : { type: String, default: '' },
gender : { type: String, default: '' },
ethnicity : { type: String, default: '' },
age : { type: String, default: '' }
},
contacts : {
email : { type: String, default: '' },
phone : { type: String, default: '' },
website : { type: String, default: '' }
},
location : {
formattedAddress : { type: String, default: '' },
country : { type: String, default: '' },
countryCode : { type: String, default: '' },
state : { type: String, default: '' },
city : { type: String, default: '' },
postcode : { type: String, default: '' },
lat : { type: String, default: '' },
lng : { type: String, default: '' }
}
});
In Homepage I have a filter for location where you can browse Users from Country or City.
All the fields contains also the number of users in there:
United Kingdom
All Cities (300)
London (150)
Liverpool (80)
Manchester (70)
France
All Cities (50)
Paris (30)
Lille (20)
Nederland
All Cities (10)
Amsterdam (10)
Etc...
This in the Homepage, then I have also the Students and Teachers pages where I wish to have information only about how many teachers there are in those Countries and Cities...
What I'm trying to do is to create a query to MongoDB to retrieve all these information with a single query.
At the moment the query looks like this:
User.aggregate([
{
$group: {
_id: { city: '$location.city', country: '$location.country', genre: '$genre' },
count: { $sum: 1 }
}
},
{
$group: {
_id: '$_id.country',
count: { $sum: '$count' },
cities: {
$push: {
city: '$_id.city',
count: '$count'
}
},
genres: {
$push: {
genre: '$_id.genre',
count: '$count'
}
}
}
}
], function(err, results) {
if (err) return next();
res.json({
res: results
});
});
The problem is that I don't know how to get all the information I need.
I don't know how to get the length of the total users in every Country.
I have the users length for each Country.
I have the users length for each city.
I don't know how to get the same but for specific genre.
Is it possible to have all these information with a single query in Mongo?
Otherwise:
Creating few promises with 2, 3 different requests to Mongo like this:
getSomething
.then(getSomethingElse)
.then(getSomethingElseAgain)
.done
I'm sure it would be easier storing every time specified data but: is it good for performance when there are more than 5000 / 10000 users in the DB?
Sorry but I'm still in the process of learning and I think these things are crucial to understand MongoDB performance / optimisation.
Thanks

What you want is a "faceted search" result where you hold the statistics about the matched terms in the current result set. Subsequently, while there are products that "appear" to do all the work in a single response, you have to consider that most generic storage engines are going to need multiple operations.
With MongoDB you can use two queries to get the results themselves and another to get the facet information. This would give similar results to the faceted results available from dedicated search engine products like Solr or ElasticSearch.
But in order to do this effectively, you want to include this in your document in a way it can be used effectively. A very effective form for what you want is using an array of tokenized data:
{
"otherData": "something",
"facets": [
"country:UK",
"city:London-UK",
"genre:Student"
]
}
So "factets" is a single field in your document and not in multiple locations. This makes it very easy to index and query. Then you can effectively aggregate across your results and get the totals for each facet:
User.aggregate(
[
{ "$unwind": "$facets" },
{ "$group": {
"_id": "$facets",
"count": { "$sum": 1 }
}}
],
function(err,results) {
}
);
Or more ideally with some criteria in $match:
User.aggregate(
[
{ "$match": { "facets": { "$in": ["genre:student"] } } },
{ "$unwind": "$facets" },
{ "$group": {
"_id": "$facets",
"count": { "$sum": 1 }
}}
],
function(err,results) {
}
);
Ultimately giving a response like:
{ "_id": "country:FR", "count": 50 },
{ "_id": "country:UK", "count": 300 },
{ "_id": "city:London-UK", "count": 150 },
{ "_id": "genre:Student": "count": 500 }
Such a structure is easy to traverse and inspect for things like the discrete "country" and the "city" that belongs to a "country" as that data is just separated consistently by a hyphen "-".
Trying to mash up documents within arrays is a bad idea. There is a BSON size limit of 16MB to be respected also, from which mashing together results ( especially if you are trying to keep document content ) is most certainly going to end up being exceeded in the response.
For something as simple as then getting the "overall count" of results from such a query, then just sum up the elements of a particular facet type. Or just issue your same query arguments to a .count() operation:
User.count({ "facets": { "$in": ["genre:Student"] } },function(err,count) {
});
As said here, particularly when implementing "paging" of results, then the roles of getting "Result Count", "Facet Counts" and the actual "Page of Results" are all delegated to "separate" queries to the server.
There is nothing wrong with submitting each of those queries to the server in parallel and then combining a structure to feed to your template or application looking much like the faceted search result from one of the search engine products that offers this kind of response.
Concluding
So put something in your document to mark the facets in a single place. An array of tokenized strings works well for this purpose. It also works well with query forms such as $in and $all for either "or" or "and" conditions on facet selection combinations.
Don't try and mash results or nest additions just to match some perceived hierarchical structure, but rather traverse the results received and use simple patterns in the tokens. It's very simple to
Run paged queries for the content as separate queries to either facets or overall counts. Trying to push all content in arrays and then limit out just to get counts does not make sense. The same would apply to a RDBMS solution to do the same thing, where paging result counts and the current page are separate query operations.
There is more information written on the MongoDB Blog about Faceted Search with MongoDB that also explains some other options. There are also articles on integration with external search solutions using mongoconnector or other approaches.