Develop Reference

How can i improve my query speed in MongoDB, NodeJS?

I have one collection who include some value coming from sensor. My collection look like this.
const MainSchema: Schema = new Schema(
{
deviceId: {
type: mongoose.Types.ObjectId,
required: true,
ref: 'Device',
},
sensorId: {
type: mongoose.Types.ObjectId,
default: null,
ref: 'Sensor',
},
value: {
type: Number,
},
date: {
type: Date,
},
},
{
versionKey: false,
}
);
I want to get data from this collection with my endpoint. This collection should has more 300.000 documents. I want to get data from this collection with sensor data. (like name and desc. to "Sensor")
My Sensor Collection:
const Sensor: Schema = new Schema(
{
name: {
type: String,
required: true,
min: 3,
},
description: {
type: String,
default: null,
},
type: {
type: String,
},
},
{
timestamps: true,
versionKey: false,
}
);
I use 2 method for get data from MainSchema. First approach is look like this (Include aggregate):
startDate, endDate and _sensorId are passed by parameter for this functions.
const data= await MainSchema.aggregate([
{
$lookup: {
from: 'Sensor',
localField: 'sensorId',
foreignField: '_id',
as: 'sensorDetail',
},
},
{
$unwind: '$sensorDetail',
},
{
$match: {
$and: [
{ sensorId: new Types.ObjectId(_sensorId) },
{
date: {
$gte: new Date(startDate),
$lt: new Date(endDate),
},
},
],
},
},
{
$project: {
sensorDetail: {
name: 1,
description: 1,
},
value: 1,
date: 1,
},
},
{
$sort: {
_id: 1,
},
},
]);
Second approach look like this (Include find and populate):
const data= await MainSchema.find({
sensorId: _sensorId,
date: {
$gte: new Date(startDate),
$lte: new Date(endDate),
},
})
.lean()
.sort({ date: 1 })
.populate('sensorId', { name: 1, description: 1});
Execution time for same data set:
First approach: 25 - 30 second
Second approach: 11 - 15 second
So how can i get this data more faster. Which one is best practise?
And how can i do extras for improve the query speed?

Overall #NeNaD's answer touches on a lot of the important points. What I'm going to say in this one should be considered in addition to that other information.
Index
Just to clarify, the ideal index here would be a compound index of { sensorId: 1, date: 1 }. This index follows the ESR Guidance for index key ordering and will provide the most efficient retrieval of the data according to the query predicates specified in the $match stage.
If the index: true annotation in Mongoose creates two separate single field indexes, then you should go manually create this index in the collection. MongoDB will only use one of those indexes to execute this query which will not be as efficient as using the compound index described above.
Also regarding the existing approach, what is the purpose of the trailing $sort?
If the application (a chart in this situation) does not need sorted results then you should remove this stage entirely. If the client does need sorted results then you should:
Move the $sort stage earlier in the pipeline (behind the $match), and
Test if including the sort field in the index improves performance.
As written, the $sort is currently a blocking operation which is going to prevent any results from being returned to the client until they are all processed. If you move the $sort stage up and can change it to sort on date (which probably makes sense for sensor data) the it should automatically use the compound index that we mentioned earlier to provide the sort in a non-blocking manner.
Stage Ordering
Ordering of aggregation stages is important, both for semantic purposes as well as for performance reasons. The database itself will attempt to do various things (such as reordering stages) to improve performance so long as it does not logically change the result set in any way. Some of these optimizations are described here. As these are version specific anyway, you can always take a look at the explain plan to get a better indication of what specific changes the database has applied. The fact that performance did not improve when you manually moved the $match to the beginning (which is generally a best practice) could suggest that the database was able to automatically do that on your behalf.
Schema
I'm a little curious about the schema itself. Is there any reason that there are two separate collections here?
My guess is that this is mostly a play at 'normalization' to help reduce data duplication. That is mostly fine, unless you find yourself constantly performing $lookups like this for most of your read operations. You could certainly consider testing what performance (and storage) looks like if you combine them.
Also for this particular operation, would it make sense to just issue two separate queries, one to get the measurements and one to get the sensor data (a single time)? The aggregation matches on sensorId and the value of that field is what is then used to match against the _id field from the other collection. Unless I'm doing the logic wrong, this should be the same data for each of the source documents.
Time Series Collections
Somewhat related to schema, have you looked into using Time Series Collections? I don't know what your specific goals or pain points are, but it seems that you may be working with IoT data. Time Series collections are purpose-built to help handle use cases like that. Might be worth looking into as they may help you achieve your goals with less hassle or overhead.

Frist step
Create index for sensorId and date properties in the collection. You can do it by specifying index: true in your model:
const MainSchema: Schema = new Schema(
{
deviceId: { type: mongoose.Types.ObjectId, required: true, ref: 'Device' },
sensorId: { type: mongoose.Types.ObjectId, default: null, ref: 'Sensor', index: true },
value: { type: Number },
date: { type: Date, index: true },
},
{
versionKey: false,
}
);
Second step
Aggregation queries can take leverage of indexes only if your $match stage is the first stage in the pipeline, so you should change the order of the items in your aggregation query:
const data= await MainSchema.aggregate([
{
$match: {
{ sensorId: new Types.ObjectId(_sensorId) },
{
date: {
$gte: new Date(startDate),
$lt: new Date(endDate),
},
},
},
},
{
$lookup: {
from: 'Sensor',
localField: 'sensorId',
foreignField: '_id',
as: 'sensorDetail',
},
},
{
$unwind: '$sensorDetail',
},
{
$project: {
sensorDetail: {
name: 1,
description: 1,
},
value: 1,
date: 1,
},
},
{
$sort: {
_id: 1,
},
},
]);

how to return values from joined tables in sequelize at the same level as master table

I am trying to find a way how to put all joined tables at the same level as my master table... so far it only results in the nested values in my final object ..
Here is what I have
Orders.findAll({
include: [
{model: Products, attributes: ['product_name']}
],
attributes: ['id_order', 'dtime_order', 'amount']
})
what I am getting is:
[
{
id_order: 1,
dtime_order: '2021-05-24T22:00:00.000Z',
amount: 20,
products: {
product_name: 'Picture'
}
}
]
but what I wanna get is:
[
{
id_order: 1,
dtime_order: '2021-05-24T22:00:00.000Z',
amount: 20,
product_name: 'Picture'
}
]
I tried this How to return result from include model in same level of main model in Sequelize? but unfortunately when I did:
Orders.findAll({
include: [
{model: Products, attributes: []}
],
attributes: ['id_order', 'dtime_order', 'amount', ['products.product_name', 'product_name']]
})
doesn't work for me saying
column "products.product_name" does not exist
There might be a hacky way to modify the object before sending it back in the response .. but I would rather do it within Sequelize ..
any idea is very welcome... thank you guys!
EDIT: adding the generated SQL
Executing (default): SELECT "orders"."id_order", "orders"."dtime_order", "orders"."amount", "products.product_name", FROM "orders" AS "orders" LEFT OUTER JOIN "products" AS "products" ON "orders"."id_order" = "products"."ir_order";
error: Get dashboard data error: column "products.product_name" does not exist
SOLUTION:
I had to use an alias in my association
Orders.hasOne(Products, {as: 'products', ....})
And then use that EXACTLY SAME alias in my include and referencing
include: [{model: Products, attributes: [], as: 'products'}]
And
attributes: [ ... , [Sequelize.col('products.product_name', 'product_name')]
without the raw: true works like a charm :) Thank you #Emma !!!

Please use Sequelize.col to wrap the nested column so that Sequelize can properly alias the column.
Orders.findAll({
include: [
{model: Products, attributes: []}
],
attributes: ['id_order', 'dtime_order', 'amount', [Sequelize.col('products.product_name'), 'product_name']],
raw: true
})

Need to count a joined table rows sequelize

I am using sequelize to query on a sqlserver database. I have two tables:
data: columns - id, name, team, type
history:columns - id, node, date, status, data_id(foreignkey)
and a relation
history.belongsTo(data, {foreignKey: 'data_id'}
data.hasMany(history, {foreignKey: 'data_id'})
My query is:
dataModel.findAll({
attributes: ['name'],
include: [{
model:historyModel
}]
})
My result looks like this:
[
{
name: "1",
history: [
{
...
}
]
},
{
name: "2",
history: [
{
...
}
]
}
]`
I want that instead of the history array I will have the count of history objects in each one. The query in sql is:
select data.name, count(history.data_id) count
from history
inner join data on data.id=history.data_id
group by history.data_id, data.name

You can do it this way:
dataModel.findAll({
attributes: {
include: [[Sequelize.fn("COUNT", Sequelize.col("history.data_id")), "historyModelCount"]]
},
include: [{
model: historyModel, attributes: []
}],
group: ['data.id']
});

sequelize - get AVG of included model

I have a a schema that is like product: { ... ,ratings: [ {rating: 3} ] }, and using sequalize, I would like to add on a property product.avg_rating using sequelize.
Here is my code:
sequelize.models.product.findAll({
include: [{
model: sequelize.models.rating,
as: 'ratings',
attributes: {
include: ['rating',[sequelize.fn('AVG', 'ratings.rating'), 'avg_rating']]
},
group: ['rating.rating'],//also tried ratings.rating, and just rating
}],
}).then(products=>{...})
But I keep getting this error:
In aggregated query without GROUP BY, expression #1 of SELECT list
contains nonaggregated column 'text_rewriter.languageCombination.id';
this is incompatible with sql_mode=only_full_group_by
Goal Output:
products: [
{product: 'product name',
...
ratings: [...],
avg_rating: 3.7 //THIS AVG IS WHAT I WANT
},{...}
]
any ideas what I am missing? I have seen many examples, but none of them use include like I did that I found.

sequelize.models.product.findAll({
include: [{
model: sequelize.models.rating,
as: 'ratings',
attributes: ['avg_rating'] //this is column name here
}],
}).then(products=>{...})
This will return :-
products: [
{product: 'product name',
ratings: [...],
rating : { //here all the attributes you wanted, in this case only 'avg_rating' }
},
{...}
]
But you have to define relastionship between products table and rating table before using this.
Table : Product have id, name
Table : Rating have id, rating, product_id
In above case relationship will be 'rating.belongsTo(product) OR product.hasMay(rating)'

MySQL implements detection of functional dependence. If the ONLY_FULL_GROUP_BY SQL mode is enabled (which it is by default), MySQL rejects queries for which the select list, HAVING condition, or ORDER BY list refer to non-aggregated columns that are neither named in the GROUP BY clause nor are functionally dependent on them.
The error is related to sql_mode, you need to execute the following command on your database console.
SET GLOBAL sql_mode = '';

Sequelize conditional inclusion of where clause nodejs

I have this code, which has multiple where clause:
Time_Sheet_Details.findAll({
include: [
{
model: timesheetNotesSubcon,
required: false,
attributes:["note","file_name", "id", "working_hrs", "timestamp", "has_screenshot", "notes_category"]
},
{
model: Timesheet,
attributes:["id","leads_id","userid"],
where: {leads_id: filters.leads_id}, // Client
include:[
{
model: Lead_Info, attributes:["id","fname","lname","email","hiring_coordinator_id","status"],
where: {hiring_coordinator_id: {$in: filters.sc_id}}, // SC
include:[{
model: adminInfoSchema,
required: false,
attributes:["admin_id","admin_fname", "admin_lname", "admin_email", "signature_contact_nos", "signature_company"],
}]
},
{
model:Personal_Info,attributes:["userid","fname","lname","email"],
where: {userid: filters.subcon_id}, // Subcon
}
]
}],
where: {
reference_date: filters.reference_date
},
order:[
["id","DESC"]
],
offset:((page-1)*limit),
limit : limit,
subQuery:false
}).then(function(foundObject){
willFulfillDeferred.resolve(foundObject);
});
The where clause is the one with the comment Client, SC and Subcon. However, what is the best approach if those where clause is optional? I am using that for search filter. So if filters.leads_id is null then the where: {leads_id: filters.leads_id}, // Client should not be included in the query. Same with the others. The only solution I can think of is repeat those code blocks for each scenario of not null parameters but that's to repetitive and not practical.
Any other approach or solutions?

If I understand correctly, I think as a first step, you should define your respective where clauses, conditionally upon wether or not each specific search criteria is set:
const clientWhere = filters.leads_id ? {leads_id: filters.leads_id} : {}
const scWhere = filters.sc_id ? {hiring_coordinator_id: {$in: filters.sc_id}} : {}
const subconWhere = filters.subcon_id ? {userid: filters.subcon_id} : {}
So at this point if a search option isn't set, there'll just be an empty object as the where clause.
Next, use those pre-defined where clause objects in your query:
Time_Sheet_Details.findAll({
include: [
{
model: timesheetNotesSubcon,
required: false,
attributes:["note","file_name", "id", "working_hrs", "timestamp", "has_screenshot", "notes_category"]
},
{
model: Timesheet,
attributes:["id","leads_id","userid"],
where: clientWhere, // Client
include:[
{
model: Lead_Info, attributes:["id","fname","lname","email","hiring_coordinator_id","status"],
where: scWhere, // SC
include:[{
model: adminInfoSchema,
required: false,
attributes:["admin_id","admin_fname", "admin_lname", "admin_email", "signature_contact_nos", "signature_company"],
}]
},
{
model:Personal_Info,attributes:["userid","fname","lname","email"],
where: subconWhere, // Subcon
}
]
}],
where: {
reference_date: filters.reference_date
},
order:[
["id","DESC"]
],
offset:((page-1)*limit),
limit : limit,
subQuery:false
}).then(function(foundObject){
willFulfillDeferred.resolve(foundObject);
});