Develop Reference

Sequelize - Can't limit properly query with includes, using hasMany and belongsToMany associations

Issue explanation
I want to do a query with pagination that limits to 12 lines each query, to be more specific, that limits to 12 Batches each query. Actually the amount of lines get smaller because a belongsToMany association with a join table i got in this query. The join order to this query is: Offer > hasMany > Batch > belongsToMany > BatchFile > belongsTo > File. The problem is, when i have many registries in the File as 'gallery' association, it brings me duplicated registries of batch.
The query i'm trying to do
const { id } = req.params;
const { page = 1 } = req.query;
const offer = await Offer.findAndCountAll({
attributes: [ 'id', 'name', 'canceled_at'],
where: { id, canceled_at: null },
order: [['offer_batches', 'name', 'ASC']],
include: [
{
/* hasMany association */
model: Batch,
as: 'offer_batches',
attributes: [ 'id', 'name'],
include: [
{ /* Other includes... */ },
{
/* belongsToMany association */
model: File,
as: 'gallery',
attributes: ['id', 'path', 'url'],
},
],
},
],
subQuery: false,
limit: 12,
offset: (page - 1) * 12,
});
Model associations
Offer model
this.hasMany(Batch, { foreignKey: 'offer_id', as: 'offer_batches' });
Batch model
this.belongsToMany(File, { through: models.BatchFile, as: 'gallery' });
BatchFile model (join table)
this.belongsTo(Batch, { foreignKey: 'batch_id', as: 'batch' });
this.belongsTo(File, { foreignKey: 'file_id', as: 'file' });
What i already tried
Giving duplicating: false option to any included Model doesn't worked;
Giving separate: true to the Batch model doesn't worked too;
Giving required: true option to any included model doesn't worked too;
If i remove subQuery: false it doesn't respect the setted limit of lines, and i already tried with all of the above combinations;
I thought sequelize would deal with this situation without problems, maybe i'm doint something wrong.
If helps, here's the raw generated SQL:
SELECT
"Offer"."id",
"Offer"."name",
"Offer"."canceled_at",
"offer_batches"."id"
AS "offer_batches.id", "offer_batches"."name"
AS "offer_batches.name", "offer_batches->gallery"."id"
AS "offer_batches.gallery.id", "offer_batches->gallery"."path"
AS "offer_batches.gallery.path", "offer_batches->gallery->BatchFile"."created_at"
AS "offer_batches.gallery.BatchFile.createdAt", "offer_batches->gallery->BatchFile"."updated_at"
AS "offer_batches.gallery.BatchFile.updatedAt", "offer_batches->gallery->BatchFile"."file_id"
AS "offer_batches.gallery.BatchFile.FileId", "offer_batches->gallery->BatchFile"."batch_id"
AS "offer_batches.gallery.BatchFile.BatchId", "offer_batches->gallery->BatchFile"."batch_id"
AS "offer_batches.gallery.BatchFile.batch_id", "offer_batches->gallery->BatchFile"."file_id"
AS "offer_batches.gallery.BatchFile.file_id"
FROM "offer" AS "Offer"
LEFT OUTER JOIN "batch" AS "offer_batches" ON "Offer"."id" = "offer_batches"."offer_id"
LEFT OUTER JOIN (
"batch_file" AS "offer_batches->gallery->BatchFile"
INNER JOIN "file" AS "offer_batches->gallery"
ON "offer_batches->gallery"."id" = "offer_batches->gallery->BatchFile"."file_id"
)
ON "offer_batches"."id" = "offer_batches->gallery->BatchFile"."batch_id"
WHERE "Offer"."id" = '1' AND "Offer"."canceled_at" IS NULL
ORDER BY "offer_batches"."name"
ASC LIMIT 12 OFFSET 0;
Environment
Node: v14.18.0
package.json dependencies
pg: 8.7.1
pg-hstore: 2.3.4
sequelize: 6.9.0
Trying to find any solution on GitHub issues or stackoverflow, nothing solved this problem. Maybe i'm doing this query wrong, any help would be grateful and welcome :)

Well, i didn't found a quite solution for this, so i resolved to use Lazy loading for this situation instead of Eager loading, as mentioned on Sequelize Docs.
I splitted the query into two new queries.
First one, on the "master" model Offer, with a simple findOne():
const offer = await Offer.findOne({
[/* My attributes */],
where: { /* My conditions */ }
});
And a second one, selecting from model Batch, without subQuery: false option, because is not needed anymore.
const batches = await Batch.findAndCountAll({
attributes: [
'id',
'name',
],
where: { offer_id: offer.id },
order: [/* My ordenation */],
include: [
{
model: File,
as: 'gallery',
attributes: ['id', 'path', 'url'],
},
],
limit: 12,
offset: (page - 1) * 12,
});

Sequelize get results when aggregate function is null

I have the following code which returns customers. The problem is that I get results only for customers that have placed an order and have created an address.
CustomerModel.findAll({
attributes: {
include: [
[db.sequelize.fn("COUNT", db.sequelize.col("orders.id")), "orderCount"],
[db.sequelize.literal(`(
SELECT o.createdDate
FROM orders AS o
WHERE
o.customerId = customer.id
ORDER BY o.createdDate DESC
LIMIT 1
)`),
'latestOrderDate'],
[db.sequelize.fn('sum', db.sequelize.col('orders.amount')), 'totalAmount'],
]
},
include: [
{ model: UserModel, as: 'user' },
{ model: AddressModel, required:false },
{ model: OrderModel, attributes: [], required:false }
]
})
Is it possible to get all customers regardless of having an order or an Address?
Thank you!

I ended up changing the two sequelize.fn to literal and it worked like a charm!

Sequelize order by id for root table when we are using join method

this below Sequelize work fine for me without using order, i'm wondering why i can't use order for root table as posts model? when i use this below code i get this error:
Unhandled rejection Error: 'posts' in order / group clause is not valid association
but that work fine on other models such as channelVideoContainer
models.posts.findAll({
where: {
channelId: 1
},
include: [
{
model: models.channelVideoContainer,
include: [models.fileServerSetting]
}, {
model: models.channelMusicContainer,
include: [models.fileServerSetting]
}, {
model: models.channelImageWithTextContainer,
include: [models.fileServerSetting]
}, {
model: models.channelFilesContainer,
include: [models.fileServerSetting]
},
models.channelPlainTextContainer
], order: [
[{model: models.posts}, 'id', 'DESC'],
], limit: 5
}).then(function (result) {
console.log(JSON.stringify(result));
});

You are getting this error because you are querying the posts table/model, and then sorting by a column on the posts model, however you are specifying a "joined" table in your order. This works for your other models because they are in fact joined (using the include option). Since you are querying the posts model you just need to pass in the name of the column you want to order by. See some of the ORDER examples in the documentation.
// just specify the 'id' column, 'post' is assumed because it is the queried Model
order: [['id', 'DESC']],
As a side note, you may want to specify required: false on your include'd models to perform a LEFT JOIN so that rows come back even if there are no matches in the joined table. If you know that rows will be returned (or they are actually required) then leave it as is.
{
model: models.channelFilesContainer,
include: [models.fileServerSetting],
required: false, // LEFT JOIN the channelFilesContainer model
},

Sequelize: how to do a WHERE condition on joined table with left outer join

My database model is as follows:
An employee drives one or zero vehicles
A vehicle can be driven by one or more employees
A vehicle has a model type that tells us it's fuel type amongst other things.
I'd like sequelize to fetch me all employees where they don't drive a vehicle, or if they do then the vehicle is not diesel.
So where VehicleID is null OR Vehicle.VehicleModel.IsDiesel = false
My current code is as follows:
var employee = sequelize.define('employee', {
ID: Sequelize.INTEGER,
VehicleID: Sequelize.INTEGER
});
var vehicle = sequelize.define('vehicle', {
ID: Sequelize.INTEGER,
ModelID: Sequelize.INTEGER
});
var vehicleModel = sequelize.define('vehicleModel', {
ID: Sequelize.INTEGER,
IsDiesel: Sequelize.BOOLEAN
});
employee.belongsTo(vehicle);
vehicle.belongsTo(vehicleModel);
If I run the following:
options.include = [{
model: model.Vehicle,
attributes: ['ID', 'ModelID'],
include: [
{
model: model.VehicleModel,
attributes: ['ID', 'IsDiesel']
}]
}];
employee
.findAll(options)
.success(function(results) {
// do stuff
});
Sequelize does a left outer join to get me the included tables. So I get employees who drive vehicles and who don't.
As soon as I add a where to my options:
options.include = [{
model: model.Vehicle,
attributes: ['ID', 'ModelID'],
include: [
{
model: model.VehicleModel,
attributes: ['ID', 'IsDiesel']
where: {
IsDiesel: false
}
}]
}];
Sequelize now does an inner join to get the included tables.
This means that I only get employees who drive a vehicle and the vehicle is not diesel. The employees who don't drive a vehicle are excluded.
Fundamentally, I need a way of telling Sequelize to do a left outer join and at the same time have a where condition that states the column from the joined table is false or null.
EDIT:
It turns out that the solution was to use required: false, as below:
options.include = [{
model: model.Vehicle,
attributes: ['ID', 'ModelID'],
include: [
{
model: model.VehicleModel,
attributes: ['ID', 'IsDiesel']
where: {
IsDiesel: false
},
required: false
}],
required: false
}];
I had already tried putting the first 'required:false' but I missed out on putting the inner one. I thought it wasn't working so I gave up on that approach. Dajalmar Gutierrez's answer made me realise I needed both for it to work.

When you add a where clause, sequelize automatically adds a required: true clause to your code.
Adding required: false to your include segment should solve the problem
Note: you should check this issue iss4019

Eager loading
When you are retrieving data from the database there is a fair chance that you also want to get associations with the same query - this is called eager loading. The basic idea behind that, is the use of the attribute include when you are calling find or findAll.
when you set
required: false
will do
LEFT OUTER JOIN
when
required: true
will do
INNER JOIN
for more detail docs.sequelizejs eager-loading

allow null value for sequelize foreignkey?

How do I allow null for a foreignkey? I have a through and belongsToMany association:
Shelf
belongsToMany(Book, { through: Library, as: "Books"});
Book
belongsToMany(Shelf, { through: Library, as: "Shelves"});
Library
section: DataTypes.STRING,
// ... other fields
I would like to record a null on the bookId in Library:
Library.create({
section: "blah",
bookId: null,
shelfId: 3
});
Since Sequelize automatically creates the bookId and shelfId in the join table Library, how would I specify that I want to allow a null on the bookId foreignkey in Library?

I am no expert on node.js nor sequelize.js, but with one search in google I got the official documentation. You might need to pay some attention to this part of the code in the documentation:
Library.hasMany(Picture, {
foreignKey: {
name: 'uid',
allowNull: false
}
});
It seems you need to specify the {foreignKey: {name: 'bookId', allowNull: true} } in Library.

By default, the Sequelize association is considered optional. If you would like to make the foreign key required, you can set allowNull to be false.
Foo.hasOne(Bar, {
foreignKey: {
allowNull: false
}
});
See the docs