I'm using sequelize and mysql2 and need to select all the distinct values from multiple columns.
For example I've a 'Title' and a 'Location' column.
At the moment this:
Job.findAll({
attributes: [
[Sequelize.fn('DISTINCT', Sequelize.col('title')), 'title']
]
})
works, returning all the job titles that are distinct. But if I add a second line for the location column I get a syntax error
Job.findAll({
attributes: [
[Sequelize.fn('DISTINCT', Sequelize.col('title')), 'title'],
[Sequelize.fn('DISTINCT', Sequelize.col('location')), 'location']
]
})
It feels like it should work, from what I've read in the docs at least - but I'm not exactly experienced or confident with sequelize, so any help would be appreciated.
Thanks
I haven't yet found a way to do this less-hackily, but you can exploit the fact that DISTINCT isn't really a function, but a statement and always operates on the entirety of the selected column set:
Job.findAll({
attributes: [
[Sequelize.fn('DISTINCT', Sequelize.col('title')), 'title'],
'location'
]
})
would generate SQL similar to
SELECT DISTINCT(`title`) AS `title`, `location` FROM `Jobs`;
but since DISTINCT is not a function, this is really the same as
SELECT DISTINCT (`title`) AS `title`, `location` FROM `Jobs`;
which does do what you want because parenthesis around column names are optional, that query up there would be the same as
SELECT DISTINCT `title`, `location` FROM `Jobs`;
Source: https://dzone.com/articles/sql-distinct-is-not-a-function (the article also talks about the DISTINCT ON PostgreSQL extension, if someone happens to need that too)
Please note that this feels very hacky and sort of fragile and could/might break if Sequelize ever decides to change the way functions are applied when generating the SQL, but it works in the version of Sequelize I use (5.22.3).
Related
I am going off this small example off the sequelize docs
https://sequelize.org/docs/v6/advanced-association-concepts/creating-with-associations/#belongsto--hasmany--hasone-association
and I am not 100% following, for our project, we have a table that has a one to many association, but also two, one to many, associations within a one to many (I know a bit hard to word that)
so Table 1 has a one to many to table 2 and table 2 has a one to many to table 3 and a one to many to table 4
and I am trying to understand how to translate that to the example in that site (url) above into something workable for myself.
I just dont fully understand how to do it.
for example, looking at the example,
include: [{
association: Product.User,
include: [ User.Addresses ]
}]
I am not sure what include means vs association means. Other examples on that page seem to use association in the same way as where include is used (I think). I also am not sure how this works with multiple layers as I described above vs this more simple example.
Now I know this is a bigger-ish type of create but we are wondering if its a pluasble thing to do vs multiple calls to create data for those other tables when creating a largr dataset vs just doing a single create like this exmaple says is possible :)
if anyone has any advice I would aapperciate it, thanks!
association option is just another alternative to indicate an associated model: you either use include and indicate both a model ans an alias OR use association and indicate an association which already stores an associated model and its alias.
As for 4 tables each of which in its turn is linked as one-to-many to the next one you just need to use nested include/association options and that's all. Also if you want to get all-in-one then don't forget to indicate separate: true on each level of include/association options otherwise a generated SQL query might have too many records and you'll get the out of memory error.
const allInOneItems = Table1.findAll({
include: [{
model: Table2,
separate: true,
include: [{
model: Table3,
separate: true,
include: [{
model: Table4,
separate: true,
}]
}]
}]
})
I have a Firestore collection named channels, and I'd like to get the list of channels based on an array of IDs and order it by the createdAt field, this is my function :
const getChannels = () => {
const q = query(
collection(db, "channels"),
where(documentId(), "in", [
"F0mnR5rNdhwSLPZ57pTP",
"G8p6TWSopLN4dNHJLH8d",
"wMWMlJwa3m3lYINNjCLT",
]),
orderBy("createdAt")
);
const unsubscribe = onSnapshot(q, (snapshot) => {
snapshot.docs.map((doc) => {
console.log(doc.data());
});
});
return unsubscribe;
};
But I'm getting this error
FirebaseError: inequality filter property and first sort order must be the same: __name__ and createdAt.
It only works if I orderBy documentId().
I'm aware there is a limitation in the docs about this, but I'm wondering if there is a workaround for this type of situation.
Also the answer for this question isn't working anymore I guess.
The title of your question indicates that you are trying to use where and orderBy for different fields. But note that you are using documentId() in the where condition to filter, which is not a field in the Firestore document.
So if you filter is based on documentId(), you can use only documentId() in orderBy() clause, that also in ascending order because currently Firestore does not support sorting in descending order of documentId() which is mentioned in this answer.
Let’s take a look at the following examples -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).orderBy(admin.firestore.FieldPath.documentId()).get();
The above will work and sort the documents based on documentId() after filtering based on documentId().
But it is not relevant to apply an orderBy() clause based on the documentId(), because without applying the orderBy() clause also yields the same result as, by default, Firestore query gives documents in ascending order of documentId(). That means the following also yields the same result -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).get();
Now Firestore doesn’t support to sort in descending order of documentId() which means the following will not work -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).orderBy(admin.firestore.FieldPath.documentId(),"desc").get();
This will ask to create an index -
The query requires an index. You can create it here:
But if you go there to create an index it will say -
__name__ only indexes are not supported.
Now let's come to your query. What you are trying to do is to filter based on documentId() and then orderBy() based on createdAt field which is not possible and it will give the following error-
inequality filter property and first sort order must be the same.
You may think to use two orderBy() clauses, something like this -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).orderBy(admin.firestore.FieldPath.documentId()).orderBy(“createdAt”
).get();
Which will not work and give the following error
order by clause cannot contain more fields after the key
I am not sure of your use case but it’s not a great idea to filter based on documentId(). If it is required to filter based on documentId(), I would suggest creating a field in the Firestore document which will contain the documentIds and filter based on that.
Now considering the title of the question, yes it is possible to use where() and orderBy() clauses for different fields in Firestore. There are some limitations and you need to stick to that -
If you include a filter with a range comparison (<, <=, >, >=), your first ordering must be on the same field.
const data=await db.collection("users").where(“number”,">=", “101”).orderBy(“createdAt”).get();
The above query doesn't work.
const data=await db.collection("users").where(“number”,">=", “101”).orderBy(“number”).get();
The above query works and you can still use further orderBy() on different fields, something like following -
const data=await db.collection("users").where(“number”,">=", “101”).orderBy(“number”).orderBy(“createdAt”).get();
You cannot order your query by any field included in an equality (=) or in clause.
const data=await db.collection("users").where(“number”,"in",["104","102","101"]).orderBy(“number”).get();
const data=await db.collection("users").where(“number”,"==", “101”).orderBy(“number”).get();
The above two don’t work.
Firestore's speed and efficiency comes almost ENTIRELY from it's use of indexes. Inequalities (INCLUDING in and not-in) are accomplished by sorting by the index, and using the value as a "cut-off" - thus REQUIRING (whether you want it or not) the orderby() to be on the same field as the inequality.
The "answer not working anymore" was never really working in the first place, as the above shows. If you aren't trying to paginate, do the obvious and "filter" by the document ID's and sort on the client.
BUT...
...more importantly, it is ALMOST NEVER useful nor performant to use documentId's to select from the database, unless you both copy it to a field, AND are looking for a SPECIFIC id. In almost all cases, it would be FAR better to use a query on another field (however you got the list of documentId's in the first place), then orderBy. Yes, the inequality/orderBy is a limitation, but it's there for a reason.
Going forward, an important design decision is to understand what questions you want your data to answer, and design your entire database schema to support those queries - this is the fundamental nature of NoSQL.
Problem:The other link that you have shared before perfectly works and the only solutions available is to create an index. However the reason you are not able to do a where and order with the above example is because you cannot create an index with the document id and createdAt.
Solution: To do so add the document id as one of the field say docID in the document then create an index with the fields docID and createdAt. This should be working for you.
Note: I have not physically tested this. Will update once I have checked it
I need to select some rows in a PostgreSQL database using Sequelize where an id exists in joined table 1 OR it exists in joined table 2.
Using Sequelize I tried querying the joined tables inside of 'include' but realized that in some cases, if an id didn't exist then that table was excluded from the resulting rows which isn't what I want. I think that what I want is something more like the following:
Model.findAll({
include: [ Association1, Association2 ],
where: { [Op.or]: [
{ 'Association1.ModelId': id },
{ 'Association2.ModelId': id }
]}
})
The code actually produces an error, stating that Model.Assocation1.ModelId doesn't exist. I'm not sure how I'm supposed to accomplish the above code example.
In pure SQL, this will be an OUTER JOIN case.
In Sequelize, you probably need to add
required: false
… to your parameters.
I am using ReactiveSearch and the CategorySearch component. I have multiple indexes I am searching against. I need about about 8 or so dataFields. However, some of the indexes don't have the same data fields. This ends up just giving me an empty result set even though my search term does exist in at least one of the indexes.
<ReactiveBase app="*" url={elasticSearch}>
<CategorySearch
componentId="searchbox"
dataField={[
'docId',
'name',
'email',
...
]}
If I remove all dataFields except docId, and name I am able to search across indexes.
Is there a way around this?
I'm trying to generate a query like the following via Knex.js:
INSERT INTO table ("column1", "column2")
SELECT "someVal", 12345
WHERE NOT EXISTS (
SELECT 1
FROM table
WHERE "column2" = 12345
)
Basically, I want to insert values only if a particular value does not already exist. But Knex.js doesn't seem to know how to do this; if I call knex.insert() (with no values), it generates an "insert default values" query.
I tried the following:
pg.insert()
.into(tableName)
.select(_.values(data))
.whereNotExists(
pg.select(1)
.from(tableName)
.where(blah)
);
but that still just gives me the default values thing. I tried adding a .columns(Object.keys(data)) in hopes that insert() would honor that, but no luck.
Is it possible to generate the query I want with knex, or will I just have to build up a raw query, without Knex.js methods?
I believe the select needs to be passed into the insert:
pg.insert(knex.select().from("tableNameToSelectDataFrom")).into("tableToInsertInto");
Also in order to select constant values or expressions you'll need to use a knex.raw expression in the select:
knex.select(knex.raw("'someVal',12345")).from("tableName")
This is my first post and I didn't test your specific example but I've done similar things like what you're asking using the above techniques.
The most comprehensive answer I've found (with explicit column names for INSERT, and custom values in the SELECT-statement) is here:
https://github.com/knex/knex/issues/1056#issuecomment-156535234
by Chris Broome
here is a copy of that solution:
const query = knex
// this part generates this part of the
// query: INSERT "tablename" ("field1", "field2" ..)
.into(knex.raw('?? (??, ??)', ['tableOrders', 'field_user_id', 'email_field']))
// and here is the second part of the SQL with "SELECT"
.insert(function() {
this
.select(
'user_id', // select from column without alias
knex.raw('? AS ??', ['jdoe#gmail.com', 'email']), // select static value with alias
)
.from('users AS u')
.where('u.username', 'jdoe')
});
console.log(query.toString());
and the SQL-output:
insert into "orders" ("user_id", "email")
select "user_id", 'jdoe#gmail.com' AS "email"
from "users" as "u"
where "u"."username" = 'jdoe'
another one approach (by Knex developer): https://github.com/knex/knex/commit/e74f43cfe57ab27b02250948f8706d16c5d821b8#diff-cb48f4af7c014ca6a7a2008c9d280573R608 - also with knex.raw
I've managed to make it work in my project and it doesn't look all that bad!
.knex
.into(knex.raw('USER_ROLES (ORG_ID, USER_ID, ROLE_ID, ROLE_SOURCE, ROLE_COMPETENCE_ID)'))
.insert(builder => {
builder
.select([
1,
'CU.USER_ID',
'CR.ROLE_ID',
knex.raw(`'${ROLES.SOURCE_COMPETENCE}'`),
competenceId,
])
.from('COMPETENCE_USERS as CU')
.innerJoin('COMPETENCE_ROLES as CR', 'CU.COMPETENCE_ID', 'CR.COMPETENCE_ID')
.where('CU.COMPETENCE_ID', competenceId)
.where('CR.COMPETENCE_ID', competenceId);
Note that this doesn't seem to work properly with the returning clause on MSSQL (it's just being ignored by Knex) at the moment.