I have an update query using typeorm on a postgresql database, like the one below, which is performed on a list of 20+ items frequently (once every 30 sec). It takes approx. 12 seconds for the update, which is a lot for my limits.
for (item of items) {
await getConnection().createQueryBuilder().update(ItemEntity)
.set({status: item.status, data: item.data})
.whereInIds(item.id).execute();
}
Is it possible to perform such a bulk update in a single query, instead of iterating other the items? If so - how?
item.status and item.data are unique for each item.
There is a way to do a workaround for this through upsert
Using an array of data that is already on the db and using ON CONFLICT to update it.
const queryInsert = manager
.createQueryBuilder()
.insert()
.into(Entity)
.values(updatedEntities)
.orUpdate(["column1", "column2", "otherEntityId"], "PK_table_entity")
.execute();
will run something like:
INSERT INTO entity (
"id", "column1", "column2", "otherEntityId"
) VALUES
($1, $2, $3, $4),
($5, $6, $7, $8),
ON CONFLICT
ON CONSTRAINT "PK_table_entity"
DO UPDATE SET
"column1" = EXCLUDED."column1",
"column2" = EXCLUDED."column2",
"otherEntityId" = EXCLUDED."otherEntityId"
But you need to be aware that orUpdate does not support using Entity relations, you will need to pass the id column of a relation entity. It also doesnt do any manipulation for the naming strategy. Another problem is that it only works if you're not using #PrimaryGeneratedColumn for your pk (you can use #PrimaryColumn instead)
Using pure psql this can be done as described in the answers to: Update multiple rows in same query using PostgreSQL
However, the UpdateQueryBuilder from Typeorm does not support a from clause.
For now, I think that a raw query is the only way, i.e. getManager().query("raw sql ...").
Related
How would I go about filtering a set of records based on their child records.
Let's say I have a collection Item that has a field to another collection Bag called bagId. I'd like to find all Items where a field on Bags matches some clause.
I.e. db.Items.find( { "where bag.type:'Paper' " }) . How would I go about doing this in MongoDB. I understand I'd have to join on Bags and then link where Item.bagId == Bag._id
I used Studio3T to convert a SQL GROUP BY to a Mongo aggregate. I'm just wondering if there's any defacto way to do this.
Should I perform a data migration to simply include Bag.type on every Item document (don't want to get into the habit of continuously making schema changes everytime I want to sort/filter Items by Bag fields).
Use something like https://github.com/meteorhacks/meteor-aggregate (No luck with that syntax yet)
Grapher https://github.com/cult-of-coders/grapher I played around with this briefly and while it's cool I'm not sure if it'll actually solve my problem. I can use it to add Bag.type to every Item returned, but I don't see how that could help me filter every item by Bag.type.
Is this just one of the tradeoffs of using a NoSQL dbms? What option above is recommended or are there any other ideas?
Thanks
You could use the $in functionality of MongoDB. It would look something like this:
const bagsIds = Bags.find({type: 'paper'}, {fields: {"_id": 1}}).map(function(bag) { return bag._id; });
const items = Items.find( { bagId: { $in: bagsIds } } ).fetch();
It would take some testing if the reactivity of this solution is still how you expect it to work and if this would still be suitable for larger collections instead of going for your first solution and performing the migration.
I have a dynamodb table that has a Global secondary Index with a range key (email, hashedPassword ).
i want to save an item if the email is not duplicated,
i used attribute_not_exists but it doesn't work, i also used :
ConditionExpression: "#email <> :email",
ExpressionAttributeNames: {"#email": "email"},
ExpressionAttributeValues: {":email": userInfo.email}
without success.
Can anyone help me please,
Thank you.
The condition expression for DynamoDB only works on the item it is working with, and not across items.
In other words, condition expression does not get evaluated against other items.
For example, if you are creating a new item, you can only enforce the email constraint if you use the Primary Key (Partition + Sort Key if you have one) as the unique constraint.
Some options you have:
Perform a read before the insert. This is not going to guarantee uniqueness of the email, but should catch a lot of duplicates.
Use Email as the Primary Key.
Perform a consistent read after the insert, which rolls back the creation
HTH
I am using Bookshelf.js and Knex.js to query my database for data, however it seems that the ORM is not returning all rows that match my query. My query is written below.
MyModel.where({id: req.params.id})
.fetchAll({withRelated: ['children.children']})
.then(result => {
res.send(JSON.stringify({myData: result}));
});
This is returning some data that I need but leaving out rows that while containing the same data in each column, have a different unique ID. So, they are different entries that simply contain the same data. I need this function to return all rows and not filter out any rows that have the same data in them.
Now I understand that it would be preferred to avoid duplicate data but I need all of it for my specific use case. Is there a way that Bookshelf.js or Knex.js could return all rows and not filter out what it "thinks" is duplicate data.
Knex / bookshelf does not filter out any data that "it thinks" being duplicate. If you are querying all rows where id = req.params.id knex will return all the rows that has requested id.
Try to run the code with DEBUG=knex:* environment variable to see the query that bookshelf sends to DB. If query seems to be wrong please add it to question. If just results seems to be wrong check DB contents and add that information to the question.
I'm trying to generate a query like the following via Knex.js:
INSERT INTO table ("column1", "column2")
SELECT "someVal", 12345
WHERE NOT EXISTS (
SELECT 1
FROM table
WHERE "column2" = 12345
)
Basically, I want to insert values only if a particular value does not already exist. But Knex.js doesn't seem to know how to do this; if I call knex.insert() (with no values), it generates an "insert default values" query.
I tried the following:
pg.insert()
.into(tableName)
.select(_.values(data))
.whereNotExists(
pg.select(1)
.from(tableName)
.where(blah)
);
but that still just gives me the default values thing. I tried adding a .columns(Object.keys(data)) in hopes that insert() would honor that, but no luck.
Is it possible to generate the query I want with knex, or will I just have to build up a raw query, without Knex.js methods?
I believe the select needs to be passed into the insert:
pg.insert(knex.select().from("tableNameToSelectDataFrom")).into("tableToInsertInto");
Also in order to select constant values or expressions you'll need to use a knex.raw expression in the select:
knex.select(knex.raw("'someVal',12345")).from("tableName")
This is my first post and I didn't test your specific example but I've done similar things like what you're asking using the above techniques.
The most comprehensive answer I've found (with explicit column names for INSERT, and custom values in the SELECT-statement) is here:
https://github.com/knex/knex/issues/1056#issuecomment-156535234
by Chris Broome
here is a copy of that solution:
const query = knex
// this part generates this part of the
// query: INSERT "tablename" ("field1", "field2" ..)
.into(knex.raw('?? (??, ??)', ['tableOrders', 'field_user_id', 'email_field']))
// and here is the second part of the SQL with "SELECT"
.insert(function() {
this
.select(
'user_id', // select from column without alias
knex.raw('? AS ??', ['jdoe#gmail.com', 'email']), // select static value with alias
)
.from('users AS u')
.where('u.username', 'jdoe')
});
console.log(query.toString());
and the SQL-output:
insert into "orders" ("user_id", "email")
select "user_id", 'jdoe#gmail.com' AS "email"
from "users" as "u"
where "u"."username" = 'jdoe'
another one approach (by Knex developer): https://github.com/knex/knex/commit/e74f43cfe57ab27b02250948f8706d16c5d821b8#diff-cb48f4af7c014ca6a7a2008c9d280573R608 - also with knex.raw
I've managed to make it work in my project and it doesn't look all that bad!
.knex
.into(knex.raw('USER_ROLES (ORG_ID, USER_ID, ROLE_ID, ROLE_SOURCE, ROLE_COMPETENCE_ID)'))
.insert(builder => {
builder
.select([
1,
'CU.USER_ID',
'CR.ROLE_ID',
knex.raw(`'${ROLES.SOURCE_COMPETENCE}'`),
competenceId,
])
.from('COMPETENCE_USERS as CU')
.innerJoin('COMPETENCE_ROLES as CR', 'CU.COMPETENCE_ID', 'CR.COMPETENCE_ID')
.where('CU.COMPETENCE_ID', competenceId)
.where('CR.COMPETENCE_ID', competenceId);
Note that this doesn't seem to work properly with the returning clause on MSSQL (it's just being ignored by Knex) at the moment.
I'm quite new with couchDB and I need a little support.
In MySQL I could run simply this query:
SELECT `name`, `id`, `desc` FROM `table`
WHERE `name`="jack" OR `cat` LIKE "%|52224|%";
And here are my two problems:
I started to create a view (still without Like option and everything):
function(doc) {
emit([doc.name, doc.cat], {
"name" : doc.name,
"desc" : doc.desc,
"id" : doc._id
});
}
1. When I use "emit([doc.name" the string must match 100% (also case sensitive).
-> How I make this option case un-sesnsitive?
That I can ask for ("Jack, jack, jAck, JAck,...) like in mysql?
2. How I create the OR option?
When I use [doc.name, doc.cat], I'm also forced to ask for both vars.
But when I have just one of them,
how I can query without creating for each option an own view?
To implement case insensitive search you just have to convert keys to lowercase:
emit([doc.name.toLowerCase(), doc.cat.toLowerCase()])
Now, if you convert your query to lowercase you'll have case insensitive matching.
The problem with this solution is that if you want both case sensitive and case insensitive search you can either emit both values:
[doc.name.toLowerCase(), doc.name, doc.cat]
and use startkey and endkey to filter results, or create a separate view.
The second question is a bit more tricky to implement.
First of all, if you need to filter by doc.name only you can send request with startkey=["jack",0] and endkey=["jack",'zzzzzz'] which will return all documents with doc.name="jack" and doc.cat between 0 and 'zzzzzz' (there probably is a better way to say "any cat", unfortunately I can't find it right now).
If you need a real OR, then you should emit two rows for each document:
emit(doc.name, doc); emit(doc.cat, doc);
This way you can POST needed keys with your request: {"keys": ["jack", "cat_name"]}
This will return every document with either "jack" OR "cat_name" key. However documents that have both will be returned twice, so you have to filter duplicates in your application code.
You can also use couchdb-lucene which will solve both of your problems and probably a lot more. It is a popular choice among couchdb users for implementing advanced queries.