I am learning postgres/SQL and am wondering what the best approach is here.
I have an invoice db design that has recipients, drafts, items.
Drafts are the invoices, items are the lines on the invoices, and recipients are who it's going to.
When creating an invoice I have to insert into each table accordingly. Which approach is considered best practice?
There is this one where it's one giant complicated query.
db.query('
WITH new_recipient AS (
INSERT INTO
recipients(...)
VALUES (...)
RETURNING id AS recipient_id, user_id
), new_draft AS (
INSERT INTO drafts(user_id, recipient_id)
SELECT user_id, recipient_id FROM new_recipient
RETURNING id AS draft_id
)
SELECT new_recipient.*, new_draft.* FROM new_draft, new_recipient'
,[...])
OR:
const data = await pool.query(
`INSERT INTO recipients (...) VALUES (...) RETURNING *`,
[....]
);
const draft = await pool.query(
`INSERT INTO drafts (recipient_id, ...) VALUES ($1, ...) RETURNING *`,
[data.rows[0].recipient_id, ...data]
);
I am inclined to use the second approach for readability and simplicity. Is there any reason one should use the first one instead? Perhaps performance is slower if you break it up into multiple queries?
Related
In the Firestore documentation, it states clearly the limitations of support for query filters with logical OR.
For example:
const userPostsQuery = query(postsRef, where("author", "==", uid);
const publicPostsQuery = query(postsRef, where("public", "==", true);
If as in the above example, we need to get a list of both, user posts and public posts all sorted together by date, ie: Both queries need to be OR-ed together, such a feature is not available in Firestore and we will have to run both queries separately, and then merge and sort the results on the client-side.
I'm fine with such a sad workaround. but what if the total number of posts can be huge? thus we need to implement a pagination system where each page shows 50 posts max. How can this be done with such a sad workaround?
Firestore has very limited operators and aggregation options. However, it has limited OR support with an Array type.
A solution that could simplify your use case is to introduce a new field of type array in your post document. Let's say this field is named a. When you create your document, a is equal to [authorId, 'public'] if the post is public, [authorId] otherwise.
Then, you can query your need using the array-contains-any operator:
const q = query(postRef, where('a', 'array-contains-any', [authorId, 'public']));
You can easily add pagination with limit, orderBy, startAt, and startAfter functions.
I have a PostgreSQL tables, first contains basic info about order and second table contains products that are inside the referenced order id. My question is how to post data to this two tables in a correct way? Or should I change my scheme into something more well-established?
Tables look like this:
There are no table design issues here, it's a normal one-to-many relationship. But just to ensure data integrity here you should use transactions in the DB.
To properly protect data integrity when inserting records into multiple tables, you have two options:
using transactions
write single query (one statement) to insert into both tables at the same time
In PostgreSQL, any functions are performed with transactions, so one function = one transaction. For example:
CREATE OR REPLACE FUNCTION inser_all_order_data(
)
RETURNS void
LANGUAGE plpgsql
AS $function$
declare
orderid integer;
begin -- begin transaction
insert into orders (created_at, type, status) values (now(), 'mytype', 'mystatus')
returning id into orderid;
insert into ordercontent (order_id, code, name, content) values (orderid, '001', 'myname', 'some text');
end; -- end transaction
$function$
;
In here, both insert statements are in the same transaction.
Example for writing single query:
with tb as (
insert into orders (created_at, type, status) values (now(), 'mytype', 'mystatus')
returning id
)
insert into ordercontent (order_id, code, name, content)
select id, '001', 'myname', 'some text' from tb;
When you write single query you needed using transactions, because one statement = one transaction.
If you don't need to insert records into both tables at the same time, then you can use insert statements as usual.
I have an update query using typeorm on a postgresql database, like the one below, which is performed on a list of 20+ items frequently (once every 30 sec). It takes approx. 12 seconds for the update, which is a lot for my limits.
for (item of items) {
await getConnection().createQueryBuilder().update(ItemEntity)
.set({status: item.status, data: item.data})
.whereInIds(item.id).execute();
}
Is it possible to perform such a bulk update in a single query, instead of iterating other the items? If so - how?
item.status and item.data are unique for each item.
There is a way to do a workaround for this through upsert
Using an array of data that is already on the db and using ON CONFLICT to update it.
const queryInsert = manager
.createQueryBuilder()
.insert()
.into(Entity)
.values(updatedEntities)
.orUpdate(["column1", "column2", "otherEntityId"], "PK_table_entity")
.execute();
will run something like:
INSERT INTO entity (
"id", "column1", "column2", "otherEntityId"
) VALUES
($1, $2, $3, $4),
($5, $6, $7, $8),
ON CONFLICT
ON CONSTRAINT "PK_table_entity"
DO UPDATE SET
"column1" = EXCLUDED."column1",
"column2" = EXCLUDED."column2",
"otherEntityId" = EXCLUDED."otherEntityId"
But you need to be aware that orUpdate does not support using Entity relations, you will need to pass the id column of a relation entity. It also doesnt do any manipulation for the naming strategy. Another problem is that it only works if you're not using #PrimaryGeneratedColumn for your pk (you can use #PrimaryColumn instead)
Using pure psql this can be done as described in the answers to: Update multiple rows in same query using PostgreSQL
However, the UpdateQueryBuilder from Typeorm does not support a from clause.
For now, I think that a raw query is the only way, i.e. getManager().query("raw sql ...").
I am facing a little bit of a mental block in terms of how to do some relational queries with firestore while adhering to the best practices. I am creating a feed feature where you can see a feed of posts from your friends. Essentially my data structure is as follows:
Friends (collection)
-friend_doc
...data
friends_uid: [uid1, uid2]
Posts (collection)
-post_doc
...data
posted_by: uid2
Basically I am making a query to get all of the friends where the friends_uid contains my uid (uid1 in this case). And then once I mapped all of the friends uid's to an array, I want to make a firestore query to get posts where the posted_by field is equal to any of the uid's in that array of friends uid's. I haven't been able to make something that does anything like that yet.
I know that it seems most convenient to loop through the string array of friends uid's and make a query for each one like:
listOfUids.forEach(async (item) => {
const postQuerySnapshot = await firestore()
.collection('posts')
.where('uid', '==', item)
.get();
results.push(postQuerySnapshot.docs);
});
but this is extremely problematic for paging and limiting data as I could possibly receive tons of posts. I may just be too deep into this code and missing an obvious solution or maybe my data structure is somewhat flawed. Any insight would be greatly appreciated.
TLDR - how can I make a firestore query that gets all docs that have a value that exists in an array of strings?
You can use an "in" query for this:
firestore()
.collection('posts')
.where('uid', 'in', [uid1, uid2, ...])
But you are limited to 10 elements in that array. So you are probably going to have to stick to what you have now. You will not be able to use Firestore's pagination API.
Your only real alternatives for this case is to create a new collection that contains all of the data you want to query in one place, as there are no real join operations. Duplicating data like this is common for nosql type databases.
I'm trying to write a query using knex to SUM the votes for each question but am not getting the correct sum. I can write the subquery in SQL but can't seem to piece it all together. I am a student and not sure if I'm doing something wrong with Knex or if my underlying logic is wrong. Thanks in advance for any help!
My knex query looks like this
return knex
.from('question')
.select(
'question.id AS question_id',
knex.raw(
`count(DISTINCT vote) AS number_of_votes`, //this returns the number_of_votes for each question_id as expected
),
knex.raw(
`sum(vote.vote) AS sum_of_votes`, //something wrong here... E.g., question_id 1 has 3 down votes so the sum should be -3, however I am getting -9
),
)
.leftJoin('user', 'question.user_id', 'user.id')
.leftJoin('vote', 'question.id', 'vote.question_id')
.groupBy('question.id', 'user.id');
There are 3 tables that look like:
user
id
user_name
question
id
title
body
user_id (FK references user.id)
vote
question_id (FK references question.id)
user_id (FK references user.id)
vote (-1 or 1)
PRIMARY KEY (question_id, user_id)
I did manage to write the query as a stand-alone SQL query and verified that it works as expected. This is what I am trying to accomplish in the above knex query:
SELECT question.id, sum(vote.vote) AS sum_of_votes FROM question LEFT JOIN vote ON question.id = vote.question_id GROUP BY question.id;
So, broadly your SQL query is correct (after fixing a couple of typos) although as #felixmosh points out it has no user information in it: might be tricky to figure out who voted for what! But perhaps you don't need that for your purposes.
Your posted solution will do the trick, but is perhaps not the most efficient query for the job as it involves a subquery and several joins. Here's the SQL it generates:
SELECT "question"."id" AS "question_id",
count(DISTINCT vote) AS number_of_votes,
(
SELECT sum(vote) FROM vote
WHERE question_id = question.id
GROUP BY question_id
) AS sum_of_votes
FROM "question"
LEFT JOIN "user" ON "question"."user_id" = "user"."id"
LEFT JOIN "vote" ON "question"."id" = "vote"."question_id"
GROUP BY "question"."id", "user"."id";
We can take a simpler approach to get the same information. How about this?
SELECT question_id,
count(vote) AS number_of_votes,
sum(vote) AS sum_of_votes
FROM vote
GROUP BY question_id;
This gets all the information you were looking for, without joining any tables or using subqueries. It also avoids DISTINCT, which could lead to incorrectly counting the number of votes. The Knex to generate such a query looks like this:
knex("vote")
.select("question_id")
.count("vote AS number_of_votes")
.sum("vote AS sum_of_votes")
.groupBy("question_id")
You only really need to join tables here if you were looking for further information from those tables (such as the user's name or the question's title).
After hours of trying to figure this out I finally got it. Here is solution:
return knex
.from('question')
.select(
'question.id AS question_id',
knex.raw(
`count(DISTINCT vote) AS number_of_votes`,
),
knex.raw(
`SELECT sum(vote) from vote WHERE question_id = question.id GROUP BY question_id) AS sum_of_votes`
)
.leftJoin('user', 'question.user_id', 'user.id')
.leftJoin('vote', 'question.id', 'vote.question_id')
.groupBy('question.id', 'user.id');