I have a PostgreSQL tables, first contains basic info about order and second table contains products that are inside the referenced order id. My question is how to post data to this two tables in a correct way? Or should I change my scheme into something more well-established?
Tables look like this:
There are no table design issues here, it's a normal one-to-many relationship. But just to ensure data integrity here you should use transactions in the DB.
To properly protect data integrity when inserting records into multiple tables, you have two options:
using transactions
write single query (one statement) to insert into both tables at the same time
In PostgreSQL, any functions are performed with transactions, so one function = one transaction. For example:
CREATE OR REPLACE FUNCTION inser_all_order_data(
)
RETURNS void
LANGUAGE plpgsql
AS $function$
declare
orderid integer;
begin -- begin transaction
insert into orders (created_at, type, status) values (now(), 'mytype', 'mystatus')
returning id into orderid;
insert into ordercontent (order_id, code, name, content) values (orderid, '001', 'myname', 'some text');
end; -- end transaction
$function$
;
In here, both insert statements are in the same transaction.
Example for writing single query:
with tb as (
insert into orders (created_at, type, status) values (now(), 'mytype', 'mystatus')
returning id
)
insert into ordercontent (order_id, code, name, content)
select id, '001', 'myname', 'some text' from tb;
When you write single query you needed using transactions, because one statement = one transaction.
If you don't need to insert records into both tables at the same time, then you can use insert statements as usual.
Related
I have a talend job that goes like the above one and i am using a select statement to access the data from oracle
select id,name,details from employee ;
and send the results via email.However for a particular column details it has 10 different lines and that are dynamic and has various other unwanted/irrelevant information and i would like to precisely filter few lines and print them to the mail using tjava component.
How can i do that?
for eg:details column has values like:
"hi i am john
i work at abc
i go to oxford
i reside at 1st street
goodbye"
so the lines are totally random,i would want to match based on a specific keyword lets say "reside at" alone and filter that for different records and capture that in that output?
In order to do this you need to create a custom SQL query in the tDBinput.
"select id,name,details from employee
where details like '%reside at%'"
(Where the % are any text check the SQL "LIKE" function for more details)
With this you will filter the row you want.
If you need more filter you can modify you SQL query like you want.
I am using Bookshelf.js and Knex.js to query my database for data, however it seems that the ORM is not returning all rows that match my query. My query is written below.
MyModel.where({id: req.params.id})
.fetchAll({withRelated: ['children.children']})
.then(result => {
res.send(JSON.stringify({myData: result}));
});
This is returning some data that I need but leaving out rows that while containing the same data in each column, have a different unique ID. So, they are different entries that simply contain the same data. I need this function to return all rows and not filter out any rows that have the same data in them.
Now I understand that it would be preferred to avoid duplicate data but I need all of it for my specific use case. Is there a way that Bookshelf.js or Knex.js could return all rows and not filter out what it "thinks" is duplicate data.
Knex / bookshelf does not filter out any data that "it thinks" being duplicate. If you are querying all rows where id = req.params.id knex will return all the rows that has requested id.
Try to run the code with DEBUG=knex:* environment variable to see the query that bookshelf sends to DB. If query seems to be wrong please add it to question. If just results seems to be wrong check DB contents and add that information to the question.
I have a requirement to get data from SQL database for the list of items given by the user.
Example, if a user uploads a .csv file containing 1000 item ids, making 1000 SQL queries by item_id would be a better option ? or getting whole database table data in a single SQL query and looping over that data would be a better option ?
As already stated in the comments:
The best way would be to create a single-column item_searchtable in the database; put the 1000 item identifiers into a text file, one identifier per row, and then to use MySQL's load functionality to load that file into your search table.
Then use:
SELECT
item_id,
, name
, <any other columns>
FROM items i
INNER JOIN item_search s
ON i.item_id = s.item_id
to export the data.
The most expensive part in a SQL connection is always accepting SQL requests and initiating and ending SQL requests. This way, you have two requests; the load and the export; the other way, you have 1000 requests.
I'm trying to generate a query like the following via Knex.js:
INSERT INTO table ("column1", "column2")
SELECT "someVal", 12345
WHERE NOT EXISTS (
SELECT 1
FROM table
WHERE "column2" = 12345
)
Basically, I want to insert values only if a particular value does not already exist. But Knex.js doesn't seem to know how to do this; if I call knex.insert() (with no values), it generates an "insert default values" query.
I tried the following:
pg.insert()
.into(tableName)
.select(_.values(data))
.whereNotExists(
pg.select(1)
.from(tableName)
.where(blah)
);
but that still just gives me the default values thing. I tried adding a .columns(Object.keys(data)) in hopes that insert() would honor that, but no luck.
Is it possible to generate the query I want with knex, or will I just have to build up a raw query, without Knex.js methods?
I believe the select needs to be passed into the insert:
pg.insert(knex.select().from("tableNameToSelectDataFrom")).into("tableToInsertInto");
Also in order to select constant values or expressions you'll need to use a knex.raw expression in the select:
knex.select(knex.raw("'someVal',12345")).from("tableName")
This is my first post and I didn't test your specific example but I've done similar things like what you're asking using the above techniques.
The most comprehensive answer I've found (with explicit column names for INSERT, and custom values in the SELECT-statement) is here:
https://github.com/knex/knex/issues/1056#issuecomment-156535234
by Chris Broome
here is a copy of that solution:
const query = knex
// this part generates this part of the
// query: INSERT "tablename" ("field1", "field2" ..)
.into(knex.raw('?? (??, ??)', ['tableOrders', 'field_user_id', 'email_field']))
// and here is the second part of the SQL with "SELECT"
.insert(function() {
this
.select(
'user_id', // select from column without alias
knex.raw('? AS ??', ['jdoe#gmail.com', 'email']), // select static value with alias
)
.from('users AS u')
.where('u.username', 'jdoe')
});
console.log(query.toString());
and the SQL-output:
insert into "orders" ("user_id", "email")
select "user_id", 'jdoe#gmail.com' AS "email"
from "users" as "u"
where "u"."username" = 'jdoe'
another one approach (by Knex developer): https://github.com/knex/knex/commit/e74f43cfe57ab27b02250948f8706d16c5d821b8#diff-cb48f4af7c014ca6a7a2008c9d280573R608 - also with knex.raw
I've managed to make it work in my project and it doesn't look all that bad!
.knex
.into(knex.raw('USER_ROLES (ORG_ID, USER_ID, ROLE_ID, ROLE_SOURCE, ROLE_COMPETENCE_ID)'))
.insert(builder => {
builder
.select([
1,
'CU.USER_ID',
'CR.ROLE_ID',
knex.raw(`'${ROLES.SOURCE_COMPETENCE}'`),
competenceId,
])
.from('COMPETENCE_USERS as CU')
.innerJoin('COMPETENCE_ROLES as CR', 'CU.COMPETENCE_ID', 'CR.COMPETENCE_ID')
.where('CU.COMPETENCE_ID', competenceId)
.where('CR.COMPETENCE_ID', competenceId);
Note that this doesn't seem to work properly with the returning clause on MSSQL (it's just being ignored by Knex) at the moment.
Below is a snipet of code that I am having trouble with. The purpose is to check duplicate entries in the database and return "h" with a boolean if true or false. For testing purposes I am returning a true boolean for "h" but by the time the alert(duplicate_count); line gets executed the duplicate_count is still 0. Even though the alert for a +1 gets executed.
To me it seems like the function updateUserFields is taking longer to execute so it's taking longer to finish before getting to the alert.
Any ideas or suggestions? Thanks!
var duplicate_count = 0
for (var i = 0; i < skill_id.length; i++) {
function updateUserFields(h) {
if(h) {
duplicate_count++;
alert("count +1");
} else {
alert("none found");
}
}
var g = new cfc_mentoring_find_mentor();
g.setCallbackHandler(updateUserFields);
g.is_relationship_duplicate(resource_id, mentee_id, section_id[i], skill_id[i], active_ind,table);
};
alert(duplicate_count);
There is no reason whatsoever to use client-side JavaScript/jQuery to remove duplicates from your database. Security concerns aside (and there are a lot of those), there is a much easier way to make sure the entries in your database are unique: use SQL.
SQL is capable of expressing the requirement that there be no duplicates in a table column, and the database engine will enforce that for you, never letting you insert a duplicate entry in the first place. The syntax varies very slightly by database engine, but whenever you create the table you can specify that a column must be unique.
Let's use SQLite as our example database engine. The relevant part of your problem is right now probably expressed with tables something like this:
CREATE TABLE Person(
id INTEGER PRIMARY KEY ASC,
-- Other fields here
);
CREATE TABLE MentorRelationship(
id INTEGER PRIMARY KEY ASC,
mentorID INTEGER,
menteeID INTEGER,
FOREIGN KEY (mentorID) REFERENCES Person(id),
FOREIGN KEY (menteeID) REFERENCES Person(id)
);
However, you can make enforce uniqueness i.e. require that any (mentorID, menteeID) pair is unique, by changing the pair (mentorID, menteeID) to be the primary key. This works because you are only allowed one copy of each primary key. Then, the MentorRelationship table becomes
CREATE TABLE MentorRelationship(
mentorID INTEGER,
menteeID INTEGER,
PRIMARY KEY (mentorID, menteeID),
FOREIGN KEY (mentorID) REFERENCES Person(id),
FOREIGN KEY (menteeID) REFERENCES Person(id)
);
EDIT: As per the comment, alerting the user to duplicates but not actually removing them
This is still much better with SQL than with JavaScript. When you do this in JavaScript, you read one database row at a time, send it over the network, wait for it to come to your page, process it, throw it away, and then request the next one. With SQL, all the hard work is done by the database engine, and you don't lose time by transferring unnecessary data over the network. Using the first set of table definitions above, you could write
SELECT mentorID, menteeID
FROM MentorRelationship
GROUP BY mentorID, menteeID
HAVING COUNT(*) > 1;
which will return all the (mentorID, menteeID) pairs that occur more than once.
Once you have a query like this working on the server (and are also pulling out all the information you want to show to the user, which is presumably more than just a pair of IDs), you need to send this over the network to the user's web browser. Essentially, on the server side you map a URL to return this information in some convenient form (JSON, XML, etc.), and on the client side you read this information by contacting that URL with an AJAX call (see jQuery's website for some code examples), and then display that information to the user. No need to write in JavaScript what a database engine will execute orders of magnitude faster.
EDIT 2: As per the second comment, checking whether an item is already in the database
Almost everything I said in the first edit applies, except for two changes: the schema and the query. The schema should become the second of the two schemas I posted, since you don't want the database engine to allow duplicates. Also, the query should be simply
SELECT COUNT(*) > 0
FROM MentorRelationship
WHERE mentorID = #mentorID AND menteeID = #menteeID;
where #mentorID and #menteeID are the items that the user selected, and are inserted into the query by a query builder library and not by string concatenation. Then, the server will get a true value if the item is already in the database, and a false value otherwise. The server can send that back to the client via AJAX as before, and the client (that's your JavaScript page) can alert the user if the item is already in the database.