Develop Reference

Is it possible to Import a csv-file to a SQLite table via node.js [duplicate]

I'm using the npm sqlite3 package in my web application for a mock-up restaurant (for learning purposes). In my current implementation, when my local server is started, I'm creating a menuItems table, like so:
var db = new sqlite3.Database(':memory:');
db.serialize(function() {
db.run('CREATE TABLE menuItems ('
+ 'itemName VARCHAR(255),'
+ 'itemDescription VARCHAR(255),'
+ 'unitPrice REAL'
+ ');')
.run("INSERT INTO menuItems (itemName, itemDescription, unitPrice) VALUES"
+ " ('Fish Filet', 'Yummy fish in a sandwich.', 9.95)")
});
However, I would like for my menuItems table to not be hard-coded in the js file and would like for the actual items to be separate from this js file. There are a few solutions that I can think of:
Use current implementation and hard-code commands to insert individual items into the table.
Use SQLite3 to import the csv file and insert it into the table.*
Read in items from a csv file and insert them via a loop.
Establish database before the server starts up and pass it into the sqlite3.Database() constructor.
*I'd like to go with option 2. However, since this is for learning purposes, I am open to any suggestions, including but not limited to using a different database-managing package or using a different kind of file (maybe just a txt file or something).
I know there are some ways to import the csv file using SQLite3. I'm trying to do the same thing but from the sqlite3 npm package. When I try to import the file via the same command (which seems to be a sqlite3-specific command) from the linked page,
db.run('.import "C:/Users/path/to/csv/file.csv"'
+ 'INTO TABLE menuItems'
+ 'FIELDS TERMINATED BY ","'
+ 'ENCLOSED BY "\'"'
+ 'LINES TERMINATED BY \'\\n\''
+ 'IGNORE 1 ROWS'
+ ');');
I receive the error
events.js:183
throw er; // Unhandled 'error' event
^
Error: SQLITE_ERROR: near ".": syntax error
Trying to use normal SQL syntax,
.run('BULK INSERT menuItems'
+ 'FROM "C:/Users/path/to/csv/file.csv" '
+ 'WITH '
+ '{ '
+ 'FIRSTROW = 2,'
+ 'FIELDTERMINATOR = ","'
+ 'ROWTERMINATOR = "\n",'
+ 'ERRORFILE = "C:/Users/path/to/csv/error_file.csv" '
+ 'TABLOCK'
+ '}')
I receive the error
events.js:183
throw er; // Unhandled 'error' event
^
Error: SQLITE_ERROR: near "BULK": syntax error
Is something in my syntax incorrect? Is there a better/working/more efficient method to do this? Option 3 seems like it would work, but I haven't tried it yet.

I ended up going with Option 3. Namely, I used the fs package to read in the CSV file, did some parsing, returned the results within a Promise object, and did the database insertions within the then of said Promise object.

try
db.run('.import "C:/Users/path/to/csv/file.csv"'
+ 'INTO TABLE menuItems'
+ 'FIELDS TERMINATED BY ","'
+ 'ENCLOSED BY "\'"'
+ 'LINES TERMINATED BY \'\\n\''
+ 'IGNORE 1 ROWS'
+ ')');
commands beginning with . don't need semicolon at end

There is a little example for read your csv into sqlite3 db.
const sql3 = require('better-sqlite3');
const db = new sql3( 'memory.db' );
const csv = require('csv-parser');
const fs = require('fs');
// create table
db.exec( 'CREATE TABLE IF NOT EXISTS menuItems ( itemName TEXT, itemDescription TEXT, unitPrice REAL );' );
//db.exec( 'DROP TABLE menuItems;' );
const insrow = db.prepare( 'insert into menuItems ( itemName, itemDescription, unitPrice ) VALUES (?, ?, ?)' );
fs.createReadStream('C:/Users/path/to/csv/file.csv')
.pipe(csv({"separator":";"}))
.on('data', (row) => {
insrow.run( row.itemName, row.itemDescription, row.unitPrice );
console.log(row);
})
.on('end', () => {
console.log('CSV file successfully processed');
db.close();
});
The database in example is memory.db, and the csv format is not comma-separated, but semicolon separated, change separator, if it necessary.
The csv in this case must be started with itemName;itemDescription;unitPrice header, and a row looks like Fish Filet;Yummy fish in a sandwich.;9.95 and so on.

Getting column names in SQL query

How can I access column's from SELECT, in my WHERE statement? I'm probably missing quotes. For context, this is in a controller, in Strapi CMS, which runs on a node.js server.
Problem:
Occurs at AND statement (mainly the first st_geomfromtext line):
const rawBuilder = strapi.connections.default.raw(
`
SELECT
locations.id as Location_ID,
locations.Title as Location_Title,
locations.Latitude as Location_Latitude,
locations.Longitude as Location_Longitude,
things.id,
things.Title,
things.Location
FROM locations
RIGHT JOIN things
ON locations.id = things.Location
WHERE things.Style = ` + ctx.query['Style.id'] + `
AND round(st_distance_sphere(
st_geomfromtext(CONCAT('POINT(',locations.Longitude, ' ', locations.Latitude,')')),
st_geomfromtext(CONCAT('POINT(` + ctx.query.Longitude + ` ` + ctx.query.Latitude + `)'))
)) <= ` + 5000
)
Test works:
Just for fun, same as above, but just passed request variables for both st_geomfromtext lines, and the response works; no SQL error:
AND round(st_distance_sphere(
st_geomfromtext(CONCAT('POINT(` + ctx.query.Longitude1 + ` ` + ctx.query.Latitude1 + `)')),
st_geomfromtext(CONCAT('POINT(` + ctx.query.Longitude2 + ` ` + ctx.query.Latitude2 + `)'))
)) <= ` + 5000
So as far as I can tell, the first st_geomfromtext line is the culprit, however it (the 1st line) works fine in a Go server... another clue that this is just a syntax problem.

Below is a working example in SQL Server that should help you resolve this.
Please try these steps:
Remove the "AND" statement from your where clause and save it somewhere
Add some filter criteria that will give you just few known locations
Add new output fields in your select criteria for each function so you will know what you are comparing.
Select CONCAT('POINT(',locations.Longitude, ' ', locations.Latitude,')') from locations
Select st_geomfromtext(CONCAT('POINT(',locations.Longitude, ' ', locations.Latitude,')')) from locations
Note: the output to the geo functions this will probably look cryptic like 0xE6100000010C75931804564253C042CF66D5E7724340
Once the values line up the way you expect then add a new version of the where clause with the adjustments you have made.
Check the precision of the st_distance_sphere function. In SQL Server this is defaulted to meters.
Example in SQL Server
CREATE TABLE #locations (id INT, Title VARCHAR(50), Latitude DECIMAL(10,4), Longitude DECIMAL(10,4))
CREATE TABLE #things (id INT, Title VARCHAR(50), LocationId INT)
INSERT INTO #locations (id, Title, Latitude, Longitude) Values (1,'WH', 38.8977, -77.0365)
INSERT INTO #locations (id, Title, Latitude, Longitude) Values (2,'CB', 38.8899, -77.0091)
INSERT INTO #things (id, Title, LocationId) Values (100,'White House',1)
INSERT INTO #things (id, Title, LocationId) Values (101,'United States Capitol',2)
--My Location at the Washington Monument
DECLARE #myLat DECIMAL(10,4) = 38.8895;
DECLARE #myLong DECIMAL(10,4) = -77.0353
SELECT
loc.id as Location_ID,
loc.Title as Location_Title,
loc.Latitude as Location_Latitude,
loc.Longitude as Location_Longitude,
th.id,
th.Title,
th.LocationId,
geometry::STGeomFromText(CONCAT('POINT(',loc.Longitude, ' ', loc.Latitude,')'),4326) as ItemPoint,
geometry::STGeomFromText(CONCAT('POINT(',#myLat,' ',#myLong,')'),4326) as MyPoint,
geometry::STGeomFromText(CONCAT('POINT(',loc.Longitude, ' ', loc.Latitude,')'),4326).STDistance(geometry::STGeomFromText(CONCAT('POINT(',#myLat,' ',#myLong,')'),4326))
FROM #locations loc
RIGHT JOIN #things th ON loc.id = th.LocationId
DROP TABLE #locations
DROP TABLE #things

Error: ER_PARSE_ERROR: You have an error in your SQL syntax;

I'm trying to insert values using mysql in nodejs. I had written the following code and installed MySQL support via npm,But canot to INSERT INTO the table due to this problem.
My code;
var mysql = require('mysql');
var values=randomValueHex(8);
var sql = "INSERT INTO `activationkeys`(`activationKey`, `productId`)
VALUES ( values ,'3')";
con.query(sql, function (err, result) {
if (err) throw err;
console.log("1 record inserted");
});
My Error on terminal:
Error: ER_PARSE_ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''3')'
How can i solve this problem?

Why are you using back quote for the column names? We do not need that in column names. You can simply create your dynamic sql query by using + operator on the column values like this:
var sql = "INSERT INTO activationkeys (activationKey, productId) VALUES ( " + values + " ,'3')";

Instead of
var sql = "INSERT INTO `activationkeys`(`activationKey`, `productId`)
VALUES ( values ,'3')";
Please try this
var sql = "INSERT INTO `activationkeys`(`activationKey`, `productId`)
VALUES ( " + values + " ,'3')";
provided values is a string

values currently means nothing the way you're using it in the SQL query.
What is values? Is it an integer, or a string?
Nevertheless, you need to concatenate the values variable within the string.
var sql = "INSERT INTO `activationkeys`(`activationKey`, `productId`) VALUES (" + values + ",'3')";

And one more correction values variable have to give like this '" + values + "' . This is the most correct way of define as a variables. Otherwise you give like this " + values + " , you may be have an error accure like this Error: ER_BAD_FIELD_ERROR: Unknown column 'xxxxxx' in 'field list'. And my code is
var sql = "INSERT INTO `activationkeys`(`activationKey`, `productId`) VALUES ( '" + values + "' , 3 )";

This is simple way to write SQL query in you JavaScript code.
Try It Once
const QUERY = INSERT INTO users (firstName, lastName) VALUES ( "${firstName}", "${lastName}")
Note: Please wrap your complete query into the back ticks.

How do I pass a table as parameter on Ajax callback

I want to pass a table as parameter on an ajax callback procedure in Oracle APEX 5, because I need to make an SQL query on that table.
The SQL process is stored as shared component inside the Apex 5 application. Screenshot
My procedure is like this
(procedure name: THIS_PROCESS)
declare
v_tablename varchar(128);--max table_name lenght
v_ID number;
v_somevar
BEGIN
SELECT Columname,
INTO v_somevar
FROM v_tablename
WHERE ID = v_ID;
--Do stuff
END;
This code (FROM v_tablename) gives me a compilation error:
ORA-00942: table or view does not exist ORA-06550: line 9, column 5:
PL/SQL: SQL Statement ignored
I'm a total newbie. I had been reading that I should call that procedure with this javascript:
apex.server.process ( "THIS_PROCESS", {
x01: "TABLENAME",
x02: "Row_ID",
pageItems: "#P1_Item,#P2_Item"
},{
success: function( pData )
// do something here
}
} );
I do not understand why I should pass x01 and x02 instead of v_tablename and v_ID
Do x01 and x02 automatically are assigned to v_tablename and v_ID?

Here's an example page process THIS_PROCESS of type "Ajax Callback". Note that you need Dynamic SQL to select from a table name that isn't hardcoded.
declare
v_table varchar2(128) := apex_application.g_x01;
v_id number := apex_application.g_x02;
v_somevar varchar2(100);
v_sql varchar2(4000);
begin
-- validate v_table parameter to avoid sql injection. will throw exception if it fails
select table_name into v_table from all_tables where table_name = v_table;
v_sql := 'SELECT Columname
FROM ' || v_table || '
WHERE ID = :A1';
execute immediate v_sql into v_somevar using v_id;
-- do something with v_somevar
end;
Do be careful with this sort of thing - this design will allow a malicious user to write their own javascript function which can pass any table name that it likes to your procedure.

You need to use dynamic sql:
declare
v_tablename varchar(128);--max table_name lenght
v_sql varchar2(1000);
v_ID number;
v_somevar varchar2(100);
BEGIN
v_sql := 'SELECT Columname FROM ' || v_tablename || ' where ID = :1';
EXECUTE IMMEDIATE v_sql INTO v_somevar USING v_ID;
--Do stuff
END;
/

Postgresql9.3 query optimization

I am using postgresql 9.3 in my node.js application. In my database i have some 7lakhs records now. Also in my database i have json datatype column.
My query is as following:
EXPLAIN ANALYSE select id_0, name_0, id_1, name_1, id_2, name_2, id_3, name_3, id_4, name_4, latitude, longitude, to_char(collecteddate, 'dd/mm/yyyy') as collecteddate, key, value->>'xxxxx' as value from table where
CAST(value->'xxxxx'->> 'aaaaa' as INTEGER)BETWEEN 1 and 43722 and value->'PCA_2011'->> 'aaaaa' NOT LIKE ' ' and
CAST(value->'xxxxx'->> 'bbbbb' as INTEGER)BETWEEN 1 and 100 and value->'xxxx'->> 'bbbbb' NOT LIKE ' '
and leveltype = 'nnnn' and id_1= 'ww' and id_0 = 'uuu' and collecteddate = '2011-03-31';
This query will retrieve almost 1lakh records and takes 3 secs to be executed. I have created index for the json column and also the columns in where conditions. But i think its very long time to execute. Is there any way to reduce the execution time. I am new to this database optimization concepts, is there any optimization techniques to reduce my execution time to some milli seconds. Thanks in advance..
EDIT:
My index definition:
CREATE INDEX index_pop on table (id_0, id_1, collecteddate, leveltype, key, (value->'xxxxx'->>'aaaa'));
My Explain analyses result:
"Bitmap Heap Scan on table (cost=1708.27..59956.46 rows=1 width=132) (actual time=880.576..5137.266 rows=93615 loops=1)"
" Recheck Cond: (((id_0)::text = '356'::text) AND ((id_1)::text = '9'::text) AND (collecteddate = '2011-03-31'::date) AND ((leveltype)::text = 'pppp'::text))"
" Filter: ((((value -> 'xxxx'::text) ->> 'aaaa'::text) !~~ ' '::text) AND (((value -> 'xxxxx'::text) ->> 'bbbb'::text) !~~ ' '::text) AND ((((value -> 'xxxxx'::text) ->> 'aaaaa'::text))::integer >= 1) AND ((((value -> 'PCA (...)"
" Rows Removed by Filter: 4199"
" -> Bitmap Index Scan on index_name (cost=0.00..1708.27 rows=37856 width=0) (actual time=828.856..828.856 rows=97814 loops=1)"
" Index Cond: (((id_0)::text = '356'::text) AND ((id_1)::text = '9'::text) AND (collecteddate = '2011-03-31'::date) AND ((leveltype)::text = 'ppppp'::text))"
"Total runtime: 5211.271 ms"
ALso 1 more thing: Bitmap Index Scan on index_name is different index other than in my where condition index, also y only 1 index is earched??