I am working on a web application that parse multiple text files, store the data in a SQLite3 database and display the data in a browser. Then I make a HTTP request to my Node.js server to query the database. The data is converted to JSON and send back to the browser. Finally, the JSON is converted back to string and parse it for display.
My program works for small files size under 200MB, but very slowly in terms of parsing and displaying. When it gets 300MB+, I can not parse the files and got this error.
ProgrammingError: You must not use 8-bit bytestrings unless you use a
text_factory that can interpret 8-bit bytestrings (like text_factory =
str). It is highly recommended that you instead just switch your
application to Unicode strings.
I tried the text_factory, it gets rid of the error code. When I query the database, it returns nothing and the program hangs in there. I am a newbie. What am I doing wrong? What can I do to make it parse larger size files and faster?
#parsing some files and insert them into a sqlite database
#create database in python
conn = sqlite3.connect('MyDB.db')
conn.text_factory = str # didn't work, the program hang
c = conn.cursor()
c.execute('''CREATE TABLE actions (dates text, names text, methods text)''')
c.execute('''CREATE TABLE people (names text, ID text)''')
# insert in different function
dataB.execute("INSERT INTO people VALUES(?, ?)", (value[index], id))
dataB.execute("INSERT INTO action VALUES(?, ?, ?)", (date, value[index], someaction ))
//seperate file
//retrieve the data from javascript
var sqlite3 = require("sqlite3");
var logDB = new sqlite3.Database("log.db");
query = "SELECT DISTINCT id, name FROM people p, action a where p.name = a.name and p.id Not IN (SELECT DISTINCT id FROM someothertable);";
function queryTheServer(query, response) {
logDB.all(query, function(error, thing){
serveFromDatabase(error, thing, query, response);
});
}
//pass by as JSON
response.writeHead(200, {"Content-Type": "application/json"});
response.write(JSON.stringify(result));
response.end();
Related
I want to run a query in Node SQL Server which is using IN clause. This is the string used for querying 'a','b','c'. This code works fine, but user is passing data so, I can't use it. May lead to attacks:
const dbResult = await request.query(`
SELECT OrderID, ParentSKURefNum, SKURefNum, OrderCompleteTime
FROM ${tables.ORDERS}
WHERE OrderID IN (${idsWithQuotes})
`);
I want to use request.input('OrderIDs', ids) and then code will be like this:
request.input('OrderIDs', ids);
const dbResult = await request.query(`
SELECT OrderID, ParentSKURefNum, SKURefNum, OrderCompleteTime
FROM ${tables.ORDERS}
WHERE OrderID IN (#OrderIDs)
`);
But the code above always shows: No data found. What am I doing wrong? In second situation I also tried removing first and last quote from the string assuming request automatically adds it.
Thanks for your help!
I'm using SQL Server 2012 which doesn't support STRING_SPLIT function to split CSV into some sort of table which then IN operator operates on.
I found it on stack overflow that we can split the values using XML which I didn't really understand but did the trick.
SELECT OrderID, ParentSKURefNum, SKURefNum, OrderCompleteTime
FROM ${tables.ORDERS}
WHERE OrderID IN (
SELECT Split.a.value('.', 'NVARCHAR(MAX)') DATA
FROM
(
SELECT CAST('<X>'+REPLACE(#OrderIDs, ',', '</X><X>')+'</X>' AS xml) AS STRING
) AS A
CROSS APPLY String.nodes('/X') AS Split(a)
)
I have to execute an INSERT INTO query using mysql or mysql2 in node.js
The values passed into the query utilizes a spatial geometry function ST_GeomFromGeoJSON() on geoMap column which is of type GEOMETRY.
Here is the simplified code:
const insertQuery = `INSERT INTO maps (name, geoMap) VALUES ?`;
const insertData = [];
geoMapList.forEach((geoMap) => {
insertData.push([
geoMap.name,
`ST_GeomFromGeoJSON(${JSON.stringify(geoMap.map)}, 2)`
]);
});
this.connection.query(insertQuery, [insertData]);
The above code does not work and throws the error
Cannot get geometry object from data you send to the GEOMETRY field
I believe this is because the ST_GeomFromGeoJSON() is not parsed as a function but as a string by MySQL.
How can this be resolved?
You can't put MySQL function calls in the parameter array, that data is all treated as literals.
Call the function in the SQL, with the JSON string as its parameter.
I don't think you can use node-mysql's bulk-insert for this, though, so you'll need to insert each row separately.
const insertQuery = `INSERT INTO maps(name, geoMap) VALUES (?, ST_GeomFromGeoJSON(?))`
geoMapList.forEach((geomap) =>
this.connection.query(insertQuery, [geomap.name, JSON.stringify(geomap.map)])
);
To avoid calling query() thousands of times, you can put multiple values lists in the query.
const values = Array(geoMapList.length).fill('(?, ST_GeomFromGeoJSON(?))).join(',');
const insertQuery = `INSERT INTO maps(name, geoMap) VALUES ${values}`;
const params = geoMapList.flatMap(({name, map}) => [name, JSON.stringify(map)]);
this.connection.query(insertquery, params);
Since this will create a very long INSERT query, make sure you increase the MySQL max_allowed_packet setting. You can also split geoMapList into smaller batches, rather than doing everything in a single query.
On my website whenever a user enters a mobile emoji like 😀 into an input field it will be saved as ?? in my database.
Those emojis are encoded in utf8mb4, so I already updated my database collation to utf8mb4_general_ci.
While the emoticons can be saved successfully now when transfering the message containing a emoji from a client to my server, it still get's somewhere changed into ?? and I am now trying to figure out where and how to solve it.
Sending the message to my server happens in this ajax call:
function updateStatus() {
var status = $("#status").val();
jsRoutes.controllers.Userdata.updateStatus( status ).ajax({
success : function(data) {
$("#profil").cftoaster({content: data});
},
error : function(err) {
$("#profil").cftoaster({content: err.responseText});
}
});
}
On serverside I am using java based Play Framework 2.4.4.
This is the beginning of the method which is called in the ajax call:
public static Result updateStatus(String status) {
String receivedName = Application.getSessionUser();
Logger.debug("RecvStatus: " + status);
...
}
The Logger output already is ?? for an emoticon.
The route looks like this:
PUT /status/ controllers.Userdata.updateStatus(status: String)
EDIT:
To make sure the transfer from client to server is alright I am now transferring the actual unicode values, I change my server function like this
Logger.debug("RecvStatus: " + status);
status = status.replace("\\","");
String[] arr = status.split("u");
status = "";
for(int i = 1; i < arr.length; i++){
int hexVal = Integer.parseInt(arr[i], 16);
status += (char)hexVal;
}
Logger.debug("RecvStatus: " + status);
and get the following output:
[debug] application - RecvStatus: \ud83d\ude01
[debug] application - RecvStatus: ?
which means the problem is probably with java
So for anyone having a similiar problem here my workaround.
First I tried to convert the String in java to base64 and store it this way. Unfortunately after decoding the base64 the unicode information still got lost and ?? was shown instead of emoticons.
What I than did was converting the received String into unicode and than into base64, and when I load the Strings from the database I first decode base64 and then convert the unicode information into an actual String. This way the emoticons are stored and afterwards shown correctly.
I m actually using the mysql module to query my database and I m facing a problem. In fact, I need to escape a full object and parse it in a query that mysql could understand.
Actually, I have :
getByCriteria: (collectionName, criteria)->
sql = 'SELECT * FROM ?? WHERE ?'
inserts = [collectionName, criteria]
sql = MySql.format(sql, inserts)
deferred = Q.defer()
MysqlAdapter.CONNECTION.query(sql, (err, rows, fields)->
console.log err
if err
deferred.reject(new Error(err))
else
deferred.resolve(rows)
)
return deferred.promise
But the console.log(sql) prints :
SELECT * FROM user WHERE username = 'admin', id=1
So I have guessed that the "Mysql.format" returns a string for the INSERT/UPDATE SQL actions.
How can I do it with mysql, without parsing the entire string with a homemade solution ?
THanks for advance
I'm afraid that MySql.format() is very simple (https://github.com/felixge/node-mysql/blob/master/lib/protocol/SqlString.js#L67) so you'll need to do your own formatting.
Just remember to escape values with MySql.escape() and identifiers with MySql.escapeId()
You'll need something similar to SqlString.objectToValues() https://github.com/felixge/node-mysql/blob/master/lib/protocol/SqlString.js#L109 but with values joined with AND instead of ,
Im new with mongo and node. I was trying to upload a csv into the mongodb.
Steps include:
Reading the csv.
Converting it into JSON.
Pushing it to the mongodb.
I used 'csvtojson' module to convert csv to json and pushed it using code :
MongoClient.connect('mongodb://127.0.0.1/test', function (err, db) { //connect to mongodb
var collection = db.collection('qr');
collection.insert(jsonObj.csvRows, function (err, result) {
console.log(JSON.stringify(result));
console.log(JSON.stringify(err));
});
console.log("successfully connected to the database");
//db.close();
});
This code is working fine with csv upto size 4mb; more than that its not working.
I tried to console the error
console.log(JSON.stringify(err));
it returned {}
Note: Mine is 32 bit system.
Is it because there a document limit of 4mb for 32-bit systems?
I'm in a scenario where I can't restrict the size and no.of attributes in the csv file (ie., the code will be handling various kinds of csv files). So how to handle that? I there any modules available?
If you are not having a problem on the parsing the csv into JSON, which presumably you are not, then perhaps just restrict the list size being passed to insert.
As I can see the .csvRows element is an array, so rather than send all of the elements at once, slice it up and batch the elements in the call to insert. It seems likely that the number of elements is the cause of the problem rather than the size. Splitting the array up into a few inserts rather than 1 should help.
Experiment with 500, then 1000 and so on until you find a happy medium.
Sort of coding it:
var batchSize = 500;
for (var i=0; i<jsonObj.csvRows.length; i += batchSize) {
var docs = jsonObj.csvRows.slice(i, i+(batchSize -1));
db.collection.insert( docs, function(err, result) {
// Also don't JSON covert a *string*
console.log(err);
// Whatever
}
}
And doing it in chunks like this.
You can make those data as an array of elements , and then simply use the MongoDB insert function, passing this array to the insert function