My goal is simply to convert a JavaScript object to JSON and insert it into a MySQL row that stores the MySQL JSON data type. This process must happen as part of a large batch insert, so I cannot edit the data manually. When call JSON.stringify on the object, I get valid JSON, but, because the object contains song lyrics that often contain single quotes, when I try to run the SQL query, MySQL throws a parse error.
This works fine
const validJson = '{"foo": "bar", "buzz": 1}';
INSERT INTO table_name ( songid, json )
VALUES ( ${song.songid}, ${validJson} );
But, this doesn’t
const validJsonWithASingleQuote = {"foo's": "bar", "buzz": 1}';
INSERT INTO table_name ( songid, json )
VALUES ( ${song.songid}, ${validJsonWithASingleQuote} );
I also tried using a prepared statement with no luck
PREPARE myInsert FROM 'INSERT INTO table_name ( songid, json ) VALUES ( ?, ? )';
SET #a = ${song.songid};
SET #b = ${JSON.stringify(r)};
EXECUTE myInsert USING #a, #b;
I should also mention, that the original JavaScript objects contain strings that have single quotes escaped with "\". The JSON.stringify method, decodes those backslashes.
Unfortunately, all of the SO questions I have found on this topic either recommend escaping single quotes with "\" manually or they have gone unresolved. Is there a programatic way to accomplish this? Perhaps a JavaScript or MySQL method that would generate valid JSON and leave the "\'" sequences in?
I finally found my way to this answer:
https://stackoverflow.com/a/49525488/1359529
Turns out the Node.js driver for mysql contains an escape method. So, something like this works:
PREPARE myInsert FROM 'INSERT INTO table_name ( songid, json ) VALUES ( ?, ? )';
SET #a = ${song.songid};
SET #b = ${mysql.escape(JSON.stringify(sr))};
EXECUTE myInsert USING #a, #b;
This tripped me up for a couple of days!
I'm using the package and was having trouble with a single quote as a prop value:
{
name: "Jade's Palace",
}
I struggled to escape the single quote for mysql and could not create "Jade's Palace" because JS uses \ as its escape char.
The solution was a prepared statement with the escape method.
const query = 'INSERT INTO Places(id, data) VALUES ?';
const params = results.data?.map((data: any) => [
data.id,
{
toSqlString: () => connection?.escape(JSON.stringify(data))
},
]);
await connection.query(query, [params]);
Related
I have an sql database from node red. The table of the database contains a column with javascript date.now() timestamps e.g. 1641154320892. Another column of the table contains temperature values. I'd like to select temperature values of a specific time period.
I tried the following code:
import sqlite3
conn = sqlite3.connect('/home/ktm/Developing/test/home_automation.db')
print ("Opened database successfully")
conn.row_factory = lambda cursor, row: row[0]
c = conn.cursor()
ids = c.execute('SELECT Buero_temp FROM home_automation WHERE Zeitstempel BETWEEN '2022-01-05' AND '2022-01-07';').fetchall()
for i in ids:
print (i)
Unfortunately, I get "SyntaxError: invalid syntax"
What am I doing wrong?
Firstly there is a syntax error.
A string in Python can be formed using both single and double quotes. If you want to use single quote within a string then outer string has to encapsulated with double quotes.
For eg:
my_string = "My string has 'single quotes'"
Now coming to your query:
As per your question the data stored in DB is in the form of timestamps, eg: 1641154320892. Javascript uses 13 digits timestamp.
So you should query with timestamps.
import sqlite3
def str_date_to_epoch(s):
print(s)
return int(datetime.strptime(s, '%Y-%m-%d').timestamp() * 1000)
conn = sqlite3.connect('/home/ktm/Developing/test/home_automation.db')
print("Opened database successfully")
conn.row_factory = lambda cursor, row: row[0]
c = conn.cursor()
statement = f"SELECT Buero_temp FROM home_automation WHERE Zeitstempel BETWEEN {str_date_to_epoch('2022-01-05')} AND {str_date_to_epoch('2022-01-07')};"
ids = c.execute(statement).fetchall()
for i in ids:
print(i)
I would like to know if it is possible to convert the ndjson data from this API: https://lichess.org/api/team/overberg-chess-group/users and turn it into an HTML table. I have found some javascript snippets that will convert normal json into an html table but not ndjson. Any help would be appreciated
Looks like the API pulls a file that's very close to JSON. Assuming that the API data is in var sourceData, just a few tweaks are required...
// Add commas between the list of objects...
data = sourceData.split( '}\n{' ).join( '},{' );
// ...remove all carriage return line feeds...
data = data.split( '\r\n' ).join( '|' );
// ...and there's one instance of a double quoted string,
// eg, "bio":""I am committed..."". This appears to be
// the only occurrence of this anomaly, so the following
// currently works, but you might run into issues in the
// future, for example, if an attribute is a null string "".
data = data.split( '""' ).join( '"' );
let dataObject = JSON.parse( '[' + data + ']' );
At this point, dataObject contains the object data representing the ndjson data pulled via the API. Note that when you go to place this object into a HTML table, you'll need to convert the pipe characters "|" back into line feeds... This should help you along your way...
I'm trying to pass a string to my mongodb query, but when it goes inside the query it adds double quotes to the string.
I'm passing this as a string to the criteria, because it is formed dynamically as a string:
str1={AccId:8,WSId:24237,CaseID:{$in:[4697516]},MEId:{$in:[4697523]},ConfigID:{$in:[4697520]}}
var criteria = { str1 },
So when I'm passing this final criteria to db.coll.aggregate, it appends str1 and double quotes to the query.
query forming={"$match":{"str1":"{AccId:8,WSId:24237,CaseID:{$in:[4697516]},MEId:{$in:[4697523]},ConfigID:{$in:[4697520]}}"}
query desired= {"$match":{AccId:8,WSId:24237,CaseID:{$in:[4697516]},MEId:{$in:[4697523]},ConfigID:{$in:[4697520]}}}
How can I get the desired query ?
What you really want to do is create the pipeline using object initializing methods like the bracket notation. Also, there's no need to use the $in operator with a single array element in your query, just query the field directly on the value.
Follow this example to get the concept:
var criteria = {
'AccId': 8,
'WSId': 24237,
'CaseID': 4697516,
'MEId': 4697523,
'ConfigID': 4697520
},
match = {};
match["$match"] = criteria;
db.collection.aggregate([match]);
I have a textarea created for a personal message, with a subject. It is passed to a Javascript/jQuery function, which passes it to a PHP file to store in the database. However, when special characters such as the ampersand, less than, greater than, apostrophe, plus sign, and quotations are used, it doesn't store correctly in my database. So without saying, when I retrieve the data, the data is not displayed properly.
Here is the HTML:
<input id="pmsubject" placeholder="Subject"><br />
<textarea id="pmtext" placeholder="Send a private message"></textarea>
<button id="pmBtn" onclick="postPm(pmsubject,pmtext)">Send</button>
Here is the Javascript/jQuery (partial):
function postPm(subject,textarea){
var data = $("#textarea").val();
var data2 = $("#subject").val();
I do some error checking and handling then send my information with AJAX:
type: "POST",
url: "pm_system.php",
data:"data="+data+"&data2="+data2,
So far so good right? Here is the pm_system.php portion where I store the code:
$data = htmlentities($_POST['data']);
$data = mysqli_real_escape_string($db_con, $data);
$data2 = htmlentities($_POST['data2']);
$data2 = mysqli_real_escape_string($db_con, $data2);
$sql = "INSERT INTO pm(subject, message)
VALUES('$data2','$data')";
$query = mysqli_query($db_con, $sql);
mysqli_close($db_con);
So if I write a message that says, I'm a big fan of cats + dogs & "sometimes" birds. My output would be:
I\'m a big fan of cats dogs
It always puts slashes in front of quotations and apostrophes, always replaces + sign with a space, and nothing follows after an ampersand sign. I've tried replacing the characters like this in Javascript:
data = data.replace(/\"/g, '"'); //Just using one for example
But that doesn't work either. How do I save these characters from a textarea in a database, unaltered?
I would guess, that the data you receive through your JavaScript function is already escaped. So when you enter I'm a big fan of cats dog you get I\'m a big fan of cats dogs in your PHP script. When you than use mysqli_real_escape() you are adding another escape character.
So you might want to replace the escape character before:
$data = stripslashes($_POST['data']);
$data = mysqli_real_escape_string($db_con, $data);
$data2 = stripslashes($_POST['data2']);
$data2 = mysqli_real_escape_string($db_con, $data2);
$sql = "INSERT INTO pm(subject, message)
VALUES('$data2','$data')";
$query = mysqli_query($db_con, $sql);
mysqli_close($db_con);
I would not recommend to use htmlentities() but save the data "as is" into the database as otherwise things like full text searches don't work correctly.
The issue with the + sign is probably because you send the values as a query string data:"data="+data+"&data2="+data2 and in a URL, a + sign is used for a space. To fix that, you should rather pass the data as an object:
type: "POST",
url: "pm_system.php",
data: { "data": data, "data2": data2 },
That should fix also most of the other problematic characters.
I'd suggest trying htmlspecialchars() instead of htmlentities(). I've had some troubles with htmlentities() and outputting the data in the past. Using htmlspecialchars() solved it.
I'm parsing the body text from incoming emails, looking for key/value pairs.
Example Email Body
First Name: John
Last Name:Smith
Email : john#example.com
Comments = Just a test comment that
may span multiple lines.
I tried using a RegEx ([\w\d\s]+)\s?[=|:]\s?(.+) in multiline mode. This works for most emails, but fails when there's a line break that should be part of the value. I don't know enough about RegEx to go any further.
I have another parser that goes line-by-line looking for the key/value pairs and simply folds a line into the last matched value if a key/value pair is NOT found. It's implemented in Scala.
val lines = text.split("\\r?\\n").toList
var lastLabelled: Int = -1
val linesBuffer = mutable.ListBuffer[(String, String)]()
// only parse lines until the first blank line
// null_? method is checks for empty strings and nulls
lines.takeWhile(!_.null_?).foreach(line => {
line.splitAt(delimiter) match {
case Nil if line.nonEmpty => {
val l = linesBuffer(lastLabelled)
linesBuffer(lastLabelled) = (l._1, l._2 + "\n" + line)
}
case pair :: Nil => {
lastLabelled = linesBuffer.length
linesBuffer += pair
}
case _ => // skip this line
}
})
I'm trying to use RegEx so that I can save the parser to the db and change it on a per-sender basis at runtime (implement different parsers for different senders).
Can my RegEx be modified to match values that contain newlines?
Do I need to just forget about using RegEx and use some JavaScript? I already have a JavaScript parser that lets me store the JS in the DB and essentially do everything that I want to do with the RegEx parser.
I think this should work...
((.+?)((\s*)(:|=)(\s*)))(((.|\n)(?!((.+?)(:|=))))+)
...as tested here http://regexpal.com/. If you loop through the matches you should be able to pull out the key and value.