Develop Reference

Storing Regex as strings in database and then retrieving for use in Javascript and PHP

Main Question: Should escaped backslashes also be stored in the database for Javascript and how well that would play with PHP's regex engine?
Details
I have a number of regex patterns which can be used to classify strings into various categories. An example is as below:
(^A)|(\(A)
This can recognize for example an "A" in the start of the string or if it is immediately after an opening bracket ( but not if it is anywhere else in the string.
DBC(ABC)AA
ABC(DBC)AA
My project uses these regex patterns in two languages PHP and Javascript.
I want to store these patterns in a MySQL database and since there is no datatype for regex, I thought I could store it as VARCHAR or TEXT.
The issue arises if I directly use strings in Javascript, the \( is counted only as ( as the \ backslash is used as an escape character. if this is used to create new RegExp it gives an error:
Uncaught SyntaxError: unterminated parenthetical
For example:
let regexstring = "(^A)|(\(A)";
console.log(regexstring); // outputs => "(^A)|((A)"
let regex = new RegExp(regexstring); // Gives Uncaught SyntaxError: unterminated parenthetical
Based on this answer in StackOverflow, the solution is to escape the backslashes like:
let regexstring = "(^A)|(\\(A)";
console.log(regexstring); // Outputs => "(^A)|(\\(A)"
regex = new RegExp(regexstring);
The question is therefore, should escaped backslashes also be stored in the database and how well that would play with PHP's regex engine?

I would store the raw regular expression.
The additional escape character is not actually part of the regex. It's there for JS to process the string correctly, because \ has a special meaning. You need to specify it when writing the string as "hardcoded" text. In fact, it would also be needed in the PHP side, if you were to use the same assignment technique in PHP, you would write it with the escape backslash:
$regexstring = "(^A)|(\\(A)";
You could also get rid of it if you changed the way you initialize regexstring in your JS:
<?
...
$regexstring = $results[0]["regexstring"];
?>
let regexstring = decodeURIComponent("<?=rawurlencode($regexstring);?>");
console.log(regexstring);
Another option is to just add the escaping backslashes in the PHP side:
<?
...
$regexstring = $results[0]["regexstring"];
$escapedRegexstring = str_replace('\', '\\', $regexstring);
?>
let regexstring = "<?=$escapedRegexstring;?>";
However, regardless of escaping, you should note that there are other differences in syntax between PHP's regex engine and the one used by JS, so you may end up having to maintain two copies anyway.
Lastly, if these regex expressions are meant to be provided by users, then keep in mind that outputting them as-is into JS code is very dangerous as it can easily cause an XSS vulnerability. The first method, of passing it through rawurlencode (in the PHP side) and decodeURIComponent (in the JS side) - should eliminate this risk.

Passing parameter PHP to JavaScript results in "Unexpected Token error"

I need to pass a parameter from PHP to JavaScript, so I do this:
var title='<?php echo ($home->title); ?>';
console.log(title);
but I obtain
Uncaught SyntaxError: Invalid or unexpected token
Can anyone help me?
This is the output:
var titolo_it='<p><strong>ddddddddddd</strong></p>
';

The line break in $home->title is breaking the JavaScript syntax. You cannot include a literal line break in a JavaScript string that way.
To fix this, you need to be sure the data is properly encoded, so any apostrophes, line breaks, etc. are in proper format for JavaScript. Use the built-in function json_encode(), like this:
var title=<?php echo json_encode($home->title); ?>;
console.log(title);

json_encode() gives special charactors like '\n'..how can i replace them in PHP?

I encoded an array using json_encode() function and it gave me a string like this..
"[{"details":"power - 2000w \nac-220-240v \/ 50-60hz\n369 degree cordless base\n","model_id":"MC-EK3428 \/ MC-EK3328"}]"
as you can see it contains special characters like "\n"..I want these special characters to be replaced with "" because in javascript I am using the JSON.parse(); function to convert this string to an object..
but it gives me an error
syntaxerror : missing ) after argument list
I think this is because of the special characters in the string..how can I escape these?
Edit
php :
$view->jsonencoded_array = json_encode($array);
javascript :
var products = JSON.parse('<?php echo $jsonencoded_array; ?>');//this line gives me the error
update :
found out that the error is given in this :
'<?php echo $jsonencoded_array; ?>'

The problem here is that \n (and various other combinations) have special meaning inside a JavaScript string, and you are dumping your JSON into a JavaScript string without doing any conversion of those characters.
Since JSON is heavily inspired by JavaScript literal syntax, you can use json_encode to convert a PHP string into a JavaScript string.
There are some gotchas, the main one being that </script> can appear in a JSON text without causing any problems, but having that in the middle of your JavaScript <script> element is going to cause the HTML parser to cut off your JavaScript in the middle of the string … but PHP's default encoding rules will generate <\/script> which solves that problem.
So:
<?php
$json_array = json_encode($array);
$javascript_string = $json_encode($json_array);
?>
var products = JSON.parse(<?php echo $javascript_string; ?>);
That said. A JSON array is also a JavaScript array, so you can skip that step entirely.
<?php
$json_array = json_encode($array);
?>
var products = <?php echo $json_array; ?>;

There must something that you are missing or there is some other reason for your issue while parsing in JavaScript; because json_encode handles \n and other special characters such " \ etc. very well and escape them properly without any explicit work.
I would suggest you to check the JSON produced and you are supplying to JavaScript and see if there is something missing in between.
Note: You can do a str_replace but it is not advised. Better stick to json_encodesince its s standard function and it works well.
Edit:
You should be echoing $view->jsonencoded_array not just $jsonencoded_array, no need to parse already JSON object.
php :
$view->jsonencoded_array = json_encode($array);
javascript :
var products = <?php echo $view->jsonencoded_array; ?>;

json_encode() twice helped me to solve this issue..
$view->jsonencoded = json_encode(json_encode($array));

URI escape / unescape from Javascript to Perl

I have an URL that javascript reads from an user input.
Here is a part of javascript code:
document.getElementById("Snd_Cont_AddrLnk_BG").value=encodeURI(document.getElementById("Con_AddresWeb_BG").value.toString());
Then I post the value of the string through CGI to a Perl Script (here is a part of perl code):
#!/usr/bin/perl -w
##
##
use strict;
use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use URI::Escape;
my $C_AddrLnk_BG=$query->param("Snd_Cont_AddrLnk_BG");
my $lst_upload_dir="../data";
my $lst_file_bg=$lst_upload_dir."/contacts_bg.js";
open(JSBG,">$lst_file_bg") || die "Failed to open $lst_file_bg\n";
printf JSBG "var GMapLink_bg=\"".uri_unescape($C_AddrLnk_BG)."\";\n";
close JSBG;
system("chmod 777 $lst_file_bg");
Somewhere in uri_unescape a problem occurs:
The original string from input is:
https://www.google.bg/maps/place/42%C2%B044'15.0%22N+23%C2%B019'04.2%22E/#42.7368454,23.317962,16z/data=!4m2!3m1!1s0x0:0x0
The string after javascript encodeURI() is:
https://www.google.bg/maps/place/42%25C2%25B044'15.0%2522N+23%25C2%25B019'04.2%2522E/#42.7368454,23.317962,16z/data=!4m2!3m1!1s0x0:0x0
And the script after perl uri_unescape() that is printed in file is:
https://www.google.bg/maps/place/42%C2%B044'15.0%22N+23%C2%B019'04.2 0.000000E+00/#42.7368454,23.317962,16z/data=!4m2!3m1!1s0x0:0x0
I can not ascertain whether the problem is in unescaping or printing, but the part
%2522E
is interpreted as
0.000000E+00
(with 10 leading spaces).
Can anyone help me with an idea of what I was doing wrong?

There are numerous problems with your code.
document.getElementById("Snd_Cont_AddrLnk_BG").value =
encodeURI(document.getElementById("Con_AddresWeb_BG").value.toString());
I can't figure out when you think encodeURI here. All you should have is the following:
document.getElementById("Snd_Cont_AddrLnk_BG").value =
document.getElementById("Con_AddresWeb_BG").value;
printf JSBG "var GMapLink_bg=\"".uri_unescape($C_AddrLnk_BG)."\";\n";
Now the erroneous encodeURI is removed, uri_unescape needs to be removed too.
Furthermore, adding quotes around text doesn't always make it a valid JavaScript literal. The easiest way to do that is as follows:
use JSON qw( );
my $json = JSON->new()->allow_nonref();
$json->encode($C_AddrLnk_BG)
That snippet also misuses printf. printf takes a format parameter, so you want
printf FH "%s", ...
or simply
print FH ...
So what you end up with is:
use JSON qw( );
my $json = JSON->new()->allow_nonref();
$json->encode($C_AddrLnk_BG)
print JSBG "var GMapLink_bg=" . $json->encode($C_AddrLnk_BG) ."\n";

You are using printf instead of print to output the result of uri_unescape. It is interpreting %22E as an engineering-format floating point field with a width of 22. Presumable you have nothing else in the printf parameter list, so undef is being evaluated as zero, resulting in 0.000000E+00.
If you had use warnings in place as you should, you would see messages like Missing argument in printf

jQuery.parseJSON throws “Invalid JSON” error due to escaped single quote in JSON

I’m making requests to my server using jQuery.post() and my server is returning JSON objects (like { "var": "value", ... }). However, if any of the values contains a single quote (properly escaped like \'), jQuery fails to parse an otherwise valid JSON string. Here’s an example of what I mean (done in Chrome’s console):
data = "{ \"status\": \"success\", \"newHtml\": \"Hello \\\'x\" }";
eval("x = " + data); // { newHtml: "Hello 'x", status: "success" }
$.parseJSON(data); // Invalid JSON: { "status": "success", "newHtml": "Hello \'x" }
Is this normal? Is there no way to properly pass a single quote via JSON?

According to the state machine diagram on the JSON website, only escaped double-quote characters are allowed, not single-quotes. Single quote characters do not need to be escaped:
Update - More information for those that are interested:
Douglas Crockford does not specifically say why the JSON specification does not allow escaped single quotes within strings. However, during his discussion of JSON in Appendix E of JavaScript: The Good Parts, he writes:
JSON's design goals were to be minimal, portable, textual, and a subset of JavaScript. The less we need to agree on in order to interoperate, the more easily we can interoperate.
So perhaps he decided to only allow strings to be defined using double-quotes since this is one less rule that all JSON implementations must agree on. As a result, it is impossible for a single quote character within a string to accidentally terminate the string, because by definition a string can only be terminated by a double-quote character. Hence there is no need to allow escaping of a single quote character in the formal specification.
Digging a little bit deeper, Crockford's org.json implementation of JSON for Java is more permissible and does allow single quote characters:
The texts produced by the toString methods strictly conform to the JSON syntax rules. The constructors are more forgiving in the texts they will accept:
...
Strings may be quoted with ' (single quote).
This is confirmed by the JSONTokener source code. The nextString method accepts escaped single quote characters and treats them just like double-quote characters:
public String nextString(char quote) throws JSONException {
char c;
StringBuffer sb = new StringBuffer();
for (;;) {
c = next();
switch (c) {
...
case '\\':
c = this.next();
switch (c) {
...
case '"':
case '\'':
case '\\':
case '/':
sb.append(c);
break;
...
At the top of the method is an informative comment:
The formal JSON format does not allow strings in single quotes, but an implementation is allowed to accept them.
So some implementations will accept single quotes - but you should not rely on this. Many popular implementations are quite restrictive in this regard and will reject JSON that contains single quoted strings and/or escaped single quotes.
Finally to tie this back to the original question, jQuery.parseJSON first attempts to use the browser's native JSON parser or a loaded library such as json2.js where applicable (which on a side note is the library the jQuery logic is based on if JSON is not defined). Thus jQuery can only be as permissive as that underlying implementation:
parseJSON: function( data ) {
...
// Attempt to parse using the native JSON parser first
if ( window.JSON && window.JSON.parse ) {
return window.JSON.parse( data );
}
...
jQuery.error( "Invalid JSON: " + data );
},
As far as I know these implementations only adhere to the official JSON specification and do not accept single quotes, hence neither does jQuery.

If you need a single quote inside of a string, since \' is undefined by the spec, use \u0027 see http://www.utf8-chartable.de/ for all of them
edit: please excuse my misuse of the word backticks in the comments. I meant backslash. My point here is that in the event you have nested strings inside other strings, I think it can be more useful and readable to use unicode instead of lots of backslashes to escape a single quote. If you are not nested however it truly is easier to just put a plain old quote in there.

I understand where the problem lies and when I look at the specs its clear that unescaped single quotes should be parsed correctly.
I am using jquery`s jQuery.parseJSON function to parse the JSON string but still getting the parse error when there is a single quote in the data that is prepared with json_encode.
Could it be a mistake in my implementation that looks like this (PHP - server side):
$data = array();
$elem = array();
$elem['name'] = 'Erik';
$elem['position'] = 'PHP Programmer';
$data[] = json_encode($elem);
$elem = array();
$elem['name'] = 'Carl';
$elem['position'] = 'C Programmer';
$data[] = json_encode($elem);
$jsonString = "[" . implode(", ", $data) . "]";
The final step is that I store the JSON encoded string into an JS variable:
<script type="text/javascript">
employees = jQuery.parseJSON('<?=$marker; ?>');
</script>
If I use "" instead of '' it still throws an error.
SOLUTION:
The only thing that worked for me was to use bitmask JSON_HEX_APOS to convert the single quotes like this:
json_encode($tmp, JSON_HEX_APOS);
Is there another way of tackle this issue? Is my code wrong or poorly written?
Thanks

When You are sending a single quote in a query
empid = " T'via"
empid =escape(empid)
When You get the value including a single quote
var xxx = request.QueryString("empid")
xxx= unscape(xxx)
If you want to search/ insert the value which includes a single quote in a query
xxx=Replace(empid,"'","''")

Striking a similar issue using CakePHP to output a JavaScript script-block using PHP's native json_encode. $contractorCompanies contains values that have single quotation marks and as explained above and expected json_encode($contractorCompanies) doesn't escape them because its valid JSON.
<?php $this->Html->scriptBlock("var contractorCompanies = jQuery.parseJSON( '".(json_encode($contractorCompanies)."' );"); ?>
By adding addslashes() around the JSON encoded string you then escape the quotation marks allowing Cake / PHP to echo the correct javascript to the browser. JS errors disappear.
<?php $this->Html->scriptBlock("var contractorCompanies = jQuery.parseJSON( '".addslashes(json_encode($contractorCompanies))."' );"); ?>

I was trying to save a JSON object from a XHR request into a HTML5 data-* attribute. I tried many of above solutions with no success.
What I finally end up doing was replacing the single quote ' with it code ' using a regex after the stringify() method call the following way:
var productToString = JSON.stringify(productObject);
var quoteReplaced = productToString.replace(/'/g, "'");
var anchor = '<a data-product=\'' + quoteReplaced + '\' href=\'#\'>' + productObject.name + '</a>';
// Here you can use the "anchor" variable to update your DOM element.

Interesting. How are you generating your JSON on the server end? Are you using a library function (such as json_encode in PHP), or are you building the JSON string by hand?
The only thing that grabs my attention is the escape apostrophe (\'). Seeing as you're using double quotes, as you indeed should, there is no need to escape single quotes. I can't check if that is indeed the cause for your jQuery error, as I haven't updated to version 1.4.1 myself yet.