Develop Reference

Decoding/reading json part of complex text file

I am starting to develop a desktop application using Electron. This app will parse some files and datas will be shown from these files. These files are containing complex data.
Now, I am trying to get json data from a complex text file. This text file contains some string and json objects. Sample file looks like that:
...strings that I'm not interested in...
{
"partOneA":0,
"partOneB":7,
....
}
...randomly strings may stand between json sections...
{
"partTwoA":7,
"partTwoB":4,
"partTwoC":4,
...
}
{
"differentPartA":3,
"differentPartB":5,
"differentPartC":6,
...
}
...somemoretext....
The problem is that, how can I get the json parts from this complex file using javascript? Performance of the solution should be considered.
Additionaly, Consider that json structure is nested like that:
{
"partOneA":0,
"partOneB" :{
"partOneBnode1":0,
"partOneBnode2":7,
}
}
Resolving with regular expressions is not applicable for this issue.
Now, I am trying to find a javascript based solution.

As long as you can rely on { and } as starting and closing tags you could use a regex like:
var jsonRegex = new RegExp(/({(?:(.|\n)*?(?:[^\\])){0,1}?})/g);
var result = jsonRegex.exec(text);
var firstMatch= result[1];
As a result you should get the first piece with the subsequent matches at the subsequent indexes. You can read the docs here on mdn.
You can play around with regex on sites like http://regexr.com/
Note
This approach does not work with nested JSON because you would require to match the same amount of opening and closing brackets (see this answer).

"Fixing" JSON coming out of MySQL

I'm fetching JSON code stored in MySQL and it has extra slashes, which I have to remove in order to parse it in JavaScript, after I print it on the page. Right now I'm doing the following:
$save = str_replace("\n", "<br>", $save); // Replace new line characters with <br>
$save = str_replace('\\"', '"', $save); // top-level JSON
$save = str_replace('\\\\"', '\"', $save); // HTML inside top level JSON
$save = str_replace('\\\\\\\\\\"', '\\\\\"', $save); // HTML inside second level JSON
Here is an example JSON code, as it comes out from MySQL:
{\"id\":2335,\"editor\":{\"selected_shape\":\"spot-7488\"},\"general\":{\"name\":\"HTML Test\",\"shortcode\":\"html-test\",\"width\":1280,\"height\":776},\"spots\":[{\"id\":\"spot-7488\",\"x\":9.9,\"y\":22.6,\"default_style\":{\"use_icon\":1},\"tooltip_content\":{\"content_type\":\"content-builder\",\"plain_text\":\"<p class=\\\"test\\\">Test</p>\",\"squares_json\":\"{\\\"containers\\\":[{\\\"id\\\":\\\"sq-container-293021\\\",\\\"settings\\\":{\\\"elements\\\":[{\\\"settings\\\":{\\\"name\\\":\\\"Paragraph\\\",\\\"iconClass\\\":\\\"fa fa-paragraph\\\"},\\\"options\\\":{\\\"text\\\":{\\\"text\\\":\\\"<p class=\\\\\\\"test\\\\\\\">Test</p>\\\"}}}]}}]}\"}}]}
And here is how it's supposed to look in order to get parsed correctly (using jsonlint.com to test):
{"id":2335,"editor":{"selected_shape":"spot-7488"},"general":{"name":"HTML Test","shortcode":"html-test","width":1280,"height":776},"spots":[{"id":"spot-7488","x":9.9,"y":22.6,"default_style":{"use_icon":1},"tooltip_content":{"content_type":"content-builder","plain_text":"<p class=\"test\">Test</p>","squares_json":"{\"containers\":[{\"id\":\"sq-container-293021\",\"settings\":{\"elements\":[{\"settings\":{\"name\":\"Paragraph\",\"iconClass\":\"fa fa-paragraph\"},\"options\":{\"text\":{\"text\":\"<p class=\\\"test\\\">Test</p>\"}}}]}}]}"}}]}
Please note that I have HTML code inside JSON, which is inside another JSON and this is where it gets a bit messy.
My question - is there a function or library for PHP (for JS will work too) which covers all those corner cases, because I'm sure someone will find a way to break the script.
Thanks!

The short answer, which is woefully inadequate, is for you to use stripslashes. The reason this answer is not adequate is that your JSON string might have been escaped or had addslashes called on it multiple times and you would have to call stripslashes precisely once for each time this had happened.
The proper solution is to find out where the slashes are being added and either a) avoid adding the slashes or b) understand why the slashes are there and respond accordingly. I strongly believe that the process that creates that broken JSON is where the problem lies.
Slashes are typically added in PHP in a few cases:
magic_quotes are turned on. This is an old PHP feature which has been removed. The basic idea is that PHP used to auto-escape quotes in incoming requests to let you just cram incoming strings into a db. Guess what? NOT SAFE.
add_slashes has been called. Why call this? Some folks use it as an incorrect means of escaping data before sticking stuff in a db. Others use it to keep HTML from breaking when echoing variables out (htmlspecialchars should probably be used instead). It can also come in handy in a variety of other meta situations when you are defining code in a string.
When escaping data input. The most common escaping function is mysqli_real_escape_string. It's very important to escape values before inserting them in a db to prevent sql injection and other exploits but you should never escape things twice.
So there's a possibility that your code is double-escaping things or that addslashes is getting called or something like magic_quotes is causing the problem, but I suspect it is another problem: some JS code might be supplying this JSON not as a proper JSON string, but one that has been escaped so to define a string within javascript.
If you take your example JSON string above, and slap some quotes around it:
var myJSON = "<put your string here>";
then SURPRISE your javascript is not broken and the var myJSON contains a string that is actually valid JSON and can be parsed into an a valid JSON object:
var myJSON = "{\"id\":2335,\"editor\":{\"selected_shape\":\"spot-7488\"},\"general\":{\"name\":\"HTML Test\",\"shortcode\":\"html-test\",\"width\":1280,\"height\":776},\"spots\":[{\"id\":\"spot-7488\",\"x\":9.9,\"y\":22.6,\"default_style\":{\"use_icon\":1},\"tooltip_content\":{\"content_type\":\"content-builder\",\"plain_text\":\"<p class=\\\"test\\\">Test</p>\",\"squares_json\":\"{\\\"containers\\\":[{\\\"id\\\":\\\"sq-container-293021\\\",\\\"settings\\\":{\\\"elements\\\":[{\\\"settings\\\":{\\\"name\\\":\\\"Paragraph\\\",\\\"iconClass\\\":\\\"fa fa-paragraph\\\"},\\\"options\\\":{\\\"text\\\":{\\\"text\\\":\\\"<p class=\\\\\\\"test\\\\\\\">Test</p>\\\"}}}]}}]}\"}}]}";
console.log(JSON.parse(myJSON)); // this is an actual object
The key here is to examine the point of entry where this JSON arrives in your system. I suspect some AJAX request has created some object and rather than sending valid JSON Of that object, it is sending instead an escaped string of a JSON object.
EDIT:
Here's a simple example of what happens when you have too many encodings. Try running this JS in your browser and observe the console output:
var myObj = {"key":"here is my value"};
console.log(myObj);
var myJSON = JSON.stringify(myObj);
console.log(myJSON);
var doubleEncoded = JSON.stringify(myJSON);
console.log(doubleEncoded);

Dynamic Unicode Generation into the DOM

I have a function linking to a method for a JavaScript library I'm working on. Basically taking romanized Korean and converting it to specific Unicode sequences and re-inserting it into the DOM. The resulting strings that it generates are correct, but the re-insertion back into the DOM seems off...
For example: If I have the following in my DOM:
<ko>hangug-eo</ko>
The function is meant to convert it accordingly, replacing hangug-eo with 한국어 to show on the browser:
한국어 within the <ko> tags...
The function that does the string setting within the DOM is as follows:
function (){
var z=document.getElementsByTagName('ko');
for(x=z.length;x--;){
z[x].childNodes[0].data=kimchi.go(z[x].childNodes[0].data);
}
}
However, it seems that all this seems to be doing is just placing the &# Unicode entities straight into the DOM without it converting to their respective character equivalents... So all I'm seeing is 한국어
Can anyone please point out what I may be doing wrong?
kimchi.go() is the function that ultimately provides the Unicoded string...

You can always just set the text directly using textContent without having to use HTML entities:
z[x].textContent = '한국어';
But if you need to use HTML entities, just use innerHTML instead
z[x].innerHTML = kimchi.go(z[x].childNodes[0].data);
You can see the latter in the example below.
https://jsfiddle.net/nmL3to8w/1/

Difference between angular.fromJson and $scope.$eval when applied to JSON string

In my angularjs apps, I usually parse a JSON string by using angular.fromJson, like so:
var myObject=angular.fromJSON(jsonString);
However, it seems that I would obtain the same result by using $scope.$eval:
var myObject=$scope.$eval(jsonString);
See this fiddle
Or by using vanilla javaScript, like so:
var myObject=JSON.parse(jsonString);
Is there any particular reason to use angular.fromJSON rather than JSON.parse?
Is there any possible issue when using $scope.$eval to parse a JSON string?

Check out the source code:
function fromJson(json) {
return isString(json)
? JSON.parse(json)
: json;
}
They're just passing through to JSON.parse.
As for $eval it shells out to $parse:
// $scope.$eval source:
$eval: function(expr, locals) {
return $parse(expr)(this, locals);
},
$parse source is too long to post, but it is essentially capable of converting inline (stringified) objects to real Objects and so it makes sense that in this case, it will actually convert your JSON as well.
(I did not know this until reading through the $parse source just now.)
Is there any particular reason to use angular.fromJSON rather than JSON.parse?
Nope, not really. Although they do check to you to ensure that you don't double-parse a JSON string, like so:
var jsonString = '{"foo":"bar"}';
var json = JSON.parse(jsonString); // Parsing once is good :)
JSON.parse(json); // Parsing twice is bad :(
Is there any possible issue when using $scope.$eval to parse a JSON string?
I don't think so off the top of my head, other than that you're doing more work than is necessary. So if you know you have JSON, there's no reason to use the heavier $parse function.

The above answer is almost correct. However, there is a potential issue with using $scope.$eval() to parse a JSON string, which does not exist with either JSON.parse() or angular.fromJson(): security. Angular allows an expression to contain complex JavaScript including function calls, conditionals with ?:, variable assignments, and so on. All of these are recognised and processed if you use $scope.$eval(), even if they were added by a malicious end-user.
JSON does not support any of those more complex JavaScript features, nor anything else potentially "dangerous". If you use a true JSON parser like JSON.parse() or angular.fromJson(), there is no chance of malicious code being injected and executed.
Since Angular expressions are isolated and evaluate only in the current $scope, the risk of code injection is somewhat mitigated - $scope.$eval() is far less dangerous than JavaScript's native eval() for parsing JSON. However there is still no reason to use either function for this purpose, since there is a potential security risk and using a proper JSON parser is likely to be faster.

How can I insert JSON into a script tage in a zope chameleon template?

I am building a pyramid/python app where my view callable for a certain template passes in a value called data. This variable is an array in the form of [[[x1,y1,z1,],...],[[v1,v2,v3],...]]
in my viewcallable I have
import json
jsdata = json.dumps(data)
I want to put it into a javascript script tag section of my template so:
<script>
data=${jsdata}
</script>
but i'm pretty sure that syntax is incorrect. How can I do this?
Edit: from this: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/templates.html it seems that Genshi style replacements are the way to go, which I take to mean that what I have above is correct. I am still unsure however, about whether this should be treated differently because it is going inside a javascript tag. Is this true?

You want to insert a JavaScript array, not a Python list.
The easiest way to convert between Python and JavaScript formats is to use the json module. JSON is a JavaScript subset for data after all:
import json
jsdata = (json.dumps(data)
.replace(u'<', u'\\u003c')
.replace(u'>', u'\\u003e')
.replace(u'&', u'\\u0026')
.replace(u"'", u'\\u0027'))
then pass jsdata to your template instead. The str.replace() calls ensure that the data remains HTML safe.
In the template, interpolate this without escaping:
<script>
var data = ${structure:jsdata};
</script>

I'm not sure about Chameleon, but "classical" Zope Page Templates did not allow to do anything inside script tags - if you don't see your variables interpolated it is likely that Chameleon behaves the same. The reason for this, as I understand, is to avoid exactly this type of code generation (you're generating JavaScript from Python via the template). Also, ZPT is an XML-based templating language and the content of <script> tags does not have to be a valid XML.
To work around the problem, you could do something like
jsdata = '<script type="text/javascript">var data = ' + json.dumps(data) + ';</script>'
and in your template insert the whole thing:
<tal:myscript replace="structure jsdata" />
Alternatively, you could do something like
<tal:lt replace="string:<" />script>
var data = <tal:script replace="structure jsdata" />;
<tal:lt replace="string:<" />/script>
which would hide the script tags from Chameleon.
It would be a good practice to try to keep the amount of generated JavaScript in your pages as minimal as possible.