Develop Reference

JSON coming from server with quotes and regular expression

I'm analysing the way a server of a search page works (by inspecting element) and I could conclude that the request is sent with a POST with JSON as parameters.
Then I simulated the same POST (with the same parameters) using Insomnia. It was successful, but the response JSON came as string and inside the JSON, variables that uses quotes now uses \" instead.
An example of the JSON response:
"{\"AudienceRefiner\":{\"ItemCount\":0}}"
How can I read this on Python?

It is just an escape character, Consider using the ast library in order to parse it into an object.
literal_eval:
Safely evaluate an expression node or a Unicode or Latin-1 encoded
string containing a Python literal or container display.
import ast
st = "{\"AudienceRefiner\":{\"ItemCount\":0}}"
obj = ast.literal_eval(st)
print (obj)
>>> {'AudienceRefiner': {'ItemCount': 0}}
For more information about it, read ast.literal_eval

I am not sure where you got "{\"AudienceRefiner\":{\"ItemCount\":0}}" I guess that its not is python but webbrowser. Anyhow just use json if the json is in a string object
import json
di = json.loads("{\"AudienceRefiner\":{\"ItemCount\":0}}")

Error Parsing JSON with escaped quotes

I am getting the following json object when I call the URL from Browser which I expect no data in it.
"{\"data\":[], \"SkipToken\":\"\", \"top\":\"\"}"
However, when I tried to call it in javascript it gives me error Parsing Json message
dspservice.callService(URL, "GET", "", function (data) {
var dataList = JSON.parse(data);
)};
This code was working before I have no idea why all of a sudden stopped working and throwing me error.

You say the server is returning the JSON (omitting the enclosing quotes):
{\"data\":[], \"SkipToken\":\"\", \"top\":\"\"}
This is invalid JSON. The quote marks in JSON surrounding strings and property names should not be preceded by a backslash. The backslash in JSON is strictly for inserting double quote marks inside a string. (It can also be used to escape other characters inside strings, but that is not relevant here.)
Correct JSON would be:
{"data":[], "SkipToken":"", "top":""}
If your server returned this, it would parse correctly.
The confusion here, and the reports by other posters that it seems like your string should work, lies in the fact that in a simple-minded test, where I type this string into the console:
var x = "{\"data\":[], \"SkipToken\":\"\", \"top\":\"\"}";
the JavaScript string literal escaping mechanism, which is entirely distinct from the use of escapes in JSON, results in a string with the value
{"data":[], "SkipToken":"", "top":""}
which of course JSON.parse can handle just fine. But Javascript string escaping applies to string literals in source code, not to things coming down from the server.
To fix the server's incorrectly-escaped JSON, you have two possibilities. One is to tell the server guys they don't need to (and must not) put backslashes before quote marks (except for quote marks inside strings). Then everything will work.
The other approach is to undo the escaping yourself before handing it off to JSON.parse. A first cut at this would be a simple regexp such as
data.replace(/\\"/g, '"')
as in
var dataList = JSON.parse(data.replace(/\\"/g, '"')
It might need additional tweaking depending on how the server guys are escaping quotes inside strings; are they sending \"\\"\", or possibly \"\\\"\"?
I cannot explain why this code that was working suddenly stopped working. My best guess is a change on the server side that started escaping the double quotes.

Since there is nothing wrong with the JSON string you gave us, the only other explanation is that the data being passed to your function is something other than what you listed.
To test this hypothesis, run the following code:
dspservice.callService(URL, "GET", "", handler(data));
function handler(data) {
var goodData = "{\"data\":[], \"SkipToken\":\"\", \"top\":\"\"}";
alert(goodData); // display the correct JSON string
var goodDataList = JSON.parse(goodData); // parse good string (should work)
alert(data); // display string in question
var dataList = JSON.parse(data); // try to parse it (should fail)
}
If the goodData JSON string can be parsed with no issues, and data appears to be incorrectly-formatted, then you have the answer to your question.
Place a breakpoint on the first line of the handler function, where goodData is defined. Then step through the code. From what you told me in your comments, it is still crashing during a JSON parse, but I'm willing to wager that it is failing on the second parse and not the first.

Did you mean that your JSON is like this?
"{\"data\":[], \"SkipToken\":\"\", \"top\":\"\"}"
Then data in your callback would be like this:
'"{\"data\":[], \"SkipToken\":\"\", \"top\":\"\"}"'
Because data is the fetched text content string.
You don't have to add extra quotes in your JSON:
{"data":[], "SkipToken":"", "top":""}

Why are endline characters illegal in HTML string sent over ajax?

Within HTML, it is okay to have endline characters. But when I try to send HTML strings that have endline characters over AJAX to have them operated with JavaScript/jQuery, it returns an error that says that endline characters are illegal. For example, if I have a Ruby string:
"<div>Hello</div>"
and jsonify it with Ruby by to_json, and send it over ajax, parse it within JavaScript by JSON.parse, and insert that in jQuery like:
$('body').append('<div>Hello</div>');
then it does not return an error, but if I do a similar thing with a string like
"<div>Hello\n</div>"
it returns an error. Why are they legal in HTML and illegal in AJAX? Are there any other differences between a legal HTML string loaded as a page and legal HTML string sent over ajax?

string literals can contain line breaks, they just need to be escaped with a backslash like so:
var string = "hello\
world!";
However, this does not create a line break in the string, as it must be an explicit \n escape sequence. This would technically become helloworld. Doing
var string = "hello"
+ "world"
would be much cleaner

Specify the type of the ajax call as 'html'. Jquery will try to infer the type when parsing the response.
If the response is json, newlines should be escaped.
I'd recommend using a library to serialize json. You're unlikely to handle all the edge cases if you roll your own.

Strings in JavaScript MUST appear on a single line, with the exception of escaping that line:
var str = "abc \
def";
However note that the newline is escaped and will not appear in the string itself.
The best option is \n, but note that if it is already going through something that parses \n then you will need to double-escape it as \\n.

Seeing how you're already escaping the JSON properly by using to_json in Ruby, I do believe the bug is in jQuery; when there are newlines in the string it has trouble determining whether you meant to create a single element or a document fragment. This would work just fine:
var str = "<div>Hello\n</div>";
var wrapper = document.createElement('div');
wrapper.innerHTML = str;
$('body').append(wrapper);
Demo

Decode JSON string in Mojolicious that was encoded with JSON.stringify

I am trying to send javascript variable as JSON string to Mojolicious and I am having problems with decoding it on perl side. My page uses utf-8 encoding.
The json string (value of $self->param('routes_jsonstr')) seems to have correct value but Mojo::JSON can't decode it. The code is working well when there are no utf-8 characters. What am I doing wrong?
Javascript code:
var routes = [ {
addr1: 'Škofja Loka', // string with utf-8 character
addr2: 'Kranj'
}];
var routes_jsonstr = JSON.stringify(routes);
$.get(url.on_route_change,
{
routes_jsonstr: routes_jsonstr
}
);
Perl code:
sub on_route_change {
my $self = shift;
my $routes=j( $self->param('routes_jsonstr') );
warn $self->param('routes_jsonstr');
warn Dumper $routes;
}
Server output
Wide character in warn at /opt/mojo/routes/script/../lib/Routes/Homepage.pm line 76.
[{"addr1":"Škofja Loka","addr2":"Kranj"}] at /opt/mojo/routes/script/../lib/Routes/Homepage.pm line 76.
$VAR1 = undef;
Last line above shows that decoding of json string didn't work. When there are no utf-8 characters to decode on perl side everything works fine and $routes contain expected data.

Mojolicious style solution can be found here:
http://showmetheco.de/articles/2010/10/how-to-avoid-unicode-pitfalls-in-mojolicious.html
In Javascript I only changed $.get() to $.post().
Updated and working Perl code now looks like this:
use Mojo::ByteStream 'b';
sub on_route_change {
my $self = shift;
my $routes=j( b( $self->param('routes_jsonstr') )->encode('UTF-8') );
}
Tested with many different utf8 strings.

Wide character warnings happen when you print. This is not due to how you decode your unicode but your STDOUT encoding. Try use utf8::all available from CPAN which will set all your IO handles to utf8. Avoiding decoding probably isn't fixing the problem, but rather making it worse. The only reason it appears to work is your terminal is fixing things up for you.

You can take away at least some of the pain by escaping the problematic characters; see https://stackoverflow.com/a/4901205/17389.

javascript ampersand (&) in return data will not show as value

I have this bit of code:
...
var aData = request.responseXML.getElementsByTagName('data')[0];
var sDescription = aData.getElementsByTagName('description')[0].firstChild.data;
alert(escape(sDescription));
document.getElementById('tempLabourLineDescription').value = sDescription;
...
sDescription is outputting: SUPPORT ASSY-FUEL TANK MOUNTING, R&R (LH) (L-ENG)
I think it is obvious what i want to do here (get the sDescription in to a field called tempLabourLineDescription but that just will not work.
However, if i in my php script replace or delete the &-char from that string it all works fine. So i thought, just escape the darn string. But that will just not work.
alerting the string doesn't work either until i remove the &-character.
What is doing this? Is sDescription not a string when it comes out of the xml file?
How can i solve this?

The answer is in this snippet:
var aData = request.responseXML...
You're expecting XML. An & by itself is not legal XML. You need to output your result like this:
SUPPORT ASSY-FUEL TANK MOUNTING, R&R (LH) (L-ENG)

It's very difficult to tell without seeing your output script, but the first thing to try is to mask the ampersand: &
The neater way, though, would be to add CDATA to your XML output:
<data><![CDATA[SUPPORT ASSY-FUEL TANK MOUNTING, R&R (LH) (L-ENG)]]></data>
your XML parser on client side should understand it no problem.

You escape the ampersand by using the HTML eqv. &

If you are unable to alter the XML output from the server (it's not your app or some other issue), a "hack" fix would be:
function htmlizeAmps(s){
return s.replace(/\x26/g,"&"); //globalreplace "&" (hex 26) with "&"
}
document.getElementById('tempLabourLineDescription').value = htmlizeAmps(sDescription);