Develop Reference

JSON unicode characters conversion

I came across this strange JSON which I can't seem to decode.
To simplify things, let's say it's a JSON string:
"\uffffffe2\uffffff94\uffffff94\uffffffe2\uffffff94\uffffff80\uffffffe2\uffffff94\uffffff80 mystring"
After decoding it should look as following:
└── mystring
JS or PHP doesn't seem to convert it correctly.
js> JSON.parse('"\uffffffe2\uffffff94\uffffff94\uffffffe2\uffffff94\uffffff80\uffffffe2\uffffff94\uffffff80 mystring"')
ffe2ff94ff94ffe2ff94ff80ffe2ff94ff80 mystring
PHP behaves the same
php> json_decode('"\uffffffe2\uffffff94\uffffff94\uffffffe2\uffffff94\uffffff80\uffffffe2\uffffff94\uffffff80 mystring"')
ffe2ff94ff94ffe2ff94ff80ffe2ff94ff80 mystring
Any ideas how to properly parse this JSON string would be welcome.

It is not valid JSON string - JSON supports only 4 hex digits after \u. Results from both PHP and JS are correct.
It is not possible decode this using standard functions.
Where did you get this JSON string?
About correct json for string you want to get - it should be "\u2514\u2500\u2500 mystring", or just "└── mystring" (json supports any unicode characters in strings except " and \).
Also if you need to encode some character that require more than two bytes - it will result in two escape codes for example "𩄎" would be "\ud864\udd0e" when escaped.
So, If you really need to decode string above - you can fix it before decoding, replacing \uffffffe2 by \uffff\uffe2 via regexp (for js it would be something like: s.replace(/(\\u[A-Fa-f0-9]{4})([A-Fa-f0-9]{4})/gi,'$1\\u$2') ).
But anyway character codes in string specified above does not look right.

Handling backslash when parsing json

Here is a portion of a larger JSON string that I attempting to call JSON.parse on. I am getting the infamous 'invalid character' error because (I believe) of the backslash parentheses (ie. "path:\"https://mysite.sharepoint.com/sites/Test\").
All online parsers I have tried it on works fine but in my javascript code the JSON.parse method fails.
I have attempted to clean the string like this and other ways but I have been unable to get it to parse.
var cleanData = data.replace(/\\"/, /\\\\/);
below is the partial JSON file. When I remove it from the JSON string the JSON.parse works so I think I have this isolated to just this. What type of general purpose clean method would work to get this thing to parse? Thansk
'{"Properties" : {
"GenerationId" : 9223372036854776000,
"indexSystem" : "",
"ExecutionTimeMs" : 109,
"QueryModification" : "path:\"https://mysite.sharepoint.com/sites/Test\" (IsDocument:\"True\" OR contentclass:\"STS_ListItem\") ContentTypeId:0x0120D5200098CBB075E51C8C4398ECCB4B4928912D*",
"RenderTemplateId" : "~sitecollection/_catalogs/masterpage/Display Templates/Search/Group_Default.js",
"StartRecord" : 0,
"piPageImpressionBlockType" : 2
}}
how?

The problem is that your backslash is getting swallowed as an escape character in the string:
'\"' === '"' // true
You actually need to escape the backslashes, so that the JSON parser sees them. Here's another example:
var unencoded = 'string with "quotes"';
'"string with \"quotes\""' === JSON.stringify(unencoded); // false
'"string with \\"quotes\\""' === JSON.stringify(unencoded); // true
However, where the escaping should be done depends on how the JSON is being made available to the JavaScript. If the JSON is embedded in the page by a server-side script, then there's no need to use JSON.parse, as valid JSON is valid JavaScript:
// if JsonData is valid JSON, it's also a valid JavaScript object
var data = <%= JsonData %>;

Javascript - strip out occurrences of u' in JSON string, parse is returning unexpected token

I have a textfield in a database that contains the results of a python json.dumps(list_instance) operation. As such, the internal fields have a u' prefix, and break the browser's JSON.parse() function.
An example of the JSON string is
"density": "{u'Penobscot': 40.75222856500098, u'Sagadahoc':
122.27083333333333, u'Lincoln': 67.97977755308392, u'Kennebec':
123.12237174095878, u'Waldo': 48.02117802779616, u'Cumberland':
288.9285325791363, u'Piscataquis': 3.9373586457405247, u'Hancock':
30.698239582715903, u'Washington': 12.368718341168325, u'Aroostook':
10.827378163074039, u'York': 183.47612497543722, u'Franklin':
16.89330963710371, u'Oxford': 25.171240748402518, u'Somerset':
12.425648288323485, u'Knox': 108.48302300109529, u'Androscoggin':
208.75502815768303}"
What I'd like to do is replace those occurrences of u' with a '(single-quote). I've tried
function renderValues(data){
var pop = JSON.parse(data.density.replace(/u'/g, "'"));
}
but I'm always getting a unexpected token ' exception. Since many of the possible key fields may contain a u, it is not feasable to just delete that character. How can I find all instances of u' and replace with ' without getting the exception?

The accepted solution is wrong. Your code fails because that string is not valid JSON. Fixing the pseudo-JSON string by replacing it is wrong.
What you have to do is fix the python code that is producing that broken JSON string, which I am pretty sure it is a str() or unicode() where there should be nothing. What you have as a value for the key "density" is a string instead of a dictionary, and therefore, python returns you something like the following:
{"density": "a string that looks like JSON but it is in fact a string reprensentation of a dictionary"}
The function json.dumps will return you always valid JSON strings.
Fix that and you will not have to hack around with filthy string replacements or whatever.
EDIT
Check the following snippet out. There you can see that the u'...' is just the python readable-representation of a unicode object, and has nothing to do whatsoever with a JSON serialization.
>>> import json
>>> data = {u'name': u'Manuel', u'age': 26}
>>> print data
{u'age': 26, u'name': u'Manuel'} # this is the python representation of a dictionary
>>> print json.dumps(data)
{"age": 26, "name": "Manuel"} # this is a valid JSON string
That JSON is not properly formed, as easy as that.

Updated solution: replace(/u'/g, "'")); => replace(/u'(?=[^:]+')/g, "'"));.
Tested with the following:
"{u'Penobscot': 40.75222856500098, u'Sagadahoc': 122.27083333333333, u'Lincoln': 67.97977755308392, u'Kennebec': 123.12237174095878, u'Waldo': 48.02117802779616, u'Cumberland': 288.9285325791363, u'Piscataquis': 3.9373586457405247, u'Hancock': 30.698239582715903, u'Timbuktu': 12.368718341168325, u'Aroostook': 10.827378163074039, u'York': 183.47612497543722, u'Franklin': 16.89330963710371, u'Oxford': 25.171240748402518, u'Somerset': 12.425648288323485, u'Knox': 108.48302300109529, u'Androscoggin': 208.75502815768303}".replace(/u'(?=[^:]+')/g, "'");
results in:
"{'Penobscot': 40.75222856500098, 'Sagadahoc': 122.27083333333333, 'Lincoln': 67.97977755308392, 'Kennebec': 123.12237174095878, 'Waldo': 48.02117802779616, 'Cumberland': 288.9285325791363, 'Piscataquis': 3.9373586457405247, 'Hancock': 30.698239582715903, 'Timbuktu': 12.368718341168325, 'Aroostook': 10.827378163074039, 'York': 183.47612497543722, 'Franklin': 16.89330963710371, 'Oxford': 25.171240748402518, 'Somerset': 12.425648288323485, 'Knox': 108.48302300109529, 'Androscoggin': 208.75502815768303}"

a little bit old in the answer but if there is no way to change or access the server response try with:
var strExample = {'att1':u'something with u'};
strExample.replace(/u'[\}|\,]/g, "ç'").replace(/u'/g, "'").replace(/ç'/g, "u'");
//{'att1':'something with u'};
The first replace will handle the u' that are in the trailing part of the string in the object changing it to 'ç' character
then removing the u from the phyton unicode and finally change it to u' like the original

I had a similar issue and made this regex that found all of the u's even if the values had them too.
replace(/(?!\s|:)((u)(?='))/g, "")
The accepted answer, I found, missed these occurrences.
I know the OP's doesn't have 'u' for the values and only for keys but thought this may be useful too :)

read a json string which has a "'" (malformed?)

I need to consume a service which is returning JSON. I have no influence whatsoever on that service (third party).
If I do
JSON.parse (data)
I get an
SyntaxError: Unexpected token
I know the service works because the error occurs depending on input parameters.
In other words, sometimes it definitely works! The HTTP response code is 200 so it is not some kind of access error, and it is repeatable.
Can I assume they are returning malformed JSON?
Writing the data as a text file to disk and reading it like this:
fs = require('fs')
fs.readFile 'output.json', '', (err, data) ->
if err?
console.log err
json = JSON.parse(data)
console.log json
returns
undefined:1
De L\'Embustier
^
SyntaxError: Unexpected token '
at Object.parse (native)
which is kind of strange because it seems as if the string is correctly escaped but it's nevertheless not being read correctly.
The file is 300+k; haven't seen how to attach it.
EDIT:
Response from jsonlint.com
Parse error on line 1297:
... "Address": "36 Rue De L\'Embust
-----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['
EDIT2:
Here is the whole file then:
http://pastebin.com/ACUfvPCx

The service is indeed returning malformed JSON.
There is no \' escape sequence in JSON. It's perfectly valid to escape any character in Javascript, but JSON uses a subset of the Javascript syntax and only allows escaping characters that actually could need escaping. As an apostrophe is not used as a string delimiter in JSON, it never needs escaping.
If you can't get the service fixed, you would need to request the JSON as plain text, replace the escaped apostrophes with just the apostrophe itself, then parse the text as JSON.

Is there a difference between single/double quoted strings passed to javascript's eval?

i have a server message sent via web-sockets. that message is a json (validated) string.
when it gets to the browser i check that it is a string with typeof(data) and it tells me that it is, in fact, a string. When finally i do var some_obj = eval( '(' + data + ')' );
it gives me an Uncaught SyntaxError: Unexpected token ILLEGAL error.
also, before using eval(), i console.log(data) and it displays correctly, although an alert(data) won't show anything on the dialog.
i can't understand what's happening.
i also tried var myJson = '{ "x": "Hello, World!", "y": [1, 2, 3] }'; and then var myObj = eval( '(' + myJson + ')' ); and it works, so i really can't understand why mine can't be evaluated (parsed).
the string received via web-sockets is this:
received 37 bytes » { "cmd": "setname", "params": "ok" }
where data = { "cmd": "setname", "params": "ok" } (with quotes i suppose, because of typeof(data) being = string).
any tips? thanks
edit1 » with web-sockets, you have to prepend a null char (0 ascii) and append a escape char (255 ascii) to the output string from the server. i assume the client (browser) as it implements web-sockets must deal with this and unwrap the string correctly (as the standard) and as i do in my server. thing is, there might be some escape char left and it doesn't deal with it correctly. but the problem only started when i tried to send json strings to be eval()ed. otherwise they work properly as any other string.

No, there's no difference between " and ' for quoting strings other than that you can use " without escaping it inside a string quoted with ' and vice-versa. But I don't think that (the title of your question) actually has anything to do with the problem you're having.
Re your edit, if you want to ensure that there are no characters with the value 0 or 255 in the string, you can do that like this:
data = data.replace(/[\u0000\u00ff]/g, '');
...before passing it to eval. And it sounds like you might want to do that, since your thing is saying it's received 37 bytes but the string is only 36 characters long and doesn't use any characters requiring two bytes (or perhaps it just has a space at the end I can't see).
Off-topic: It's best not to use eval to deserialize JSON. Instead, use a library that handles it directly. Crockford has two different non-eval libs on his github page, one (json_parse.js) that uses a recursive-descent parser and another (json_parse_state.js) that uses a state machine. If you really, really want to use eval to parse JSON, take a look at his implementation in json2.js, which at least takes a couple of steps to weed out malicious stuff.
Off-topic 2: Re
where data = { "cmd": "setname", "params": "ok" } (with quotes i suppose, because of typeof(data) being = string).
We only use quotes to quote string literals in code; there are no quotes around actual string data itself in memory. If I do this:
var foo = "bar";
...the string that foo points to consists entirely of the characters b, a, and r. There are no quotes; the quotes are only there in the code to tell the parser that what follows is a string literal.