Develop Reference

Going mad over encoding bugs with JSON and the differences between JS and PHP

I have a real mess at my hands with encoding-related bugs.
I have a DB with latin1 (which is close to Windows-1252 I believe), a user frontend page in Windows-1252, and an AJAX backend in Windows-1252. These can't be changed atm.
Yet, because JSON expects UTF8 data, I'm running into tons of trouble with German Umlaute.
I'm currently retrieving some escaped example data from the DB on the frontend [{"\u00f6\u00e4\u00fc\u00df"}] and using
foreach($example_array_of_objects as $k => &$v) {
foreach($v as $k2 => $v2) {
$v[$k2] = utf8_decode($v2);
}
}
which results in correct display of the data in input form fields on the frontend.
However, this is where I'm stuck. PHP's json_encode escapes Umlaute to these \u sequences, but in Javascript, JSON.stringify just doesn't.
When I JSON.stringify the input field data and send it to the AJAX script, I get only garbage from a print_r response:
öäüß
encodeURIComponent doesn't do the same type of escaping as PHP does. This is infuriating.
How can I transform "öäüß" to \u00f6\u00e4\u00fc\u00df in JS (or how can I synchronize the way data is handled between JS/PHP/mySQL somehow)?

You can not really modify how JSON.stringify works - providing a replacer function as the 2nd argument will force you to manually encode values (unpleasant thing). Your best bet is to use UTF-8 in the frontend (JavaScript code) and convert from/to CP1252 only in your PHP code.
When sending data to the frontend you should use these flags
json_encode($array, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_NUMERIC_CHECK);
which will ensure the cleanest possible UTF-8 output.
To populate your $array you should use mb_convert_encoding($original_data_from_DB, 'UTF-8', 'CP1252') and to get your data after json_decode you should use mb_convert_encoding($data_from_java_script, 'CP1252', 'UTF-8')

Faced this type of issue once but not with PHP and it was solved using encodeURIComponent. If encodeURIComponent is not working for you try using a Base64 encoding decoding on both the sides using atob and btoa.

I managed to do it by letting PHP handle the majority of it now:
JS sends to AJAX:
mydata = JSON.stringify(data);
AJAX Backend:
//decode JS way of JSON with additional slashes etc.
$tmp = json_decode(stripslashes(html_entity_decode($_POST['mydata'])), true);
//re-encode the PHP way
$json = json_encode($tmp);
//save to DB
[...]
User Frontend (Form):
//Retrieval from DB
$mydata = json_decode($db_row['mydata'], true);
//loop through, replace " with " for input fields, decode utf8
foreach($mydata as $k => &$v) {
foreach($v as $k2 => $v2) {
$v[$k2] = utf8_decode(preg_replace('~"~', '"', $v2));
}
}

PHP $_GET and underlines

I have a very short piece of PHP that I use to make HTTP requests from JavaScript.
<?php echo file_get_contents($_GET['url']); ?>
I have used it successfully in a few projects, but am running into a problem with making requests in my current project. Based on my searching, I believe it may be caused by the underscore in the request, though through my searching and not knowing PHP, I have not been able to confirm that.
Below is an example of what I am doing from JavaScript:
$.get("grabber.php?url=" + "http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW", function(forecast) {
console.log(forecast);
});
If I copy the url and put in it in a browser, I get back the JSON that I requested. When I use the code above, I end up getting an error message from NOAA:
Wrong Product : Product cannot be null or empty Wrong Time zone: Time zone cannot be null or empty Wrong Unit:Unit cannot be null or empty Wrong Format: Format cannot be null or empty Wrong Date: The beginDate cannot be null or empty
Do I need to use a regex for the underscore in PHP? Is there some other issue that I do not understand?
Thanks.

You need to send it encoded, which will convert all the underscores/spaces/ampersands etc. with their encoded equivalents:
var url = "http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW";
$.get("grabber.php?url=" + encodeURIComponent(url), function(forecast){
console.log(forecast);
}
Using encodeURIComponent() on that URL shows:
http%3A%2F%2Ftidesandcurrents.noaa.gov%2Fapi%2Fdatagetter%3Fstation%3D8573364%26begin_date%3D20160202%26end_date%3D20160203%26product%3Dpredictions%26units%3Denglish%26time_zone%3Dgmt%26format%3Djson%26application%3Dposeidonweathercom%2B%26datum%3DMLLW
Alternatively, if you just want to access the JSON data and handle it within the JavaScript function, you can retrieve the data via the URL directly, without having to encode the URL:
$.get("http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW", function(forecast) {
console.log(forecast);
});

Um why do you even need your php code ... the code below will work just fine and eliminate your server overhead.
$.get("http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW", function(forecast) {
console.log(forecast);
});

Node-red crash / JSON output sometimes returns html

My node-red is crashing multiple times per day.
One of the problems I think that may be the cause is an http request I am doing.
I am requesting the output of a webpage in JSON.
But sometimes I see in the log that returns an error in HTML.
There is probably something wrong on the webpage serverside, but my flow is erroring out on this.
SyntaxError: undefined:1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> ^ Unexpected token <
2 Jan 15:26:26 - [error] [function:Filter temperatures] SyntaxError: undefined:1
Can I edit my functions to be able to filter this out, and hopefully not crash node-red any more?
This is an example of a function I am using to parse the JSON.
datarequest = JSON.parse(msg.payload);
msg1 = {};
msg1.payload = datarequest.data.valvesetat.bypass;
return [msg1];

It looks like the web page is returning something that is not JSON (some kind of error, I guess) and the JSON.parse() fails to parse it.
Enclose the JSON.parse() part in a try/catch block and manage the response of the webpage.

If there's no { characters in the HTML around the JSON, you could try:
msg.payload = msg.payload.replace(/^.*?(\{/{.*\}).*/, '$1');
before calling JSON.parse.

Google Script: JSON.parse fails for no apparent reason

I have this function in google script that fetches a JSON from the web, but it fails when i try to execute it, citing:
SyntaxError: Unexcpected character in string: '\''
Script:
function getTheFeed(url){
var response = UrlFetchApp.fetch(url);
var json = response.getContentText();
var Jdata = JSON.parse(json);
Jdata = json;
return Jdata;
}
I've tested the URL, by importing it in a string and doing JSON.parse on it, and i get no errors in Google Chrome.
Any other ideas?
UPDATE: After doing Logger.Log turns out the JSON is being cut after 8KB of response. Nothing conflicting at the place the request ends...
Still looking for a response...

Try to use alert(url) and see what you really get.
It could be an escaping issue which sometimes browsers handle properly when you enter it directly in the address bar.
EDIT:
Different browsers have different limits. But generally the limit is around 2,000 characters for the GET method of a URL.
See:
What is the character limit on URL

Sending a JSON POST request with BSF Preprocessor

I am working with JMeter to write some performance tests. One of the things that I need to do is to construct a huge json request dynamically and send it as POST request paylod. Using BSF preprocessor, I am able to modify the payload dynamically however my javascript string is being encoded, while I want to send it without being encoded.
I am not sure how BSF preprocessor can stop it from being encoded. The command I currently use to change my POST request payload is as follows:
var jsonData = '[{"item":"value","something":"everything"}]';
sampler.addArgument("",jsonData);
I would really appreciate if you can point me to some examples which clearly explain how bsf preprocessors are expected to be used.
Any pointers to skip the encoding will also be appreciated.

Since JMeter 2.6 you can use the RAW request pane using Post Body tab.
So your solution is to do the following:
In BSF Sampler, put you JSON in a variable:
var jsonData = '[{"item":"value","something":"everything"}]';
vars.putObject("jsonData",jsonData);
In Post Body, put:
${jsonData}
Another option using your method is to put in BSFPreProcessor using Beanshell language (not javascript):
import org.apache.jmeter.protocol.http.util.HTTPArgument;
String jsonData = "[{\"item\":\"value\",\"something\":\"everything\"}]";
HTTPArgument arg =new HTTPArgument("", jsonData, null, true);
arg.setAlwaysEncoded(false);
sampler.getArguments().addArgument(arg);
Regards
Philippe M.

set property on your sampler "HTTPArgument.always_encode" to false this should disable argument encoding