Develop Reference

API Data Returning Unicode Characters in Console

I am facing a rather confusing problem since the last two days. I am working on a document management system, that uses an API that pulls in data from SOLR. The data is in tune of around ~15Mbs, and pulls records of more than 4000+ documents. The API has response in this format -
{
"documents": [
{
id: 123,
some_field: "abcd",
some_other_field: "abcdef"
},
{
id: 124,
some_field: "abcd1",
some_other_field: "abcdef1"
}
]
}
Everything works fine in browser. If I hit the endpoint in Chrome or Firefox browser, it gives me the correct output and I am able to see the JSON output.
However, if I try hitting the same API endpoint with a Java or JS code - the response code is 200, but the output in console (Terminal or Eclipse) shows unicode characters like \u0089 \u0078 U+0080 - all the output comes in this way, and since there are around 4000+ records being fetched by the API, the console kinda fills with all of these unicode characters.
The only difference that I see between the requests made from browser and the code is that in browser I can see Content-Encoding : gzip, while I cannot find this header from the code that I written . For eg - in JS code, through Chakram framework, I can check
expect(response).to.be.encoded.with.gzip
mentioned here. However, this returns a failure stating expected undefined to match gzip
What am I missing here? Is this something related to encoding/decoding or something entirely different?
Edit 1 : The Response Headers as seen in Network tab of Chrome :
cache-control: max-age=0, private, must-revalidate, max-age=315360000
content-encoding: gzip
content-type: application/json; charset=utf-8
date: Tue, 22 May 2018 06:07:26 GMT
etag: "a07eb7c1eef4ab97699afc8d61fb9c5d"
expires: Fri, 19 May 2028 06:07:26 GMT
p3p: CP="NON CUR OTPi OUR NOR UNI"
server: Apache
Set-Cookie : some_cookie
status: 200 OK
strict-transport-security:
transfer-encoding: chunked
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-request-id: abceefr4-1234-acds-100b-d2bef2413r47
x-runtime: 3.213943
x-ua-compatible: chrome=1
x-xss-protection: 1; mode=block
The Request Headers as seen in Network tab of Chrome
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: some_cookie
Host: abcd.bcd.com
IV_USER: demouser123
IV_USER_L: demouser123
MAIL: demouser#f.com
PERSON_ID: 123
Referer: http://abcd.bcd.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
X-CSRF-TOKEN: some_csrf_token
Edit 2 : The tests that I am using
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
console.log(JSON.stringify(t_resp));
expect(t_resp).to.have.header('Content-Encoding','gzip');
});
});

This has nothing to do with encoding. The web server in general compresses to gzip to save the bandwidth since its redundant to transfer the whole 15MB file as is refer this article for more about gZip and the its working ( https://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/ ). So where does it went wrong and how it worked in chrome is pretty simple chrome has an inbuilt unicode parser(even an HTML parser) in its devTools which can show you the parsed content rather showing you the wiered text (same can be seen in response tab next to preview tab). why you see wierd text is that you are stingfying the response which will escape special character if any console.log(JSON.stringify(t_resp));. You cannot use something like console.log("response", t_resp); without stringifying in terminal since the terminal doesn't have a JSON or an unicode parser it just prints in text. try removing that console since stringifying a 15mb file is a costly process.
Edit 1:-
if you still want to output in the console here whats to be done.
Since NODE cannot decode gzip by default directly (not with chakram, its just a APItesting platform) you can use zlib to do this. Please find the example snippet
const zlib = require('zlib');
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
zlib.gunzip(t_resp, function(err, dezipped) {
console.log(dezipped);
});
});
});

Try with console.dir to display your values
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
console.dir(t_resp, { depth: null });
});
});
Console.dir

javascript utf-8 encoded text being interpreted as unicode

I'm sending utf-8 encoded text to a wso2 service bus. The service bus interprets this as unicode. In the header the encoding is specified and the character type is also set as utf-8.
When sending the µ, it gets send as C2 B5. If I check this on the utf-8 codepage its the µ sign. But the bus thinks its C2 = Â and B5 = µ.
Does anyone know how to prevent or fix this?
EDIT:
The json my client send to the WSO32
Accept: application/json, text/plain, /
Content-Type: application/json;charset=utf-8
Accept-Language: nl-BE
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; MSAppHost/1.0)
Host: app1.o.esb.local
Content-Length: 156
Connection: Keep-Alive
Pragma: no-cache
{"meldingen":{"MeldingDto":[{"Id":-1,"Code":"","MeldingTijd":"2014-06-20T15:34:52.680+02:00","Melder":"ABW_PN","Commentaar":"testµ","InterventieId":381}]}}
Here in the hexview in fiddler the µ sign is C2 B5.
The WSO2 service sees this as 2 characters instead of one.
When I use soapUI and send the same character gets send as, B5 and that gets interpreted correctly. I would expect that the WSO2 would know how to handle UTF-8.
Should I edit my encoding to send B5 instead of C2 B5 or does the WSO2 esb need an aditional setting so it knows how to interpret the UTF8 encoding correctly?
Thanks in advance
Ian

My collegues and I figured out how to fix this. On the client we encoded the message in the following manner:
encodeURIComponent(comment);
In the case of the µ sign we got %C2%B5.
Then on the server side WSO2 we used a javascript function (local entry):
function decode(mc)
{
var Input = mc.getProperty('InputdecodeURIComponent').toString();
var Output = decodeURIComponent(Input);
mc.setProperty('OutputdecodeURIComponent', Output);
}
<property name="InputdecodeURIComponent" value=”ZET HIER ENCODED VALUE” />
<script language="js" key="HelperFunctions.js" function="decodeXml"/>
<property name="Output" expression="get-property('OutputdecodeURIComponent')"/>
I hope this helps.
Regards
Ian

Only GET working in cross domain API request with django-piston

I'm not able to do POST/PUT/DELETE cross-domain request on my API using django-piston, I've CORS enabled using this script (based on this):
class CORSResource(Resource):
"""
Piston Resource to enable CORS.
"""
# headers sent in all responses
cors_headers = [
('Access-Control-Allow-Origin', '*'),
('Access-Control-Allow-Headers', 'AUTHORIZATION'),
]
# headers sent in pre-flight responses
preflight_headers = cors_headers + [
('Access-Control-Allow-Methods', '*'),
('Access-Control-Allow-Credentials','true')
]
def __init__(self, handler, authentication=None):
super(CORSResource, self).__init__(handler, authentication)
self.csrf_exempt = getattr(self.handler, 'csrf_exempt', True)
def __call__(self, request, *args, **kwargs):
request_method = request.method.upper()
# intercept OPTIONS method requests
if request_method == "OPTIONS":
# preflight requests don't need a body, just headers
resp = HttpResponse()
# add headers to the empty response
for hk, hv in self.preflight_headers:
resp[hk] = hv
else:
# otherwise, behave as if we called the base Resource
resp = super(CORSResource, self).__call__(request, *args, **kwargs)
# slip in the headers after we get the response
# from the handler
for hk, hv in self.cors_headers:
resp[hk] = hv
return resp
#property
def __name__(self):
return self.__class__.__name__
In the frontend I'm using Backbone with JSONP activated. I don't have any errors, the OPTIONS request works fine then nothing happens. I tried to change the « Access-Control-Allow-Methods » but it doesn't change anything. Any idea ?
Edit:
Here is the request headers of an OPTIONS request:
OPTIONS /api/comments/ HTTP/1.1
Host: apitest.dev:8000
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Origin: http://3l-oauth.dev:1338
Access-Control-Request-Method: POST
Access-Control-Request-Headers: authorization,content-type
Pragma: no-cache
Cache-Control: no-cache
and the response headers:
HTTP/1.0 200 OK
Date: Sat, 12 May 2012 09:22:56 GMT
Server: WSGIServer/0.1 Python/2.7.3
Access-Control-Allow-Methods: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: AUTHORIZATION
Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN

JSONP is GET only:
You cannot make POST, PUT, or DELETE requests cross-domain. Cross domain JavaScript is facilitated through the use of <script> tags that send requests to your server for dynamic JavaScript. script tags are GET requests only.
However, one of the recommended methods for adjusting to this limitation when dealing with cross-domain JavaScript is to use a method query parameter that you would use in your server-side code to determine how you should handle a specific request.
For instance, if the request was
POST /api/comments/
then you could do this:
/api/comments?method=POST
Under the hood, it's still a GET request, but you can achieve your goal with slight modifications to your API.
When determining the type of request, instead of checking the HTTP Method:
if request_method == "OPTIONS":
Check the request parameter "method" instead:
if request.GET["method"] == "OPTIONS":
JSONP Requests Must return JavaScript:
The other really important point to take note of is that your JSONP requests must all return JavaScript wrapped (or padded) in a function call. Since the requests are made under the hood by script tags that expect JavaScript, your server must return content that the browser understands.
If this doesn't make sense to you or you need more information, there is a great explanation here on how JSONP and script tag remoting works under the hood.

image upload to web service in JavaScript

I need to upload an image to a webservice from javascript. I have to send a json string an a file(image). In java we have MultipartEntity. I have the followig code in java:
HttpPost post = new HttpPost( aWebImageUrl2 );
MultipartEntity entity = new MultipartEntity( HttpMultipartMode.BROWSER_COMPATIBLE );
// For File parameters
entity.addPart( "picture", new FileBody((( File ) imgPath )));
// For usual String parameters
entity.addPart( "url", new StringBody( aImgCaption, "text/plain", Charset.forName( "UTF-8" )));
post.setEntity( entity );
Now I need to do the same image upload in javascript.
But in javaScript I didn't find any equivalent of MultipartEntity. Please Suggest any solutions.

For uploading images I use either Valum's ajax upload plugin or jQuery form plugin that allows to submit a normal form in an ajax way.
If you will use POST requests then don't forget to use MAX_FILE_SIZE hidden attribute:
<input type="hidden" name="MAX_FILE_SIZE" value="20000000">
Note that it must precede the file input field. It is in bytes, so this will limit the upload to 20MB. See PHP documentation for details.

Assuming that your Java code is using Apache HttpComponents (what you really should have said then), your code, when augmented with
URI aWebImageUrl2 = new URI("http://localhost:1337/");
File imgPath = new File("…/face.png");
final String aImgCaption = "face";
// …
HttpClient httpClient = new DefaultHttpClient();
httpClient.execute(post);
submits the following example HTTP request (as tested with nc -lp 1337, see GNU Netcat):
POST / HTTP/1.1
Content-Length: 990
Content-Type: multipart/form-data; boundary=oQ-4zTK_UL007ymPgBL2VYESjvFwy4cN8C-F
Host: localhost:1337
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.2 (java 1.5)
--oQ-4zTK_UL007ymPgBL2VYESjvFwy4cN8C-F
Content-Disposition: form-data; name="picture"; filename="face.png"
Content-Type: application/octet-stream
�PNG[…]
The simplest solution to do something like this in HTML is, of course, to use a FORM element and no or minimal client-side scripting:
<form action="http://service.example/" method="POST"
enctype="multipart/form-data">
<input type="file" name="picture">
<input type="submit">
</form>
which submits (either when submitted with the submit button or the form object's submit() method) the following example request:
POST / HTTP/1.1
Host: localhost:1337
Connection: keep-alive
Content-Length: 886
Cache-Control: max-age=0
Origin: http://localhost
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryhC26St5JdG0WUaCi
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: http://localhost/scripts/test/XMLHTTP/file.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: de-CH,de;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
------WebKitFormBoundaryhC26St5JdG0WUaCi
Content-Disposition: form-data; name="picture"; filename="face.png"
Content-Type: image/png
�PNG[…]
But since you have asked explicitly about a "javascript" solution (there really is no such programming language), I presume that you want to have more client-side control over the submit process. In that case, you can use the W3C File API and XMLHttpRequest or XMLHttpRequest2 APIs as provided by recent browsers (not the programming languages):
<script type="text/javascript">
function isHostMethod(obj, property)
{
if (!obj)
{
return false;
}
var t = typeof obj[property];
return (/\bunknown\b/i.test(t) || /\b(object|function)\b/i.test(t) && obj[property]);
}
var global = this;
function handleSubmit(f)
{
if (isHostMethod(global, "XMLHttpRequest"))
{
try
{
var input = f.elements["myfile"];
var file = input.files[0];
var x = new XMLHttpRequest();
x.open("POST", f.action, false); // ¹
try
{
var formData = new FormData();
formData.append("picture", file);
x.send(formData);
return false;
}
catch (eFormData)
{
try
{
var reader = new FileReader();
reader.onload = function (evt) {
var boundary = "o" + Math.random();
x.setRequestHeader(
"Content-Type", "multipart/form-data; boundary=" + boundary);
x.send(
"--" + boundary + "\r\n"
+ 'Content-Disposition: form-data; name="picture"; filename="' + file.name + '"\r\n'
+ 'Content-Type: application/octet-stream\r\n\r\n'
+ evt.target.result
+ '\r\n--' + boundary + '--\r\n');
};
reader.readAsBinaryString(file);
return false;
}
catch (eFileReader)
{
}
}
}
catch (eFileOrXHR)
{
}
}
return true;
}
</script>
<form action="http://service.example/" method="POST"
enctype="multipart/form-data"
onsubmit="return handleSubmit(this)">
<input type="file" name="myfile">
<input type="submit">
</form>
This approach tries to use the XMLHttpRequest API. If that fails, the function returns true, so true is returned to the event handler (see the attribute value), and the form is submitted the usual way (the latter might not work with your Web service; test before use by disabling script support).
If XMLHttpRequest can be used, it is "tested"² if the file input has a files property and the object referred to by that has a 0 property (referring to the first selected File for that form control, if supported).
If yes, the XMLHttpRequest2 API is tried, which send() method can take a reference to a FormData and do all the multi-part magic by itself. If the XMLHttpRequest2 API is not supported (which should throw an exception), the File API's FileReader is tried, which can read the contents of a File as binary string (readAsBinaryString()); if that is successful (onload), the request is prepared and submitted. If one of those approaches seemingly worked, the form is not submitted (return false).
Example request submitted with this code using the FormData API:
POST / HTTP/1.1
Host: localhost:1337
Connection: keep-alive
Content-Length: 887
Origin: http://localhost
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryLIXsjWnCpVbD8FVA
Accept: */*
Referer: http://localhost/scripts/test/XMLHTTP/file.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: de-CH,de;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
------WebKitFormBoundaryLIXsjWnCpVbD8FVA
Content-Disposition: form-data; name="picture"; filename="face.png"
Content-Type: image/png
�PNG[…]
The example request looks slightly different when the FileReader API was used instead (just as proof of concept):
POST / HTTP/1.1
Host: localhost:1337
Connection: keep-alive
Content-Length: 1146
Origin: http://localhost
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Content-Type: multipart/form-data; boundary=o0.9578036249149591
Accept: */*
Referer: http://localhost/scripts/test/XMLHTTP/file.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: de-CH,de;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
--o0.9578036249149591
Content-Disposition: form-data; name="picture"; filename="face.png"
Content-Type: application/octet-stream
PNG[…]
Notice that the XMLHttpRequest2, FormData and File API are having only Working Draft status and so are still in flux. Also, this approach works if the resource submitted from and the resource submitted to are using the same protocol, domain, and port number; you may have to deal with and work around the Same Origin Policy. Add feature tests and more exception handling as necessary.
Also notice that the request made using FileReader is larger with the same file and misses the leading character, as indicated in the question referred to by Frits van Campen. This may be due to a (WebKit) bug, and you may want to remove this alternative then; suffice it for me to say that the readAsBinaryString() method is deprecated already in the File API Working Draft in favor of readAsArrayBuffer() which should use Typed Arrays.
See also "Using files from web applications".
¹ Use true for asynchronous handling; this avoids UI blocking, but requires you to do processing in the event listener, and you will always have to cancel form submission (even if XHR was unsuccessful).
² If the property access is not possible, an exception will be thrown. If you prefer a real test, implement (additional) feature-testing (instead), and be aware that not everything can be safely feature-tested.

you can actually invoke a service using javascript, there is a sample code for this here
if your requirement is to upload the image and make the webservice call from JS then it could be tricky.
you can simply upload the image to a server and have the server call the webservice, there are loads of tools which helps you to upload a file to a server.

MultipartEntity sounds like Multipart/form-data.
You can use a regular XMLHttpRequest to make a POST request. You can use the HTML 5 FormData to build your Multipart/form-data request.
Here is an example: HTML5 File API readAsBinaryString reads files as much larger, different than files on disk

I've done this before and it works, using HTML5's canvas element. I'll be using jQuery here. I'm assuming a generic image of 300px by 300px.
First, add a hidden canvas to your page :
$("body").append('<canvas id="theCanvas" style="display:none" width="300px" height="300px"></canvas>');
Then, load the image to the canvas :
var canvas = document.getElementById('theCanvas');
var context = canvas.getContext('2d');
var imageObj = new Image();
imageObj.src = "/path/to/image.jpg";
context.drawImage(imageObj, 0, 0, 300, 300);
Now, you can access what's on the canvas as a data string and post it to the webservice using jQuery's post function :
$.post("path/to/service", {'image':canvas.toDataURL("image/png"), 'url':'caption'}, function(file){
//Callback code
});

Chrome extension OATH 401 error(Unauthorized)

I am using JS to access the rdio plugin. I am using the following for Oauth http://code.google.com/chrome/extensions/tut_oauth.html.
I am able to get the signed token etc. However, when ever I try to send a signedRequest at http://api.rdio.com/1/, I receive 401, un-authorized error.
X-Mashery-Error-Code:ERR_401_INVALID_SIGNATURE
X-Mashery-Responder:mashery-web4.LAX
This is what I am trying to send:
var url = 'http://api.rdio.com/1/';
var request = {
'method': 'POST',
'headers': {
'Content-Type': 'application/x-www-form-urlencoded'
},
'parameters': {
'alt': 'json',
'method':'currentUser'
},
'body': 'Data to send'
};
bgPage.oauth.sendSignedRequest(url, mycallback, request);
I receive the following error in console.
Request URL:http://api.rdio.com/1/?alt=json&method=currentUser&oauth_consumer_key=yv8ehzehdv55**********&oauth_nonce=******&oauth_signature=**********&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1305190893&oauth_token=us6myp99p4qc86umea9p8fp*****************
Request Method:POST
Status Code:401 Unauthorized
Request Headers
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:12
Content-Type:application/x-www-form-urlencoded
Cookie:__qca=P0-158278476-1296771701175; r=eyJfdSI6IDE5MjY1LCAiX2UiOiAzMTU1NjkyNn0.SvN8xd7rIuLzTp7hxqi4eJEdvu8; __utmz=225830489.1305153361.198.18.utmcsr=rdioquiz.ianloic.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utma=225830489.281668250.1296676147.1305184513.1305187119.201; __utmc=225830489
Host:api.rdio.com
Origin:chrome-extension://oiojbkkpmcgmpnjkhjmaggajckamjkap
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24
Query String Parameters
alt:json
method:currentUser
oauth_consumer_key:yv8ehzehdv55pbb74ss9dt23
oauth_nonce:BQF0x
oauth_signature:KttF************tRO 8PL yjPF2Ktk=
oauth_signature_method:HMAC-SHA1
oauth_timestamp:1305190893
oauth_token:us6myp99p4qc86umea9p8fphbgq4dxdd76txvyn***********
Form Data
Data to send:
Response Headers
Accept-Ranges:bytes
Content-Length:30
Content-Type:text/xml
Date:Thu, 12 May 2011 09:01:33 GMT
Server:Mashery Proxy
X-Mashery-Error-Code:ERR_401_INVALID_SIGNATURE
X-Mashery-Responder:mashery-web4.LAX
*I am just trying to mimic what's mentioned here. Its an Oauth library(http://code.google.com/chrome/extensions/tut_oauth.html) from Google to make Chrome extension development easy.
They have an Oauth sample code to get your document list etc. http://code.google.com/chrome/extensions/samples.html#4e35caa9742fb82dbd628892d23a781614f6eff6
I think I am not able to get past send a POST requestto the rdio API. It gives an un-authorized error.*

We found a similar issue with the same service (rdio) and method ("currentUser").
What ended up working was:
(1) make sure you have method=currentUser in the POST body; I'm not sure from the above curl output if that is the case.
And, this is the bit that actually fixed the issue:
(2) we had to also add the method name to the signature itself.
FYI we used this library: https://oauth.googlecode.com/svn/code/javascript/
But the tricky part, as you are seeing, was figuring out how to seed the method in that library that creates the signature. Without the 'method=currentUser' being part of the signature, we experienced the same error condition.

Check your timezone, date, and time on your computer. If any one of these is wrong, OAuth will fail.