Develop Reference

API Data Returning Unicode Characters in Console

I am facing a rather confusing problem since the last two days. I am working on a document management system, that uses an API that pulls in data from SOLR. The data is in tune of around ~15Mbs, and pulls records of more than 4000+ documents. The API has response in this format -
{
"documents": [
{
id: 123,
some_field: "abcd",
some_other_field: "abcdef"
},
{
id: 124,
some_field: "abcd1",
some_other_field: "abcdef1"
}
]
}
Everything works fine in browser. If I hit the endpoint in Chrome or Firefox browser, it gives me the correct output and I am able to see the JSON output.
However, if I try hitting the same API endpoint with a Java or JS code - the response code is 200, but the output in console (Terminal or Eclipse) shows unicode characters like \u0089 \u0078 U+0080 - all the output comes in this way, and since there are around 4000+ records being fetched by the API, the console kinda fills with all of these unicode characters.
The only difference that I see between the requests made from browser and the code is that in browser I can see Content-Encoding : gzip, while I cannot find this header from the code that I written . For eg - in JS code, through Chakram framework, I can check
expect(response).to.be.encoded.with.gzip
mentioned here. However, this returns a failure stating expected undefined to match gzip
What am I missing here? Is this something related to encoding/decoding or something entirely different?
Edit 1 : The Response Headers as seen in Network tab of Chrome :
cache-control: max-age=0, private, must-revalidate, max-age=315360000
content-encoding: gzip
content-type: application/json; charset=utf-8
date: Tue, 22 May 2018 06:07:26 GMT
etag: "a07eb7c1eef4ab97699afc8d61fb9c5d"
expires: Fri, 19 May 2028 06:07:26 GMT
p3p: CP="NON CUR OTPi OUR NOR UNI"
server: Apache
Set-Cookie : some_cookie
status: 200 OK
strict-transport-security:
transfer-encoding: chunked
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-request-id: abceefr4-1234-acds-100b-d2bef2413r47
x-runtime: 3.213943
x-ua-compatible: chrome=1
x-xss-protection: 1; mode=block
The Request Headers as seen in Network tab of Chrome
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: some_cookie
Host: abcd.bcd.com
IV_USER: demouser123
IV_USER_L: demouser123
MAIL: demouser#f.com
PERSON_ID: 123
Referer: http://abcd.bcd.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
X-CSRF-TOKEN: some_csrf_token
Edit 2 : The tests that I am using
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
console.log(JSON.stringify(t_resp));
expect(t_resp).to.have.header('Content-Encoding','gzip');
});
});

This has nothing to do with encoding. The web server in general compresses to gzip to save the bandwidth since its redundant to transfer the whole 15MB file as is refer this article for more about gZip and the its working ( https://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/ ). So where does it went wrong and how it worked in chrome is pretty simple chrome has an inbuilt unicode parser(even an HTML parser) in its devTools which can show you the parsed content rather showing you the wiered text (same can be seen in response tab next to preview tab). why you see wierd text is that you are stingfying the response which will escape special character if any console.log(JSON.stringify(t_resp));. You cannot use something like console.log("response", t_resp); without stringifying in terminal since the terminal doesn't have a JSON or an unicode parser it just prints in text. try removing that console since stringifying a 15mb file is a costly process.
Edit 1:-
if you still want to output in the console here whats to be done.
Since NODE cannot decode gzip by default directly (not with chakram, its just a APItesting platform) you can use zlib to do this. Please find the example snippet
const zlib = require('zlib');
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
zlib.gunzip(t_resp, function(err, dezipped) {
console.log(dezipped);
});
});
});

Try with console.dir to display your values
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
console.dir(t_resp, { depth: null });
});
});
Console.dir

Firefox is parsing empty response payload

We're making an XHR request with the following headers (I've simplified a bit):
POST http://localhost:9001/login
Host: localhost:9001
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: application/json, text/plain, */*
Content-Type: application/json;charset=utf-8
Content-Length: 67
Then our server responds like this (again simplified):
Status code: 200 OK
Cache-Control: no-cache, no-store
Connection: close
Content-Length: 0
Date: Mon, 27 Feb 2017 17:19:53 GMT
Server: WildFly/9
Set-Cookie: JSESSIONID=123; path=/
There's no payload in the response. Note the Content-Length: 0. But Firefox still tries to parse it as XML. And outputs the following error to the console:
XML Parsing Error: no root element found
Location: http://localhost:9001/login
Line Number 1, Column 1
Note, the server doesn't send a content-type header. And according to RFC 7231 it only has to send a content-type header when there is actual content.
So is this a bug in Firefox or is my research faulty?
Reproducing it yourself
I've written a small server and client to reproduce the problem.
server.js (start with node ./server.js):
const fs = require('fs'), http = require('http');
const server = http.createServer(function (request, response) {
if (request.method === 'POST') {
// send empty response
response.end();
return;
}
// send file content
fs.readFile('.' + request.url, function (error, content) {
response.writeHead(200, { 'Content-Type': request.url === '/index.html' ? 'text/html' : 'text/javascript' });
response.end(content);
});
}).listen(8080);
index.html
<script src="client.js"></script>
client.js
var xhr = new XMLHttpRequest();
xhr.open('POST', 'http://localhost:8080/login');
xhr.send();
When opening the URL http://localhost:8080/index.html in Firefox, there will be an error in the JavaScript console.
Firefox Version 51.0.1

There's no payload in the response. Note the Content-Length: 0
That means there is a payload, and that payload is 0 bytes in size. Consider the difference between null and "" as strings. What you have here is the equivalent of "" when you want the equivalent of null.
Set the status code to 204 rather than 200 to indicate you aren't sending an entity (and remove the content-type, since there's no content-type with no entity).
(For a long time Firefox would still log an error for this case, but thankfully this is finally fixed. Even when it did log the error it would still continue to run any scripts correctly).

Judging from the POST request it seems that you are using XHR/JS to send the request.
So the problem is likely in the code that is processing the result.
(Also, the Content-Type in the request is incorrect, there's no charset parameter on application/json)

JavaScript gets null response or empty responseXML from Python SimpleXMLRPCServer

I found the answer to this question. It is resolved.
Narrative: I am accessing a Python API, a set of methodCalls on top of SimpleXMLRPCServer. Server responds to browser GET request with a html page, "web_interface.html". The HTML page is a very simple script that sends a XHR POST request of xml params to the XMLRPC server. Server responds to XHR POST with headers but empty document. Server responds to cURL with correct data. Why is JavaScript not getting any readable data in response from server?
| web_interface.html |
<!DOCTYPE html>
<html>
<head>
<script>
var xrequest = '<?xml version="1.0?"><methodCall><methodName>helloWorld<methodName><params><param><firstWord><string>hello</string><firstWord></param><param><secondWord><string>world</string></secondWord></param></params></methodCall>';
function hello() {
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if (this.readyState == XMLHttpRequest.DONE) {
alert(this.responseText);
alert(this.status);
alert(this.response);
}
}
xhr.open('POST', '/', true);
xhr.setRequestHeader("Authorization", "Basic " + "aGVsbG8=" + ":" + "dGVzdA==");
xhr.send(xrequest);
}
</script>
</head>
<body>
<div>
<h2 id="msgoutput">HelloWorld API Test</h2>
<button type="button" onclick="hello(); return false;">SAY HELLO!</button>
</div>
</body>
</html>
Note: Clicking the button produces the alert dialogs. Status dialog shows "200" while Text and response dialogs are null.
| Mozilla Inspector Data & Headers |
POST Raw Data:
<?xml version="1.0?"><methodCall><methodName>helloWorld<methodName><params><param><firstWord><string>hello</string><firstWord></param><param><secondWord><string>world</string></secondWord></param></params></methodCall>
Response Headers:
Content-Length 332
Content-Type text/html
Date Sat, 10 Dec 2016 23:51:21 GMT
Server BaseHTTP/0.3 Python/2.7.12
Request Headers:
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
AuthorizationBasic aGVsbG8=:dGVzdA== (not my actual creds, swapped fakes)
Connection keep-alive
Content-Length 218
Content-Type text/plain;charset=UTF-8
DNT1
Hostlocalhost:8442
Referer http://localhost:8442/
User-Agent Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0
| Test with cURL |
:~$ curl -i --data '<?xml version="1.0"?><methodCall><methodName>helloWorld</methodName><params><param><firstWord><string>hello</string></firstWord></param><param><secondWord><string>world</string></secondWord></param></params></methodCall>' http://username:password#localhost:8442
HTTP/1.0 200 OK
Server: BaseHTTP/0.3 Python/2.7.12
Date: Sun, 11 Dec 2016 00:00:13 GMT
Content-type: text/html
Content-length: 137
<?xml version='1.0'?>
<methodResponse>
<params>
<param>
<value><string>hello-world</string></value>
</param>
</params>
</methodResponse>
Note: No problem, cURL returns the XML response as text. I pointed cURL at a netcat socket to see exactly what it is sending to the XMLRPC server. Here's what netcat shows when cURL hits:
| cURL POST Data |
POST / HTTP/1.1
Host: localhost:8442
Authorization: Basic YWRtaW46Z2liYmVyc2g=
User-Agent: curl/7.47.0
Accept: */*
Content-Length: 220
Content-Type: application/x-www-form-urlencoded
<?xml version="1.0"?><methodCall><methodName>helloWorld</methodName><params><param><firstWord><string>hello</string></firstWord></param><param><secondWord><string>world</string></secondWord></param></params></methodCall>
It's not CORS. Already tested a GET request to xhr.responseText with same browser on same machine. Setup is using same host, same port, same directory for both the GET page and the XHR POST XMLRPC request.
What am I missing?

The Python SimpleXMLRPCServer had some code hacked into it to handle cookies. Problem is, the code couldn't handle the cookie objects when a browser makes connection. This code threw an error on each invocation. It was stopping the server from sending the response text to the browser. I learned this after writing a Python snippet to output debug information to a file. I removed the cookie code and then the response XML was successfully sent from the server to the browser.
I also discovered that sending the username/password from the XHR Send statement would throw an authentication error from the server. The server is expecting credentials to be in the POST header as Authentication: Basic base64 [btoa(username:password)].
The content-type header was not the culprit in this case. Now content-type is set to text/xml on both client and server.
Here is the modified JavaScript code that works:
<!DOCTYPE html>
<html>
<head>
<script>
var xrequest = '<?xml version="1.0"?><methodCall><methodName>helloWorld</methodName><params><param><firstWord><string>hello</string></firstWord></param><param><secondWord><string>world</string></secondWord></param></params></methodCall>';
function hello() {
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if (this.readyState == XMLHttpRequest.DONE) {
}
}
xhr.open("POST", "/", true);
xhr.setRequestHeader("Authorization", "Basic " + btoa("admin" + ":" + "1234"));
xhr.setRequestHeader("Content-Type", "text/xml")
xhr.send(xrequest);
}
</script>
</head>
<body>
<div>
<h2 id="msgoutput">HelloWorld API Test</h2>
<button type="button" onclick="hello(); return false;">SAY HELLO!</button>
</div>
</body>
</html>

HTML form submit giving 400 bad request

I'm submitting a HTML form to REST(eXist db) web service using POST method.A normal submission is giving 400 bad request
Here is my HTML code
<html>
<script type="text/javascript">
/* function createXMLHttpRequest()
{
if( typeof XMLHttpRequest == "undefined" )
XMLHttpRequest = function()
{
try
{
return new ActiveXObject("Msxml2.XMLHTTP.6.0")
}
catch(e) {}
try
{
return new ActiveXObject("Msxml2.XMLHTTP.3.0")
}
catch(e) {}
try
{
return new ActiveXObject("Msxml2.XMLHTTP")
}
catch(e) {}
try
{
return new ActiveXObject("Microsoft.XMLHTTP")
}
catch(e) {}
throw new Error( "This browser does not support XMLHttpRequest." )
};
return new XMLHttpRequest();
}
var AJAX = createXMLHttpRequest();*/
function submitForm()
{
//AJAX.open("POST",'http://localhost:8899/exist/rest/db/xql/sample.xq');
// AJAX.send(document.form.xmlData.value);
document.form.submit();
};
</script>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
</head>
<body>
<form name='form' action="http://localhost:8899/exist/rest/db/xql/sample.xq" enctype="text/plain" method="post">
<input type="text" name="xmlData"/>
<input type="button" value="Submit" onclick="submitForm()";>
</form>
</body>
</html>
The commented code is to send POST request using AJAX.
I captured the http header request and response for form submit and AJAX submit
These are the request headers:
HTML form submit header:
(Request-Line) POST /exist/rest/db/xql/sample.xq HTTP/1.1
Host localhost:8899
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Connection keep-alive
Content-Type text/plain
Content-Length 26
AJAX request header:
(Request-Line) POST /exist/rest/db/xql/sample.xq HTTP/1.1
Host localhost:8899
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Connection keep-alive
Content-Length 16
Content-Type text/plain; charset=UTF-8
Origin null
Pragma no-cache
Cache-Control no-cache
Im not getting what's wrong in my code .
Im working on this for 2 days but i din't find any solution.
Please look into this and provide a solution.
Thanks in advance.

I'm pretty sure it's because you're sending only the value in your data.
You need to send a name = value pair.

Your code sumbits data to the server as it should be. There must be some problem with your server side code.
Quoting from checkupdown.com about error 400
400 errors in the HTTP cycle
1.Any client (e.g. your Web browser or our CheckUpDown robot) goes through the following cycle:
2.Obtain an IP address from the IP name of the site (the site URL without the leading 'http://'). This lookup (conversion of IP name to IP address) is provided by domain name servers (DNSs).
3.Open an IP socket connection to that IP address.
4.Write an HTTP data stream through that socket.
5.Receive an HTTP data stream back from the Web server in response. This data stream contains status codes whose values are determined by the HTTP protocol. Parse this data stream for status codes and other useful information.
This error occurs in the final step above when the client receives an HTTP status code it recognises as '400'.

Does your target accept POST requests, or only GET?

But you aren't sending any parameters with the Ajax POST?
The Ajax code should look something like this:
var xmlData=encodeURIComponent(document.getElementById("xmlData").value);
var parameters="xmlData="+xmlData;
AJAX.open("POST", "'http://localhost:8899/exist/rest/db/xql/sample.xq", true)
AJAX.setRequestHeader("Content-type", "application/x-www-form-urlencoded")
AJAX.send(parameters)

Building a javascript client for the TFS SOAP webservice

I am trying to build a javascript client that gets information about active projects on a TFS.
Since TFS has SOAP endpoints I was thinking of using wsdl2js ( http://cxf.apache.org/docs/tools.html ) to generate a local proxy and then call the functions that i need on that proxy (like list projects, etc).
Here's my js code:
function showresponse(response)
{
alert("rasp");
}
function showerror(error)
{
alert('error');
}
var test=new ClassificationSoap();
test.url="http://192.168.48.130:8080/Services/v1.0/CommonStructureService.asmx";
test.ListAllProjects(showresponse,showerror);
However, neither of the response functions are called.
According to CommonStructureService.asmx here's how the request should look:
POST /Services/v1.0/CommonStructureService.asmx HTTP/1.1
Host: 192.168.48.130
Content-Type: text/xml; charset=utf-8
Content-Length: length
SOAPAction: "http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Classification/03/ListAllProjects"
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<ListAllProjects xmlns="http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Classification/03" />
</soap:Body>
</soap:Envelope>
I fired fiddler, and here's how my raw request looks:
OPTIONS http://192.168.48.130:8080/Services/v1.0/CommonStructureService.asmx HTTP/1.1
Host: 192.168.48.130:8080
Connection: keep-alive
Cache-Control: max-age=0
Access-Control-Request-Method: POST
Origin: null
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30
Access-Control-Request-Headers: MessageType, SOAPAction, Content-Type
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
As you can see, there is no xml sent.
Here's the raw response:
HTTP/1.1 401 Unauthorized
Content-Length: 1656
Content-Type: text/html
Server: Microsoft-IIS/6.0
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Wed, 27 Jul 2011 17:14:08 GMT
Proxy-Support: Session-Based-Authentication
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>You are not authorized to view this page</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252">
<STYLE type="text/css">
BODY { font: 8pt/12pt verdana }
H1 { font: 13pt/15pt verdana }
H2 { font: 8pt/12pt verdana }
A:link { color: red }
A:visited { color: maroon }
</STYLE>
</HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD>
<h1>You are not authorized to view this page</h1>
You do not have permission to view this directory or page using the credentials that you supplied because your Web browser is sending a WWW-Authenticate header field that the Web server is not configured to accept.
<hr>
<p>Please try the following:</p>
<ul>
<li>Contact the Web site administrator if you believe you should be able to view this directory or page.</li>
<li>Click the Refresh button to try again with different credentials.</li>
</ul>
<h2>HTTP Error 401.2 - Unauthorized: Access is denied due to server configuration.<br>Internet Information Services (IIS)</h2>
<hr>
<p>Technical Information (for support personnel)</p>
<ul>
<li>Go to Microsoft Product Support Services and perform a title search for the words <b>HTTP</b> and <b>401</b>.</li>
<li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr),
and search for topics titled <b>About Security</b>, <b>Authentication</b>, and <b>About Custom Error Messages</b>.</li>
</ul>
</TD></TR></TABLE></BODY></HTML>
So basically it says it needs authentication.
Why isn't the showerror function called? How can my client know how to ask user for credentials?
Also, how does authentication work?
I know I need to send an authorization header that contains "user:pass" encoded with base64, but I can't find any reference to this in the generated Classification.js
Thanks

You'll note that your first request was an OPTIONS request, not a POST. Are you sending your request to a cross-origin server using XMLHTTPRequest? If so, the server needs to be configured, via CORS, to return an Access-Control-Allow-Origin directive in response to that pre-flight OPTIONS request.