Develop Reference

Wondering how to send HEX buffer over HTTP with only text option to send

I am trying to send a Hexadecimal command to a TV RS232 through a Global Cache HTTP to RS232 device. Global cache can also operate device with RS232 over TCP/UDP port 4999.
Global Cache has a URL on the device that can take raw data and pass it on to the RS232 device or we can POST to the URL port 80 from file containing the HEX command but in the field, our sending device will need to create the HEX command without accessing a file and must use the HTTP POST rather than sending direct to port 4999.
Here is examples that work:
Command line:
echo -n -e "\x7F\x08\x99\xA2\xB3\xC4\x02\xFF\x01\x07\xCF" | nc 192.168.1.222 4999
Node JS TCP
hexString = "7F0899A2B3C402FF0107CF";
rawHex = Buffer.from(hexString, 'hex');
console.log(rawHex);
client.connect(4999, '192.168.1.222', () => {
console.log("Connected");
client.write(rawHex); //This will send the byte buffer over TCP
client.end();
})
Node JS HTTP
hexString = "7F0899A2B3C402FF0138CF";
rawHex = Buffer.from(hexString, 'hex');
console.log(rawHex);
client = net.createConnection({
host: '192.168.1.222',
port: 80,
path: '/api/v1/serialports/1/sendserial/'
}, () => {
console.log("Connected");
client.write(rawHex); //This will send the byte buffer over HTTP
client.end();
}
)
Insominia
First write the command line output to a file:
echo -n -e "\x7F\x08\x99\xA2\xB3\xC4\x02\xFF\x01\x07\xCF" > /tmp/hexcommand
Send that file as data with HTTP POST
> POST /api/v1/serialports/1/sendserial HTTP/1.1
> Host: 192.168.1.222
> User-Agent: insomnia/2021.7.2
> Content-Type: application/octet-stream
> Accept: */*
> Content-Length: 11
| �����8�
As mentioned, the device that will send this command strings cannot access a file and does not have node.js. So we need to get the POST data HEX buffer as a text data and send it with a HTTP POST.
FAILS:
Trying some javascript encoded HEX or other web url style hex encoder do not work. If I process the ASCII representation with some javascript examples and send the result, it is not affecting the device.
Example:
str = "7F0899A2B3C402FF0107CF";
function convertString(){
return(str.split("").reduce((hex,c)=>hex+=c.charCodeAt(0).toString(16).padStart(2,"0"),""))
}
Result of the function is more like web compatible hex. Notice the "text/plain" type in following POST example.
GlobalCache document specifies text/plain is required, so we are not sure why the application stream worked on the previous working example.
> POST /api/v1/serialports/1/sendserial HTTP/1.1
> Host: 192.168.1.222
> User-Agent: insomnia/2021.7.2
> Content-Type: text/plain
> Accept: */*
> Content-Length: 11
| 37463038393941324233433430324646303130374346
Also, sending the ASCII representation with the same POST method fails on all cases.
7F0899A2B3C402FF0107CF
7f0899a2b3c402ff0107cf
7F 08 99 A2 B3 C4 02 FF 01 07 CF
\x7F\x08\x99\xA2\xB3\xC4\x02\xFF\x01\x07\xCF
None of above are equivalent to the working examples that produce the usable HEX output.
Thanks for any advice how to change that ASCII text into a string that may go working across the HTTP Post

Content-length header doesnt match request body on chunked upload

Hello,
I have a problem with chunked uploading to google-cloud-storage (gcs) using dropzone.js.
What Im Trying To Do:
I want to upload a (bigger) file to google-cloud-storage via dropzone. Since its a quite large file I'm using dropzone's internal function to chunk (and upload) it. I'm trying to upload via signed url that I'm creating, initalizing and afterwards passing to dropzone so that it knows where to send the file. Also I'm adding a Content-Range header to the request so that gcs keeps track of the current file-upload status.
Current Status:
When the download starts, it seems to work. But after the first chunk finishes dropzone tries to reupload the first chunk, because the xhr-response status is 400.
The Problem:
When investigated the xhr request and response it showed that the Content Length and Content-Range header are NOT the same size. And that is also what the response text told me the error is.
My code in .on("sending") by dropzone:
let procChunks = file.upload.chunks; // Get all chunks processed until now
latestChunk = procChunks[procChunks.length-1]; // Select latest chunk
const chunkFirstByte = dz.options.chunkSize * latestChunk.index; // Calc first byte
const chunkLastByte = chunkFirstByte + (latestChunk.dataBlock.data.size-1); // Calc last byte
let header = "bytes "+chunkFirstByte+"-"+chunkLastByte +"/"+ (file.size-1); // Create header value
xhr.setRequestHeader("Content-Range", header); // Set header to xhr
Request Header:
PUT /upload/storage/v1/b/(myBucket)/o?uploadType=resumable&name=test2.fna&upload_id=(myUpLoadID) HTTP/2
Host: storage.googleapis.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0
Accept: application/json
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
X-Requested-With: XMLHttpRequest
Content-Range: bytes 0-8388607/13088352
Content-Type: multipart/form-data; boundary=---------------------------428770732237854445893641225227
Content-Length: 8388843
Origin: http://127.0.0.1:5000
Connection: keep-alive
Referer: http://127.0.0.1:5000/upload
TE: Trailers
The request body:
-----------------------------428770732237854445893641225227
Content-Disposition: form-data; name="file"; filename="test2.fna"
Content-Type: application/octet-stream
>NC_000913.3 Escherichia coli str. K-12 substr. MG1655, complete genome
AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACTG
GTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATACTTTAACCAATATAGGCATAGCGCACAGAC
AGATAAAAATTACAGAGTACACAACATCCATGAAACGCATTAGCACCACCATTACCACCACCATCACCATTACCACAGGT
AACGGTGCGGGCTGACGCGTACAGGAAACACAGAAAAAAGCCCGCACCTGACAGTGCGGGCTTTTTTTTTCGACCAAAGG
TAACGAGGTAACAACCATGCGAGTGTTGAAGTTCGGCGGTACATCAGTGGCAAATGCAGAACGTTTTCTGCGTGTTGCCG
ATATTCTGGAAAGCAATGCCAGGCAGGGGCAGGTGGCCACCGTCCTCTCTGCCCCCGCCAAAATCACCAACCACCTGGTG
GCGATGATTGAAAAAACCATTAGCGGCCAGGATGCTTTACCCAATATCAGCGATGCCGAACGTATTTTTGCCGAACTTTT
GACGGGACTCGCCGCCGCCCAGCCGGGGTTCCCGCTGGCGCAATTGAAAACTTTCGTCGATCAGGAATTTGCCCAAATAA
AACATGTCCTGCATGGCATTAGTTTGTTGGGGCAGTGCCCGGATAGCATCAACGCTGCGCTGATTTGCCGTGGCGAGAAA
ATGTCGATCGCCATTATGGCCGGCGTATTAGAAGCGCGCGGTCACAACGTTACTGTTATCGATCCGGTCGAAAAACTGCT
GGCAGTGGGGCATTACCTCGAATCTACCGTCGATATTGCTGAGTCCACCCGCCGTATTGCGGCAAGCCGCATTCCGGCTG
ATCACATGGTGCTGATGGCAGGTTTCACCGCCGGTAATGAAAAAGGCGAACTGGTGGTGCTTGGACGCAACGGTTCCGAC
TACTCTGCTGCGGTGCTGGCTGCCTGTTTACGCGCCGATTGTTGCGAGATTTGGACGGACGTTGACGGGGTCTATACCTG
CGACCCGCGTCAGGTGCCCGATGCGAGGTTGTTGAAGTCGATGTCCTACCAGGAAGCGATGGAGCTTTCCTACTTCGGCG
CTAAAGTTCTTCACCCCCGCACCATTACCCCCATCGCCCAGTTCCAGATCCCTTGCCTGATTAAAAATACCGGAAATCCT
CAAGCACCAGGTACGCTCATTGGTGCCAGCCGTGATGAAGACGAATTACCGGTCAAGGGCATTTCCAATCTGAATAACAT
GGCAATGTTCAGCGTTTCTGGTCCGGGGATGAAAGGGATGGTCGGCATGGCGGCGCGCGTCTTTGCAGCGATGTCACGCG
CCCGTATTTCCGTGGTGCTGATTACGCAATCATCT
Response Text:
Invalid request. There were 8388843 byte(s) (or more) in the request body. There should have been 8388608 byte(s) (starting at offset 0 and ending at offset 8388607) according to the Content-Range header.
As you can maybe see the first 4 lines are NOT from the file.
Additional Information:
The file is pure text data (utf-8 encoded i guess).
Im chunking in 8mb large chunks (recommended by google).
Somewhere on google's api guide I read that when uploading via PUT there should/must be no other data except the file data.
Content-Length header gets added automatically by dropzone.
My Questions:
Where does those 4 lines come from?
Can I (re)set the xhr request body?
Is my problem maybe caused by some formating problem of file-data? (Like bytes, strings)?
In case that's not the problem - do u have any other idea what could be the problem?
THANK YOU VERY MUCH!
Any help appricated! If u need some more information please ask!

API Data Returning Unicode Characters in Console

I am facing a rather confusing problem since the last two days. I am working on a document management system, that uses an API that pulls in data from SOLR. The data is in tune of around ~15Mbs, and pulls records of more than 4000+ documents. The API has response in this format -
{
"documents": [
{
id: 123,
some_field: "abcd",
some_other_field: "abcdef"
},
{
id: 124,
some_field: "abcd1",
some_other_field: "abcdef1"
}
]
}
Everything works fine in browser. If I hit the endpoint in Chrome or Firefox browser, it gives me the correct output and I am able to see the JSON output.
However, if I try hitting the same API endpoint with a Java or JS code - the response code is 200, but the output in console (Terminal or Eclipse) shows unicode characters like \u0089 \u0078 U+0080 - all the output comes in this way, and since there are around 4000+ records being fetched by the API, the console kinda fills with all of these unicode characters.
The only difference that I see between the requests made from browser and the code is that in browser I can see Content-Encoding : gzip, while I cannot find this header from the code that I written . For eg - in JS code, through Chakram framework, I can check
expect(response).to.be.encoded.with.gzip
mentioned here. However, this returns a failure stating expected undefined to match gzip
What am I missing here? Is this something related to encoding/decoding or something entirely different?
Edit 1 : The Response Headers as seen in Network tab of Chrome :
cache-control: max-age=0, private, must-revalidate, max-age=315360000
content-encoding: gzip
content-type: application/json; charset=utf-8
date: Tue, 22 May 2018 06:07:26 GMT
etag: "a07eb7c1eef4ab97699afc8d61fb9c5d"
expires: Fri, 19 May 2028 06:07:26 GMT
p3p: CP="NON CUR OTPi OUR NOR UNI"
server: Apache
Set-Cookie : some_cookie
status: 200 OK
strict-transport-security:
transfer-encoding: chunked
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-request-id: abceefr4-1234-acds-100b-d2bef2413r47
x-runtime: 3.213943
x-ua-compatible: chrome=1
x-xss-protection: 1; mode=block
The Request Headers as seen in Network tab of Chrome
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: some_cookie
Host: abcd.bcd.com
IV_USER: demouser123
IV_USER_L: demouser123
MAIL: demouser#f.com
PERSON_ID: 123
Referer: http://abcd.bcd.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
X-CSRF-TOKEN: some_csrf_token
Edit 2 : The tests that I am using
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
console.log(JSON.stringify(t_resp));
expect(t_resp).to.have.header('Content-Encoding','gzip');
});
});

This has nothing to do with encoding. The web server in general compresses to gzip to save the bandwidth since its redundant to transfer the whole 15MB file as is refer this article for more about gZip and the its working ( https://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/ ). So where does it went wrong and how it worked in chrome is pretty simple chrome has an inbuilt unicode parser(even an HTML parser) in its devTools which can show you the parsed content rather showing you the wiered text (same can be seen in response tab next to preview tab). why you see wierd text is that you are stingfying the response which will escape special character if any console.log(JSON.stringify(t_resp));. You cannot use something like console.log("response", t_resp); without stringifying in terminal since the terminal doesn't have a JSON or an unicode parser it just prints in text. try removing that console since stringifying a 15mb file is a costly process.
Edit 1:-
if you still want to output in the console here whats to be done.
Since NODE cannot decode gzip by default directly (not with chakram, its just a APItesting platform) you can use zlib to do this. Please find the example snippet
const zlib = require('zlib');
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
zlib.gunzip(t_resp, function(err, dezipped) {
console.log(dezipped);
});
});
});

Try with console.dir to display your values
describe('Hits required API',()=>{
before(()=>{
return chakram.wait(api_response = chakram.get(url,options));
});
it('displayes response',()=>{
return api_response.then((t_resp)=>{
console.dir(t_resp, { depth: null });
});
});
Console.dir

How do you open HTML file in javascript?

So, firstly I'd like to specific that I'm aware of the following code
window.open(url);
which has worked very well for me in the past.
The problem this time is that I'm trying to open a html file that isn't online as such. The file itself is here "http://coynesresources.weebly.com/uploads/7/6/5/3/76537449/small.html" and when I try to use the following
window.open("http://coynesresources.weebly.com/uploads/7/6/5/3/76537449/small.html");
only allows me to download the HTML file, rather than open it in my browser. Any help would be greatly appreciated.

Try this
window.open("http://coynesresources.weebly.com/uploads/7/6/5/3/76537449/small.html", "windowName", "width=200,height=100",false);
more information see here
http://www.w3schools.com/jsref/met_win_open.asp
https://developer.mozilla.org/en-US/docs/Web/API/Window/open

You can't use direct ajax query for this url (crossdomain policy). So you must use proxy with jsonp.
See: Loading cross domain endpoint with jQuery AJAX
And working example for you:
http://jsbin.com/bahigodiru/1/edit?html,js,output
But it is not good solution. There are a lot of free hostings that you can use. Or use jsfiddle/jsbin (for jsbin you can get link to page without online editor: http://jsbin.com/bahigodiru/1/ )
weebly.com is a platform to create a site. You should not upload you html as file. You should add it's content as custom html: https://www.weebly.com/blog/file-upload-new-linker-custom-html-and-a-new-upgrade-process

I'm going to guess that you can't.
The issue is the server is returning a header that tells the browser to download it instead of display it
curl -v "http://coynesresources.weebly.com/uploads/7/6/5/3/76537449/small.html"
* Trying 199.34.228.54...
* Connected to coynesresources.weebly.com (199.34.228.54) port 80 (#0)
> GET /uploads/7/6/5/3/76537449/small.html HTTP/1.1
> Host: coynesresources.weebly.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Sat, 09 Jul 2016 19:17:29 GMT
< Content-Type: text/html
< Content-Length: 904
< Last-Modified: Sat, 09 Jul 2016 15:10:54 GMT
< Connection: keep-alive
< ETag: "578113fe-388"
< Expires: Sat, 16 Jul 2016 19:17:29 GMT
< Cache-Control: max-age=604800
< Content-Disposition: attachment
< Accept-Ranges: bytes
< X-W-DC: SFO
<
That Content-Disposition: attachment part tells the browser to download instead of display
See RFC 6266
4.2. Disposition Type
If the disposition type matches "attachment" (case-insensitively),
this indicates that the recipient should prompt the user to save the
response locally, rather than process it normally (as per its media
type).

Ajax response is gzip compressed - Prototype, Firefox can't handle it

I'm trying to query a web service (with JavaScript, prototype). The server responds with XML, but compresses it; headers are set appropriately.
Under Safari 4, everything is fine. The response is decompressed and JavaScript can deal with the data.
Under Firefox 3.5.8, no data is returned to JavaScript!
Code:
var req = new Ajax.Request(this.url, {
asynchronous: false,
contentType: 'text/xml',
method: 'post',
postBody: xmlString,
onSuccess: function(t) {
// debug, place response into textarea to show
$('responseText').value = t.responseText;
}
});
This is the response, as I trace it on the network:
HTTP/1.1 200 OK.
Date: Fri, 05 Mar 2010 14:10:51 GMT.
Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny6 with Suhosin-Patch.
X-Powered-By: PHP/5.2.6-1+lenny6.
Vary: Accept-Encoding.
Content-Encoding: gzip.
Content-Length: 2104.
Keep-Alive: timeout=15, max=100.
Connection: Keep-Alive.
Content-Type: text/xml.
.
............]s......W`.3...H&A.$.Q.^[.:....... (and so on ...)
Any idea why this is happening? What can I do about it?
I tried setting the 'Accept-Encoding' header in the request, can't get it working properly. Besides, the response can be rather large, meaning: it's good that it is compressed by the server.