I'm trying to send a MessagePack-encoded message from Cowboy to a browser over WebSocket, and received data is always empty or invalid. I'm able to send binary data from JS to my cowboy handler, but not vice versa.
I'm using Cowboy 1.0.4 with official msgpack-erlang application. I also use msgpack-lite for my in-browser javascript.
Examples:
websocket_handler:
websocket_handle({text, <<"return encoded">>}, Req, State) ->
%% sends encoded message to client. Client is unable to decode and fails
{reply, {binary, msgpack:pack(<<"message">>)}, Req, State};
websocket_handle({binary, Encoded}, Req, State) ->
%% Works as expected
lager:info("Received encoded message: ~p", [msgpack:unpack(Encoded)]),
{ok, Req, State};
JS:
var host = "ws://" + window.location.host + "/websocket";
window.socket = new WebSocket(host);
socket.binaryType = 'arraybuffer';
socket.onmessage = function(event) {
var message = msgpack.decode(event.data);
console.log(message);
};
Browser returns an error inside msgpack.min.js:
Error: Invalid type: undefined
...ion n(t){var r=i(t),e=f[r];if(!e)throw new Error("Invalid type: "+(r?"0x"+r.toSt...
If I try to output raw event.data to console, here's what I'm getting:
ArrayBuffer {}
It seems to be empty for some reason. I'm new both to erlang and msgpack, and don't know what is going wrong. Thanks for your help!
Found the reason of my problem.
The way how I tried to decode message on the client was wrong:
socket.onmessage = function(event) {
var message = msgpack.decode(event.data);
console.log(message);
};
The right way:
socket.onmessage = function(event) {
var raw_binary_data = new Uint8Array(event.data);
var message = msgpack.decode(raw_binary_data);
console.log(message);
};
It seems like msgpack-lite doesn't support binary type. Try pack your data as a string.
{binary, msgpack:pack("message", [{enable_str, true}])}
Using Uint8Array is a valid solution for client side issue. On server to pack strings use:
msgpack:pack(<<"message">>,[{pack_str,from_binary}])
Source : Article
Related
Perhaps the question is not worded in the greatest way but here's some more context. Using GridFSBucket, I'm able to store a file in mongo and obtain a download stream for that file. Here's my question. Let's say I wanted to send that file back as a response to my http request.
I do:
downloadStream.pipe(res);
On the client side now when I print the responseText, I get some long string with some funky characters that look to be encrypted. What is the format/type of this string/stream? How do I setup my response so that I can get the streamed data as an ArrayBuffer on my client side?
Thanks
UPDATE:
I haven't solved the problem yet, however the suggestion by #Jekrb, gives exactly the same output as doing console.log(this.responseText). It looks like the string is not a buffer. Here is the output from these 2 lines:
console.log(this.responseText.toString('utf8'))
var byteArray = new Uint8Array(arrayBuffer);
UPDATE 2 - THE CODE SNIPPETS
Frontend:
var savePDF = function(blob){
//fs.writeFile("test.pdf",blob);
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if (this.readyState === XMLHttpRequest.DONE && this.status === 200){
//TO DO: Handle the file in the response which we will be displayed.
console.log(this.responseText.toString('utf8'));
var arrayBuffer = this.responseText;
if (arrayBuffer) {
var byteArray = new Uint8Array(arrayBuffer);
}
console.log(arrayBuffer);
}
};
xhr.open("POST","/pdf",true);
xhr.responseType = 'arrayBuffer';
xhr.send(blob);
};
BACKEND:
app.post('/pdf',function(req,res){
MongoClient.connect("mongodb://localhost:27017/test", function(err, db) {
if(err) return console.dir(err);
console.log("Connected to Database");
var bucket = new GridFSBucket(db, { bucketName: 'pdfs' });
var CHUNKS_COLL = 'pdfs.chunks';
var FILES_COLL = 'pdfs.files';
// insert file
var uploadStream = bucket.openUploadStream('test.pdf');
var id = uploadStream.id;
uploadStream.once('finish', function() {
console.log("upload finished!")
var downloadStream = bucket.openDownloadStream(id);
downloadStream.pipe(res);
});
// This pipes the POST data to the file
req.pipe(uploadStream);
});
});
My guess is that either the response is being outputted as plain binary which is not base64 encoded (still a buffer) or it is a compressed (gzip) response that needs to be uncompressed first.
Hard to pinpoint the issue without seeing the code though.
UPDATE:
Looks like you're missing the proper response headers.
Try setting these headers before the downloadStream.pipe(res):
res.setHeader('Content-disposition', 'attachment; filename=test.pdf');
res.set('Content-Type', 'application/pdf');
Your stream is likely already a buffer. You might be able to call responseText.toString('utf8') to convert the streamed data into readable string.
I solved it!!!
Basically preset the response type to "arraybuffer" before you make the request using
req.responseType = "arraybuffer"
Now, once you receive the response, don't use responseText, instead use response. response contains the arraybuffer with the data for the file.
I have a websocket connection between
• Nodejs server (+ socket.io)
• Browser
A message is emitted from the server to the browser (Udp message from an Arduino)
udpSocket.on("message", function (msg, rinfo) {
for (i = 0; i < msg.length; i++) {
console.log(msg.readInt8(i));
}
console.log("");
io.sockets.emit('message', msg);
});
Code in the browser (index.html):
A)
I started with:
socket.on('message', function(message) {
document.getElementById("ko1").innerHTML = message;
})
<p> Arduino sends: </p>
<p id="ko1"> <br>
Result in browser : [object ArrayBuffer]
My conclusion : message is coming in, but must be handled in another way.
B)
Research learned that I have to use a DataView on the Array Buffer (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays). My code:
socket.on('message', function(message) {
var uint8 = new Uint8Array(message);
document.getElementById("ko1").innerHTML = uint8[0];
})
Result : no data printed to the browser
Question: what am I doing wrong? I also tried other Data Views, but no success. It’ s driving me mad. Please help.
I have a node.js client which downloads and decrypts an AES encrypted file from another host.
var base64 = require('base64-stream');
var crypto = require('crypto');
var aes = crypto.createDecipher('aes-256-cbc', crypto.createHash('sha256').update(pass).digest('hex'));
// file stream
var file = fs.createWriteStream(params.target);
var base64reader = base64.decode();
response.pipe(base64reader) // decode base64
.pipe(aes) // decrypt
.pipe(file); // write in file
// on last data chunk received: file load complete
aes.on('end', function (chunk) {
if (typeof params.success !== 'undefined')
params.success();
});
If the other host close his connection unexpectedly before finishing the request, the code above throws this error:
TypeError: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length
at Decipher.Cipher._flush (crypto.js:262:27)
at Decipher.eval (_stream_transform.js:130:12)
at Decipher.g (events.js:187:16)
at Decipher.EventEmitter.emit (events.js:95:17)
at prefinish (_stream_writable.js:427:12)
at finishMaybe (_stream_writable.js:435:7)
at afterWrite (_stream_writable.js:317:3)
at onwrite (_stream_writable.js:307:7)
at WritableState.onwrite (_stream_writable.js:100:5)
at afterTransform (_stream_transform.js:99:5)
at TransformState.afterTransform (_stream_transform.js:74:12)
at Decipher.Cipher._transform (crypto.js:258:3)
at Decipher.Transform._read (_stream_transform.js:179:10)
at Decipher.Readable.read (_stream_readable.js:334:10)
at flow (_stream_readable.js:743:26)
at WriteStream.eval (_stream_readable.js:601:7)
I tried to add an aes.on('error', function(() {...}); handler but it will not be called. I also tried adding
response.on('end', function() { aes.emit('close'); });
response.on('close', function() { aes.emit('close'); });
but then aes.on('end', ...); will not be called. Adding aes.emit('end') to this statements make no sense, because this will be also called in case of an error which leads to the error above.
response.on('end', function() { aes.emit('end'); aes.emit('close'); });
response.on('close', function() { aes.emit('end'); aes.emit('close'); });
Does anybody have an idea how I can catch this error?
Thanks a lot!!
Its a bug in node.js v0.11.9 which is solved in v0.11.13. Then aes.on('error', ...) will be called correctly.
I have server receiving data from a client [GPS Device]. I have problem presenting the data (i.e the results obtained from the client) in a readable format. Below are the things I have tried.
Doing:
console.log(data)
I get
<Buffer d0 d7 3d 00 c4 56 7e 81>
Also tried
console.log(data.toString())
But I get unwanted results:See below:
��A�V~�
Here is my full code:
var net = require('net');
var fs = require('fs');
var server = net.createServer(function (socket) {
console.log('Server started: Waiting for client connection ...');
console.log('Client connected:port,address: '+socket.remotePort, socket.remoteAddress);
socket.on('data', function (data) {
var date = new Date();
var today = date.getDate()+'_'+date.getMonth();
fs.appendFile(today+'_log.txt', data, function (err) {
if (err) throw err;
console.log(data.toString())
});
});
});
server.listen(my_port, my_ip);
Thanks for your input.
According to the documentation, you must specify an encoding to get a String instead of a Buffer:
Event: 'data'#
Buffer object
Emitted when data is received. The argument data will be a Buffer or String. Encoding of data is set by socket.setEncoding().
You could configure the socket to get the data in UTF-8, for example, with:
socket.setEncoding('utf8');
Assuming the data in buffer is 7 bit ASCII,
console.log(data.toString('ascii'))
would resolve the problem.
Do it like this
socket.on('data', function (data) {
var buff = Buffer.from(data);
But remember, lots of GPS devices use little-endian, so later you're going to have to decode the data as well.
I'm trying to manage a bunch of socket connections. My app is basically an http server that receives posts and passes these along to a socket. When clients open a socket connection, they send a connect message with an id:
{"m":"connect","id":"1"}
The app then saves this id and socket in the id2socket and socket2id maps. On disconnect, the socket/id pair is deleted from the maps.
A post will also contain an id, which indicates the post data should be sent to the socket with that id.
That's great, and this works fine for a single open socket. However, when I have more than one socket open, and then I close a socket, that disconnect wipes everything from the map. I think my understanding of sockets in node is incomplete- is there only a single socket object that is used in the callback? Is there a better way to manage my open socket connections and ids?
start server:
>>node server.js
TCP server listening on 127.0.0.1:5280
HTTP server listening on 127.0.0.1:9002
telnet in:
>>telnet localhost 5280
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
{"m":"connect","id":"123"}
{"m":"connect","id":"123","success":"true"}
server after connection:
>>Connection from 127.0.0.1:57572
received data: {"m":"connect","id":"123"}
id: 1
m: connect
associating uid 1 with socket [object Object]
do a post:
python post.py {"foo":"bar"}
So this works fine for several open sockets (as long as 1 device is id 123, server has this hardwired for now). However, as soon as you close one connection all the socket connections are removed from the map.
Here's my code:
python script to do post:
import sys
import json
import httplib, urllib, urllib2
values = json.loads('{"foo":"bar"}')
headers = {"Content-type": "application/json"}
conn = httplib.HTTPConnection('127.0.0.1', 9002)
headers = {"Content-type": "application/json"}
conn.request("POST", "", json.dumps(values), headers)
response = conn.getresponse()
print "response.status: "+response.status
print "response.reason: "+response.reason
print "response.read: "+response.read()
conn.close()
node server (http and tcp), hardwired to send data to device '123' on post:
var net = require('net'); // tcp-server
var http = require("http"); // http-server
var qs = require('querystring'); // http-post
// Map of sockets to devices
var id2socket = new Object;
var socket2id = new Object;
// Setup a tcp server
var server_plug = net.createServer(function(socket) {
// Event handlers
socket.addListener("connect", function(conn) {
console.log("Connection from " + socket.remoteAddress + ":" + socket.remotePort );
});
socket.addListener("data", function(data) {
console.log("received data: " + data);
try {
request = JSON.parse(data);
response = request;
if(request.m !== undefined && request['id'] !== undefined){ // hack on 'id', id is js obj property
console.log("id: "+request['id']);
console.log("m: "+request.m);
if(request.m == 'connect'){
console.log("associating uid " + request['id'] + " with socket " + socket);
id2socket[request['id']] = socket;
socket2id[socket] = request['id'];
response.success = 'true';
} else {
response.success = 'true';
}
}
socket.write(JSON.stringify(response));
} catch (SyntaxError) {
console.log('Invalid JSON:' + data);
socket.write('{"success":"false","response":"invalid JSON"}');
}
});
socket.on('end', function() {
id = socket2id[socket]
console.log("socket disconnect by id " + id);
// wipe out the stored info
console.log("removing from map socket:"+socket+" id:"+id);
delete id2socket[id];
delete socket2id[socket];
});
socket.on('timeout', function() {
console.log('socket timeout');
});
});
// Setup http server
var server_http = http.createServer(
// Function to handle http:post requests, need two parts to it
// http://jnjnjn.com/113/node-js-for-noobs-grabbing-post-content/
function onRequest(request, response) {
request.setEncoding("utf8");
request.addListener("data", function(chunk) {
request.content += chunk;
});
request.addListener("end", function() {
console.log("post received!");
//console.log("Request received: "+request.content);
if (request.method == 'POST') {
//var json = qs.parse(request.content);
//console.log("Post: "+json);
// HACK TO TEST STUFF:
// send a message to one of the open sockets
try {
var socket = id2socket['123']; //hardwired
socket.write('{"m":"post"}');
} catch (Error) {
console.log("Cannot find socket with id "+'123');
}
}
});
}
);
// Fire up the servers
var HOST = '127.0.0.1';
var PORT = 5280;
var PORT2 = 9002;
server_plug.listen(PORT, HOST);
console.log("TCP server listening on "+HOST+":"+PORT);
server_http.listen(PORT2);
console.log("HTTP server listening on "+HOST+":"+PORT2);
Objects only take strings as keys for their properties. As your log shows, a socket object is converted into the string "[object Object]". As a result, socket #2 overwrites the id from socket #1 in the object, because all sockets are converted into the same string key. So, there is only one property in the object at all times, because all sockets come down to the same key. When you try to remove the id for socket #2, the single property is deleted and the object is empty.
You seem to want a custom property for each separate socket when used as a key. You can use WeakMaps for this. WeakMaps do allow objects as keys (as opposed to string-only keys), but as they're relatively new they may contain bugs at the moment.
(Note that the id2socket map can just be a plain object, because numbers are converted into strings just fine, and each number has its own, distinct string representation*.)
Using WeakMaps is as follows:
var socket2id = new WeakMap; // as if you were doing: var socket2id = {};
socket2id.set(socket, id); // as if you were doing: socket2id[socket] = id;
socket2id.get(socket); // as if you were doing: socket2id[socket];
socket2id.delete(socket); // as if you were doing: delete socket2id[socket];
Make sure to run with node --harmony (>= 0.7) or node --harmony_weakmaps (<= 0.6).
* 0 and -0 are exceptions, but you shouldn't be using -0 anyway because 0 === -0, so it's difficult to differ between them.