i'm not 100% sure but from what i read when i send a blob (binary data) over websocket, the blob does not contain any file information. (Also the official specification states that wesockets only send the raw binary)
the filesize
the mimetype
user info (explain later)
i'm using https://github.com/websockets/ws
Testing:
Sending directly the blob from an input file.
ws.send(this.files[0]) //this should already contain the info
Creating a new blob with the native javascript api from file setting the proper mimetype.
ws.send(new Blob([this.files[0]],{type:this.files[0].type})); //also this
on both sides you can get only the effective blob without any other information.
Is it possible to append let's say a 4kb predefined json data converted also to binary that contains important information like the mimetype and the filesize,
and then just split off the 4kb when needed?
{"mime":"txt/plain","size":345}____________4KB_REST_OF_THE_BINARY
OR
ws.send({"mime":"txt\/plain","size":345})
ws.send(this.files[0])
Even if the first one is the worst solution ever it would allow me to send everything in one time.
The second one has a big problem:
it's a chat that allows to send also files like documents,images,music videos.
i could write some sort of handshaking system when sending the file/user info before i send the binary data.
BUT
if another person sends also a file, as it's async, the handshaking system has no chance to determine wich file is the right one for the correct user and mimetype.
So how do you properly send a binary file in a multiuser async envoirement?
i know i can convert to base64 but thats 30% bigger.
btw. Totally disappointed with Apple... while chrome shows every binary data properly, my ios devices are not able to handle blob's, only images will show in blob or base64 format, not even a simple txt file. Basically only a <img> tag can read dynamic files.
How everything works (now):
user sends a file
nodejs gets the binary data, also user info... but not mimetype,filename,size.
nodejs broadcasts the raw binary file to all the users.(can't specify user & file info)
clients create a bloburl (who send that? XD).
EDIT
what i have now:
client 1 (sends a file)CHROME
fileInput.addEventListener('change',function(e){
var file=this.files[0];
ws.send(new Blob([file],{
type:file.type //<- SET MIMETYPE
}));
//file.size
},false);
note: file is already a blob ... but this is how you would normally create a new blob specifying the mimetype.
server (broadcasts the binary data to the other clients)NODEJS
aaaaaand the mimetype is gone...
ws.addListener('message',function(binary){
var b=0,c=wss.clients.length;
while(b<c){
wss.clients[b++].send(binary)
}
});
client 2 (recieves the binary)CHROME
ws.addEventListener('message',function(msg){
var blob=new Blob([msg.data],{
type:'application/octet-stream' //<- LOST
});
var file=window.URL.createObjectURL(blob);
},false);
note: m.data is already a blob ... but this is how you would normally create a new blob specifying the mimetype witch is lost.
In client 2 i need the mimetype and naturally i also need the info about the user, wich can be retrieved from client 1 or the server (not a good choice)...
You're a bit out of luck with this because Node doesn't support the Blob interface and so any data you send or receive in Binary with Node is just Binary. You would have to have something that knew how to interpret a Blob object.
Here's an idea, and let me know if this works. Reading through the documentation for websockets\ws it says it supports sending and receiving ArrayBuffers. Which means you can use TypedArrays.
Here's where it gets nasty. You set a certain fixed n number of bytes at the beginning of every TypedArray to signal the mime type encoded in utf8 or what have you, and the rest of your TypedArray contains your file's bytes.
I would recommend using UInt8Array because utf8 characters are 8 bits long and your text will probably be readable when encoded that way. As for the file bits you'll probably just end up writing those down somewhere and appending an ending to it.
Also note, this method of interpretation works both ways whether from Node or in the Browser.
This solution is really just a form of type casting and you might get some unexpected results. The fixed length of your mime type field is crucial.
Here it is illustrated. Copy, paste, set the image file to whatever you want and then run that. You'll see the mime type I set pop out.
var fs = require('fs');
//https://stackoverflow.com/questions/8609289/convert-a-binary-nodejs-buffer-to-javascript-arraybuffer
function toUint8Array(buffer) {
var ab = new ArrayBuffer(buffer.length);
var array = new Uint8Array(ab);
for (var i = 0; i < buffer.length; ++i) {
array[i] = buffer[i];
}
return array;
}
//data is a raw Buffer object
fs.readFile('./ducklings.png', function (err, data) {
var mime = new Buffer('image/png');
var allBuffed = Buffer.concat([mime, data]);
var array = toUint8Array(allBuffed);
var mimeBytes = array.subarray(0,9); //number of characters in mime Buffer
console.log(String.fromCharCode.apply(null, mimeBytes));
});
Here's how you do it on the client side:
SOLUTION A: GET A PACKAGE
Get buffer, an implementation of Node's Buffer API for browsers. The solution to concatenate Byte buffers will work exactly as before. You can append fields like To: and what not as well. The way you format your headers in order to best serve your clients will be an evolving process I'm sure.
SOLUTION B: OLD SCHOOL
STEP 1: Convert your Blob to an ArrayBuffer
Notes: How to convert a String to an ArrayBuffer
var fr = new FileReader();
fr.addEventListener('loadend', function () {
//Asynchronous action in part 2.
var message = concatenateBuffers(headerStringAsBuffer, fr.result);
ws.send(message);
});
fr.readAsArrayBuffer(blob);
STEP 2: Concatenate ArrayBuffers
function concatenateBuffers(buffA, buffB) {
var byteLength = buffA.byteLength + buffB.byteLength;
var resultBuffer = new ArrayBuffer(byteLength);
//wrap ArrayBuffer in a typedArray/view
var resultView = new Uint8Array(resultBuffer);
var viewA = new Uint8Array(resultBuffer);
var viewB = new Uint8Array(resultBuffer);
//Copy 8 bit integers AKA Bytes
resultView.set(viewA);
resultView.set(viewB, viewA.byteLength);
return resultView.buffer
}
STEP 3: Receive and Reblob
I'm not going to repeat how to convert the concatenated String bytes back into a string because I've done it in the server example, but for turning the file bytes into a blob of your mime type is fairly simple.
new Blob(buffer.slice(offset, buffer.byteLength), {type: mimetype});
This Gist by robnyman goes into further details on how you would use an image transmitted via XHR, put it into localstorage, and use it in an image tag on your page.
I liked #Breedly's idea of prepending a fixed length byte array to indicate mime type of the ArrayBuffer so I created this npm package that I use when dealing with websockets but maybe others' might find it useful.
Example usage
const {
arrayBufferWithMime,
arrayBufferMimeDecouple
} = require('arraybuffer-mime')
// some image array buffer
const uint8 = new Uint8Array(1)
uint8[0] = 1
const ab = uint8.buffer
const mime = 'image/png'
const abWithMime = arrayBufferWithMime(ab, mime)
const {mime, arrayBuffer} = arrayBufferMimeDecouple(abWithMime)
console.log(mime) // "image/png"
console.log(arrayBuffer) // ArrayBuffer
Related
Working on a Javascript-based file encryption program. It works great for text files, but I'm trying to do other data (images, etc) and the decrypted file isn't matching. In my program, I use the following code to read the file that the user provides:
let reader = new FileReader();
let file = UI.file.input.files[0];
reader.readAsText(file, 'UTF-8');
The reader then adds that to a Javascript object that contains other information, like the date and file name. Then, that's stringified using JSON.stringify() and encrypted. When the encryption is completed, the file gets saved as a different file format with the encrypted JSON inside of it. The code for that is:
let message = // ENCRYPTED FILE STRING
let file = new Blob([message], {
type: 'text/plain'
});
let url = URL.createObjectURL(file);
The url is then attached to a link element on the page. That's all working fine.
To decrypt it, the file is again provided by the user and runs through the same reader as used above. It's decrypted successfully and the resulting string is again put into an object using JSON. Up to that point, it works exactly as it's supposed to. It works fine if what I'm decrypting is a text file, but if I do an image file it all goes bad.
To save the decrypted file, I use this code:
let message = // DECRYPTED DATA CONVERTED TO AN OBJECT
let fileName = message.name;
let fileType = message.type;
let file = new Blob([message.data], {
type: fileType
});
let url = URL.createObjectURL(file);
The original file and and original file type are both pulled from the file before it's encrypted and added to the object that has the file data. Like I said, there's no problem with the encryption or decryption process. I've used a HEX viewer to check and the string that is being put into the encryption process is identical to the one coming out. I'm guessing the issue is somewhere in my final block of code.
My best guess is something to do with encoding, although I'm not sure exactly what. Any help would be greatly appreciated.
You can use FileReader.readAsDataURL and strip the data:*/*;base64, from the string. Then you'll have a base64 encoded string that you can put into JSON.
I'm having a hard time figuring out how to send a file over a WebSocket using Google Protocol buffers.
My message structure for the buffer is
message FileData_m {
required uint32 block = 1; // File starting offset
required bytes data = 2; // Size of 65536 for blocks
}
The idea is to break up the file into blocks and send it over a WebSocket. Currently I'm running a Node.js server that handles interactions between host and server I just don't know how to properly send the file in a binary manner.
Any help and/or pointing me in the right direction would be very helpful!
The solution to this was to make sure that my data was being sent using an array buffer
var dataToSend = new ArrayBuffer(65536);
dataToSend = file; // File that was uploaded from file chooser
Then when creating your protobuf message just use that array buffer as your source of data
var FileData = new FileData({
"block": 0,
"data": dataToSend
});
I have a base64 string which I decoded and wishes to allow the user to save this as a file. In particular, when I check the length of decodedContent, it's 11271 bytes.
var content = messageObj['data'];
var decodedContent = atob(content);
console.log(decodedContent.length);
Then I used
var blob = new Blob([decodedContent], {type: 'application/octet-stream'});
window.open((window.URL || window.webkitURL).createObjectURL(blob));
To prompt the user to save decodedContent. When I check the file size saved, it says 16892 bytes, which is different from what is stated above. Any idea why?
Content is a base64 encoded tar-ball file sent from the server.
for i ,l in enumerate(to_download):
if i == 1:
break
last_index = l.rfind('|')
download_path = l[last_index+1:].strip()
mda_url = '%s/%s'%(root_url, download_path)
logger.debug('Downloading file %s/%s at %s', i, len(to_download), mda_url)
mda_req = urllib2.Request(mda_url)
mda_response = urllib2.urlopen(mda_req)
f = StringIO.StringIO(mda_response.read())
replace_path = mda_url.replace('/', '_')
ti = TarInfo("%s.txt" %replace_path)
ti.size = f.len
tar.addfile(ti, f)
tar.close()
tar_file.close()
with open("/Users/carrier24sg/Desktop/my.tar", 'rb') as f:
tar_str = f.read()
logger.info("Completed downloading all the requested files..")
return tar_str
UPDATE:
Narrowed down to the problem being with either var decodedContent = atob(content); or var blob = new Blob([decodedContent], {type: 'application/octet-stream'});
Finally I managed to use the #Jeremy Bank's answer here. His first answer solves the issue of content length being different, but when I check the checksum, the content doesn't seem to tally. Only using his second answer's function b64toBlob did I get to resolve this. However, I'm still not sure what is wrong here, so I'm hoping someone can shed some light to this.
I think the problem is that atomb() gives back a basic base64 version of the file. when you ask for the size of it, it will return the bytes it contains.
When you make a blob from the base64 variable and ask for its size, it will return how much space it will fill up on your computer.
The two things are different because file storing size and coded size are not the same thing. And they differ on different platforms as well.
I had a problem when try to saving a binary file by IE, ADODB.Stream on client.
Assume the the client's browser has full permission to write file to hard disk.
Here is my client code witten by javascript.:
var base64_encoded_string = "base64 string of a binary file";
var data = window.atob(base64_encoded_string);
var stream = new ActiveXObject("ADODB.Stream");
stream.Type = 1; // text
stream.Open();
stream.Write(data); //Problem in here
stream.SaveToFile("D:\\Whatever.pfx", 2);
stream.Close();
As I marked problem come from writing binary data. I always got error:
"Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another"
Whatever I format or change data variable to array of bytes, blob ...
Please help me how to input data to write binary file in this circumstance.
You need to use WriteText
stream.WriteText(data);
I'm trying to read and write raw data from files using Mozilla's add-on SDK. Currently I'm reading data with something like:
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available(),{charset:"UTF-8"});
callback(data, status, nsiFile);
});
}
This works for text files, but when I start messing with raw bytes outside of Unicode's normal range, it doesn't work. For example, if a file contains the byte 0xff, then that byte and anything past that byte isn't read at all. Is there any way to read (and write) raw data using the SDK?
You've specified an explicit charset in the options to NetUtil.readInputStream.
When you omit the charset option, then the data will be read as raw bytes. (Source)
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
// Do not specify a charset at all!
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available());
callback(data, status, nsiFile);
});
}
The suggestion to use io/byte-streams is OK as well, but keep in mind that that SDK module is still marked experimental, and that using ByteReader via io/file as the example suggests is not a good idea because this would be sync I/O on the main thread.
I don't really see the upside, as you'd use NetUtil anyway.
Anyway, this should work:
const {ByteReader} = require("sdk/io/byte-streams");
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var reader = new ByteReader(inputStream);
var data = reader.read(inputStream);
reader.close();
callback(data, status, nsiFile);
});
}
Also, please keep in mind that reading large files like this is problematic. Not only will the whole file buffered in memory, obviously, but:
The file is read as a char (byte) array first, so there will be a temporary buffer in the stream of at least file.size length (via asyncFetch).
Both NetUtil.readInputStreamToString and ByteReader will use another char (byte) array to read the result into from the inputStream, but ByteReader will do that in 32K chunks, while NetUtil.readInputStreamToString, will use a big buffer of file.length.
The data is then read into the resulting jschar/wchar_t (word) array aka. Javascript string, i.e. you need file.size * 2 bytes in memory at least.
E.g., reading a 1MB file would require more than fileSize * 4 = 4MB memory (NetUtil.readInputStreamToString) and/or more than fileSize * 3 = 3MB memory (ByteReader) during the read operation. After the operation, 2MB of that memory will be still alive to store the resulting data in a Javascript string.
Reading a 1MB file might be OK, but a 10MB file might be already problematic on mobile (Firefox for Android, Firefox OS) and a 100MB would be problematic even on desktop.
You can also read the data directly into an ArrayBuffer (or Uint8Array), which has more efficient storage for byte arrays than a Javascript string and avoid the temporary buffers of NetUtil.readInputStreamToString and/or ByteReader.
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var bs = Cc["#mozilla.org/binaryinputstream;1"].
createInstance(Ci.nsIBinaryInputStream);
bs.setInputStream(inputStream);
var len = inputStream.available();
var data = new Uint8Array(len);
reader.readArrayBuffer(len, data.buffer);
bs.close();
callback(data, status, nsiFile);
});
}
PS: The MDN documentation might state something about "iso-8859-1" being the default if the charset option is omitted in the NetUtil.readInputStreamToString call, but the documentation is wrong. I'll fix it.