Develop Reference

Send XMLHttpRequest data in chunks or as ReadableStream to reduce memory usage for large data

I've been trying to use JS's XMLHttpRequest Class for file uploading. I initially tried something like this:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const rawFileData = await file.arrayBuffer();
request.send(rawFileData);
The above code works (yay!), and sends the raw binary data of the file to my server.
However...... It uses a TON of memory (because the whole file gets stored in memory, and JS isn't particulary memory friendly)... I found out that on my machine (16GB RAM), I couldn't send files larger than ~100MB, because JS would allocate too much memory, and the Chrome tab would crash with a SIGILL code.
So, I thought it would be a good idea to use ReadableStreams here. It has good enough browser compatibility in my case (https://caniuse.com/#search=ReadableStream) and my TypeScript compiler told me that request.send(...) supports ReadableStreams (I later came to the conclusion that this is false). I ended up with code like this:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const fileStream = file.stream();
request.send(fileStream);
But my TypeScript compiler betrayed me (which hurt) and I recieved "[object ReadableStream]" on my server ಠ_ಠ.
I still haven't explored the above method too much, so I'm not sure if there might be a way to do this. I'd also appreciate help on this very much!
Splitting the request in chunk would be an optimal solution, since once a chunk has been sent, we can remove it from memory, before the whole request has even been recieved.
I have searched and searched, but haven't found a way to do this yet (which is why I'm here...). Something like this in pseudocode would be optimal:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const fileStream = file.stream();
const fileStreamReader = fileStream.getReader();
const sendNextChunk = async () => {
const chunk = await fileStreamReader.read();
if (!chunk.done) { // chunk.done implies that there is no more data to be read
request.writeToBody(chunk.value); // chunk.value is a Uint8Array
} else {
request.end();
break;
}
}
sendNextChunk();
I'd like to expect this code to send the request in chunks and end the request when all chunks are sent.
The most helpful resource I tried, but didn't work:
Method for streaming data from browser to server via HTTP
Didn't work because:
I need the solution to work in a single request
I can't use RTCDataChannel, it must be in a plain HTTP request (is there an other way to do this than XMLHttpRequest?)
I need it to work in modern Chrome/Firefox/Edge etc. (no IE support is fine)
Edit: I don't want to use multipart-form (FormData Class). I want to send actual binary data read from the filestream in chunks.

You can't do this with XHR afaik. But the more modern fetch API does support passing a ReadableStream for the request body. In your case:
const file = thisFunctionReturnsAFileObject();
const response = await fetch('/upload-file', {
method: 'POST',
body: file.stream(),
});
However, I'm not certain whether this will actually use chunked encoding.

You are facing a Chrome bug where they do set an hard-limit of 256MB to the size of the ArrayBuffer that can be sent.
But anyway, sending an ArrayBuffer will create a copy of the data, so you should rather send your data as a File directly, since this will only read the File exactly like you wanted it to be, as a stream by small chunks.
So taking your first code block that would give
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
request.send(file);
Ans this will work in Chrome too, even with few Gigs files. The only limit you would face here would be before, when you'd do whatever processing you are doing on that File.
Regarding posting ReadableStreams, this will eventually come, but as of today July the 13th of 2020, only Chrome has started working on its implementation, and we web-devs still can't play with it, and specs are still having hard times to come with something stable.
But it's not a problem for you, since you would not win anything doing so anyway. Posting a ReadableStream made from a static File is useless, both fetch and xhr will do this internally already.

Javascript: AES encryption is slow

I use forge as encryption library for both the gulp script (that performs encryption) and the front-end one (where in-browser decryption happens).
The computer is a i5-6200U w/ 16GB ram and takes about 10 seconds for either symmetric enc. or decryption of a 15MB json file.
My real issue is the decryption time being too long for users (multiple files to load and decrypt take 30s+ on this system).
I'm certainly missing some key element (buffer or... whatever my lack of experience in the domain might miss). Is there something obviously wrong in the following code? Thanks for your attention.
Obtaining the data
function logic(url){
return new Promise( (resolve, reject) => {
var xhr = new XMLHttpRequest();
xhr.onload = function (event) {
resolve(xhr.response);
};
xhr.onreject = function (err) {
reject(err);
}
xhr.open('GET', url);
xhr.send();
});
}
Decrypting the data
load('data/dicom.json').then( bytes => {
const tIn = new Date().getTime();
const forge = getForge();
const pwd = "aStringPassword";
const iv = getInitVector();
const salt = getSalt();
const key = forge.pkcs5.pbkdf2(pwd, salt, 100, 16);
var decipher = forge.cipher.createDecipher('AES-CBC', key);
decipher.start({iv: iv});
decipher.update(forge.util.createBuffer(bytez));
decipher.finish();
const clear = decipher.output.getBytes();
const tOut = new Date().getTime();
console.log(`decrypted in ${(tOut - tIn) / 1000 }s`); // 10s for 15MB json file
return clear ;
});

Forge up to at least 0.7.1 uses strings for its internal buffer implementation. (This code pre-dates modern buffer APIs and future Forge versions will use newer APIs.) This has some consequences when processing large inputs. As output string buffers get larger during processing the internal JavaScript VM can slow down just doing string processing. One way to avoid this is to use streaming capabilities of the Forge API such that string buffer manipulations use larger data chunks. The input can be processed in chunks with update() and the output manually built during this process. Getting the output chunks with getBytes() will clear the output buffer and allow the Forge internals to operate more efficiently. Building your own output with those chunks does not have the same performance impact.
A test was written to check decrypting large buffers with a single update() call, many update() calls, and with native node APIs. As the input size increases from 1M to 20M the slowdown of a single update() call vs native node APIs goes from ~8x to well over 50x! But if you use streaming processing the slowdown can be only ~4.6x and not noticeably dependent on input size! For your 15M input size this equates to ~0.75s vs ~10.31s. For comparison node is ~0.15s and the WebCrypto API is likely similar. (Timing from a i7-4790K)
A test was also written to see how the chunk size effected the results. When processing large inputs it seems ~64k is about optimal using node.js. This could be different depending on the JavaScript VM and other factors. The key takeaway is that using streaming with any chunk size (even 1M!) offers improvements to avoid linear buffer slowdowns as input size increases.
An example with improved and more constant performance:
const decipher = forge.cipher.createDecipher('AES-CBC', key);
decipher.start({iv: iv});
const length = bytes.length;
const chunkSize = 1024 * 64;
let index = 0;
let clear = '';
do {
clear += decipher.output.getBytes();
const buf = forge.util.createBuffer(bytes.substr(index, chunkSize));
decipher.update(buf);
index += chunkSize;
} while(index < length);
const result = decipher.finish();
assert(result);
clear += decipher.output.getBytes();
A secondary issue with the code is that you want to avoid doing CPU intensive code on the main JS thread. The streaming API will allow you to run each update() call via setImmediate() (if available) or setTimeout(). This will allow the user to interact with the browser while processing is going on. If you can also stream the input data fetch then you could start processing while data is coming over the network. Updating the original code to do this is left as an exercise for the reader. A smaller chunk size might help UI interactivity in this situation.
Lastly it should be noted that native APIs are likely to always be higher performance than Forge. The current WebCrypto API does not offer a streaming API but its performance may be high enough that it may not be an issue in this use case. It's worth trying and seeing what works best.
Also note you should check the decipher.finish() return value.
Encryption has the same buffer issues with large inputs and can use the same pattern as the code above.
For those reading this in the future: newer web APIs and Forge improvements may have greatly changed performance results.

No clue so far, but the code run on the benchmark page, as shown below, runs way slower:
(source available here)
/*** encrypt */
var input = forge.util.createBuffer("plaintext");
var cipher = forge.aes.startEncrypting(key, iv);
cipher.update(input);
var status = cipher.finish();
var ciphertext = cipher.output.data;
Test run via web page: 35ms, same data in my gulp script: 165ms. No clue why, so far.

Is it possible to retrieve parts of a large binary file via Javascripts xhr based on offset and length?

I'm writing a handler for accessing binary data. Currently I'm doing this:
...
d = new FileReader();
d.onload = function (e) {
var response_buffer, data, byte_len, temp_float;
response_buffer = e.target.result;
data = new DataView(response_buffer);
byte_len = data.byteLength;
// create empty placeholder for binary data received
tmp_data = new Float32Array(byte_len / Float32Array.BYTES_PER_ELEMENT);
len = tmp_data.length;
// Pre-parse
for (i = 0; i < len; i += 1) {
tmp_data[i] = data.getFloat32(i * Float32Array.BYTES_PER_ELEMENT, true);
}
...
Which works fine and "pre-processes" my fetched data (file/http) into my tmp_data array.
However I need to be able to handle large files as well (like 1GB+). My idea was to try and fetch only part of the file because from the binary structure I would know exactly what offsets I would have to fetch.
Question:
Is it possible to XHR for a "range" of bytes from a large file if I know offset and length instead of having to fetch the whole file?
Thanks!

If your server respects it, you can set the Range: request header, which does exactly that.

You could also implement your own polling strategy where you ask for the data in chunks of x bytes and pass along an offset. Similar to how an endless scroll works on a web page.

Detecting JPG CMYK image clientside with jquery or js

Is there a way to use jQuery or Javascript to detect the colorspace of a JPG image that is being uploaded via file input? To be more accurate I need a way to kick back a response when a user attempts to upload a CMYK colorspaced JPG. I would like to catch this on the client side if at all possible. I would prefer to do this using jQuery/js, but I'm definitely open to libraries that may have this functionality. Thanks in advance for any help.

I've looked around a bit, and it seems like JavaScript could be powerful enough to do this, albeit possibly very slowly.
Here is a stackoverflow question that deals with using JavaScript for face detection: Any library for face recognition in JavaScript?
I know that doesn't answer the question specifically, but it does demonstrate that JavaScript could potentially be leveraged for what you want.
Here is a question that deals with using JS along with the canvas object to get the average color of an image: Get average color of image via Javascript
There are some answers there that may be helpful for you. One of the answers there links to a library called Color Thief that could do what you want: https://github.com/lokesh/color-thief
If no one has already built what you want, my takeaway here is that it would be a laborious task, which may or may not yield results fast enough to be useful.

I had a comment that you could do a jquery ajax call to get the raw data. I don't think that is actually true, as there is no way to get the raw data. However, many browsers support the "arraybuffer" responseType of an XML http request, and you can use that. Here is a code snippet:
var img_url = "/images/testimg.jpg";
var req = new XMLHttpRequest();
req.open("GET", img_url, true);
req.responseType = "arraybuffer";
req.onload = function (event)
{
var buf = req.response;
var num_channels = null;
if (buf)
{
var img = new Uint8Array(buf);
for (var i=0; i<img.length-9; i++)
{
if (img[i]==0xFF && (img[i+1]==0xC0 || img[i+1]==0xC1))
{
var num_channels = img[i+9];
break;
}
}
}
if (num_channels==4)
console.log("This is a four-channel image.");
};
req.send(null);
Obviously, you'd want to actually do something with the result, and there should be better checking to make sure this IS a JPEG and not a random collection of bytes. Furthermore, this only will work on Baseline and Extended JPEGs, not progressive, arithmetic encoding, JPEG 2000, or anything else.
This doesn't really test if the colorspace is CMYK (or YCCK), just that it has four components, but the basic idea is there.

Can I save images or load images from the client, instead of sending them every time?

I'm using node.js and socket.io, and I'm making a game. There are quite a few sprites I need to send to the client, so to lower the bandwidth I thought about storing the data on the clients computer. For now, the basic idea would be that it just downloads once, and then gets saved to the clients computer. But how do I gain control of the client's files? And how can I save pictures/general data on the client's computer?
Here is what I'm doing:
self.sources =
[
[//trees, 0
"res/images/enviornment/trees/tree_1_1.png",//0
"res/images/enviornment/trees/tree_1_2.png",//1
"res/images/enviornment/trees/tree_1_3.png",//2
"res/images/enviornment/trees/tree_1_4.png",//3
], //there are many other pictures, but this is just the concept
];
self.images = new Array();
for (i = 0; i < self.sources.length; i++)
{
self.images.push([]);
for(t = 0; t < self.sources[i].length; t++)
{
self.images[i][t] = new Image();
self.images[i][t].src = self.sources[i][t];
}
}

Browsers will cache the images, but the type of expiration is chosen by the server, and depending on the type, it may also be up to the server to properly handle requests for files where the client's cache is up to date.
If you want it to "just work," use Express.
To learn more about http caching, try this: http://betterexplained.com/articles/how-to-optimize-your-site-with-http-caching/

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9

You should take a look at the html5 offline applications spec. Beware, it's not supported everywhere yet.