I use forge as encryption library for both the gulp script (that performs encryption) and the front-end one (where in-browser decryption happens).
The computer is a i5-6200U w/ 16GB ram and takes about 10 seconds for either symmetric enc. or decryption of a 15MB json file.
My real issue is the decryption time being too long for users (multiple files to load and decrypt take 30s+ on this system).
I'm certainly missing some key element (buffer or... whatever my lack of experience in the domain might miss). Is there something obviously wrong in the following code? Thanks for your attention.
Obtaining the data
function logic(url){
return new Promise( (resolve, reject) => {
var xhr = new XMLHttpRequest();
xhr.onload = function (event) {
resolve(xhr.response);
};
xhr.onreject = function (err) {
reject(err);
}
xhr.open('GET', url);
xhr.send();
});
}
Decrypting the data
load('data/dicom.json').then( bytes => {
const tIn = new Date().getTime();
const forge = getForge();
const pwd = "aStringPassword";
const iv = getInitVector();
const salt = getSalt();
const key = forge.pkcs5.pbkdf2(pwd, salt, 100, 16);
var decipher = forge.cipher.createDecipher('AES-CBC', key);
decipher.start({iv: iv});
decipher.update(forge.util.createBuffer(bytez));
decipher.finish();
const clear = decipher.output.getBytes();
const tOut = new Date().getTime();
console.log(`decrypted in ${(tOut - tIn) / 1000 }s`); // 10s for 15MB json file
return clear ;
});
Forge up to at least 0.7.1 uses strings for its internal buffer implementation. (This code pre-dates modern buffer APIs and future Forge versions will use newer APIs.) This has some consequences when processing large inputs. As output string buffers get larger during processing the internal JavaScript VM can slow down just doing string processing. One way to avoid this is to use streaming capabilities of the Forge API such that string buffer manipulations use larger data chunks. The input can be processed in chunks with update() and the output manually built during this process. Getting the output chunks with getBytes() will clear the output buffer and allow the Forge internals to operate more efficiently. Building your own output with those chunks does not have the same performance impact.
A test was written to check decrypting large buffers with a single update() call, many update() calls, and with native node APIs. As the input size increases from 1M to 20M the slowdown of a single update() call vs native node APIs goes from ~8x to well over 50x! But if you use streaming processing the slowdown can be only ~4.6x and not noticeably dependent on input size! For your 15M input size this equates to ~0.75s vs ~10.31s. For comparison node is ~0.15s and the WebCrypto API is likely similar. (Timing from a i7-4790K)
A test was also written to see how the chunk size effected the results. When processing large inputs it seems ~64k is about optimal using node.js. This could be different depending on the JavaScript VM and other factors. The key takeaway is that using streaming with any chunk size (even 1M!) offers improvements to avoid linear buffer slowdowns as input size increases.
An example with improved and more constant performance:
const decipher = forge.cipher.createDecipher('AES-CBC', key);
decipher.start({iv: iv});
const length = bytes.length;
const chunkSize = 1024 * 64;
let index = 0;
let clear = '';
do {
clear += decipher.output.getBytes();
const buf = forge.util.createBuffer(bytes.substr(index, chunkSize));
decipher.update(buf);
index += chunkSize;
} while(index < length);
const result = decipher.finish();
assert(result);
clear += decipher.output.getBytes();
A secondary issue with the code is that you want to avoid doing CPU intensive code on the main JS thread. The streaming API will allow you to run each update() call via setImmediate() (if available) or setTimeout(). This will allow the user to interact with the browser while processing is going on. If you can also stream the input data fetch then you could start processing while data is coming over the network. Updating the original code to do this is left as an exercise for the reader. A smaller chunk size might help UI interactivity in this situation.
Lastly it should be noted that native APIs are likely to always be higher performance than Forge. The current WebCrypto API does not offer a streaming API but its performance may be high enough that it may not be an issue in this use case. It's worth trying and seeing what works best.
Also note you should check the decipher.finish() return value.
Encryption has the same buffer issues with large inputs and can use the same pattern as the code above.
For those reading this in the future: newer web APIs and Forge improvements may have greatly changed performance results.
No clue so far, but the code run on the benchmark page, as shown below, runs way slower:
(source available here)
/*** encrypt */
var input = forge.util.createBuffer("plaintext");
var cipher = forge.aes.startEncrypting(key, iv);
cipher.update(input);
var status = cipher.finish();
var ciphertext = cipher.output.data;
Test run via web page: 35ms, same data in my gulp script: 165ms. No clue why, so far.
Related
I am using native javascript websocket in browser and we have an application hosted on AWS where every request goes through API gateway.
In some cases, request data is going upto 60kb, and then my websocket connection is closing automatically. In AWS documentation, I found out below explanation of this issue
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-known-issues.html
API Gateway supports message payloads up to 128 KB with a maximum frame size of 32 KB. If a message exceeds 32 KB, you must split it into multiple frames, each 32 KB or smaller. If a larger message is received, the connection is closed with code 1009.
I tried to find how I can split a message in multiple frames using native javascript websocket but could not find any config related to frames in documentation or anywhere else
Although I find something related to message fragmentation but it seems like a custom solution that I need to implement at both frontend and backend
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#message_fragmentation
As far as I know, you cannot do this using the JS AWS SDK "postToConnection" API. Best you can do is write your own poor's man fragmentation and send the chunks as independent messages.
const splitInChunks =
(sizeInBytes: number) =>
(buffer: Buffer): Buffer[] => {
const size = Buffer.byteLength(buffer);
let start = 0;
let end = sizeInBytes;
const chunks: Buffer[] = [];
do {
chunks.push(buffer.subarray(start, end));
start += sizeInBytes;
end += sizeInBytes;
} while (start < size);
return chunks;
};
Where sizeInBytes must be smaller than 32KB. Then you iterate over the chunks:
await Promise.all(chunks.map(c => apiGatewayClient.postToConnection({ data: JSON.stringify(c), connectionId: myConnectionId })
Which may run into rate limits depending on the number of chunks, so consider sending the requests serially and not in parallel
Final remark: Buffer.prototype.subarray is very efficient because it does not reallocate memory: the new chunks point at the same memory space of the original buffer. Think pointer arithmetic in C.
I have a vanilla JS script running in-browser (Chrome, and only Chrome needs to be supported - if browser support is of any importance).
I want to offload a 15 MB gzip file to a web worker and unzip that file asynchronously, then return the uncompressed data back to the main thread, in order not to freeze the main application thread during the decompression procedure.
When unzipping in the main thread, I'm using the JSXCompressor library, and that works fine. However, as this library references the window object, which isn't accessible from a worker context, I can't use the same library injected into the worker code (running the decompression raises an exception on the first line of the library mentioning "window", saying it's undefined).
The same is true for other JS libraries I've managed to dig up in an afternoon of googling, like zlib or the more modern Pako. They all in one way or another seem to reference a DOM element, which raises exceptions when used in a web worker context.
So my question is - is anyone aware of a way I can pull this off, either by explaining to me what I seem to be getting wrong, through a hack, or by providing me with a link to a JS library that can function in this use case (I need only decompression, standard gzip)?
Edit: I'm also interested in any hack that can leverage built-in browser capabilities for ungzipping, as is done for HTTP requests.
Thanks a bunch.
I've authored a library fflate to accomplish exactly this task. It offers asynchronous versions of every compression/decompression method it supports, but rather than running in an event loop, the library delegates the processing to a separate thread. You don't need to manually create a worker or specify paths to the package's internal workers, since it generates them on-the-fly.
import { gunzip } from 'fflate';
// Let's suppose you got a File object (from, say, an input)
const reader = new FileReader();
reader.onloadend = () => {
const typedArrayUncompressed = new Uint8Array(reader.result);
gunzip(typedArrayUncompressed, (err, gzippedResult) => {
// This is a Uint8Array
console.log('Compressed output:', gzippedResult);
});
}
reader.readAsArrayBuffer(fileObject);
Effectively, you need to convert the input format to a Uint8Array, then convert the output format to whatever you want to use. For instance, FileReader is the most cross-platform solution for files, fflate.strToU8 and fflate.strFromU8 work for string conversions.
P.S. This is actually still about as fast as the native CompressionStream solution from my tests, but will work in more browsers. If you want streaming support, use fflate's AsyncGunzip stream class.
There is a new web API Compression streams proposal, which is already implemented in Chrome and which does exactly this: asynchronously compress/decompress data.
It should support both deflate and gzip algorithms, and should use native implementations -> faster than any lib.
So in Chrome you can simply do:
if( "CompressionStream" in window ) {
(async () => {
// To be able to pass gzipped data in stacksnippet we host it as a data URI
// that we do convert to a Blob.
// The original file is an utf-8 text "Hello world"
// which is way bigger once compressed, but that's an other story ;)
const compressed_blob = await fetch("data:application/octet-stream;base64,H4sIAAAAAAAAE/NIzcnJVyjPL8pJAQBSntaLCwAAAA==")
.then((r) => r.blob());
const decompressor = new DecompressionStream("gzip");
const decompression_stream = compressed_blob.stream().pipeThrough(decompressor);
const decompressed_blob = await new Response(decompression_stream).blob();
console.log("decompressed:", await decompressed_blob.text());
})().catch(console.error);
}
else {
console.error("Your browser doesn't support the Compression API");
}
Obviously, this is also be available in Web Workers, but since the API is designed as entirely asynchronous, and making use of Streams, browsers should theoretically already be able to outsource all the hard work on an other thread on their own anyway.
Now, this is still a bit of a future solution, and for today you still might want to use a library instead.
However, we don't do library recommendations here, but I should note that I personally do use pako in Web Workers in a daily basis, with no problem and I don't see why a compression library would ever need the DOM, so I supsect you are doing something wrong™.
(async() => {
const worker_script = `
importScripts("https://cdnjs.cloudflare.com/ajax/libs/pako/1.0.11/pako_inflate.min.js");
self.onmessage = async (evt) => {
const file = evt.data;
const buf = await file.arrayBuffer();
const decompressed = pako.inflate(buf);
// zero copy
self.postMessage(decompressed, [decompressed.buffer]);
};
`;
const worker_blob = new Blob([worker_script], { type: "application/javascript" });
const worker_url = URL.createObjectURL(worker_blob);
const worker = new Worker(worker_url);
const compressed_blob = await fetch("data:application/octet-stream;base64,H4sIAAAAAAAAE/NIzcnJVyjPL8pJAQBSntaLCwAAAA==")
.then((r) => r.blob());
worker.onmessage = ({ data }) => {
console.log("received from worker:", new TextDecoder().decode(data));
};
worker.postMessage(compressed_blob);
})().catch(console.error);
I've been trying to use JS's XMLHttpRequest Class for file uploading. I initially tried something like this:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const rawFileData = await file.arrayBuffer();
request.send(rawFileData);
The above code works (yay!), and sends the raw binary data of the file to my server.
However...... It uses a TON of memory (because the whole file gets stored in memory, and JS isn't particulary memory friendly)... I found out that on my machine (16GB RAM), I couldn't send files larger than ~100MB, because JS would allocate too much memory, and the Chrome tab would crash with a SIGILL code.
So, I thought it would be a good idea to use ReadableStreams here. It has good enough browser compatibility in my case (https://caniuse.com/#search=ReadableStream) and my TypeScript compiler told me that request.send(...) supports ReadableStreams (I later came to the conclusion that this is false). I ended up with code like this:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const fileStream = file.stream();
request.send(fileStream);
But my TypeScript compiler betrayed me (which hurt) and I recieved "[object ReadableStream]" on my server ಠ_ಠ.
I still haven't explored the above method too much, so I'm not sure if there might be a way to do this. I'd also appreciate help on this very much!
Splitting the request in chunk would be an optimal solution, since once a chunk has been sent, we can remove it from memory, before the whole request has even been recieved.
I have searched and searched, but haven't found a way to do this yet (which is why I'm here...). Something like this in pseudocode would be optimal:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const fileStream = file.stream();
const fileStreamReader = fileStream.getReader();
const sendNextChunk = async () => {
const chunk = await fileStreamReader.read();
if (!chunk.done) { // chunk.done implies that there is no more data to be read
request.writeToBody(chunk.value); // chunk.value is a Uint8Array
} else {
request.end();
break;
}
}
sendNextChunk();
I'd like to expect this code to send the request in chunks and end the request when all chunks are sent.
The most helpful resource I tried, but didn't work:
Method for streaming data from browser to server via HTTP
Didn't work because:
I need the solution to work in a single request
I can't use RTCDataChannel, it must be in a plain HTTP request (is there an other way to do this than XMLHttpRequest?)
I need it to work in modern Chrome/Firefox/Edge etc. (no IE support is fine)
Edit: I don't want to use multipart-form (FormData Class). I want to send actual binary data read from the filestream in chunks.
You can't do this with XHR afaik. But the more modern fetch API does support passing a ReadableStream for the request body. In your case:
const file = thisFunctionReturnsAFileObject();
const response = await fetch('/upload-file', {
method: 'POST',
body: file.stream(),
});
However, I'm not certain whether this will actually use chunked encoding.
You are facing a Chrome bug where they do set an hard-limit of 256MB to the size of the ArrayBuffer that can be sent.
But anyway, sending an ArrayBuffer will create a copy of the data, so you should rather send your data as a File directly, since this will only read the File exactly like you wanted it to be, as a stream by small chunks.
So taking your first code block that would give
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
request.send(file);
Ans this will work in Chrome too, even with few Gigs files. The only limit you would face here would be before, when you'd do whatever processing you are doing on that File.
Regarding posting ReadableStreams, this will eventually come, but as of today July the 13th of 2020, only Chrome has started working on its implementation, and we web-devs still can't play with it, and specs are still having hard times to come with something stable.
But it's not a problem for you, since you would not win anything doing so anyway. Posting a ReadableStream made from a static File is useless, both fetch and xhr will do this internally already.
I've been trying to create polyphonic WAV playback with node.js on raspberry pi 3 running latest raspbian:
shelling out to aplay/mpg123/some other program - allows me to only play single sound at once
I tried combination of https://github.com/sebpiq/node-web-audio-api and https://github.com/TooTallNate/node-speaker (sample code below) but audio quality is very low, with a lot of distortions
Is there anything I'm missing here? I know I could easily do it in another programming language (I was able to write C++ code with SDL, and Python with pygame), but the question is if it's possible with node.js :)
Here's my current web-audio-api + node-speaker code:
var AudioContext = require('web-audio-api').AudioContext;
var Speaker = require('speaker');
var fs = require('fs');
var track1 = './tracks/1.wav';
var track2 = './tracks/1.wav';
var context = new AudioContext();
context.outStream = new Speaker({
channels: context.format.numberOfChannels,
bitDepth: context.format.bitDepth,
sampleRate: context.format.sampleRate
});
function play(audioBuffer) {
if (!audioBuffer) { return; }
var bufferSource = context.createBufferSource();
bufferSource.connect(context.destination);
bufferSource.buffer = audioBuffer;
bufferSource.loop = false;
bufferSource.start(0);
}
var audioData1 = fs.readFileSync(track1);
var audioData2 = fs.readFileSync(track2);
var audioBuffer1, audioBuffer2;
context.decodeAudioData(audioData1, function(audioBuffer) {
audioBuffer1 = audioBuffer;
if (audioBuffer1 && audioBuffer2) { playBoth(); }
});
context.decodeAudioData(audioData2, function(audioBuffer) {
audioBuffer2 = audioBuffer;
if (audioBuffer1 && audioBuffer2) { playBoth(); }
});
function playBoth() {
console.log('playing...');
play(audioBuffer1);
play(audioBuffer2);
}
audio quality is very low, with a lot of distortions
According to the WebAudio spec (https://webaudio.github.io/web-audio-api/#SummingJunction):
No clipping is applied at the inputs or outputs of the AudioNode to allow a maximum of dynamic range within the audio graph.
Now if you're playing two audio streams, it's possible that summing them results in a value that's beyond the acceptable range, which sounds like - distortions.
Try lowering the volume of each audio stream by first piping them through a GainNode as so:
function play(audioBuffer) {
if (!audioBuffer) { return; }
var bufferSource = context.createBufferSource();
var gainNode = context.createGain();
gainNode.gain.value = 0.5 // for instance, find a good value
bufferSource.connect(gainNode);
gainNode.connect(context.destination);
bufferSource.buffer = audioBuffer;
bufferSource.loop = false;
bufferSource.start(0);
}
Alternatively, you could use a DynamicsCompressorNode, but manually setting the gain gives you more control over the output.
This isn't exactly answer-worthy but I can't post comments at the moment ><
I had a similar problem with an app made using js audio api and the, rather easy fix, was lowering the quality of the audio and changing the format.
In your case what I could think of is setting the bit depth&sampling frequency as low as possible without affecting the listener's experience (e.g. 44.1kHz and 16 bit depth).
You might also try changing the format, wav, in theory, should be quite good at the job of not being CPU intensive, however, there are other uncompressed formats (e.g. .aiff)
You may try using multiple cores of the pi:
https://nodejs.org/api/cluster.html
Although this may prove a bit complicated, if you are doing the audio-streaming in parallel with other unrelated processes, you could try moving the audio on a separate CPU.
An (easy) thing you could try would be running node with more RAM, although, in your case, I doubt that I possible.
The biggest problem, however, might be the code, sadly enough I am not experienced with the modules you are using and as such can give to real advice on that (hence, why I said this is not answer worthy :p)
when you create Speaker instant, set parameter like this
channels = 1 // you can try with 1 or 2 and get the best quantity
bitDepth = 16
sampleRate = 48000 // normally 44100 for speaking and higher for music playing
You can spawn from node 2 aplay processes each playing one file. Use detached: true to allow node to continue running.
I'm trying to read and write raw data from files using Mozilla's add-on SDK. Currently I'm reading data with something like:
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available(),{charset:"UTF-8"});
callback(data, status, nsiFile);
});
}
This works for text files, but when I start messing with raw bytes outside of Unicode's normal range, it doesn't work. For example, if a file contains the byte 0xff, then that byte and anything past that byte isn't read at all. Is there any way to read (and write) raw data using the SDK?
You've specified an explicit charset in the options to NetUtil.readInputStream.
When you omit the charset option, then the data will be read as raw bytes. (Source)
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
// Do not specify a charset at all!
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available());
callback(data, status, nsiFile);
});
}
The suggestion to use io/byte-streams is OK as well, but keep in mind that that SDK module is still marked experimental, and that using ByteReader via io/file as the example suggests is not a good idea because this would be sync I/O on the main thread.
I don't really see the upside, as you'd use NetUtil anyway.
Anyway, this should work:
const {ByteReader} = require("sdk/io/byte-streams");
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var reader = new ByteReader(inputStream);
var data = reader.read(inputStream);
reader.close();
callback(data, status, nsiFile);
});
}
Also, please keep in mind that reading large files like this is problematic. Not only will the whole file buffered in memory, obviously, but:
The file is read as a char (byte) array first, so there will be a temporary buffer in the stream of at least file.size length (via asyncFetch).
Both NetUtil.readInputStreamToString and ByteReader will use another char (byte) array to read the result into from the inputStream, but ByteReader will do that in 32K chunks, while NetUtil.readInputStreamToString, will use a big buffer of file.length.
The data is then read into the resulting jschar/wchar_t (word) array aka. Javascript string, i.e. you need file.size * 2 bytes in memory at least.
E.g., reading a 1MB file would require more than fileSize * 4 = 4MB memory (NetUtil.readInputStreamToString) and/or more than fileSize * 3 = 3MB memory (ByteReader) during the read operation. After the operation, 2MB of that memory will be still alive to store the resulting data in a Javascript string.
Reading a 1MB file might be OK, but a 10MB file might be already problematic on mobile (Firefox for Android, Firefox OS) and a 100MB would be problematic even on desktop.
You can also read the data directly into an ArrayBuffer (or Uint8Array), which has more efficient storage for byte arrays than a Javascript string and avoid the temporary buffers of NetUtil.readInputStreamToString and/or ByteReader.
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var bs = Cc["#mozilla.org/binaryinputstream;1"].
createInstance(Ci.nsIBinaryInputStream);
bs.setInputStream(inputStream);
var len = inputStream.available();
var data = new Uint8Array(len);
reader.readArrayBuffer(len, data.buffer);
bs.close();
callback(data, status, nsiFile);
});
}
PS: The MDN documentation might state something about "iso-8859-1" being the default if the charset option is omitted in the NetUtil.readInputStreamToString call, but the documentation is wrong. I'll fix it.