I have read that transferable objects can be transferred really fast using postmessage of web worker. According to this transferable objects are either arraybuffer or messageport.
Question is, how do I convert say an arbitrary object that is of large size (30 mb) to a transferable object and pass it as an argument to postmessage. From what I understand I can convert my array to json string and then convert json string to raw byte data and store that inside of an array object. However, this seems to defeat the purpose of fast transferring.
could someone enlighten me to pass an object as transferable object or if it's even possible?
Thanks in advance!
This misconception is quite recurring here. You're imagining that it's possible to write some fast javascript code to convert your large object into transferable. But indeed, any conversion code you write defeats the purpose, just as you said. And the more complex data, the more speed you lose.
Objects are normally (when not transfering) converted by native structured clone algorithm (which uses implementation defined format and sure is optimal). Any javascript code you write will most likely be slower than structured clone, while achieving the same goal - transferring data as binary.
The purpose of transferable objects is to allow transfer for binary data, such as images (from canvas), audio or video. These kinds of data can be transferred without being processed by structured clone algorithm, which is why transferable interface was added. And the effect is insignificant even for these - see an answer about transferable speed.
As a last note, I wrote a prototype based library that converts javascript objects to ArrayBuffer and back. It's slower, especially for JSON like data. It's advantages (and advantages of any similar code you write) are:
You can define custom object conversions
You can use inheritance (eg. sending your own types, like Foo)
Code to transfer JSON like object
If your data is like JSON, just stick to structured clone and do not transfer. If you don't trust me, test it with this code. You will see it's slower than normal postMessage.
var object = {dd:"ddd", sub:{xx:"dd"}, num:666};
var string = JSON.stringify(object);
var uint8_array = new TextEncoder(document.characterSet.toLowerCase()).encode(string);
var array_buffer = uint8_array.buffer;
// now transfer array buffer
worker.postMessage(array_buffer, [array_buffer])
The opposite conversion, considering you have some ArrayBuffer:
// Let me just generate some array buffer for the simulation
var array_buffer = new Uint8Array([123,34,100,100,34,58,34,100,100,100,34,44,34,115,117,98,34,58,123,34,120,120,34,58,34,100,100,34,125,44,34,110,117,109,34,58,54,54,54,125]).buffer;
// Now to the decoding
var decoder = new TextDecoder("utf-8");
var view = new DataView(array_buffer, 0, array_buffer.byteLength);
var string = decoder.decode(view);
var object = JSON.parse(string);
Should have looked up Tomas's answer earlier.
Proof, although not specifically the way Tomas suggested.
Version A
Version B
I manually converted to a stringified json obejct to a Uint8Array like so:
function stringToUintArray(message) {
var encoded = self.btoa(message);
var uintArray = Array.prototype.slice.call(encoded).map(ch => ch.charCodeAt(0));
var uarray = new Uint8Array(uintArray);
return uarray;
}
and transferred it like so from the web worker to main thread:
console.time('generate');
var result = generate(params.low, params.high, params.interval, params.size);
var uarr = stringToUintArray(JSON.stringify(result));
console.timeEnd('generate');
self.postMessage(uarr.buffer, [uarr.buffer]);
and on the main thread I did something like this:
var uarr = new Uint8Array(e.data);
var json = UintArrayToString(uarr);
var result = JSON.parse(json);
Related
I have an external json file which I am using to initialize a js object in a web app (using webpack).
To read the file im doing this:
var myObject = require('json-loader!constants/myfile.json')
The application needs to modify the object over time and occasionally needs to return to the original state. Due to this, I've found this to be the most performant way to initialize and reinitialize (deep clone) the object:
var clonedObject = JSON.parse(JSON.stringify(myObject))
This approach seems redundant - First load a json object then stringify the object only to load it again.
Is there a way to read the JSON file as a string and then JSON.parse the object (thus omitting the JSON.stringify step)?
You can use raw-loader to read a file as string:
var jsonString = require('raw-loader!constants/myfile.json');
var obj1 = JSON.parse(jsonString);
var obj2 = JSON.parse(jsonString);
You'll want to read the file contents first using a blob perhaps.
Get the text content, then use the JSON.parse(jsonString).
Honestly I am too ignorant to keep libraries in my head, and thus I read JSON files with fs like this:
var fs=require("fs");
var stuff=JSON.parse(fs.readFileSync("filename","utf8"));
And then it is pretty sure that someone could store the intermediate result of fs.readFileSync() if they wanted to:
var fs=require("fs");
var originaljson=fs.readFileSync("filename","utf8");
var stuff=JSON.parse(originaljson);
I have an ArrayBuffer which looks like:
This buffer is placed under variable named myBuffer and what I'm interested in, is to get the Uint8Array from this object.
I tried to loop as:
myBuffer.forEach(function(element) {
console.log(element);
});
and to directly access to the Array as:
console.log(myBuffer['[[Uint8Array]]']);
console.log(myBuffer['Uint8Array']);
but seems none of this is the correct approach
Those pseudo-properties you are seeing are something the developer console is putting there for your benefit. They aren't really there at all, as a property or a symbol (AFAIK), and even if they were it would be non-standard.
You can easily get a Uint8Array view of your buffer the standard way like this though:
new Uint8Array(myBuffer)
You will first need to convert the array buffer into a typed array, then from there you can use the spread operator to create an array
const typedArray = new Uint8Array(myBuffer);
const array = [...typedArray];
Lets say I have a C API as follows:
void get_result_buffer(context* ctx, void** result, size_t* result_size);
Where context is some arbitrary opaque context type holding state. The intended way to call this is
context* ctx = ...;
do_something_with_context(ctx, ...);
void* result_buffer = 0;
size_t result_buffer_size = 0;
get_result_buffer(ctx, &result_buffer, &result_buffer_size);
/* Now result_buffer and result_buffer_size are meaningful and populated with the results of having called `do_something_with_context`. */
The result_buffer is owned by the context object, so the caller doesn't need to free it. Now I'd like to be able to call get_result_buffer from Emscripten. I can easily enough set up cwrap for this, it looks something like:
wrap_get_result_buffer = something.cwrap(
'get_result_buffer',
null,
['number', 'number', 'number']
)
But I'm unclear how I can set things up so that the out parameters "work" in JS. Ideally, at the end, I'd have something that looks like a byte buffer containing a copy of the data pointed to by the result out parameter, with a length as described by the result_size out parameter.
It seems that the values that I pass in need to be allocated somehow, and then I would pass the resulting allocation handle in as the number type parameters, but I have no idea how to do that in the JS/Emscripten layer. Similarly, after the call, I'd expect that those values have now been updated by the transpiled C code, but I'm unclear on how to extract the now populated data into some sort of JS byte array.
Any guidance on how to do this or pointers to example code?
OK. I figured this out. For future emscripteners, you want to do something like the following.
var out_data_ptr = Module._malloc(8)
var out_data_array = new Uint32Array(Module.HEAPU32.buffer, out_data_ptr, 2)
wrap_get_result_buffer(context, out_data_ptr, out_data_ptr + 4)
var response_uint8_array = new Uint8Array(Module.HEAPU8.buffer, out_data_array[0], out_data_array[1])
Module._free(out_data_ptr)
The theory of operations here is that we create a two element array that will store the 'slots' to be filled in by calling get_result_buffer, and then construct a view over that exposing it as two number compatible elements. We then pass those in to our get_result_buffer function as lifted above with cwrap. After that, the heap memory that the context refers to is reachable from those slots, which can then be used to construct a Uint8Array that provides JS level access to the bytes in the result.
Let's suppose we have two arrays.
var normal = new Array(1,2,3,4);
var float32 = new Float32Array(4);
Now its possible to empty normal array by
normal.length = 0;
But, In the case of float32 I am unable to empty array by
float32.lenght = 0;
Array remains same. why??
Because your Float32Array is just a view over an underlying ArrayBuffer, which itself is a fixed-length raw binary data buffer.
At the risk of simplifying a bit, once an ArrayBuffer has been assigned memory slots, it will stay in the same slots, and you won't be able to modify its byteLength*.
*Actually, you can empty the object which holds the data, by transferring its data, even if transferring it just for emptying the ArrayBuffer object makes no sense since you won't be able to change its length again (no push):
var arr = new Float32Array(56);
console.log('before transfer', arr.length);
postMessage(arr, '*', [arr.buffer]);
console.log('after transfer', arr.length);
Array remains same. why??
Float32Array is a TypedArray and as per docs
The length property is an accessor property whose set accessor
function is undefined, meaning that you can only read this property.
hence even after setting the length property to 0, its value remain as is
var arr = new Float32Array([21,31]);
arr.length = 0;
console.log(arr.length) //2
This might be a bit controversial but...
This state of affairs exists because the typed arrays are not generally meant for use by JavaScript developers. They are there to make JavaScript a more attractive target for compilers like emscripten.
Arrays in C/C++ are fixed-size (which is why you declare the size in bytes of those arrays when you create them in JS) and can only hold elements of a single type, so the idea of altering their length makes no sense and would invalidate a lot of the assumptions that C/C++ compilers get to make because of it.
The whole point of having typed arrays is to allow faster computations for expensive processes (e.g. 3D graphics) and if you had to check every access for an out-of-bounds access it would be too slow.
I have two JSON Objects:
var JSonData = {"li_ZPI":{},"li_ZIN":{"li_ISD":{},"li_AAH":{"li_AAD":{}},"li_EAH":{"li_EAD":{}},"li_REG":{},"li_PSC":{},"li_IMC":{},"li_GSP":{},"li_IES":{}}};
and
var additionalJSonData = {"li_AAH_1":{"li_AAD_1":{}}};
I want to add additionalJSonData object after JSONData "li_AAH" Object.
If you are using jQuery, you can easily do it using $.extend() function:
$.extend(true, JSonData,additionalJSonData);
In any other frameworks, there are similar functions.
The problem is that there's no after. These "JSON" objects (Javascript objects, really, JSON would be text strings that could convert to JS objects) are hashes, not arrays. Hashes have no order - that is,
for ( var f in JSonData ) {
}
there's no rule that the various keys ("li_ZPI", etc.) would show up in any particular order, or even the same order each time.
So you can certainly add the new item:
JSonData["li_AAH_1"] = {"li_AAD_1":{}};
but don't expect the eventual JSON string to be ordered the way you're hoping.