JavaScript: Most efficient and performant way to partition ArrayBuffer memory - javascript

I would compare what I am doing to what JavaScript runtimes already do, yet I'm doing it in JavaScript and Wasm. JavaScript implementations store JavaScript objects and values in actual computer heap memory, yet performing operations such as attempting to read/write out of bounds memory don't actually modify the memory (ex: arrays perform a no-op and return undefined respectively).
I'll give an example of my specific situation:
Let's say that I have an array buffer of 1000 bytes, we'll name the variable memory.
I want to split apart the buffer specifically into Int32Arrays of size 4. Each partition from the ArrayBuffer must do two things:
a) Refer to the original buffer (so that, when the original data is manipulated, the partition will update its values automaticially)
b) Not expose the original buffer (as the partition could then be used to corrupt the other partitions)
I have a function that determines which section is available for usage, we'll call it findPartition. It returns an integer acting as a pointer to a set of available bytes. (like C's malloc)
Each partition is expected to always remain the same type, that is, they will always be Int32Arrays if they start as an Int32Array, and their size will always be constant.
The script operating on the partition may both, write to, and read from, its partitioned array.
Originally, I was thinking that I could just call the Int32Array constructor on my array buffer, simply like so:
const createPartition = () => new Int32Array( memory, findPartition(), 4 );
The problem is that the buffer is exposed, so I could either delete the buffer property.
But... the buffer property is readonly, so delete fails when used on the array.
I then thought that I could make a class to do this:
class Partition {
#source = new Int32Array( memory, findPartition(), 4 );
get 0() { return this.#source[0]; }
set 0(x) { this.#source[0] = x; }
get 1() { return this.#source[1]; }
set 1(x) { this.#source[1] = x; }
get 2() { return this.#source[2]; }
set 2(x) { this.#source[2] = x; }
get 3() { return this.#source[3]; }
set 3(x) { this.#source[3] = x; }
get length() { return 4; }
};
Well, that works, but it's much more verbose, thus harder to maintain later, and, as the partitions are not given direct access to the indexes' values, because they have to go through getters and setters, I feel that performance could be lost.
Ideally, the Int32Array.prototype is also on the object, so I would have to wrap everything, which would be annoying and unmaintainable. If the spec updates the methods of the prototype, then I would have to update the wrappers too.
Does anyone have a better way to segment the array buffer, while maintaining safety between the segments?

Simplest way is to extend chosen typed array like that:
// Seal and freeze hidden Object (TypedArray.prototype) that has methods
// that can leak original buffer if attacked using defineProperty tricks
// Since we can't directly access hidden Object on which `subarray`
// and many other methods defined we use this workaround
// Paranoid: More checks required to make sure that: `subarray` method; 'byteOffset',
// 'byteLength', 'buffer' getters; are not modified beforehand
Object.seal(Int8Array.__proto__.prototype);
Object.freeze(Int8Array.__proto__.prototype);
class customUint32Array extends Uint32Array {
get buffer(){
// copy! viewed array buffer segment
// test if `super.` is faster/slower than `this.` access
return super.buffer.slice(this.byteOffset, this.byteOffset + this.byteLength);
// return super.buffer.slice(super.byteOffset, super.byteOffset + super.byteLength);
}
}
// var customUint32ArrayOverWholeBufferCached = new customUint32Array(memory);
function Partition(){
// test performance of `new` vs `customUint32ArrayOverWholeBufferCached.subarray`
// for fastest array buffer view creation
return new customUint32Array(memory, findPartitionByteOffset(), 4);
// return customUint32ArrayOverWholeBufferCached.subarray(findPartitionIndex(), 4);
}
By the way 'private' class properties in most JS environments are exposed as any other property and will leak original buffer.
Prototype chains forged manually instead of class X extends Y are welcome in comments.
If one will pass my original buffer leak tests, I'll include it here.
Current instance' prototype chain looks something like: customUint32Array.Uint32Array.TypedArray.prototype.Object

Related

Handling output buffers in emscripten

Lets say I have a C API as follows:
void get_result_buffer(context* ctx, void** result, size_t* result_size);
Where context is some arbitrary opaque context type holding state. The intended way to call this is
context* ctx = ...;
do_something_with_context(ctx, ...);
void* result_buffer = 0;
size_t result_buffer_size = 0;
get_result_buffer(ctx, &result_buffer, &result_buffer_size);
/* Now result_buffer and result_buffer_size are meaningful and populated with the results of having called `do_something_with_context`. */
The result_buffer is owned by the context object, so the caller doesn't need to free it. Now I'd like to be able to call get_result_buffer from Emscripten. I can easily enough set up cwrap for this, it looks something like:
wrap_get_result_buffer = something.cwrap(
'get_result_buffer',
null,
['number', 'number', 'number']
)
But I'm unclear how I can set things up so that the out parameters "work" in JS. Ideally, at the end, I'd have something that looks like a byte buffer containing a copy of the data pointed to by the result out parameter, with a length as described by the result_size out parameter.
It seems that the values that I pass in need to be allocated somehow, and then I would pass the resulting allocation handle in as the number type parameters, but I have no idea how to do that in the JS/Emscripten layer. Similarly, after the call, I'd expect that those values have now been updated by the transpiled C code, but I'm unclear on how to extract the now populated data into some sort of JS byte array.
Any guidance on how to do this or pointers to example code?
OK. I figured this out. For future emscripteners, you want to do something like the following.
var out_data_ptr = Module._malloc(8)
var out_data_array = new Uint32Array(Module.HEAPU32.buffer, out_data_ptr, 2)
wrap_get_result_buffer(context, out_data_ptr, out_data_ptr + 4)
var response_uint8_array = new Uint8Array(Module.HEAPU8.buffer, out_data_array[0], out_data_array[1])
Module._free(out_data_ptr)
The theory of operations here is that we create a two element array that will store the 'slots' to be filled in by calling get_result_buffer, and then construct a view over that exposing it as two number compatible elements. We then pass those in to our get_result_buffer function as lifted above with cwrap. After that, the heap memory that the context refers to is reachable from those slots, which can then be used to construct a Uint8Array that provides JS level access to the bytes in the result.

Why we can not empty Float32Array() array by assigning length equal to zero, like other normal array? In Javascript

Let's suppose we have two arrays.
var normal = new Array(1,2,3,4);
var float32 = new Float32Array(4);
Now its possible to empty normal array by
normal.length = 0;
But, In the case of float32 I am unable to empty array by
float32.lenght = 0;
Array remains same. why??
Because your Float32Array is just a view over an underlying ArrayBuffer, which itself is a fixed-length raw binary data buffer.
At the risk of simplifying a bit, once an ArrayBuffer has been assigned memory slots, it will stay in the same slots, and you won't be able to modify its byteLength*.
*Actually, you can empty the object which holds the data, by transferring its data, even if transferring it just for emptying the ArrayBuffer object makes no sense since you won't be able to change its length again (no push):
var arr = new Float32Array(56);
console.log('before transfer', arr.length);
postMessage(arr, '*', [arr.buffer]);
console.log('after transfer', arr.length);
Array remains same. why??
Float32Array is a TypedArray and as per docs
The length property is an accessor property whose set accessor
function is undefined, meaning that you can only read this property.
hence even after setting the length property to 0, its value remain as is
var arr = new Float32Array([21,31]);
arr.length = 0;
console.log(arr.length) //2
This might be a bit controversial but...
This state of affairs exists because the typed arrays are not generally meant for use by JavaScript developers. They are there to make JavaScript a more attractive target for compilers like emscripten.
Arrays in C/C++ are fixed-size (which is why you declare the size in bytes of those arrays when you create them in JS) and can only hold elements of a single type, so the idea of altering their length makes no sense and would invalidate a lot of the assumptions that C/C++ compilers get to make because of it.
The whole point of having typed arrays is to allow faster computations for expensive processes (e.g. 3D graphics) and if you had to check every access for an out-of-bounds access it would be too slow.

WeakMap, WeakSet, what's the point? [duplicate]

What are the actual uses of the WeakMap data structure introduced in ECMAScript 6?
Since a key of a weak map creates a strong reference to its corresponding value, ensuring that a value which has been inserted into a weak map will never disappear as long as its key is still alive, it can't be used for memo tables, caches or anything else that you would normally use weak references, maps with weak values, etc. for.
It seems to me that this:
weakmap.set(key, value);
...is just a roundabout way of saying this:
key.value = value;
What concrete use cases am I missing?
Fundamentally
WeakMaps provide a way to extend objects from the outside without interfering with garbage collection. Whenever you want to extend an object but can't because it is sealed - or from an external source - a WeakMap can be applied.
A WeakMap is a map (dictionary) where the keys are weak - that is, if all references to the key are lost and there are no more references to the value - the value can be garbage collected. Let's show this first through examples, then explain it a bit and finally finish with real use.
Let's say I'm using an API that gives me a certain object:
var obj = getObjectFromLibrary();
Now, I have a method that uses the object:
function useObj(obj){
doSomethingWith(obj);
}
I want to keep track of how many times the method was called with a certain object and report if it happens more than N times. Naively one would think to use a Map:
var map = new Map(); // maps can have object keys
function useObj(obj){
doSomethingWith(obj);
var called = map.get(obj) || 0;
called++; // called one more time
if(called > 10) report(); // Report called more than 10 times
map.set(obj, called);
}
This works, but it has a memory leak - we now keep track of every single library object passed to the function which keeps the library objects from ever being garbage collected. Instead - we can use a WeakMap:
var map = new WeakMap(); // create a weak map
function useObj(obj){
doSomethingWith(obj);
var called = map.get(obj) || 0;
called++; // called one more time
if(called > 10) report(); // Report called more than 10 times
map.set(obj, called);
}
And the memory leak is gone.
Use cases
Some use cases that would otherwise cause a memory leak and are enabled by WeakMaps include:
Keeping private data about a specific object and only giving access to it to people with a reference to the Map. A more ad-hoc approach is coming with the private-symbols proposal but that's a long time from now.
Keeping data about library objects without changing them or incurring overhead.
Keeping data about a small set of objects where many objects of the type exist to not incur problems with hidden classes JS engines use for objects of the same type.
Keeping data about host objects like DOM nodes in the browser.
Adding a capability to an object from the outside (like the event emitter example in the other answer).
Let's look at a real use
It can be used to extend an object from the outside. Let's give a practical (adapted, sort of real - to make a point) example from the real world of Node.js.
Let's say you're Node.js and you have Promise objects - now you want to keep track of all the currently rejected promises - however, you do not want to keep them from being garbage collected in case no references exist to them.
Now, you don't want to add properties to native objects for obvious reasons - so you're stuck. If you keep references to the promises you're causing a memory leak since no garbage collection can happen. If you don't keep references then you can't save additional information about individual promises. Any scheme that involves saving the ID of a promise inherently means you need a reference to it.
Enter WeakMaps
WeakMaps mean that the keys are weak. There are no ways to enumerate a weak map or to get all its values. In a weak map, you can store the data based on a key and when the key gets garbage collected so do the values.
This means that given a promise you can store state about it - and that object can still be garbage collected. Later on, if you get a reference to an object you can check if you have any state relating to it and report it.
This was used to implement unhandled rejection hooks by Petka Antonov as this:
process.on('unhandledRejection', function(reason, p) {
console.log("Unhandled Rejection at: Promise ", p, " reason: ", reason);
// application specific logging, throwing an error, or other logic here
});
We keep information about promises in a map and can know when a rejected promise was handled.
This answer seems to be biased and unusable in a real world scenario. Please read it as is, and don't consider it as an actual option for anything else than experimentation
A use case could be to use it as a dictionary for listeners, I have a coworker who did that. It is very helpful because any listener is directly targetted with this way of doing things. Goodbye listener.on.
But from a more abstract point of view, WeakMap is especially powerful to dematerialize access to basically anything, you don't need a namespace to isolate its members since it is already implied by the nature of this structure. I'm pretty sure you could do some major memory improvements by replacing awkwards redundant object keys (even though deconstructing does the work for you).
Before reading what is next
I do now realize my emphasize is not exactly the best way to tackle the problem and as Benjamin Gruenbaum pointed out (check out his answer, if it's not already above mine :p), this problem could not have been solved with a regular Map, since it would have leaked, thus the main strength of WeakMap is that it does not interfere with garbage collection given that they do not keep a reference.
Here is the actual code of my coworker (thanks to him for sharing)
Full source here, it's about listeners management I talked about above (you can also take a look at the specs)
var listenableMap = new WeakMap();
export function getListenable (object) {
if (!listenableMap.has(object)) {
listenableMap.set(object, {});
}
return listenableMap.get(object);
}
export function getListeners (object, identifier) {
var listenable = getListenable(object);
listenable[identifier] = listenable[identifier] || [];
return listenable[identifier];
}
export function on (object, identifier, listener) {
var listeners = getListeners(object, identifier);
listeners.push(listener);
}
export function removeListener (object, identifier, listener) {
var listeners = getListeners(object, identifier);
var index = listeners.indexOf(listener);
if(index !== -1) {
listeners.splice(index, 1);
}
}
export function emit (object, identifier, ...args) {
var listeners = getListeners(object, identifier);
for (var listener of listeners) {
listener.apply(object, args);
}
}
WeakMap works well for encapsulation and information hiding
WeakMap is only available for ES6 and above. A WeakMap is a collection of key and value pairs where the key must be an object. In the following example, we build a WeakMap with two items:
var map = new WeakMap();
var pavloHero = {first: "Pavlo", last: "Hero"};
var gabrielFranco = {first: "Gabriel", last: "Franco"};
map.set(pavloHero, "This is Hero");
map.set(gabrielFranco, "This is Franco");
console.log(map.get(pavloHero));//This is Hero
We used the set() method to define an association between an object and another item (a string in our case). We used the get() method to retrieve the item associated with an object. The interesting aspect of the WeakMaps is the fact that it holds a weak reference to the key inside the map. A weak reference means that if the object is destroyed, the garbage collector will remove the entire entry from the WeakMap, thus freeing up memory.
var TheatreSeats = (function() {
var priv = new WeakMap();
var _ = function(instance) {
return priv.get(instance);
};
return (function() {
function TheatreSeatsConstructor() {
var privateMembers = {
seats: []
};
priv.set(this, privateMembers);
this.maxSize = 10;
}
TheatreSeatsConstructor.prototype.placePerson = function(person) {
_(this).seats.push(person);
};
TheatreSeatsConstructor.prototype.countOccupiedSeats = function() {
return _(this).seats.length;
};
TheatreSeatsConstructor.prototype.isSoldOut = function() {
return _(this).seats.length >= this.maxSize;
};
TheatreSeatsConstructor.prototype.countFreeSeats = function() {
return this.maxSize - _(this).seats.length;
};
return TheatreSeatsConstructor;
}());
})()
๐— ๐—ฒ๐˜๐—ฎ๐—ฑ๐—ฎ๐˜๐—ฎ
Weak Maps can be used to store metadata about DOM elements without interfering with garbage collection or making coworkers mad at your code. For example, you could use them to numerically index all of the elements in a webpage.
๐—ช๐—ถ๐˜๐—ต๐—ผ๐˜‚๐˜ ๐—ช๐—ฒ๐—ฎ๐—ธ๐— ๐—ฎ๐—ฝ๐˜€ ๐—ผ๐—ฟ ๐—ช๐—ฒ๐—ฎ๐—ธ๐—ฆ๐—ฒ๐˜๐˜€:
var elements = document.getElementsByTagName('*'),
i = -1, len = elements.length;
while (++i !== len) {
// Production code written this poorly makes me want to cry:
elements[i].lookupindex = i;
elements[i].elementref = [];
elements[i].elementref.push( elements[(i * i) % len] );
}
// Then, you can access the lookupindex's
// For those of you new to javascirpt, I hope the comments below help explain
// how the ternary operator (?:) works like an inline if-statement
document.write(document.body.lookupindex + '<br />' + (
(document.body.elementref.indexOf(document.currentScript) !== -1)
? // if(document.body.elementref.indexOf(document.currentScript) !== -1){
"true"
: // } else {
"false"
) // }
);
๐—จ๐˜€๐—ถ๐—ป๐—ด ๐—ช๐—ฒ๐—ฎ๐—ธ๐— ๐—ฎ๐—ฝ๐˜€ ๐—ฎ๐—ป๐—ฑ ๐—ช๐—ฒ๐—ฎ๐—ธ๐—ฆ๐—ฒ๐˜๐˜€:
var DOMref = new WeakMap(),
__DOMref_value = Array,
__DOMref_lookupindex = 0,
__DOMref_otherelement = 1,
elements = document.getElementsByTagName('*'),
i = -1, len = elements.length, cur;
while (++i !== len) {
// Production code written this well makes me want to ๐Ÿ˜Š:
cur = DOMref.get(elements[i]);
if (cur === undefined)
DOMref.set(elements[i], cur = new __DOMref_value)
cur[__DOMref_lookupindex] = i;
cur[__DOMref_otherelement] = new WeakSet();
cur[__DOMref_otherelement].add( elements[(i * i) % len] );
}
// Then, you can access the lookupindex's
cur = DOMref.get(document.body)
document.write(cur[__DOMref_lookupindex] + '<br />' + (
cur[__DOMref_otherelement].has(document.currentScript)
? // if(cur[__DOMref_otherelement].has(document.currentScript)){
"true"
: // } else {
"false"
) // }
);
๐—ง๐—ต๐—ฒ ๐——๐—ถ๐—ณ๐—ณ๐—ฒ๐—ฟ๐—ฒ๐—ป๐—ฐ๐—ฒ
The difference may look negligible, aside from the fact that the weakmap version is longer, however there is a major difference between the two pieces of code shown above. In the first snippet of code, without weak maps, the piece of code stores references every which way between the DOM elements. This prevents the DOM elements from being garbage collected. (i * i) % len may seem like an oddball that nobody would use, but think again: plenty of production code has DOM references that bounce all over the document. Now, for the second piece of code, because all the references to the elements are weak, when you a remove a node, the browser is able to determine that the node is not used (not able to be reached by your code), and thus delete it from memory. The reason for why you should be concerned about memory usage, and memory anchors (things like the first snippet of code where unused elements are held in memory) is because more memory usage means more browser GC-attempts (to try to free up memory to avert a browser crash) means slower browsing experience and sometimes a browser crash.
As for a polyfill for these, I would recommend my own library (found here # github). It is a very lightweight library that will simply polyfill it without any of the way-overly-complex frameworks you might find in other polyfills.
~ Happy coding!
I use WeakMap for the cache of worry-free memoization of functions that take in immutable objects as their parameter.
Memoization is fancy way of saying "after you compute the value, cache it so you don't have to compute it again".
Here's an example:
// using immutable.js from here https://facebook.github.io/immutable-js/
const memo = new WeakMap();
let myObj = Immutable.Map({a: 5, b: 6});
function someLongComputeFunction (someImmutableObj) {
// if we saved the value, then return it
if (memo.has(someImmutableObj)) {
console.log('used memo!');
return memo.get(someImmutableObj);
}
// else compute, set, and return
const computedValue = someImmutableObj.get('a') + someImmutableObj.get('b');
memo.set(someImmutableObj, computedValue);
console.log('computed value');
return computedValue;
}
someLongComputeFunction(myObj);
someLongComputeFunction(myObj);
someLongComputeFunction(myObj);
// reassign
myObj = Immutable.Map({a: 7, b: 8});
someLongComputeFunction(myObj);
<script src="https://cdnjs.cloudflare.com/ajax/libs/immutable/3.8.1/immutable.min.js"></script>
A few things to note:
Immutable.js objects return new objects (with a new pointer) when you modify them so using them as keys in a WeakMap is guarantees the same computed value.
The WeakMap is great for memos because once the object (used as the key) gets garbage collected, so does the computed value on the WeakMap.
I have this simple feature based use case/Example for WeakMaps.
MANAGE A COLLECTION OF USERS
I started off with a User Object whose properties include a fullname, username, age, gender and a method called print which prints a human readable summary of the other properties.
/**
Basic User Object with common properties.
*/
function User(username, fullname, age, gender) {
this.username = username;
this.fullname = fullname;
this.age = age;
this.gender = gender;
this.print = () => console.log(`${this.fullname} is a ${age} year old ${gender}`);
}
I then added a Map called users to keep a collection of multiple users which are keyed by username.
/**
Collection of Users, keyed by username.
*/
var users = new Map();
Addition of the Collection also required helper functions to add, get, delete a User and even a function to print all the users for sake of completeness.
/**
Creates an User Object and adds it to the users Collection.
*/
var addUser = (username, fullname, age, gender) => {
let an_user = new User(username, fullname, age, gender);
users.set(username, an_user);
}
/**
Returns an User Object associated with the given username in the Collection.
*/
var getUser = (username) => {
return users.get(username);
}
/**
Deletes an User Object associated with the given username in the Collection.
*/
var deleteUser = (username) => {
users.delete(username);
}
/**
Prints summary of all the User Objects in the Collection.
*/
var printUsers = () => {
users.forEach((user) => {
user.print();
});
}
With all of the above code running in, say NodeJS, only the users Map has the reference to the User Objects within the entire process. There is no other reference to the individual User Objects.
Running this code an interactive NodeJS shell, just as an Example I add four users and print them:
ADD MORE INFO TO USERS WITHOUT MODIFYING EXISTING CODE
Now say a new feature is required wherein each users Social Media Platform (SMP) links need to be tracked along with the User Objects.
The key here is also that this feature must be implemented with minimum intervention to the existing code.
This is possible with WeakMaps in the following manner.
I add three separate WeakMaps for Twitter, Facebook, LinkedIn.
/*
WeakMaps for Social Media Platforms (SMPs).
Could be replaced by a single Map which can grow
dynamically based on different SMP names . . . anyway...
*/
var sm_platform_twitter = new WeakMap();
var sm_platform_facebook = new WeakMap();
var sm_platform_linkedin = new WeakMap();
A helper function, getSMPWeakMap is added simply to return the WeakMap associated with the given SMP name.
/**
Returns the WeakMap for the given SMP.
*/
var getSMPWeakMap = (sm_platform) => {
if(sm_platform == "Twitter") {
return sm_platform_twitter;
}
else if(sm_platform == "Facebook") {
return sm_platform_facebook;
}
else if(sm_platform == "LinkedIn") {
return sm_platform_linkedin;
}
return undefined;
}
A function to add a users SMP link to the given SMP WeakMap.
/**
Adds a SMP link associated with a given User. The User must be already added to the Collection.
*/
var addUserSocialMediaLink = (username, sm_platform, sm_link) => {
let user = getUser(username);
let sm_platform_weakmap = getSMPWeakMap(sm_platform);
if(user && sm_platform_weakmap) {
sm_platform_weakmap.set(user, sm_link);
}
}
A function to print only the users who are present on the given SMP.
/**
Prints the User's fullname and corresponding SMP link of only those Users which are on the given SMP.
*/
var printSMPUsers = (sm_platform) => {
let sm_platform_weakmap = getSMPWeakMap(sm_platform);
console.log(`Users of ${sm_platform}:`)
users.forEach((user)=>{
if(sm_platform_weakmap.has(user)) {
console.log(`\t${user.fullname} : ${sm_platform_weakmap.get(user)}`)
}
});
}
You can now add SMP links for the users, also with the possibility of each user having a link on multiple SMPs.
...continuing with the earlier Example, I add SMP links to the users, multiple links for users Bill and Sarah and then print the links for each SMP separately:
Now say a User is deleted from the users Map by calling deleteUser. That removes the only reference to the User Object. This in turn will also clear out the SMP link from any/all of the SMP WeakMaps (by Garbage Collection) as without the User Object there is no way to access any of its SMP link.
...continuing with the Example, I delete user Bill and then print out the links of the SMPs he was associated with:
There is no requirement of any additional code to individually delete the SMP link separately and the existing code before this feature was not modified in anyway.
If there is any other way to add this feature with/without WeakMaps please feel free to comment.
WEAKMAP: keep in mind weakMap is all about memory allocation and garbage collection and only related to key of object type
in javascript when u store values in key-value pair array, map, set, etc... a memory allocated to all key-value pair and this memory will not be free even if you delete or set null to that key consider this as a strongmap keys are strongly attache to memory below is example
let john = { name: "yusuf" };
let map = new Map();
map.set(yusuf, "xyz"); //here "yusuf" is the key and "xyz" is value
yusuf= null; // overwrite the reference
// the object previously referenced by yusuf is stored inside the array
// therefore it won't be garbage-collected
// we can get it using map.keys()
but this is not the case with weakMap in here memory will be free
let john = { name: "yusuf" };
let map = new WeakMap();
map.set(yusuf, "...");
yusuf= null; // overwrite the reference
// yusuf is removed from memory!
USE CASE : you will use it in javascript where u want to manage memory in more efficient way
If weโ€™re working with an object that โ€œbelongsโ€ to another code, maybe even a third-party library, and would like to store some data associated with it, that should only exist while the object is alive โ€“ then WeakMap is exactly whatโ€™s needed.
We put the data to a WeakMap, using the object as the key, and when the object is garbage collected, that data will automatically disappear as well.
weakMap.set(yusuf, "secret documents");
// if yusuf dies, secret documents will be destroyed automatically
I took reference from this great article : https://javascript.info/weakmap-weakset
I think it's very helpful for checking a connection income in applications socket.
The other case, the 'Weak Collection' are useful: https://javascript.info/task/recipients-read

What are the actual uses of ES6 WeakMap?

What are the actual uses of the WeakMap data structure introduced in ECMAScript 6?
Since a key of a weak map creates a strong reference to its corresponding value, ensuring that a value which has been inserted into a weak map will never disappear as long as its key is still alive, it can't be used for memo tables, caches or anything else that you would normally use weak references, maps with weak values, etc. for.
It seems to me that this:
weakmap.set(key, value);
...is just a roundabout way of saying this:
key.value = value;
What concrete use cases am I missing?
Fundamentally
WeakMaps provide a way to extend objects from the outside without interfering with garbage collection. Whenever you want to extend an object but can't because it is sealed - or from an external source - a WeakMap can be applied.
A WeakMap is a map (dictionary) where the keys are weak - that is, if all references to the key are lost and there are no more references to the value - the value can be garbage collected. Let's show this first through examples, then explain it a bit and finally finish with real use.
Let's say I'm using an API that gives me a certain object:
var obj = getObjectFromLibrary();
Now, I have a method that uses the object:
function useObj(obj){
doSomethingWith(obj);
}
I want to keep track of how many times the method was called with a certain object and report if it happens more than N times. Naively one would think to use a Map:
var map = new Map(); // maps can have object keys
function useObj(obj){
doSomethingWith(obj);
var called = map.get(obj) || 0;
called++; // called one more time
if(called > 10) report(); // Report called more than 10 times
map.set(obj, called);
}
This works, but it has a memory leak - we now keep track of every single library object passed to the function which keeps the library objects from ever being garbage collected. Instead - we can use a WeakMap:
var map = new WeakMap(); // create a weak map
function useObj(obj){
doSomethingWith(obj);
var called = map.get(obj) || 0;
called++; // called one more time
if(called > 10) report(); // Report called more than 10 times
map.set(obj, called);
}
And the memory leak is gone.
Use cases
Some use cases that would otherwise cause a memory leak and are enabled by WeakMaps include:
Keeping private data about a specific object and only giving access to it to people with a reference to the Map. A more ad-hoc approach is coming with the private-symbols proposal but that's a long time from now.
Keeping data about library objects without changing them or incurring overhead.
Keeping data about a small set of objects where many objects of the type exist to not incur problems with hidden classes JS engines use for objects of the same type.
Keeping data about host objects like DOM nodes in the browser.
Adding a capability to an object from the outside (like the event emitter example in the other answer).
Let's look at a real use
It can be used to extend an object from the outside. Let's give a practical (adapted, sort of real - to make a point) example from the real world of Node.js.
Let's say you're Node.js and you have Promise objects - now you want to keep track of all the currently rejected promises - however, you do not want to keep them from being garbage collected in case no references exist to them.
Now, you don't want to add properties to native objects for obvious reasons - so you're stuck. If you keep references to the promises you're causing a memory leak since no garbage collection can happen. If you don't keep references then you can't save additional information about individual promises. Any scheme that involves saving the ID of a promise inherently means you need a reference to it.
Enter WeakMaps
WeakMaps mean that the keys are weak. There are no ways to enumerate a weak map or to get all its values. In a weak map, you can store the data based on a key and when the key gets garbage collected so do the values.
This means that given a promise you can store state about it - and that object can still be garbage collected. Later on, if you get a reference to an object you can check if you have any state relating to it and report it.
This was used to implement unhandled rejection hooks by Petka Antonov as this:
process.on('unhandledRejection', function(reason, p) {
console.log("Unhandled Rejection at: Promise ", p, " reason: ", reason);
// application specific logging, throwing an error, or other logic here
});
We keep information about promises in a map and can know when a rejected promise was handled.
This answer seems to be biased and unusable in a real world scenario. Please read it as is, and don't consider it as an actual option for anything else than experimentation
A use case could be to use it as a dictionary for listeners, I have a coworker who did that. It is very helpful because any listener is directly targetted with this way of doing things. Goodbye listener.on.
But from a more abstract point of view, WeakMap is especially powerful to dematerialize access to basically anything, you don't need a namespace to isolate its members since it is already implied by the nature of this structure. I'm pretty sure you could do some major memory improvements by replacing awkwards redundant object keys (even though deconstructing does the work for you).
Before reading what is next
I do now realize my emphasize is not exactly the best way to tackle the problem and as Benjamin Gruenbaum pointed out (check out his answer, if it's not already above mine :p), this problem could not have been solved with a regular Map, since it would have leaked, thus the main strength of WeakMap is that it does not interfere with garbage collection given that they do not keep a reference.
Here is the actual code of my coworker (thanks to him for sharing)
Full source here, it's about listeners management I talked about above (you can also take a look at the specs)
var listenableMap = new WeakMap();
export function getListenable (object) {
if (!listenableMap.has(object)) {
listenableMap.set(object, {});
}
return listenableMap.get(object);
}
export function getListeners (object, identifier) {
var listenable = getListenable(object);
listenable[identifier] = listenable[identifier] || [];
return listenable[identifier];
}
export function on (object, identifier, listener) {
var listeners = getListeners(object, identifier);
listeners.push(listener);
}
export function removeListener (object, identifier, listener) {
var listeners = getListeners(object, identifier);
var index = listeners.indexOf(listener);
if(index !== -1) {
listeners.splice(index, 1);
}
}
export function emit (object, identifier, ...args) {
var listeners = getListeners(object, identifier);
for (var listener of listeners) {
listener.apply(object, args);
}
}
WeakMap works well for encapsulation and information hiding
WeakMap is only available for ES6 and above. A WeakMap is a collection of key and value pairs where the key must be an object. In the following example, we build a WeakMap with two items:
var map = new WeakMap();
var pavloHero = {first: "Pavlo", last: "Hero"};
var gabrielFranco = {first: "Gabriel", last: "Franco"};
map.set(pavloHero, "This is Hero");
map.set(gabrielFranco, "This is Franco");
console.log(map.get(pavloHero));//This is Hero
We used the set() method to define an association between an object and another item (a string in our case). We used the get() method to retrieve the item associated with an object. The interesting aspect of the WeakMaps is the fact that it holds a weak reference to the key inside the map. A weak reference means that if the object is destroyed, the garbage collector will remove the entire entry from the WeakMap, thus freeing up memory.
var TheatreSeats = (function() {
var priv = new WeakMap();
var _ = function(instance) {
return priv.get(instance);
};
return (function() {
function TheatreSeatsConstructor() {
var privateMembers = {
seats: []
};
priv.set(this, privateMembers);
this.maxSize = 10;
}
TheatreSeatsConstructor.prototype.placePerson = function(person) {
_(this).seats.push(person);
};
TheatreSeatsConstructor.prototype.countOccupiedSeats = function() {
return _(this).seats.length;
};
TheatreSeatsConstructor.prototype.isSoldOut = function() {
return _(this).seats.length >= this.maxSize;
};
TheatreSeatsConstructor.prototype.countFreeSeats = function() {
return this.maxSize - _(this).seats.length;
};
return TheatreSeatsConstructor;
}());
})()
๐— ๐—ฒ๐˜๐—ฎ๐—ฑ๐—ฎ๐˜๐—ฎ
Weak Maps can be used to store metadata about DOM elements without interfering with garbage collection or making coworkers mad at your code. For example, you could use them to numerically index all of the elements in a webpage.
๐—ช๐—ถ๐˜๐—ต๐—ผ๐˜‚๐˜ ๐—ช๐—ฒ๐—ฎ๐—ธ๐— ๐—ฎ๐—ฝ๐˜€ ๐—ผ๐—ฟ ๐—ช๐—ฒ๐—ฎ๐—ธ๐—ฆ๐—ฒ๐˜๐˜€:
var elements = document.getElementsByTagName('*'),
i = -1, len = elements.length;
while (++i !== len) {
// Production code written this poorly makes me want to cry:
elements[i].lookupindex = i;
elements[i].elementref = [];
elements[i].elementref.push( elements[(i * i) % len] );
}
// Then, you can access the lookupindex's
// For those of you new to javascirpt, I hope the comments below help explain
// how the ternary operator (?:) works like an inline if-statement
document.write(document.body.lookupindex + '<br />' + (
(document.body.elementref.indexOf(document.currentScript) !== -1)
? // if(document.body.elementref.indexOf(document.currentScript) !== -1){
"true"
: // } else {
"false"
) // }
);
๐—จ๐˜€๐—ถ๐—ป๐—ด ๐—ช๐—ฒ๐—ฎ๐—ธ๐— ๐—ฎ๐—ฝ๐˜€ ๐—ฎ๐—ป๐—ฑ ๐—ช๐—ฒ๐—ฎ๐—ธ๐—ฆ๐—ฒ๐˜๐˜€:
var DOMref = new WeakMap(),
__DOMref_value = Array,
__DOMref_lookupindex = 0,
__DOMref_otherelement = 1,
elements = document.getElementsByTagName('*'),
i = -1, len = elements.length, cur;
while (++i !== len) {
// Production code written this well makes me want to ๐Ÿ˜Š:
cur = DOMref.get(elements[i]);
if (cur === undefined)
DOMref.set(elements[i], cur = new __DOMref_value)
cur[__DOMref_lookupindex] = i;
cur[__DOMref_otherelement] = new WeakSet();
cur[__DOMref_otherelement].add( elements[(i * i) % len] );
}
// Then, you can access the lookupindex's
cur = DOMref.get(document.body)
document.write(cur[__DOMref_lookupindex] + '<br />' + (
cur[__DOMref_otherelement].has(document.currentScript)
? // if(cur[__DOMref_otherelement].has(document.currentScript)){
"true"
: // } else {
"false"
) // }
);
๐—ง๐—ต๐—ฒ ๐——๐—ถ๐—ณ๐—ณ๐—ฒ๐—ฟ๐—ฒ๐—ป๐—ฐ๐—ฒ
The difference may look negligible, aside from the fact that the weakmap version is longer, however there is a major difference between the two pieces of code shown above. In the first snippet of code, without weak maps, the piece of code stores references every which way between the DOM elements. This prevents the DOM elements from being garbage collected. (i * i) % len may seem like an oddball that nobody would use, but think again: plenty of production code has DOM references that bounce all over the document. Now, for the second piece of code, because all the references to the elements are weak, when you a remove a node, the browser is able to determine that the node is not used (not able to be reached by your code), and thus delete it from memory. The reason for why you should be concerned about memory usage, and memory anchors (things like the first snippet of code where unused elements are held in memory) is because more memory usage means more browser GC-attempts (to try to free up memory to avert a browser crash) means slower browsing experience and sometimes a browser crash.
As for a polyfill for these, I would recommend my own library (found here # github). It is a very lightweight library that will simply polyfill it without any of the way-overly-complex frameworks you might find in other polyfills.
~ Happy coding!
I use WeakMap for the cache of worry-free memoization of functions that take in immutable objects as their parameter.
Memoization is fancy way of saying "after you compute the value, cache it so you don't have to compute it again".
Here's an example:
// using immutable.js from here https://facebook.github.io/immutable-js/
const memo = new WeakMap();
let myObj = Immutable.Map({a: 5, b: 6});
function someLongComputeFunction (someImmutableObj) {
// if we saved the value, then return it
if (memo.has(someImmutableObj)) {
console.log('used memo!');
return memo.get(someImmutableObj);
}
// else compute, set, and return
const computedValue = someImmutableObj.get('a') + someImmutableObj.get('b');
memo.set(someImmutableObj, computedValue);
console.log('computed value');
return computedValue;
}
someLongComputeFunction(myObj);
someLongComputeFunction(myObj);
someLongComputeFunction(myObj);
// reassign
myObj = Immutable.Map({a: 7, b: 8});
someLongComputeFunction(myObj);
<script src="https://cdnjs.cloudflare.com/ajax/libs/immutable/3.8.1/immutable.min.js"></script>
A few things to note:
Immutable.js objects return new objects (with a new pointer) when you modify them so using them as keys in a WeakMap is guarantees the same computed value.
The WeakMap is great for memos because once the object (used as the key) gets garbage collected, so does the computed value on the WeakMap.
I have this simple feature based use case/Example for WeakMaps.
MANAGE A COLLECTION OF USERS
I started off with a User Object whose properties include a fullname, username, age, gender and a method called print which prints a human readable summary of the other properties.
/**
Basic User Object with common properties.
*/
function User(username, fullname, age, gender) {
this.username = username;
this.fullname = fullname;
this.age = age;
this.gender = gender;
this.print = () => console.log(`${this.fullname} is a ${age} year old ${gender}`);
}
I then added a Map called users to keep a collection of multiple users which are keyed by username.
/**
Collection of Users, keyed by username.
*/
var users = new Map();
Addition of the Collection also required helper functions to add, get, delete a User and even a function to print all the users for sake of completeness.
/**
Creates an User Object and adds it to the users Collection.
*/
var addUser = (username, fullname, age, gender) => {
let an_user = new User(username, fullname, age, gender);
users.set(username, an_user);
}
/**
Returns an User Object associated with the given username in the Collection.
*/
var getUser = (username) => {
return users.get(username);
}
/**
Deletes an User Object associated with the given username in the Collection.
*/
var deleteUser = (username) => {
users.delete(username);
}
/**
Prints summary of all the User Objects in the Collection.
*/
var printUsers = () => {
users.forEach((user) => {
user.print();
});
}
With all of the above code running in, say NodeJS, only the users Map has the reference to the User Objects within the entire process. There is no other reference to the individual User Objects.
Running this code an interactive NodeJS shell, just as an Example I add four users and print them:
ADD MORE INFO TO USERS WITHOUT MODIFYING EXISTING CODE
Now say a new feature is required wherein each users Social Media Platform (SMP) links need to be tracked along with the User Objects.
The key here is also that this feature must be implemented with minimum intervention to the existing code.
This is possible with WeakMaps in the following manner.
I add three separate WeakMaps for Twitter, Facebook, LinkedIn.
/*
WeakMaps for Social Media Platforms (SMPs).
Could be replaced by a single Map which can grow
dynamically based on different SMP names . . . anyway...
*/
var sm_platform_twitter = new WeakMap();
var sm_platform_facebook = new WeakMap();
var sm_platform_linkedin = new WeakMap();
A helper function, getSMPWeakMap is added simply to return the WeakMap associated with the given SMP name.
/**
Returns the WeakMap for the given SMP.
*/
var getSMPWeakMap = (sm_platform) => {
if(sm_platform == "Twitter") {
return sm_platform_twitter;
}
else if(sm_platform == "Facebook") {
return sm_platform_facebook;
}
else if(sm_platform == "LinkedIn") {
return sm_platform_linkedin;
}
return undefined;
}
A function to add a users SMP link to the given SMP WeakMap.
/**
Adds a SMP link associated with a given User. The User must be already added to the Collection.
*/
var addUserSocialMediaLink = (username, sm_platform, sm_link) => {
let user = getUser(username);
let sm_platform_weakmap = getSMPWeakMap(sm_platform);
if(user && sm_platform_weakmap) {
sm_platform_weakmap.set(user, sm_link);
}
}
A function to print only the users who are present on the given SMP.
/**
Prints the User's fullname and corresponding SMP link of only those Users which are on the given SMP.
*/
var printSMPUsers = (sm_platform) => {
let sm_platform_weakmap = getSMPWeakMap(sm_platform);
console.log(`Users of ${sm_platform}:`)
users.forEach((user)=>{
if(sm_platform_weakmap.has(user)) {
console.log(`\t${user.fullname} : ${sm_platform_weakmap.get(user)}`)
}
});
}
You can now add SMP links for the users, also with the possibility of each user having a link on multiple SMPs.
...continuing with the earlier Example, I add SMP links to the users, multiple links for users Bill and Sarah and then print the links for each SMP separately:
Now say a User is deleted from the users Map by calling deleteUser. That removes the only reference to the User Object. This in turn will also clear out the SMP link from any/all of the SMP WeakMaps (by Garbage Collection) as without the User Object there is no way to access any of its SMP link.
...continuing with the Example, I delete user Bill and then print out the links of the SMPs he was associated with:
There is no requirement of any additional code to individually delete the SMP link separately and the existing code before this feature was not modified in anyway.
If there is any other way to add this feature with/without WeakMaps please feel free to comment.
WEAKMAP: keep in mind weakMap is all about memory allocation and garbage collection and only related to key of object type
in javascript when u store values in key-value pair array, map, set, etc... a memory allocated to all key-value pair and this memory will not be free even if you delete or set null to that key consider this as a strongmap keys are strongly attache to memory below is example
let john = { name: "yusuf" };
let map = new Map();
map.set(yusuf, "xyz"); //here "yusuf" is the key and "xyz" is value
yusuf= null; // overwrite the reference
// the object previously referenced by yusuf is stored inside the array
// therefore it won't be garbage-collected
// we can get it using map.keys()
but this is not the case with weakMap in here memory will be free
let john = { name: "yusuf" };
let map = new WeakMap();
map.set(yusuf, "...");
yusuf= null; // overwrite the reference
// yusuf is removed from memory!
USE CASE : you will use it in javascript where u want to manage memory in more efficient way
If weโ€™re working with an object that โ€œbelongsโ€ to another code, maybe even a third-party library, and would like to store some data associated with it, that should only exist while the object is alive โ€“ then WeakMap is exactly whatโ€™s needed.
We put the data to a WeakMap, using the object as the key, and when the object is garbage collected, that data will automatically disappear as well.
weakMap.set(yusuf, "secret documents");
// if yusuf dies, secret documents will be destroyed automatically
I took reference from this great article : https://javascript.info/weakmap-weakset
I think it's very helpful for checking a connection income in applications socket.
The other case, the 'Weak Collection' are useful: https://javascript.info/task/recipients-read

Can I make a "Virtual Array" in JavaScript?

I'm calling a JavaScript function that wants an array of things to display. It displays a count, and displays the items one by one. Everything works when I pass it a normal JavaScript array.
But I have too many items to hold in memory at once. What I'd like to do, is pass it an object with the same interface as an array, and have my method(s) be called when the function tries to access the data. And in fact, if I pass the following:
var featureArray = {length: count, 0: func(0)};
then the count is displayed, and the first item is correctly displayed. But I don't want to assign all the entries, or I'll run out of memory. And the function currently crashes when the user tries to display the second item. I want to know when item 1 is accessed, and return func(1) for item 1, and func(2) for item 2, etc. (i.e., delaying the creation of the item until it is requested).
Is this possible in JavaScript?
If I understand correctly, this would help:
var object = {length: count, data: function (whatever) {
// create your item
}};
Then, instead of doing array[1], array[2], et cetera, you'd do object.data(1), object.data(2), and so on.
Since there seems to be a constraint that the data must be accessed using array indexing via normal array indexing arr[index] and that can't be changed, then the answer is that NO, you can't override array indexing in Javascript to change how it works and make some sort of virtual array that only fetches data upon demand. It was proposed for ECMAScript 4 and rejected as a feature.
See these two other posts for other discussion/confirmation:
How would you overload the [] operator in Javascript
In javascript, can I override the brackets to access characters in a string?
The usual way to solve this problem would be to switch to using a method such as .get(n) to request the data and then the implementor of .get() can virtualize however much they want.
P.S. Others indicate that you could use a Proxy object for this in Firefox (not supported in other browsers as far as I know), but I'm not personally familiar with Proxy objects as it's use seems rather limited to code that only targets Firefox right now.
Yes, generating items on the go is possible. You will want to have a look at Lazy.js, a library for producing lazily computed/loaded sequences.
However, you will need to change your function that accepts this sequence, it will need to be consumed differently than a plain array.
If you really need to fake an array interface, you'd use Proxies. Unfortunately, it is only a harmony draft and currently only supported in Firefox' Javascript 1.8.5.
Assuming that the array is only accessed in an iteration, i.e. starting with index 0, you might be able to do some crazy things with getters:
var featureArray = (function(func) {
var arr = {length: 0};
function makeGetter(i) {
arr.length = i+1;
Object.defineProperty(arr, i, {
get: function() {
var val = func(i);
Object.defineProperty(arr, i, {value:val});
makeGetter(i+1);
return val;
},
configurable: true,
enumerable: true
});
}
makeGetter(0);
return arr;
}(func));
However, I'd recommend to avoid that and rather switch the library that is expecting the array. This solution is very errorprone if anything else is done with the "array" but accessing its indices in order.
Thank you to everyone who has commented and answered my original question - it seems that this is not (currently) supported by JavaScript.
I was able to get around this limitation, and still do what I wanted. It uses an aspect of the program that I did not mention in my original question (I was trying to simplify the question), so it is understandable that other's couldn't recommend this. That is, it doesn't technically answer my original question, but I'm sharing it in case others find it useful.
It turns out that one member of the object in each array element is a callback function. That is (using the terminology from my original question), func(n) is returning an object, which contains a function in one member, which is called by the method being passed the data. Since this callback function knows the index it is associated with (at least, when being created by func(n)), it can add the next item in the array (or at least ensure that it is already there) when it is called. A more complicated solution might go a few ahead, and/or behind, and/or could cleanup items not near the current index to free memory. This all assumes that the items will be accessed consecutively (which is the case in my program).
E.g.,
1) Create a variable that will stay in scope (e.g., a global variable).
2) Call the function with an object like I gave as an example in my original question:
var featureArray = {length: count, 0: func(0)};
3) func() can be something like:
function func(r) {
return {
f : function() {featureArray[r + 1] = func(r + 1); DoOtherStuff(r); }
}
}
Assuming that f() is the member with the function that will be called by the external function.

Categories

Resources