Learning about Node.js event loop and partitioning - javascript

I have written 2 types of sum functions.
Loop sum
const sum = (n) => {
let sum = 0;
for (let i = 1; i <= n; i++) {
sum += i;
}
return sum;
};
console.time('sum');
console.log(sum(1_00_000));
console.timeEnd('sum');
// Output:
// 5000050000
// sum: 11.172ms
Partitioned sum (Partitioning of the event loop)
const sum = (n, cb) => {
let total = 0;
const help = (i, cb) => {
if (i > n) {
return cb(total);
}
total += i;
setImmediate(help, i + 1, cb);
};
help(0, cb);
};
console.time('sum');
sum(1_00_000, (result) => {
console.log(result);
console.timeEnd('sum');
});
// Output:
// 5000050000
// sum: 1.225s
Now, the Loop sum takes way less time than the Partitioned sum.
But, partitioning and off-loading are the recommended ways of handling computationally intensive tasks as mentioned here: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/#don-t-block-the-event-loop
My question are:
How is partitioning of event loop good for large scale operations?
Am I right here to compare the two using the time taken?
What would be a better metric in this case?
Edit 1: Replaced screenshots with code snippets and output

Related

Function that sums up all integers from a deeply nested array

I have an issue here, I'm trying to create a function that sums up all integers from a deeply nested array, but it's failing an unit test, which means something is not right.
Here is my function:
export const arraySum = (arr) => {
let sum = 0;
for (let i = 0; i < arr.length; i++) {
if (typeof arr[i] === "number") sum = sum + arr[i];
else if (Array.isArray(arr[i])) sum = sum + arraySum(arr[i]);
}
return sum;
};
And here is my unit test which is failing:
test("it should sum up from deeply nested arrays", () => {
type ValueOrArray = number | Array<ValueOrArray>;
const createDeeplyNestedArray = (depth: number): ValueOrArray => {
let retval: ValueOrArray = [1];
for (let i = 0; i < depth - 1; i++) {
retval = [1, retval];
}
return retval;
};
const NUMBER_OF_ELEMENTS = 100000;
const arr = createDeeplyNestedArray(NUMBER_OF_ELEMENTS);
expect(arraySum(arr)).toEqual(NUMBER_OF_ELEMENTS);
});
Function memory is stored on something called a "call stack". Whenever you call a function, all of its variables are allocated and pushed onto the 'stack' and when the function returns, they are popped off the stack. Given the following code:
const a = () => {
}
const b = () => {
a()
// some code
}
const c = () => {
b()
}
c()
When c is called, your call stack will contain all the memory for variables used in c. When c calls b, all the memory for variables used in b are added to the stack. When b calls a all the memory for variables used in a are added to the stack. When a finishes executing (so when you get to 'some code'), variables related to a are deallocated and removed from the stack.
The problem you have here is that every time your function recursively calls itself, more memory is being allocated onto the stack. to stop this kind of code using up all the system memory, the runtime limits how big the stack can get - which is why you are hitting this error.
To pass this test, you need a solution which doesn't call itself every time it hits an array within an array. Here's my solution, effectively using an array as a buffer; each time I hit a nested array I add it to the buffer. Once I finish processing the outer array, I then check if there is any arrays left in the buffer.
export const arraySum = (arr) => {
let sum = 0;
const buffer = [arr];
while (buffer.length > 0) {
const next = buffer.shift();
for (let i = 0; i < next.length; i++) {
if (typeof next[i] === "number") sum = sum + next[i];
else if (Array.isArray(next[i])) buffer.push(next[i]);
}
}
return sum;
};
That doesn't work bc like Keith said you are reaching the maximum call stack size.
RangeError: Maximum call stack size exceeded is a type of error thrown in JavaScript when a function call is made that exceeds the program's stack size. The call stack is a data structure that keeps track of which functions have been called, and in what order.
Maybe you can try to solve it in a iterative way like this:
const arraySum = (arr) => {
if (!Array.isArray(arr)) return 0;
let sum = 0;
while (Array.isArray(arr)) {
let arrCopy = [];
for (let i = 0; i < arr.length; i++) {
if (typeof arr[i] === "number") sum = sum + arr[i];
else if (Array.isArray(arr[i])) arrCopy = arrCopy.concat(arr[i]);
}
arr = arrCopy.length > 0 ? arrCopy : null;
}
return sum;
};
We can use a stack or queue in place of recursion. Also, order doesn't matter.
function f(A) {
let sum = 0;
let stack = [A];
while (stack.length > 0) {
stack.pop().forEach(e => {
if (Array.isArray(e))
stack.push(e);
else if (typeof e === "number")
sum += e;
});
}
return sum;
}
console.log(f([1,2,[3,[4]]]));
The approach of combining a flat based flattening function which circumvents call stack errors of very deeply nested arrays of a nesting depth close to 10_000 and higher and a reduce based function which does sum-up number types only, does solve the OP's problem. And the following implementation does prove it ...
// Implementation
function flatAlmostInfiniteNestedArray(arr) {
while (arr.length < (arr = arr.flat(1000)).length) {
}
return arr;
}
function totalNestedNumberValues(arr) {
return flatAlmostInfiniteNestedArray(arr)
.reduce((total, value) => (
'number' === typeof value
? total + value
: total
), 0);
}
// Test
const createDeeplyNestedArray = depth => {
let retval = [1];
for (let i = 0; i < depth - 1; i++) {
retval = [1, retval];
}
return retval;
};
const NUMBER_OF_ELEMENTS = 100000;
const arr = createDeeplyNestedArray(NUMBER_OF_ELEMENTS);
console.log(
'(totalNestedNumberValues(arr) === NUMBER_OF_ELEMENTS) ?..',
(totalNestedNumberValues(arr) === NUMBER_OF_ELEMENTS),
);
.as-console-wrapper { min-height: 100%!important; top: 0; }
Edit ... due to following comments
"Very interesting. I hadn't realized that Array.flat was implemented in a way that would lead to a call stack overflow. It makes sense that it would be, but I didn't know, and have never had a real-world structure where it would matter. Thanks!" – Scott Sauyet
"#ScottSauyet ...the poor man's approach of (mis)using flat as kind of a callstack-safe solution looses if it comes to performance ... jsbench.me :: sum up deeply nested array's number values" – Peter Seliger
One of the above linked performance tests features a better, much more performant, stack based, solution for flattening deeply nested arrays which are critical of being natively flattened due to possible overflowing call stacks.
Thus the formerly provided code example would change to ...
// Implementation
function flatCallstackCriticalNestedArray(nested) {
const stack = [nested];
const flat = [];
let value;
while (stack.length) {
if (Array.isArray(value = stack.pop())) {
stack.push(...value);
} else {
flat.push(value);
}
}
return flat;
}
function totalNestedNumberValues(arr) {
return flatCallstackCriticalNestedArray(arr)
.reduce((total, value) => (
'number' === typeof value
? total + value
: total
), 0);
}
// Test
const createDeeplyNestedArray = depth => {
let retval = [1];
for (let i = 0; i < depth - 1; i++) {
retval = [1, retval];
}
return retval;
};
const NUMBER_OF_ELEMENTS = 100000;
const arr = createDeeplyNestedArray(NUMBER_OF_ELEMENTS);
console.log(
'(totalNestedNumberValues(arr) === NUMBER_OF_ELEMENTS) ?..',
(totalNestedNumberValues(arr) === NUMBER_OF_ELEMENTS),
);
.as-console-wrapper { min-height: 100%!important; top: 0; }
Have you considered a using library? This might not be as fast as a vanilla solution, but it should still be plenty fast and much more readable.
.as-console-wrapper {max-height: 100% !important; top: 0}
<script type="module">
import objectScan from 'https://cdn.jsdelivr.net/npm/object-scan#18.3.0/lib/index.min.js';
const createDeeplyNestedArray = (depth) => {
let retval = [1];
for (let i = 0; i < depth - 1; i += 1) {
retval = [1, retval];
}
return retval;
};
const NUMBER_OF_ELEMENTS = 100000;
const arr = createDeeplyNestedArray(NUMBER_OF_ELEMENTS);
const arraySum = objectScan(['**'], {
rtn: 'sum',
filterFn: ({ value }) => typeof value === 'number'
});
console.log(arraySum(arr));
// => 100000
</script>
Disclaimer: I'm the author of object-scan
Note that this will traverse arrays and objects. If you only want to traverse arrays you could use ['**{[*]}'] as search needles.

How to find the number of subarrays in an array with given sum?

My program should be as following:
Input : {1,2,3,2,1,8,-3}, sum = 5
Output should be 3 example combinations ({2,3}, {3,2}, {8,-3}) have sum
exactly equal to 5.
I tried to do it in JavaScript but I'm confused.
function findSubarraySum(arr, sum) {
var res = 0;
var currentSum = 0;
for (var i = 0; i < arr.length; i++) {
currentSum += arr[i];
if (currentSum == sum)
res++;
}
return res;
}
console.log(findSubarraySum([1, 2, 3, 4], 10));
You first need a way to iterate over all the unique ways you can choose a start and and of your subarray boundaries (your slice definition).
In my code below, I use a combinations function to get all possible combinations of two indexes for the array supplied. You could do something else, like a simple doubly nested for loop.
Next you need to take the slice of the array according to the slice definition and reduce the elements into a sum. The Array.prototype.reduce function works well for that.
Finally, you want to include the subArray in the list of results only if the reduced sum matched the desired sum.
// Input : {1,2,3,2,1,8,-3}, sum = 5
const { combinations, range } = (() => {
const _combinations = function*(array, count, start, result) {
if (count <= 0) {
yield [...result]; // Yes, we want to return a copy
return;
}
const nextCount = count - 1;
const end = array.length - nextCount; // leave room on the array for the next remaining elements
for (let i = start; i < end; i += 1) {
// we have already used the element at (start - 1)
result[result.length - count] = array[i];
const nextStart = i + 1; // Always choose the next element from the ones following the last chosen element
yield* _combinations(array, nextCount, nextStart, result);
}
};
function* combinations(array, count) {
yield* _combinations(array, count, 0, Array(count));
}
function* range(l) {
for (let i = 0; i < l; i += 1) {
yield i;
}
}
return {
combinations,
range,
};
})();
const subArraysBy = (predicate, array) => {
const result = [];
for (const [beg, end] of combinations([...range(array.length+1)], 2)) {
const subArray = array.slice(beg, end);
if (predicate(subArray)) {
result.push(subArray);
}
}
return result;
};
const sum = array => array.reduce((sum, e) => sum + e);
console.log(
subArraysBy(
a => sum(a) === 5,
[1, 2, 3, 2, 1, 8, -3],
),
);
References:
MDN: Array.prototype.reduce
MDN: function* -- not required for your solution
Lodash: _.range -- implemented this in my code rather than use the lodash one. They work similarly.
Python Docs: combinations - My combinations implementation is inspired by python itertools.

Find averages in an array of different numbers

I feel like I didn't phrase my title very well, can someone please correct it if you understand my question.
I have an array of
arr = [1,2,3,4,5,
6,7,8,9,0,
3,4,7,2,1,
4,6,1,2,3,
5,6,8,9,3
2,3,4,5,6
]
And I want to do several things
Split it into chunks with the size of 5
Calculate the number of chunks. In this case, it should be 6 chunks.
Calculate the sum of numbers of all chunks in each position and divide it by the total number of chunks. In this case,
(1+6+3+4+5+2)/6, (2+7+4+6+6+3)/6, ..., (5+0+1+3+3+6)/6
Return results as an array
var result = [3.5, 4.66, ..., 3]
I have got the idea, but not sure how to implement it.
Thanks
I believe this code accomplishes what you want.
function averageValues (arr) {
var chunks = Math.ceil(arr.length / 5); // find the number of chunks
var sums = [0, 0, 0, 0, 0]; // keep a running tally
for (var i = 0; i < arr.length; i ++) {
sums[i % 5] += arr[i]; // add each element to the proper part of the sum
}
for (var i = 0; i < sums.length; i ++) {
sums[i] /= chunks; // divide each part of the sum by the number of chunks
}
return sums;
}
You can solve this by maintaining five separate sums to end with five separate averages.
Prepare your sums array of length 5:
var sums = [ 0, 0, 0, 0, 0 ];
For each number in your set, increment the corresponding sum by that number.
for (var x = 0; x < arr.length; x++)
sums[x % 5] += arr[x];
Divide each sum by how many numbers were used:
var numbers = arr.length / 5; // 6 numbers each
var result = sums.map(
function(s) {
return s / numbers; // divide each sum by 6
}
);
This uses the assumption that your set length is always a multiple of 5.
Here is a more functional approach to your problem. This uses the assumption that your set length is always a multiple of 5.
// add extra array helpers
Array.prototype.eachSlice = function (n, fn) {
let slices = [];
for (let i = 0; i < this.length; i += n) {
let slice = this.slice(i, i + n);
slices.push(slice);
}
if (fn) slices.forEach(fn);
return slices;
}
Array.prototype.sum = function (fn) {
let fnReduce = fn ? (acc, ...args) => acc + fn(...args) : (acc, v) => acc + v;
return this.reduce(fnReduce, 0);
}
Array.prototype.avg = function (fn) {
return this.sum(fn) / this.length;
}
// actual solution
let arr = [
1,2,3,4,5,
6,7,8,9,0,
3,4,7,2,1,
4,6,1,2,3,
5,6,8,9,3,
2,3,4,5,6,
];
let chunkSize = 5;
console.log('--- question #1 ---');
console.log('Split it into chunks with the size of 5.');
console.log('-------------------');
let chunks = arr.eachSlice(chunkSize);
console.log(chunks);
console.log('--- question #2 ---');
console.log('Calculate the number of chunks. In this case, it should be 6 chunks.');
console.log('-------------------');
console.log(chunks.length);
console.log('--- question #3 ---');
console.log('Calculate the sum of numbers of all chunks in each position and divide it by the total number of chunks.');
console.log('-------------------');
let avgChunks = new Array(chunkSize).fill()
.map((_, i) => chunks.avg(chunk => chunk[i]));
console.log('See the result under question #4.');
console.log('--- question #4 ---');
console.log('Return results as an array.');
console.log('-------------------');
console.log(avgChunks);
It could be useful:
//The average method using an array
function average(arr) {
var sum = arr.reduce(function (a,b) { return a + b; },0)
return sum/ arr.length
}
//Chunk array method, it returns an array of the sliced arrays by size
function chunkArray(arr, chunkSize){
var numChunks = arr.length / chunkSize
var chunks= []
for (let index = 0; index < numChunks; index++) {
chunks.push(arr.slice(index * chunkSize, (index * chunkSize) + chunkSize))
}
return chunks
}
//Finally, the average of arrays, it returns the average of each array
function averageArrays(arrs){
return arrs.map(function (arr) {
return average(arr)
})
}
//Example of usage
var chunks = chunkArray([
1,2,3,4,5,
6,7,8,9,0,
3,4,7,2,1,
4,6,1,2,3,
5,6,8,9,3,
2,3,4,5,6
],5)
console.log(averageArrays(chunks))
I think #Aplet123 has the most straight forward and easy to understand approach, though I changed up a little bit to suit my needs.
var chunks = Math.ceil(arr.length / 5) // Find the number of chunks
var sums = new Array(5).fill(0) // Keeps a running tally and fill values 0
arr.map((x, i) => sums[i%5] += arr[i]) // add each element to the proper part of the sum
var avgs = sums.map((x) => x/chunks /divide each part of the sum by the number of chunks

Deferred assignment in pure javascript

In this question I encountered the following simplified problem:
We start with an array of Objects with a value attribute. We want to calculate for each value what percentage of the sum of values it is, and add it to the structure as a property. To do this, we need to know the sum of values, but this sum is not calculated beforehand.
//Original data structure
[
{ "value" : 123456 },
{ "value" : 12146 }
]
//Becomes
[
{
"value" : 123456,
"perc" : 0.9104
},
{
"value" : 12146 ,
"perc" : 0.0896
}
]
An easy, and probably most readable, solution is to go through the data structure twice. First we calculate the sum, then we calculate the percentage and add it to the data structure.
var i;
var sum = 0;
for( i = 0; i < data.length; i++ ) {
sum += data[i].value;
}
for( i = 0; i < data.length; i++ ) {
data[i].perc = data[i].value / sum;
}
Can we instead just go through the data structure once, and somehow tell that the percentage expression should only be evaluated once the entire sum is known?
I am primarily interested in answers that address pure javascript. That is: Without any libraries.
A solution with self-modifying code.
It moves the function f for the calculation to the end of the iteration and then it goes through the chained functions for the assignments of the percentage of the single items.
var data = [
{ "value": 123456 },
{ "value": 12146 },
];
data.reduceRight(function (f, a, i) { // reduceRight, i=0 is at the end of reduce required
var t = f; // temporary save previous value or function
f = function (s) { // get a new function with sum as parameter
a.perc = a.value / s; // do the needed calc with sum at the end of reduce
t && t(s); // test & call the old func with sum as parameter
};
f.s = (t.s || 0) + a.value; // add value to sum and save sum in a property of f
i || f(f.s); // at the last iteration call f with sum as parameter
return f; // return the function
}, 0); // start w/ a value w/ a prop (undef/null don't work)
document.write('<pre>' + JSON.stringify(data, 0, 4) + '</pre>');
This solution uses a single loop to calculate the sum and place a computed perc property on each element using a getter:
function add_percentage(arr) {
var sum = 0;
arr.forEach(e => {
sum += e.value;
Object.defineProperty(e, "perc", {
get: function() { return this.value / sum; }
});
});
}
A straightforward deferral would just be
function add_percentage(arr) {
var sum = 0;
arr.forEach(e => {
sum += e.value;
setTimeout(() => e.perc = e.value / sum);
});
}
But, what is the point of doing this exactly?
A way to make this with one less loop is to write out the whole sum statement made up of all possible items, for instance
var sum = (data[0] ? data[0].value : 0) +
(data[1] ? data[1].value : 0) +
(data[2] ? data[2].value : 0) +
...
(data[50] ? data[50].value : 0);
for( i = 0; i < data.length; i++ ) {
data[i].perc = data[i].value / sum;
}
Not that this is actually a real solution
You could use Array's reduce function but that is still a loop in the background, and a function call for each array element:
var sum = data.reduce(function(output,item){
return output+item.value;
},0);
for( i = 0; i < data.length; i++ ) {
data[i].perc = data[i].value / sum;
}
You could use the ES6 Promise, but there you are still adding a bunch of function calls
var data = [
{ "value" : 123456 },
{ "value" : 12146 }
]
var sum = 0;
var rej = null;
var res = null;
var def = new Promise(function(resolve,reject){
rej = reject;
res = resolve;
});
function perc(total){
this.perc = this.value/total;
}
for( i = 0; i < data.length; i++ ) {
def.then(perc.bind(data[i]));
sum+=data[i].value;
}
res(sum);
Perf Tests
Addition statement
10,834,196
±0.44%
fastest
Reduce
3,552,539
±1.95%
67% slower
Promise
26,325
±8.14%
100% slower
For loops
9,640,800
±0.45%
11% slower
Looking at the problem some more, the desired effect is most easily reproduced using a stack. The easiest way of doing that here is by creating a recursive function instead of a loop. The recursive function will act as a loop, and the destacking can be used to set the percentage property.
/**
* Helper function for addPercentage
* #param arr Array of data objects
* #param index
* #param sum
* #return {number} sum
*/
function deferredAddPercentage(arr, index, sum) {
//Out of bounds
if (index >= arr.length) {
return sum;
}
//Pushing the stack
sum = deferredAddPercentage(arr, index + 1, sum + arr[index].value);
//Popping the stack
arr[index].percentage = arr[index].value / sum;
return sum;
}
/**
* Adds the percentage property to each contained object
* #param arr Array of data Objects
*/
function addPercentage(arr) {
deferredAddPercentage(arr, 0, 0);
}
// ******
var data = [{
"value": 10
}, {
"value": 20
}, {
"value": 20
}, {
"value": 50
}];
addPercentage(data);
console.log( data );
It will perform 29% worse than 2 simple for-loops. Extended Patrick's JSPerf.
The OP has already given an example of a recursive solution. Although I believe that a non-tail recursive function is the ideal approach for this task, I think their implementation has two drawbacks though:
It mutates state of its parent scope
It is very specific and thus hardly reusable
I'm trying to implement a more generic solution without mutating global state. Please note that I would usually solve this problem by combining several smaller, reusable functions. The OP's condition is, however, to have only a single loop. This is a fun challenge and my implementation isn't intended for being used in real code!
I call the function defmap that is, deferred map:
const xs = [
{ "value" : 10 },
{ "value" : 20 },
{ "value" : 20 },
{ "value" : 50 }
];
const defmap = red => map => acc => xs => {
let next = (len, acc, [head, ...tail]) => {
map = tail.length
? next(len, red(acc, head), tail)
: map([], red(acc, head), len);
return map(Object.assign({}, head));
};
return next(xs.length, acc, xs);
};
const map = f => (xs, acc, len) => o => xs.length + 1 < len
? map(f) (append(f(o, acc), xs), acc, len)
: append(f(o, acc), xs);
const append = (xs, ys) => [xs].concat(ys);
const reducer = (acc, o) => acc + o.value;
const mapper = (o, acc) => Object.assign(o, {perc: o.value / acc});
console.log(defmap(reducer) (map(mapper)) (0) (xs));
As per my comment, I can not see a way to do this without effectively looping over it twice.
To actually count the values
To evaluate each value against the total
To answer the "deferred" part of your question, one possible solution, albeit slower (Just guessing due to function call?) and probably not what you would want to use (JSFiddle):
var data = [
{ value: 10 },
{ value: 20 },
{ value: 20 },
{ value: 50 }
];
var total = 0;
for (var i = 0; i < data.length; i++) {
var current = data[i];
total += current["value"];
current.getPercent = function() { return this["value"] / total; };
}
for (var i = 0; i < data.length; i++) {
var current = data[i];
console.log(current.getPercent());
}
Outputs:
0.1
0.2
0.2
0.5
This has the added benefit of not actually calculating the value until you need it, but when it does calculate it, there will be a higher cpu cost (Due to calling the function etc).
This could be marginally optimised by changing the getPercent line to:
current.getPercent = function() {
return this["percent"] || (this["percent"] = this["value"] / total);
}
Which would ensure the calculation is only run the first time. Updated Fiddle
EDIT
I ran some tests (Forgot to save before I crashed chrome by testing too many iterations, but they are simple enough to replicate).
I was getting
Sumurai initial method (1000000 objects with value 0 -> 9999999) = 2200ms
My initial method (Same) = 3800ms
My "optimised" method (Same) = 4200ms

What is the best way to sum arrays using ECMASCRIPT 6 Generator/Functions

Is there a better way instead of adding values of arrays up using a generator function as closure?
var sumArrays = function(){
var sum = 0;
return function*(){
while(true){
var array = yield sum;
if(array.__proto__.constructor === Array){
sum += array.reduce(function(val,val2){ return val+val2; });
}
else sum=0;
}
};
};
var gen = sumArrays();
// is this step required to make a generator or could it be done at least differently to spare yourself from doing this step?
gen = gen();
// sum some values of arrays up
console.log('sum: ',gen.next()); // Object { value=0, done=false}
console.log('sum: ',gen.next([1,2,3,4])); // Object { value=10, done=false}
console.log('sum: ',gen.next([6,7])); // Object { value=23, done=false}
// reset values
console.log('sum: ',gen.next(false)); // Object { value=0, done=false}
console.log('sum: ',gen.next([5])); // Object { value=5, done=false}
This doesn't seem to be a problem generators are supposed to solve, so I would not use a generator here.
Directly using reduce (ES5) seems to be more appropriate:
let sum = [1,2,3,4].reduce((sum, x) => sum + x);
As a function:
function sum(arr) {
return arr.reduce((sum, x) => sum + x);
}
If you really want to sum multiple arrays across multiple function calls, then return a normal function:
function getArraySummation() {
let total = 0;
let reducer = (sum, x) => sum + x;
return arr => total + arr.reduce(reducer);
}
let sum = getArraySummation();
console.log('sum:', sum([1,2,3])); // sum: 6
console.log('sum:', sum([4,5,6])); // sum: 15
Keep it simple.
Here is use of for/of loop and arrow function,
const sumArray = myArray => {
let sum = 0;
for(const value of myArray)
sum+= value;
return sum;
}
const myArray = [1, 2, 5];
console.log(sumArray(myArray));

Categories

Resources