Thats how I do it:
function processArray(array, index, callback) {
processItem(array[index], function(){
if(++index === array.length) {
callback();
return;
}
processArray(array, index, callback);
});
};
function processItem(item, callback) {
// do some ajax (browser) or request (node) stuff here
// when done
callback();
}
var arr = ["url1", "url2", "url3"];
processArray(arr, 0, function(){
console.log("done");
});
Is it any good? How to avoid those spaghetti'ish code?
Checkout the async library, it's made for control flow (async stuff) and it has a lot of methods for array stuff: each, filter, map. Check the documentation on github. Here's what you probably need:
each(arr, iterator, callback)
Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the each function is immediately called with the error.
eachSeries(arr, iterator, callback)
The same as each only the iterator is applied to each item in the array in series. The next iterator is only called once the current one has completed processing. This means the iterator functions will complete in order.
As pointed in some answer one can use "async" library. But sometimes you just don't want to introduce new dependency in your code. And below is another way how you can loop and wait for completion of some asynchronous functions.
var items = ["one", "two", "three"];
// This is your async function, which may perform call to your database or
// whatever...
function someAsyncFunc(arg, cb) {
setTimeout(function () {
cb(arg.toUpperCase());
}, 3000);
}
// cb will be called when each item from arr has been processed and all
// results are available.
function eachAsync(arr, func, cb) {
var doneCounter = 0,
results = [];
arr.forEach(function (item) {
func(item, function (res) {
doneCounter += 1;
results.push(res);
if (doneCounter === arr.length) {
cb(results);
}
});
});
}
eachAsync(items, someAsyncFunc, console.log);
Now, running node iterasync.js will wait for about three seconds and then print [ 'ONE', 'TWO', 'THREE' ]. This is a simple example, but it can be extended to handle many situations.
As correctly pointed out, you have to use setTimeout, for example:
each_async = function(ary, fn) {
var i = 0;
-function() {
fn(ary[i]);
if (++i < ary.length)
setTimeout(arguments.callee, 0)
}()
}
each_async([1,2,3,4], function(p) { console.log(p) })
The easiest way to handle async iteration of arrays (or any other iterable) is with the await operator (only in async functions) and for of loop.
(async function() {
for(let value of [ 0, 1 ]) {
value += await(Promise.resolve(1))
console.log(value)
}
})()
You can use a library to convert any functions you may need which accept callback to return promises.
In modern JavaScript there are interesting ways to extend an Array into an async itarable object.
Here I would like to demonstrate a skeleton of a totally new type AsyncArray which extends the Array type by inheriting it's goodness just to become an async iterable array.
This is only available in the modern engines. The code below uses the latest gimmicks like the private instance fields and for await...of.
If you are not familiar with them then I would advise you to have a look at the above linked topics in advance.
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
So an Async Iterable Array must contain promises. Only then it can return an iterator object which with every next() call returns a promise to eventually resolve into an object like {value : "whatever", done: false} or {done: true}. So basically everything returned is a promise here. The await abstraction unpacks the value within and gives it to us.
Now as I mentioned before, this AsyncArray type, since extended from Array, allows us to use those Array methods we are familiar with. That should simplify our job.
Let's see what happens;
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
var aa = AsyncArray.from({length:10}, (_,i) => new Promise(resolve => setTimeout(resolve,i*1000,[i,~~(Math.random()*100)])));
async function getAsycRandoms(){
for await (let random of aa){
console.log(`The Promise at index # ${random[0]} gets resolved with a random value of ${random[1]}`);
};
};
getAsycRandoms();
For modern Node.js:
To iterate through a collection truly asynchronously, you can try my tiny package with zero dependencies, compatible with ESM and CJS modules with .d.ts typings. Check the code it's really tiny.
https://www.npmjs.com/package/array-to-async-iterable
You can use it just like this:
for await(const el of new AsyncTimeIterator(arrayOfObjects)){
...
}
You can't just use for await of loop because of the JavaScript engines' microtasks and macrotasks nature.
In a brief, you won't get new HTTP requests and let other timers' callbacks to be executed with this code:
for await(const el of array){
...
}
You force V8 or the other engine to execute all the microtasks (your loop iteration) and when the loop completes you'll unblock the event loop and be ready to receive HTTP connections. So this code is completely useless.
Related
I am thinking about a scenario of building up a promise queue:
//Let's assume that promises is an array of promises
var promiseQueue = [];
for (var promise of promises) {
if (promiseQueue.length) promiseQueue[promiseQueue.length - 1].then(promise);
promiseQueue.push(promise);
}
I am thinking about implementing a function called resolver:
function *resolve() {
var promise;
while (promise = yield) Promise.resolve(promise);
}
and then iterating it:
var promiseGenerator = resolve();
The problem is the for..of here which would be responsible for the actual iteration:
for (var r of promiseGenerator) {
}
At the code above the generator will be successfully iterated, but unfortunately I am not aware of a way to successfully pass a parameter to this generator at the iteration of for..of.
I would like to clarify that I do not need an alternative, I am perfectly aware that we can do something like this:
for (var p in promiseQueue) promiseGenerator.next(promiseQueue[p]);
I am specifically interested to know whether I can pass parameters to the generator when I execute a for..of cycle.
EDIT
The problem raised by amn is that in the example he/she was focusing on would always get undefined. That's true if we pass undefined to next(), but not true if we pass something else. The problem I was raising is that a for..of loop does not allow us to pass anything to yield, which is this specific question is all about, the example is a mere illustration of the problem, showing that the promises we would create will never be created in a for..of loop. However, there is life for Iterable objects outside the realm of for..of loops and we can pass defined values into the yield. An example with the criticized code chunk can look like:
function *resolve() {
var promise;
while (promise = yield) Promise.resolve(promise);
}
var responses = [];
var f = resolve();
var temp;
for (var i = 10; !(temp = f.next(i)).done; i--) responses.push(temp);
As we can see above, the yield above cannot be assumed ab ovo to be undefined. And of course we can pass some custom thenables, like
Promise.resolve({
then: function(onFulfill, onReject) { onFulfill('fulfilled!'); }
});
or even promises which were not resolved yet. The point of the example was to show that we cannot pass values to the yield using the for..of loop, which is quite a feature gap in my opinion.
No, it is not possible to pass arguments to next.
function* generateItems() { /* ... */ }
for (var item of generateItems()) {
console.log(item);
}
is mostly short for
function* generateItems() { /* ... */ }
var iterator = generateItems()[Symbol.iterator]();
do {
const result = iterator.next();
if (result.done) break;
const item = result.value;
console.log(item);
} while (true);
barring a few missing try/catch wrappers. You can see in the spec here that it calls .next with no arguments:
Let nextResult be ? Call(iteratorRecord.[[NextMethod]], iteratorRecord.[[Iterator]], « »).
e.g.
iterator.next.apply(iterator, []);
calling next() with an empty array of arguments.
I have an array of functions, as in:
funcArray = [func1, func2, func3];
When in a given function, I want to execute the next function in the array. How do I do this? Here is my basic skeleton:
function func1() {
// I get current function caller
var currentFunc = func1.caller;
// I want to execute the next function. Happens to be func2 in the example.
}
I cannot use indexOf function, as one would for an array of strings or numbers.
NOTE: This question appears to be similar to this and the one it refers to. However, it is a different question.
I want to alter the sequence of processing by merely modifying the array. That's the goal. A possibly more efficient approach would be appreciated.
Clarification: Based upon some of the comments:
funcArray is global.
The goal is to implement middleware for a Node.js HTTP module in as simple and efficient a manner as possible without using any third-party modules.
Unless func1 closes over funcArray, you cannot have it reach out and find func2 and execute it, nor should you. Even if func1 does close over funcArray, it would be poor separation of concerns for func1 to reach out and find itself in funcArray and then execute func2.
Instead, have other code that's in charge of running the functions.
If they're synchronous
If the functions complete their work synchronously, then it's simply:
funcArray.forEach(fn => fn());
or
for (const fn of funcArray) {
fn();
}
or if the result of one function should be passed to the next, you can use reduce:
const finalResult = funcArray.reduce((previousResult, fn) => fn(previousResult), undefined);
...where undefined is the value to pass to func1.
If they're asynchronous
If they don't do their work synchronously, you'll need to provide them a way to notify their caller that they've completed their work. Promises are a good, standard way to do that, but you could use simple callbacks instead.
If you make them return promises, for instance, you can use the old promise reduce trick:
funcArray.reduce((p, fn) => {
return p.then(() => {
fn();
});
}, Promise.resolve());
or if the result of one function should be passed to the next:
funcArray.reduce((p, fn) => {
return p.then(fn);
}, Promise.resolve());
You can provide an argument to Promise.resolve to set the value to pass to func1 (without one, it'll receive undefined).
You can bind to the function the index where it is in the array so you can use this index to get and call the next function:
var funcArray = [func1, func2];
var boundFuncArray = funcArray.map((f, i) => f.bind(null, i));
boundFuncArray[0]();
function func1(nextFunctionIndex) {
console.log('func1 called');
// Execute next function:
var nextFunc = boundFuncArray[nextFunctionIndex + 1];
nextFunc && nextFunc();
}
function func2(nextFunctionIndex) {
console.log('func2 called');
// Execute next function:
var nextFunc = boundFuncArray[nextFunctionIndex + 1];
nextFunc && nextFunc();
}
As T.J Crowder stated in the comment below, you can also bind the next function to the current one:
var funcArray = [func1, func2];
var boundFuncArray= funcArray.map((f, i, arr) => f.bind(null, arr[i + 1]));
boundFuncArray[0]();
function func1(nextFunc) {
console.log('func1 called');
// Execute next function:
nextFunc && nextFunc();
}
function func2(nextFunc ) {
console.log('func2 called');
// Execute next function:
nextFunc && nextFunc();
}
You can get the current function's name with arguments.callee.name, loop through the array of functions, and call the next function:
funcArray = [func1, func2, func3];
// Only func1() and func2() will be documented since the others have repeating code
function func1() {
// show the current function name
console.log(arguments.callee.name);
// loop the array of functions
for(var i = 0; i < funcArray.length; ++i)
{
// when the current array item is our current function name and
// another function exists after this then call it and break
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
function func2() {
console.log(arguments.callee.name);
// some logic which switches our next function to be func4()
funcArray[2] = func4;
for(var i = 0; i < funcArray.length; ++i)
{
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
function func3() {
console.log(arguments.callee.name);
for(var i = 0; i < funcArray.length; ++i)
{
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
function func4() {
console.log(arguments.callee.name);
for(var i = 0; i < funcArray.length; ++i)
{
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
// call the first function
funcArray[0]();
Output:
func1
func2
func4
I have solved it this way:
// Adding next options to array
function addNext(array) {
array.last = 1
Object.defineProperty(array, 'next', {get:
function() {
if(this.last < this.length) {
this.last++
return this[this.last-1]
} else {
this.last = 1
return () => {}
}
}
});
}
// The functions for array (has to be function and not arrow function)
function first(param) {
console.log('first',param)
return this.next(param)
}
function second(param) {
console.log('second',param)
return this.next(param)
}
function third(param) {
console.log('third',param)
return this.next(param)
}
// The array
let fns = [first,second,third]
// Adding next option to array
addNext(fns)
// Run first function from array
fns[0]('test')
I dont know if your functions require certain parameters but this is the first thing that came to my mind.
var functArray = [
function() {
console.log("function1 executed");
},
function() {
console.log("function2 executed");
},
function() {
console.log("function3 executed");
},
function() {
console.log("function4 executed");
}];
functArray.forEach(function(x){
x();
});
The accepted answer and other comments did help me, but the way I implemented it is as follows:
//The functions are defined as variables.
//They do not get hoisted, so must be defined first.
func1 = function (arg1, arg2) {
//Code to do whatever...
...
//Execute the next function.
//The name of the function is returned by executing nextFunc()
global[nextFunc()](arg1, arg2, arg3);
}
func2 = function (arg1) { //Note different type of args
...
}
//Note that this is an array of strings representing function names.
funcArray = ["func1", "func2", "func3",...]
//Start the execution...
func1(arg1, arg2);
function nextFunc() {
var currentFuncName = nextFunc.caller.name;
var index = funcArray.indexOf(currentFuncName);
if (index < funcArray.length)
return funcArray[index+1];
}
The sequence of functions to be executed is easily managed through the array funcArray. The number or type of arguments is not fixed for each function. Additionally, the functions control if they should stop the chain or continue with the next function.
It is very simple to understand requiring basic Javascript skills. No overheads of using Promises.
"global" gets replaced by "window" for browser. This is a Node.js implementation. The use of function names in the array will, however, break if you minify the JS code. As I am going to use it on the server, I do not expect to minify it.
You can do it in this way with promise.all if your functions to be executed in parallel.
let toBeExecutedList = [];
toBeExecutedList.push(() => this.addTwoNumber(2, 3));
toBeExecutedList.push(()=>this.square(2));
And Then wherever you want to use them, do it like this:
const resultArr = await Promise.all([
toBeExecutedList.map(func => func()),
]);
Suppose I have the the following Promise chain:
var result = Promise.resolve(filename)
.then(unpackDataFromFile)
.then(transformData)
.then(compileDara)
.then(writeData);
Now I have not only one transformData function but two or more, stored in an array. I want to try the first one, and if the compileData function fails, try the second one and so on until either compileData succeeds or the array of transformData functions is exhausted.
Can someone give me an example on how to implement this?
Running all transformData functions and give the result array to compileData is not an option, since the functions are very expensive and I want to run as few as possible of them.
transformData itself also returns a Promise, if that helps.
I would start by isolating the notion of trying a number of promises until one succeeds:
function tryMultiple([promise, ...rest]) {
if (!promise) throw new Error("no more to try");
return promise.catch(() => tryMultiple(rest));
}
Now write a handler which tries each combination of transforming and compiling:
function transformAndCompile(transformers) {
return function(data) {
return tryMultiple(transformers.map(t => t(data).then(compileData)));
};
}
Now the top level is just:
var result = Promise.resolve(filename)
.then(unpackDataFromFile)
.then(transformAndCompile(transformers))
.then(writeData);
By the way, Promise.resolve(filename).then(unpackDataFromFile) is just a roundabout way of saying unpackDataFromFile(filename).
You can do something like this:
// various transformer functions to try in order to be tried
var transformers = [f1, f2, f3, f4];
function transformFile(filename) {
// initialize tIndex to select next transformer function
var tIndex = 0;
var p = unpackDataFromFile(filename);
function run() {
return p.then(transformers[tIndex++])
.then(compileData)
.catch(function(err) {
if (tIndex < transformers.length) {
// execute the next transformer, returning
// a promise so it is linked into the chain
return run();
} else {
// out of transformers, so reject and stop
throw new Error("No transformer succeeded");
}
}).then(writeData);
}
return run();
}
transformFile("someData.txt").then(function(finalResult) {
// succeeded here
}).catch(function(err) {
// error here
});
Here's how this works:
Sets up a tIndex variable that indexes into the array of transformer functions.
Calls unpackDataFromFile(filename) and saves the resulting promise.
Then executes the sequence p.then(transformer).then(compileData) using the first transformer. If that succeeds, it calls writeData and returns the resulting promise.
If either the transformer or compileData fails, then it goes to the next transformer function and starts over. The key here to making this work is that in the .catch() handler, it returns a new promise which chains into the originally returned promise. Each new call to run() is chained onto the original promise from unpackDataFromFile() which allows you to reuse that result.
Here's a bit more generic implementation that makes an iterator for an array that iterates until the iterator callback returns a promise that fulfills.
// Iterate an array using an iterator that returns a promise
// Stop iterating as soon as you get a fulfilled promise from the iterator
// Pass:
// p - Initial promise (can be just Promise.resolve(data))
// array - array of items to pass to the iterator one at a time
// fn - iterator function that returns a promise
// iterator called as fn(data, item)
// data - fulfilled value of promise passed in
// item - array item for this iteration
function iterateAsyncUntilSuccess(p, array, fn) {
var index = 0;
function next() {
if (index < array.length) {
var item = array[index++];
return p.then(function(data) {
return fn(data, item).catch(function(err) {
// if this one fails, try the next one
return next();
});
});
} else {
return Promise.reject(new Error("End of data with no operation successful"));
}
}
return next();
}
// Usage:
// various transformer functions to try in order to be tried
var transformers = [f1, f2, f3, f4];
iterateAsyncUntil(unpackDataFromFile(filename), transformers, function(data, item) {
return item(data).then(compileData);
}).then(writeData).then(function(result) {
// successfully completed here
}).catch(function(err) {
// error here
});
The following should do what you want most idiomatically:
var transformers = [transformData, transformData2];
var result = unpackDataFromFile(filename)
.then(function transpile(data, i = 0) {
return transformers[i](data).then(compileData)
.catch(e => ++i < transformers.length? transpile(data, i) : Promise.reject(e));
})
.then(writeData);
Basically you recurse on the transformers array, using .catch().
Thats how I do it:
function processArray(array, index, callback) {
processItem(array[index], function(){
if(++index === array.length) {
callback();
return;
}
processArray(array, index, callback);
});
};
function processItem(item, callback) {
// do some ajax (browser) or request (node) stuff here
// when done
callback();
}
var arr = ["url1", "url2", "url3"];
processArray(arr, 0, function(){
console.log("done");
});
Is it any good? How to avoid those spaghetti'ish code?
Checkout the async library, it's made for control flow (async stuff) and it has a lot of methods for array stuff: each, filter, map. Check the documentation on github. Here's what you probably need:
each(arr, iterator, callback)
Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the each function is immediately called with the error.
eachSeries(arr, iterator, callback)
The same as each only the iterator is applied to each item in the array in series. The next iterator is only called once the current one has completed processing. This means the iterator functions will complete in order.
As pointed in some answer one can use "async" library. But sometimes you just don't want to introduce new dependency in your code. And below is another way how you can loop and wait for completion of some asynchronous functions.
var items = ["one", "two", "three"];
// This is your async function, which may perform call to your database or
// whatever...
function someAsyncFunc(arg, cb) {
setTimeout(function () {
cb(arg.toUpperCase());
}, 3000);
}
// cb will be called when each item from arr has been processed and all
// results are available.
function eachAsync(arr, func, cb) {
var doneCounter = 0,
results = [];
arr.forEach(function (item) {
func(item, function (res) {
doneCounter += 1;
results.push(res);
if (doneCounter === arr.length) {
cb(results);
}
});
});
}
eachAsync(items, someAsyncFunc, console.log);
Now, running node iterasync.js will wait for about three seconds and then print [ 'ONE', 'TWO', 'THREE' ]. This is a simple example, but it can be extended to handle many situations.
As correctly pointed out, you have to use setTimeout, for example:
each_async = function(ary, fn) {
var i = 0;
-function() {
fn(ary[i]);
if (++i < ary.length)
setTimeout(arguments.callee, 0)
}()
}
each_async([1,2,3,4], function(p) { console.log(p) })
The easiest way to handle async iteration of arrays (or any other iterable) is with the await operator (only in async functions) and for of loop.
(async function() {
for(let value of [ 0, 1 ]) {
value += await(Promise.resolve(1))
console.log(value)
}
})()
You can use a library to convert any functions you may need which accept callback to return promises.
In modern JavaScript there are interesting ways to extend an Array into an async itarable object.
Here I would like to demonstrate a skeleton of a totally new type AsyncArray which extends the Array type by inheriting it's goodness just to become an async iterable array.
This is only available in the modern engines. The code below uses the latest gimmicks like the private instance fields and for await...of.
If you are not familiar with them then I would advise you to have a look at the above linked topics in advance.
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
So an Async Iterable Array must contain promises. Only then it can return an iterator object which with every next() call returns a promise to eventually resolve into an object like {value : "whatever", done: false} or {done: true}. So basically everything returned is a promise here. The await abstraction unpacks the value within and gives it to us.
Now as I mentioned before, this AsyncArray type, since extended from Array, allows us to use those Array methods we are familiar with. That should simplify our job.
Let's see what happens;
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
var aa = AsyncArray.from({length:10}, (_,i) => new Promise(resolve => setTimeout(resolve,i*1000,[i,~~(Math.random()*100)])));
async function getAsycRandoms(){
for await (let random of aa){
console.log(`The Promise at index # ${random[0]} gets resolved with a random value of ${random[1]}`);
};
};
getAsycRandoms();
For modern Node.js:
To iterate through a collection truly asynchronously, you can try my tiny package with zero dependencies, compatible with ESM and CJS modules with .d.ts typings. Check the code it's really tiny.
https://www.npmjs.com/package/array-to-async-iterable
You can use it just like this:
for await(const el of new AsyncTimeIterator(arrayOfObjects)){
...
}
You can't just use for await of loop because of the JavaScript engines' microtasks and macrotasks nature.
In a brief, you won't get new HTTP requests and let other timers' callbacks to be executed with this code:
for await(const el of array){
...
}
You force V8 or the other engine to execute all the microtasks (your loop iteration) and when the loop completes you'll unblock the event loop and be ready to receive HTTP connections. So this code is completely useless.
I have a question regarding the native Array.forEach implementation of JavaScript: Does it behave asynchronously?
For example, if I call:
[many many elements].forEach(function () {lots of work to do})
Will this be non-blocking?
No, it is blocking. Have a look at the specification of the algorithm.
However a maybe easier to understand implementation is given on MDN:
if (!Array.prototype.forEach)
{
Array.prototype.forEach = function(fun /*, thisp */)
{
"use strict";
if (this === void 0 || this === null)
throw new TypeError();
var t = Object(this);
var len = t.length >>> 0;
if (typeof fun !== "function")
throw new TypeError();
var thisp = arguments[1];
for (var i = 0; i < len; i++)
{
if (i in t)
fun.call(thisp, t[i], i, t);
}
};
}
If you have to execute a lot of code for each element, you should consider to use a different approach:
function processArray(items, process) {
var todo = items.concat();
setTimeout(function() {
process(todo.shift());
if(todo.length > 0) {
setTimeout(arguments.callee, 25);
}
}, 25);
}
and then call it with:
processArray([many many elements], function () {lots of work to do});
This would be non-blocking then. The example is taken from High Performance JavaScript.
Another option might be web workers.
If you need an asynchronous-friendly version of Array.forEach and similar, they're available in the Node.js 'async' module: http://github.com/caolan/async ...as a bonus this module also works in the browser.
async.each(openFiles, saveFile, function(err){
// if any of the saves produced an error, err would equal that error
});
There is a common pattern for doing a really heavy computation in Node that may be applicable to you...
Node is single-threaded (as a deliberate design choice, see What is Node.js?); this means that it can only utilize a single core. Modern boxes have 8, 16, or even more cores, so this could leave 90+% of the machine idle. The common pattern for a REST service is to fire up one node process per core, and put these behind a local load balancer like http://nginx.org/.
Forking a child -
For what you are trying to do, there is another common pattern, forking off a child process to do the heavy lifting. The upside is that the child process can do heavy computation in the background while your parent process is responsive to other events. The catch is that you can't / shouldn't share memory with this child process (not without a LOT of contortions and some native code); you have to pass messages. This will work beautifully if the size of your input and output data is small compared to the computation that must be performed. You can even fire up a child node.js process and use the same code you were using previously.
For example:
var child_process = require('child_process');
function run_in_child(array, cb) {
var process = child_process.exec('node libfn.js', function(err, stdout, stderr) {
var output = JSON.parse(stdout);
cb(err, output);
});
process.stdin.write(JSON.stringify(array), 'utf8');
process.stdin.end();
}
Array.forEach is meant for computing stuff not waiting, and there is nothing to be gained making computations asynchronous in an event loop (webworkers add multiprocessing, if you need multi-core computation). If you want to wait for multiple tasks to end, use a counter, which you can wrap in a semaphore class.
Edit 2018-10-11:
It looks like there is a good chance the standard described below may not go through, consider pipelineing as an alternative (does not behave exactly the same but methods could be implemented in a similar manor).
This is exactly why I am excited about es7, in future you will be able to do something like the code below (some of the specs are not complete so use with caution, I will try to keep this up to date). But basically using the new :: bind operator, you will be able to run a method on an object as if the object's prototype contains the method. eg [Object]::[Method] where normally you would call [Object].[ObjectsMethod]
Note to do this today (24-July-16) and have it work in all browsers you will need to transpile your code for the following functionality:Import / Export, Arrow functions, Promises, Async / Await and most importantly function bind. The code below could be modfied to use only function bind if nessesary, all this functionality is neatly available today by using babel.
YourCode.js (where 'lots of work to do' must simply return a promise, resolving it when the asynchronous work is done.)
import { asyncForEach } from './ArrayExtensions.js';
await [many many elements]::asyncForEach(() => lots of work to do);
ArrayExtensions.js
export function asyncForEach(callback)
{
return Promise.resolve(this).then(async (ar) =>
{
for(let i=0;i<ar.length;i++)
{
await callback.call(ar, ar[i], i, ar);
}
});
};
export function asyncMap(callback)
{
return Promise.resolve(this).then(async (ar) =>
{
const out = [];
for(let i=0;i<ar.length;i++)
{
out[i] = await callback.call(ar, ar[i], i, ar);
}
return out;
});
};
This is a short asynchronous function to use without requiring third party libs
Array.prototype.each = function (iterator, callback) {
var iterate = function () {
pointer++;
if (pointer >= this.length) {
callback();
return;
}
iterator.call(iterator, this[pointer], iterate, pointer);
}.bind(this),
pointer = -1;
iterate(this);
};
There is a package on npm for easy asynchronous for each loops.
var forEachAsync = require('futures').forEachAsync;
// waits for one request to finish before beginning the next
forEachAsync(['dogs', 'cats', 'octocats'], function (next, element, index, array) {
getPics(element, next);
// then after all of the elements have been handled
// the final callback fires to let you know it's all done
}).then(function () {
console.log('All requests have finished');
});
Also another variation forAllAsync
It is possible to code even the solution like this for example :
var loop = function(i, data, callback) {
if (i < data.length) {
//TODO("SELECT * FROM stackoverflowUsers;", function(res) {
//data[i].meta = res;
console.log(i, data[i].title);
return loop(i+1, data, errors, callback);
//});
} else {
return callback(data);
}
};
loop(0, [{"title": "hello"}, {"title": "world"}], function(data) {
console.log("DONE\n"+data);
});
On the other hand, it is much slower than a "for".
Otherwise, the excellent Async library can do this: https://caolan.github.io/async/docs.html#each
These code snippet will give you better understanding of forEach and forOf comparison.
/* eslint-disable no-console */
async function forEachTest() {
console.log('########### Testing forEach ################ ')
console.log('start of forEachTest func')
let a = [1, 2, 3]
await a.forEach(async (v) => {
console.log('start of forEach: ', v)
await new Promise(resolve => setTimeout(resolve, v * 1000))
console.log('end of forEach: ', v)
})
console.log('end of forEachTest func')
}
forEachTest()
async function forOfTest() {
await new Promise(resolve => setTimeout(resolve, 10000)) //just see console in proper way
console.log('\n\n########### Testing forOf ################ ')
console.log('start of forOfTest func')
let a = [1, 2, 3]
for (const v of a) {
console.log('start of forOf: ', v)
await new Promise(resolve => setTimeout(resolve, v * 1000))
console.log('end of forOf: ', v)
}
console.log('end of forOfTest func')
}
forOfTest()
Here is a small example you can run to test it:
[1,2,3,4,5,6,7,8,9].forEach(function(n){
var sum = 0;
console.log('Start for:' + n);
for (var i = 0; i < ( 10 - n) * 100000000; i++)
sum++;
console.log('Ended for:' + n, sum);
});
It will produce something like this(if it takes too less/much time, increase/decrease the number of iterations):
(index):48 Start for:1
(index):52 Ended for:1 900000000
(index):48 Start for:2
(index):52 Ended for:2 800000000
(index):48 Start for:3
(index):52 Ended for:3 700000000
(index):48 Start for:4
(index):52 Ended for:4 600000000
(index):48 Start for:5
(index):52 Ended for:5 500000000
(index):48 Start for:6
(index):52 Ended for:6 400000000
(index):48 Start for:7
(index):52 Ended for:7 300000000
(index):48 Start for:8
(index):52 Ended for:8 200000000
(index):48 Start for:9
(index):52 Ended for:9 100000000
(index):45 [Violation] 'load' handler took 7285ms
Although Array.forEach is not asynchronous, you can get asynchronous "end result". Example below:
function delayFunction(x) {
return new Promise(
(resolve) => setTimeout(() => resolve(x), 1000)
);
}
[1, 2, 3].forEach(async(x) => {
console.log(x);
console.log(await delayFunction(x));
});
Use Promise.each of bluebird library.
Promise.each(
Iterable<any>|Promise<Iterable<any>> input,
function(any item, int index, int length) iterator
) -> Promise
This method iterates over an array, or a promise of an array, which contains promises (or a mix of promises and values) with the given iterator function with the signature (value, index, length) where the value is the resolved value of a respective promise in the input array. Iteration happens serially. If the iterator function returns a promise or a thenable, then the result of the promise is awaited before continuing with next iteration. If any promise in the input array is rejected, then the returned promise is rejected as well.
If all of the iterations resolve successfully, Promise.each resolves to the original array unmodified. However, if one iteration rejects or errors, Promise.each ceases execution immediately and does not process any further iterations. The error or rejected value is returned in this case instead of the original array.
This method is meant to be used for side effects.
var fileNames = ["1.txt", "2.txt", "3.txt"];
Promise.each(fileNames, function(fileName) {
return fs.readFileAsync(fileName).then(function(val){
// do stuff with 'val' here.
});
}).then(function() {
console.log("done");
});