I need to execute a bunch of asynchronous methods (client SQLite database), and call only one final callback.
Of course, the ugly way is:
execAll : function(callBack) {
asynch1(function() {
asynch2(function() {
...
asynchN(function() {
callBack();
})
})
});
}
But I know there are better ways to do it. Intuitively I would detect when all call back has been called with a counter to call the final callback.
I think this is a common design-pattern, so if someone could point me in the right direction...
Thanks in advance !
this is easy
var callback = (function(){
var finishedCalls = 0;
return function(){
if (++finishedCalls == 4){
//execute your action here
}
};
})();
Just pass this callback to all your methods, and once it has been called 4 times it will execute.
If you want to use factory for this then you can do the following
function createCallback(limit, fn){
var finishedCalls = 0;
return function(){
if (++finishedCalls == limit){
fn();
}
};
}
var callback = createCallback(4, function(){
alert("woot!");
});
async1(callback);
async2(callback);
async3(callback);
async4(callback);
I've written some async utilities you might find useful, allowing you to write your example as:
function(callback) {
async.series([
asynch1(),
asynch2(),
...
asynchN()
], callback);
}
Or, if you wanted to run them in parallel, as:
function(callback) {
async.parallel([
asynch1(),
asynch2(),
...
asynchN()
], callback);
}
There are loads of other useful functions like async map/reduce too:
http://caolanmcmahon.com/async.html
Hope that helps!
You should consider using Deferred pattern for asynchronous methods. You can get more information from the this StackOverflow question and answers:
What are the differences between Deferred, Promise and Future in JavaScript?
The marked answer from jnewman was good actually!
Hope this helps.
Promises can help manage this. There are two general scenarios - parallel and serial. Parallel can be accomplished using Promise.all(), serial is more complex - task B can only start when task A is done. Here's a bare-bones sample:
// returns a promise that resolves as the task is done
const wrap = (fn, delay) => new Promise(resolve => setTimeout(_ => resolve(fn()), delay));
const task = (fn, delay) => delay ? wrap(fn, delay) : Promise.resolve(fn());
// given a list of promises, execute them one by one.
const sequence = async l => l.reduce(async (a, b) => [].concat(await a, await b));
const tasks = [
task(_ => console.log("hello world")),
task(_ => console.log("hello future"), 1000)
];
sequence(tasks).then(_ => console.log("all done"));
You may need some ES6/7 translation to make this work in browsers or older node versions.
Related
I am new to node async/sync concept. these day i am confused by the sequence of async function executions.
I am kinda hard to get the benefit of the async calls of single-process node:
Here is an example:
1.Sequence of async/await
Assume an async function:
async readPathInFile(filePath) {
//open filePath, read the content and return
return file path in filePath
}
We have several calls like:
var a = 'a.txt';
var b = await readPathInFile(a); // call_a
var c = await readPathInFile(b); // call_b
var d = await readPathInFile(c); // call_c
As we know the readPathInFile is defined as asynchronous, but the calls
call_a, call_b, call_c relies on the previous ones sequentially.
For these calls, the calls have no differences with synchronous calls.
So, What is the benefit of the asynchronous definition ?
Callbacks concept has the same concerns:
For example:
readPathInFile(filePath, callback) {
//open filePath, read the content and return
callback(file path in filePath)
}
var a = 'a.txt';
readPathInFile(a, (b)=>{
//call_a
readPathInFile(b, (c)=>{
//call_b
readPathInFile(c, (d)=>{
//
});
});
}); // call_a
call_z();
//only if the call_z does not rely on a,b,c, z finishes without caring about a,b,c. This is a only benefit.
Please correct me if my assumption is wrong.
As the term implies, asynchronous means that functions run out of order. In this respect your understanding is correct. What this implies however is not at all obvious.
To answer your question is a few words, the async/await keywords are syntax sugar, meaning they're not so much intrinsic qualities of the language, rather are shortcuts for saying something else. Specifically, the await keyword translates the expression into the equivalent Promise syntax.
function someFunction(): Promise<string> {
return new Promise(function(resolve) {
resolve('Hello World');
});
}
console.log('Step 1');
console.log(await someFunction());
console.log('Step 3');
is the same as saying
...
console.log('Step 1');
someFunction().then(function (res) {
console.log(res);
console.log('Step 3');
});
Note the 3rd console.log is inside the then callback.
When you redefine someFunction using the async syntax, you'd get the following:
async function someFunction() {
return 'Hello World';
}
The two are equivalent.
To answer your question
I'd like to reiterate that async/await are syntactical sugar. their effect can be replicated by existing functionality
Their purpose therefore is to combine the worlds of asynchronous and synchronous code. If you've ever seen an older, larger JS file, you'll quickly notice how many callbacks are used throughout the script. await is the mechanism to use asynchronous functions while avoiding the callback hell, but fundamentally, a function marked async will always be async regardless of how many awaits you put in front of it.
If you're struggling to understand how an await call works, just remember that the callback you pass to .then(...) can be called at any time. This is the essence of asynchronicity. await just adds a clean way to use it.
A visual example
I like analogies. Here's my favourite;
Picture yourself at a cafe. There are 3 people waiting in line. The barista takes your order and informs you that your coffee will be ready in 5 minutes. So this means you can sit down at a table and read a magazine or what ever, instead of waiting in line for the barista to make your coffee and accept payment etc.
The advantage of that is that the people behind you in the queue can get their own things done, instead of waiting in line for the coffees to be ready.
This is fundamentally how asynchronous code works. The barista can only do one thing at a time, but their time can be optimised by allowing everyone in the queue to do other things while they wait
Hope that helps
This article (linked below in the comments) has an awesome illustration: Here it is
What is the benefit of the asynchronous definition ?
To enable standard logical-flow code to run asynchronously. Contrast your examples:
var a = 'a.txt';
var b = await readPathInFile(a); // call_a
var c = await readPathInFile(b); // call_b
var d = await readPathInFile(c); // call_c
vs.
var a = 'a.txt';
readPathInFile(a, (b)=>{
//call_a
readPathInFile(b, (c)=>{
//call_b
readPathInFile(c, (d)=>{
//
});
});
}); // call_a
The latter is often called "callback hell." It's much harder to reason about even in that case, but as soon as you add branching and such, it gets extremely hard to reason about. In contrast, branching with async functions and await is just like branching in synchronous code.
Now, you could write readPathInFileSync (a synchronous version of readPathInFile), which would not be an async function and would let you do this:
var a = 'a.txt';
var b = readPathInFileSync(a); // call_a
var c = readPathInFileSync(b); // call_b
var d = readPathInFileSync(c); // call_c
So what's the difference between that and the async version at the beginning of the answer? The async version lets other things happen on the one main JavaScript thread while waiting for those I/O calls to complete. The synchronous version doesn't, it blocks the main thread, waiting for I/O to complete, preventing anything else from being handled.
Here's an example, using setTimeout to stand in for I/O:
const logElement = document.getElementById("log");
function log(msg) {
const entry = document.createElement("pre");
entry.textContent = Date.now() + ": " + msg;
logElement.appendChild(entry);
entry.scrollIntoView();
}
// Set up "other things" to happen
const things = [
"this", "that", "the other", "something else", "yet another thing", "yada yada"
];
let starting = false; // We need this flag so we can show
// the "Doing X work" messsage before
// we actually start doing it, without
// giving the false impression in
// rare cases that other things happened
// during sync work
otherThing();
function otherThing() {
if (!starting) {
log(things[Math.floor(Math.random() * things.length)]);
}
setTimeout(otherThing, 250);
}
// Stand-in for a synchronous task that takes n milliseconds
function doSomethingSync(n) {
const done = Date.now() + n;
while (Date.now() < done) {
// Busy-wait. NEVER DO THIS IN REAL CODE,
// THIS IS SIMULATING BLOCKING I/O.
}
}
// Stand-in for an asynchronous task that takes n milliseconds
function doSomething(n) {
return new Promise(resolve => {
setTimeout(resolve, n);
});
}
function doWorkSync() {
doSomethingSync(200);
doSomethingSync(150);
doSomethingSync(50);
doSomethingSync(400);
}
async function doWorkAsync() {
await doSomething(200);
await doSomething(150);
await doSomething(50);
await doSomething(400);
}
// Do the work synchronously
document.getElementById("run-sync").addEventListener("click", (e) => {
const btn = e.currentTarget;
btn.disabled = true;
log(">>>> Doing sync work");
starting = true;
setTimeout(() => {
starting = false;
doWorkSync();
log("<<<< Done with sync work");
btn.disabled = false;
}, 50);
});
// Do the work asynchronously
document.getElementById("run-async").addEventListener("click", (e) => {
const btn = e.currentTarget;
btn.disabled = true;
log(">>>> Doing async work");
starting = true;
setTimeout(() => {
starting = false;
doWorkAsync().then(() => {
log("<<<< Done with async work");
btn.disabled = false;
});
}, 50);
});
html {
font-family: sans-serif;
}
#log {
max-height: 5em;
min-height: 5em;
overflow: auto;
border: 1px solid grey;
}
#log * {
margin: 0;
padding: 0;
}
<div>Things happening:</div>
<div id="log"></div>
<input type="button" id="run-sync" value="Run Sync">
<input type="button" id="run-async" value="Run Async">
There are times you don't care about that blocking, which is what the xyzSync functions in the Node.js API are for. For instance, if you're writing a shell script and you just need to do things in order, it's perfectly reasonable to write synchronous code to do that.
In constrast, if you're running a Node.js-based server of any kind, it's crucial not to allow the one main JavaScript thread to be blocked on I/O, unable to handle any other requests that are coming in.
So, in summary:
It's fine to write synchronous I/O code if you don't care about the JavaScript thread doing other things (for instance, a shell script).
If you do care about blocking the JavaScript thread, writing code with async/await makes it much easier (vs. callback functions) to avoid doing that with simple, straight-forward code following the logic of your operation rather than its temporality.
Javascript Promises lets asynchronous functions return values like synchronous methods instead of immediately returning the final value the asynchronous function returns a promise to provide data in the future.
Actually Promises solves the callback hell issue.
I'm kind of new to Node, but I understand that writing syncronous functions is a bad thing. I locks up the event loop or something... So it's good to write everything asyncronous.
But in some cases, writing everything async could be bad. For example I have a function that makes an API call (to a 3rd party API service) and then I have to write the result to a database. I need to do this a bunch over a short period of time, for example 500 times.
Calling this API 500 times async and then writing to the database 500 times async would be probably ban me from the API service (throttling) and overload my database server.
What is the best way to control or limit something like this? I want to keep things async so it's effecient but I simply cannot take the above approach.
I've researched some Promise throttling approaches. Is that the correct way to approach this type of problem? Is there a better more appropriate way to do it?
The async npm package is wonderful and has several solutions that can be used in this particular situation. One approach is using a queue with a set concurrency limit (example taken directly from the async README):
// create a queue object with concurrency 2
var q = async.queue(function (task, callback) {
console.log('hello ' + task.name);
callback();
}, 2);
// assign a callback
q.drain = function() {
console.log('all items have been processed');
}
// add some items to the queue
q.push({name: 'foo'}, function (err) {
console.log('finished processing foo');
});
github.com/caolan/async#queue
In your particular situation, just wait to call the callback() until whatever timing or transaction detail you are waiting for has completed.
I am not sure how Promise throttle works under the hood, I believe Promise a better approach compared to setTimeout, with promise it is more event based, my issue with that npm package is, it offers no callback option once your call is done, my implemenation would be something like:
class PromiseThrottler {
constructor(maxParallelCalls) {
this.maxParallelCalls = maxParallelCalls;
this.currentCalls = 0; // flag holding the no. of parallel calls at any point
this.queue = []; // queue maintaining the waiting calls
}
// pormiseFn - the fn that wraps some promise call the we need to make, thenChain - callback once your async call is done, args- arguments that needs to be passed to the function
add(promiseFn, thenChain, ...args) {
this.queue.push({
promiseFn, thenChain, args
});
this.call();
}
call() {
if (!this.queue.length || this.currentCalls >= this.maxParallelCalls) return;
this.currentCalls++;
let obj = this.queue.shift();
let chain = obj.args.length ? obj.promiseFn(...obj.args) : obj.promiseFn();
if (obj.thenChain) chain.then(obj.thenChain);
chain
.catch(() => {})
.then(() => {
this.currentCalls--;
this.call();
});
this.call();
}
}
//usage
let PT = new PromiseThrottler(50)
, rn = max => Math.floor(Math.random() * max) // generate Random number
, randomPromise = id => new Promise(resolve => setTimeout(() => resolve(id), rn(5000))) // random promise generating function
, i = 1
, thenCall = id => {
console.log('resolved for id:', id);
let div = document.createElement('div');
div.textContent = `resolved for id: ${id}`;
document.body.appendChild(div);
};
while (++i < 501) PT.add(randomPromise, thenCall, i);
One simple way of limiting this is to use setTimeout and do a "recursive" loop like this.
function asyncLoop() {
makeAPICall(function(result) {
writeToDataBase(result, function() {
setTimeout(asyncLoop, 1000);
});
});
}
And of course you can also use that same strategy with promises.
This question is somewhat academic in that I don't have a real need to do this.
I'm wondering if I can force the resolution of a promise into a returned value from a function such that the function callers are not aware that the functions contain promised async operations.
In .NET I can do things like this by using functions on Task[] or return Task.Result which causes the caller to await the completion of the task and callers won't know or care that the work has been done using tasks.
If you're using ES6 you can use a generator to make code like this. It essentially comes close to 'blocking' on the promise, so you have the appearance of a long-running method that just returns the value you want, but async/promises live under the covers.
let asyncTask = () =>
new Promise(resolve => {
let delay = Math.floor(Math.random() * 100);
setTimeout(function () {
resolve(delay);
}, delay);
});
let makeMeLookSync = fn => {
let iterator = fn();
let loop = result => {
!result.done && result.value.then(res =>
loop(iterator.next(res)));
};
loop(iterator.next());
};
makeMeLookSync(function* () {
let result = yield asyncTask();
console.log(result);
});
More explanation and the source available here: http://www.tivix.com/blog/making-promises-in-a-synchronous-manner/
Here is the code compiled on Babeljs.io
Suppose you maintain a library that exposes a function getData. Your users call it to get actual data:
var output = getData();
Under the hood data is saved in a file so you implemented getData using Node.js built-in fs.readFileSync. It's obvious both getData and fs.readFileSync are sync functions. One day you were told to switch the underlying data source to a repo such as MongoDB which can only be accessed asynchronously. You were also told to avoid pissing off your users, getData API cannot be changed to return merely a promise or demand a callback parameter. How do you meet both requirements?
Asynchronous function using callback/promise is the DNA of JavasSript and Node.js. Any non-trivial JS app is probably permeated with this coding style. But this practice can easily lead to so called callback pyramid of doom. Even worse, if any code in any caller in the call chain depends on the result of the async function, those code has to be wrapped in callback function as well, imposing a coding style constraint on caller. From time to time I find the need to encapsulate an async function (often provided in a 3rd party library) into a sync function in order to avoid massive global re-factoring. Searching for a solution on this subject usually ended up with Node Fibers or npm packages derived from it. But Fibers just cannot solve the problem I am facing. Even the example provided by Fibers' author illustrated the deficiency:
...
Fiber(function() {
console.log('wait... ' + new Date);
sleep(1000);
console.log('ok... ' + new Date);
}).run();
console.log('back in main');
Actual output:
wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
back in main
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)
If function Fiber really turns async function sleep into sync, the output should be:
wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)
back in main
I have created another simple example in JSFiddle and looking for code to yield expected output. I'll accept a solution that only works in Node.js so you are free to require any npm package despite not working in JSFiddle.
deasync turns async function into sync, implemented with a blocking mechanism by calling Node.js event loop at JavaScript layer. As a result, deasync only blocks subsequent code from running without blocking entire thread, nor incuring busy wait. With this module, here is the answer to the jsFiddle challenge:
function AnticipatedSyncFunction(){
var ret;
setTimeout(function(){
ret = "hello";
},3000);
while(ret === undefined) {
require('deasync').runLoopOnce();
}
return ret;
}
var output = AnticipatedSyncFunction();
//expected: output=hello (after waiting for 3 sec)
console.log("output="+output);
//actual: output=hello (after waiting for 3 sec)
(disclaimer: I am the co-author of deasync. The module was created after posting this question and found no workable proposal.)
You've got to use promises:
const asyncOperation = () => {
return new Promise((resolve, reject) => {
setTimeout(()=>{resolve("hi")}, 3000)
})
}
const asyncFunction = async () => {
return await asyncOperation();
}
const topDog = () => {
asyncFunction().then((res) => {
console.log(res);
});
}
I like arrow function definitions more. But any string of the form "() => {...}" could also be written as "function () {...}"
So topDog is not async despite calling an async function.
EDIT: I realize a lot of the times you need to wrap an async function inside a sync function is inside a controller. For those situations, here's a party trick:
const getDemSweetDataz = (req, res) => {
(async () => {
try{
res.status(200).json(
await asyncOperation()
);
}
catch(e){
res.status(500).json(serviceResponse); //or whatever
}
})() //So we defined and immediately called this async function.
}
Utilizing this with callbacks, you can do a wrap that doesn't use promises:
const asyncOperation = () => {
return new Promise((resolve, reject) => {
setTimeout(()=>{resolve("hi")}, 3000)
})
}
const asyncFunction = async (callback) => {
let res = await asyncOperation();
callback(res);
}
const topDog = () => {
let callback = (res) => {
console.log(res);
};
(async () => {
await asyncFunction(callback)
})()
}
By applying this trick to an EventEmitter, you can get the same results. Define the EventEmitter's listener where I've defined the callback, and emit the event where I called the callback.
If function Fiber really turns async function sleep into sync
Yes. Inside the fiber, the function waits before logging ok. Fibers do not make async functions synchronous, but allow to write synchronous-looking code that uses async functions and then will run asynchronously inside a Fiber.
From time to time I find the need to encapsulate an async function into a sync function in order to avoid massive global re-factoring.
You cannot. It is impossible to make asynchronous code synchronous. You will need to anticipate that in your global code, and write it in async style from the beginning. Whether you wrap the global code in a fiber, use promises, promise generators, or simple callbacks depends on your preferences.
My objective is to minimize impact on the caller when data acquisition method is changed from sync to async
Both promises and fibers can do that.
There is a npm sync module also. which is used for synchronize the process of executing the query.
When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.
Sample code
/*require sync module*/
var Sync = require('sync');
app.get('/',function(req,res,next){
story.find().exec(function(err,data){
var sync_function_data = find_user.sync(null, {name: "sanjeev"});
res.send({story:data,user:sync_function_data});
});
});
/*****sync function defined here *******/
function find_user(req_json, callback) {
process.nextTick(function () {
users.find(req_json,function (err,data)
{
if (!err) {
callback(null, data);
} else {
callback(null, err);
}
});
});
}
reference link: https://www.npmjs.com/package/sync
Nowadays this generator pattern can be a solution in many situations.
Here an example of sequential console prompts in nodejs using async readline.question function:
var main = (function* () {
// just import and initialize 'readline' in nodejs
var r = require('readline')
var rl = r.createInterface({input: process.stdin, output: process.stdout })
// magic here, the callback is the iterator.next
var answerA = yield rl.question('do you want this? ', r=>main.next(r))
// and again, in a sync fashion
var answerB = yield rl.question('are you sure? ', r=>main.next(r))
// readline boilerplate
rl.close()
console.log(answerA, answerB)
})() // <-- executed: iterator created from generator
main.next() // kick off the iterator,
// runs until the first 'yield', including rightmost code
// and waits until another main.next() happens
I can't find a scenario that cannot be solved using node-fibers. The example you provided using node-fibers behaves as expected. The key is to run all the relevant code inside a fiber, so you don't have to start a new fiber in random positions.
Lets see an example: Say you use some framework, which is the entry point of your application (you cannot modify this framework). This framework loads nodejs modules as plugins, and calls some methods on the plugins. Lets say this framework only accepts synchronous functions, and does not use fibers by itself.
There is a library that you want to use in one of your plugins, but this library is async, and you don't want to modify it either.
The main thread cannot be yielded when no fiber is running, but you still can create plugins using fibers! Just create a wrapper entry that starts the whole framework inside a fiber, so you can yield the execution from the plugins.
Downside: If the framework uses setTimeout or Promises internally, then it will escape the fiber context. This can be worked around by mocking setTimeout, Promise.then, and all event handlers.
So this is how you can yield a fiber until a Promise is resolved. This code takes an async (Promise returning) function and resumes the fiber when the promise is resolved:
framework-entry.js
console.log(require("./my-plugin").run());
async-lib.js
exports.getValueAsync = () => {
return new Promise(resolve => {
setTimeout(() => {
resolve("Async Value");
}, 100);
});
};
my-plugin.js
const Fiber = require("fibers");
function fiberWaitFor(promiseOrValue) {
var fiber = Fiber.current, error, value;
Promise.resolve(promiseOrValue).then(v => {
error = false;
value = v;
fiber.run();
}, e => {
error = true;
value = e;
fiber.run();
});
Fiber.yield();
if (error) {
throw value;
} else {
return value;
}
}
const asyncLib = require("./async-lib");
exports.run = () => {
return fiberWaitFor(asyncLib.getValueAsync());
};
my-entry.js
require("fibers")(() => {
require("./framework-entry");
}).run();
When you run node framework-entry.js it will throw an error: Error: yield() called with no fiber running. If you run node my-entry.js it works as expected.
You shouldn't be looking at what happens around the call that creates the fiber but rather at what happens inside the fiber. Once you are inside the fiber you can program in sync style. For example:
function f1() {
console.log('wait... ' + new Date);
sleep(1000);
console.log('ok... ' + new Date);
}
function f2() {
f1();
f1();
}
Fiber(function() {
f2();
}).run();
Inside the fiber you call f1, f2 and sleep as if they were sync.
In a typical web application, you will create the Fiber in your HTTP request dispatcher. Once you've done that you can write all your request handling logic in sync style, even if it calls async functions (fs, databases, etc.).
I struggled with this at first with node.js and async.js is the best library I have found to help you deal with this. If you want to write synchronous code with node, approach is this way.
var async = require('async');
console.log('in main');
doABunchOfThings(function() {
console.log('back in main');
});
function doABunchOfThings(fnCallback) {
async.series([
function(callback) {
console.log('step 1');
callback();
},
function(callback) {
setTimeout(callback, 1000);
},
function(callback) {
console.log('step 2');
callback();
},
function(callback) {
setTimeout(callback, 2000);
},
function(callback) {
console.log('step 3');
callback();
},
], function(err, results) {
console.log('done with things');
fnCallback();
});
}
this program will ALWAYS produce the following...
in main
step 1
step 2
step 3
done with things
back in main
Making Node.js code sync is essential in few aspects such as database. But actual advantage of Node.js lies in async code. As it is single thread non-blocking.
we can sync it using important functionality Fiber()
Use await() and defer ()
we call all methods using await(). then replace the callback functions with defer().
Normal Async code.This uses CallBack functions.
function add (var a, var b, function(err,res){
console.log(res);
});
function sub (var res2, var b, function(err,res1){
console.log(res);
});
function div (var res2, var b, function(err,res3){
console.log(res3);
});
Sync the above code using Fiber(), await() and defer()
fiber(function(){
var obj1 = await(function add(var a, var b,defer()));
var obj2 = await(function sub(var obj1, var b, defer()));
var obj3 = await(function sub(var obj2, var b, defer()));
});
I hope this will help. Thank You
I have a question regarding the native Array.forEach implementation of JavaScript: Does it behave asynchronously?
For example, if I call:
[many many elements].forEach(function () {lots of work to do})
Will this be non-blocking?
No, it is blocking. Have a look at the specification of the algorithm.
However a maybe easier to understand implementation is given on MDN:
if (!Array.prototype.forEach)
{
Array.prototype.forEach = function(fun /*, thisp */)
{
"use strict";
if (this === void 0 || this === null)
throw new TypeError();
var t = Object(this);
var len = t.length >>> 0;
if (typeof fun !== "function")
throw new TypeError();
var thisp = arguments[1];
for (var i = 0; i < len; i++)
{
if (i in t)
fun.call(thisp, t[i], i, t);
}
};
}
If you have to execute a lot of code for each element, you should consider to use a different approach:
function processArray(items, process) {
var todo = items.concat();
setTimeout(function() {
process(todo.shift());
if(todo.length > 0) {
setTimeout(arguments.callee, 25);
}
}, 25);
}
and then call it with:
processArray([many many elements], function () {lots of work to do});
This would be non-blocking then. The example is taken from High Performance JavaScript.
Another option might be web workers.
If you need an asynchronous-friendly version of Array.forEach and similar, they're available in the Node.js 'async' module: http://github.com/caolan/async ...as a bonus this module also works in the browser.
async.each(openFiles, saveFile, function(err){
// if any of the saves produced an error, err would equal that error
});
There is a common pattern for doing a really heavy computation in Node that may be applicable to you...
Node is single-threaded (as a deliberate design choice, see What is Node.js?); this means that it can only utilize a single core. Modern boxes have 8, 16, or even more cores, so this could leave 90+% of the machine idle. The common pattern for a REST service is to fire up one node process per core, and put these behind a local load balancer like http://nginx.org/.
Forking a child -
For what you are trying to do, there is another common pattern, forking off a child process to do the heavy lifting. The upside is that the child process can do heavy computation in the background while your parent process is responsive to other events. The catch is that you can't / shouldn't share memory with this child process (not without a LOT of contortions and some native code); you have to pass messages. This will work beautifully if the size of your input and output data is small compared to the computation that must be performed. You can even fire up a child node.js process and use the same code you were using previously.
For example:
var child_process = require('child_process');
function run_in_child(array, cb) {
var process = child_process.exec('node libfn.js', function(err, stdout, stderr) {
var output = JSON.parse(stdout);
cb(err, output);
});
process.stdin.write(JSON.stringify(array), 'utf8');
process.stdin.end();
}
Array.forEach is meant for computing stuff not waiting, and there is nothing to be gained making computations asynchronous in an event loop (webworkers add multiprocessing, if you need multi-core computation). If you want to wait for multiple tasks to end, use a counter, which you can wrap in a semaphore class.
Edit 2018-10-11:
It looks like there is a good chance the standard described below may not go through, consider pipelineing as an alternative (does not behave exactly the same but methods could be implemented in a similar manor).
This is exactly why I am excited about es7, in future you will be able to do something like the code below (some of the specs are not complete so use with caution, I will try to keep this up to date). But basically using the new :: bind operator, you will be able to run a method on an object as if the object's prototype contains the method. eg [Object]::[Method] where normally you would call [Object].[ObjectsMethod]
Note to do this today (24-July-16) and have it work in all browsers you will need to transpile your code for the following functionality:Import / Export, Arrow functions, Promises, Async / Await and most importantly function bind. The code below could be modfied to use only function bind if nessesary, all this functionality is neatly available today by using babel.
YourCode.js (where 'lots of work to do' must simply return a promise, resolving it when the asynchronous work is done.)
import { asyncForEach } from './ArrayExtensions.js';
await [many many elements]::asyncForEach(() => lots of work to do);
ArrayExtensions.js
export function asyncForEach(callback)
{
return Promise.resolve(this).then(async (ar) =>
{
for(let i=0;i<ar.length;i++)
{
await callback.call(ar, ar[i], i, ar);
}
});
};
export function asyncMap(callback)
{
return Promise.resolve(this).then(async (ar) =>
{
const out = [];
for(let i=0;i<ar.length;i++)
{
out[i] = await callback.call(ar, ar[i], i, ar);
}
return out;
});
};
This is a short asynchronous function to use without requiring third party libs
Array.prototype.each = function (iterator, callback) {
var iterate = function () {
pointer++;
if (pointer >= this.length) {
callback();
return;
}
iterator.call(iterator, this[pointer], iterate, pointer);
}.bind(this),
pointer = -1;
iterate(this);
};
There is a package on npm for easy asynchronous for each loops.
var forEachAsync = require('futures').forEachAsync;
// waits for one request to finish before beginning the next
forEachAsync(['dogs', 'cats', 'octocats'], function (next, element, index, array) {
getPics(element, next);
// then after all of the elements have been handled
// the final callback fires to let you know it's all done
}).then(function () {
console.log('All requests have finished');
});
Also another variation forAllAsync
It is possible to code even the solution like this for example :
var loop = function(i, data, callback) {
if (i < data.length) {
//TODO("SELECT * FROM stackoverflowUsers;", function(res) {
//data[i].meta = res;
console.log(i, data[i].title);
return loop(i+1, data, errors, callback);
//});
} else {
return callback(data);
}
};
loop(0, [{"title": "hello"}, {"title": "world"}], function(data) {
console.log("DONE\n"+data);
});
On the other hand, it is much slower than a "for".
Otherwise, the excellent Async library can do this: https://caolan.github.io/async/docs.html#each
These code snippet will give you better understanding of forEach and forOf comparison.
/* eslint-disable no-console */
async function forEachTest() {
console.log('########### Testing forEach ################ ')
console.log('start of forEachTest func')
let a = [1, 2, 3]
await a.forEach(async (v) => {
console.log('start of forEach: ', v)
await new Promise(resolve => setTimeout(resolve, v * 1000))
console.log('end of forEach: ', v)
})
console.log('end of forEachTest func')
}
forEachTest()
async function forOfTest() {
await new Promise(resolve => setTimeout(resolve, 10000)) //just see console in proper way
console.log('\n\n########### Testing forOf ################ ')
console.log('start of forOfTest func')
let a = [1, 2, 3]
for (const v of a) {
console.log('start of forOf: ', v)
await new Promise(resolve => setTimeout(resolve, v * 1000))
console.log('end of forOf: ', v)
}
console.log('end of forOfTest func')
}
forOfTest()
Here is a small example you can run to test it:
[1,2,3,4,5,6,7,8,9].forEach(function(n){
var sum = 0;
console.log('Start for:' + n);
for (var i = 0; i < ( 10 - n) * 100000000; i++)
sum++;
console.log('Ended for:' + n, sum);
});
It will produce something like this(if it takes too less/much time, increase/decrease the number of iterations):
(index):48 Start for:1
(index):52 Ended for:1 900000000
(index):48 Start for:2
(index):52 Ended for:2 800000000
(index):48 Start for:3
(index):52 Ended for:3 700000000
(index):48 Start for:4
(index):52 Ended for:4 600000000
(index):48 Start for:5
(index):52 Ended for:5 500000000
(index):48 Start for:6
(index):52 Ended for:6 400000000
(index):48 Start for:7
(index):52 Ended for:7 300000000
(index):48 Start for:8
(index):52 Ended for:8 200000000
(index):48 Start for:9
(index):52 Ended for:9 100000000
(index):45 [Violation] 'load' handler took 7285ms
Although Array.forEach is not asynchronous, you can get asynchronous "end result". Example below:
function delayFunction(x) {
return new Promise(
(resolve) => setTimeout(() => resolve(x), 1000)
);
}
[1, 2, 3].forEach(async(x) => {
console.log(x);
console.log(await delayFunction(x));
});
Use Promise.each of bluebird library.
Promise.each(
Iterable<any>|Promise<Iterable<any>> input,
function(any item, int index, int length) iterator
) -> Promise
This method iterates over an array, or a promise of an array, which contains promises (or a mix of promises and values) with the given iterator function with the signature (value, index, length) where the value is the resolved value of a respective promise in the input array. Iteration happens serially. If the iterator function returns a promise or a thenable, then the result of the promise is awaited before continuing with next iteration. If any promise in the input array is rejected, then the returned promise is rejected as well.
If all of the iterations resolve successfully, Promise.each resolves to the original array unmodified. However, if one iteration rejects or errors, Promise.each ceases execution immediately and does not process any further iterations. The error or rejected value is returned in this case instead of the original array.
This method is meant to be used for side effects.
var fileNames = ["1.txt", "2.txt", "3.txt"];
Promise.each(fileNames, function(fileName) {
return fs.readFileAsync(fileName).then(function(val){
// do stuff with 'val' here.
});
}).then(function() {
console.log("done");
});