I want to understand one thing about async module in node.js.
I have created a function that map an object from a form to a model object and return this object.
This object is a video with an array of tags.
My question is where can I return the video ? I know normally it is inside the async callback function but if I do that, the object returned is undefined.
Whereas If i return the video object at the end of the whole function, it works but it's not safe as I'm not sure, my async is finished...
By the way, I don't understand the callback function passed in argument to async.each and
called after video.products.push(tag); . What does this function do?
Regards
in my mapping.js :
exports.video = function(object) {
var video = new Video();
video.name = object.name;
video.products = [];
async.each(object.tags, function(tago, callback) {
tag = {
"name" : tago.name
}
video.products.push(tag);
callback();
} ,
function(err) {
if( err ) {
console.log('Error' + error);
throw err;
}
logger.debug("into async" + video);
}
);
logger.debug("end function " );
**//return video;**
}
in my video.js :
var video = mapping.video(object);
logger.debug(video); // return undefined
The simple answer is that you can't - at least not via easy or obvious approach. As its name suggests, async is a library for queuing up asynchronous function calls into the event loop. So your exports.video function simply kicks off a bunch of asynchronous functions, which execute one after the other on an unpredictable time-frame, and then returns immediately. No matter where you try to return your video object within the scope of your function calls which are instantiated by async, the exports.video function will already have returned.
In this case it doesn't really seem like you need asynchronous function calls for what you're doing. I'd suggest that you replace your use of async with something like Underscore's each method, which executes synchronously, instead.
http://documentcloud.github.io/underscore/#each
You'd need to define a callback for your exports.video function e.g..
exports.video = function(object, callback) {
// video code (snip)...
async.each(object.tags,
function eachTag(tag, done) {
// code run for each tag object (snip)...
done();
},
function finished(err) {
// code run at the end (snip)...
callback(thingThatsReturned);
});
};
...and call it like this:
var videoUtils = require('videoUtils');
var tags = getTags();
videoUtils.video({ tags: tags }, function(thingThatsReturned) {
// do something with 'thingThatsReturned'
});
By the way, I don't understand the callback function passed in
argument to async.each and called after video.products.push(tag); .
What does this function do?
The async.each function will call the 'eachTag' function above (2nd argument) for each item in your array. But because it's done asynchronously, and you might do something else async in the function (hit a database/api etc.), it needs to know when that function for that particular array item has finished. Calling done() tells async.each that the function has finished processing. Once all the functions are finished processing (they've all called done()), async.each will run the 'finished' function above (3rd argument).
This is pretty standard async stuff for Node.js, but it can be tricky to get ones head around it at first. Hang in there :-)
Edit: It looks like your code isn't doing anything asynchronous. If it was, then the above code would be the way to do it, otherwise the following code would work better:
exports.video = function(object) {
// video code (snip)...
if (Array.isArray(object.tags)) {
object.tags.forEach(function eachTag(tag) {
// code run for each tag object (snip)...
});
}
return thingThatsReturned;
};
...and call it...
var videoUtils = require('videoUtils');
var tags = getTags();
var thingThatsReturned = videoUtils.video({ tags: tags });
I rewrote this question, because the old version was obviously misleading.
Please read the text and make shure you understood what I'm asking for. If
there is still anything left in the dark I'll modify this question for clarity.
Just inform me.
One of my projects is to port a library from Python to JavaScript.
The Python library is entirely blocking/synchronous when it comes to I/O
and such. This is of course perfectly normal for Python code.
I plan to port the synchronous/blocking methods as they are to JavaScript.
This has several reasons and whether or not it's worth the effort is a
good but different question.
Additionally I wan't to add an asynchronous/non-blocking api.
Think of it like the fs module in node where there are i.e. fs.open and
fs.openSync coexisting.
The library is pure JavaScript and will run in Node and in the Browser.
The question is what a good/the best approach for the development of these two coexisting APIs would be.
I believe its good to have the same thing happening in one place only.
Hence an approach where some parts of the implementation could be shared would be preferable.
Not at any price of course, that's why I'm asking.
I had a proposal for an approach in here, but I'm going to post it as a
possible answer. However, I'm waiting for some serious discussion to happen
before I decide what I accept as an answer.
So far approaches are:
implement both apis separately and definetly use promises for the asynchronous functions.
use something like the obtain api proposal - beeing a more integrated approach
If you're talking I/O in node.js then most I/O methods have a synchronous version.
There is no direct conversion from Asynchronicity To Synchronicity. I can think of two approaches:
Have each asynchronous method run a polling loop waiting for the async task to complete before returning.
Drop the idea of mimicking synchronous code and instead invest in better coding patterns (such as promises)
To illustrate I will assume option 2 is a better choice. The following example uses Q promises (easily installed with npm install q.
The idea behind promises is that although they are asynchronous the return object is a promise for a value as if it was a normal function.
// Normal function
function foo(input) {
return "output";
}
// With promises
function promisedFoo(input) {
// Does stuff asynchronously
return promise;
}
The first function takes an input and returns a result. The second example takes an input and immediately returns a promise which will eventually resolve to a value when the async task finishes. You then manage this promise as follows:
var promised_value = promisedFoo(input);
promised_value.then(function(value) {
// Yeah, we now have a value!
})
.fail(function(reason) {
// Oh nos.. something went wrong. It passed in a reason
});
Using promises you no longer have to worry when something will happen. You can easily chain promises so things happen synchronously without insane nested callbacks or 100 named functions.
It well worth learning about. Remember promises are meant to make async code behave like sync code even though it isn't blocking.
Write lower level API using promises that takes async/sync flag.
Higher level async API returns these promises directly (while also working with async callbacks like it's 1970).
Higher level sync API unwraps the value synchronously from the promise and returns the value or throws the error.
(Examples use bluebird which is orders of magnitude faster and has more features at the cost of file size compared to Q, although that might not be ideal for browsers.)
Low level api that is not exposed:
//lowLevelOp calculates 1+1 and returns the result
//There is a 20% chance of throwing an error
LowLevelClass.prototype.lowLevelOp = function(async, arg1, arg2) {
return new Promise(function(resolve, reject) {
if (Math.random() < 0.2) {
throw new Error("random error");
}
if (!async) resolve(1+1);
else {
//Async
setTimeout(function(){
resolve(1+1);
}, 50);
}
});
};
High level exposed API that works synchronously, using promises or callbacks:
HighLevelClass.prototype.opSync = function(arg1, arg2) {
var inspection =
this.lowLevel.lowLevelOp(false, arg1, arg2).inspect();
if (inspection.isFulfilled()) {
return inspection.value();
}
else {
throw inspection.error();
}
};
HighLevelClass.prototype.opAsync = function(arg1, arg2, callback) {
//returns a promise as well as accepts callback.
return this.lowLevel.lowLevelOp(true, arg1, arg2).nodeify(callback);
};
You can automatically generate the high level api for synchronous methods:
var LowLevelProto = LowLevelClass.prototype;
Object.keys(LowLevelProto).filter(function(v) {
return typeof LowLevelProto[v] === "function";
}).forEach(function(methodName) {
//If perf is at all a concern you really must do this with a
//new Function instead of closure and reflection
var method = function() {
var inspection = this.lowLevel[methodName].apply(this.lowLevel, arguments);
if (inspection.isFulfilled()) {
return inspection.value();
}
else {
throw inspection.error();
}
};
HighLevelClass.prototype[methodName + "Sync" ] = method;
});
I implemented a library that does what I'm asking for ObtainJS.
(Yes, the Library uses Promises BUT not as others proposed in their ansewers here)
Reposting the Readme.md:
ObtainJS
ObtainJS is a micro framework to bring together asynchronous and
synchronous JavaScript code. It helps you to Don't Repeat Yourself
(DRY) if you are developing a library with interfaces for both
blocking/synchronous and non-blocking/asynchronous execution models.
As a USER
of a library that was implemented with ObtainJS you won't have to learn
a lot. Typically a function defined using ObtainJS has as first argument
the switch, that lets you choose the execution path, followed by its normal
arguments:
// readFile has an obtainJS API:
function readFile(obtainAsyncExecutionSwitch, path) { /* ... */ }
execute synchronously
If the obtainSwitch is a falsy value readFile will execute synchronously
and return the result directly.
var asyncExecution = false, result;
try {
result = readFile(asyncExecution, './file-to-read.js');
} catch(error) {
// handle the error
}
// do something with result
execute asynchronously
If the obtainSwitch is a truthy value readFile will execute asynchronously
and always return a Promise.
See Promises at MDN
var asyncExecution = true, promise;
promise = readFile(asyncExecution, './file-to-read.js');
promise.then(
function(result) {
// do something with result
},
function(error){
// handle the error
}
)
// Alternatively, use the returned promise directly:
readFile(asyncExecution, './file-to-read.js')
.then(
function(result) {
// do something with result
},
function(error){
// handle the error
}
)
You can use a callback based api, too. Note that the Promise is returned anyways.
var asyncExecution;
function unifiedCallback(error, result){
if(error)
// handle the error
else
// do something with result
}
asyncExecution = {unified: unifiedCallback}
readfile(asyncExecution, './file-to-read.js');
or with a separate callback and errback
var asyncExecution;
function callback(result) {
// do something with result
}
function errback(error) {
// handle the error
}
var asyncExecution = {callback: callback, errback: errback}
readfile(asyncExecution, './file-to-read.js');
```
As a smart ;-) LIBRARY AUTHOR
who's going to implement a API using with ObtainJS the work is a bit more.
Stay with me.
The behavior above is achieved by defining a twofold dependency tree: one
for the actions of the synchronous execution path and one for the actions
of the asynchronous execution path.
Actions are small functions with dependencies on the results of other
actions. The asynchronous execution path will fallback to synchronous
actions if there is no asynchronous action defined for a dependency.
You wouldn't define an asynchronous action if its synchronous
equivalent is non-blocking. This is where you DRY!
So, what you do, for example, is splitting your synchronous and blocking
method in small function-junks. These junks depend on the results of each
other. Then you define a non-blocking AND asynchronous junk for each
synchronous AND blocking junk. The rest does obtainJS for you. Namely:
creating a switch for synchronous or asynchronous execution
resolving the dependency tree
executing the junks in the right order
providing you with the results via:
return value when using the synchronous path
promises OR callbacks (your choice!) when using the asynchronous path
Here is the readFile function
from above, taken directly from working code at
ufoJS
define(['ufojs/obtainJS/lib/obtain'], function(obtain) {
// obtain.factory creates our final function
var readFile = obtain.factory(
// this is the synchronous dependency definition
{
// this action is NOT in the async tree, the async execution
// path will fall back to this method
uri: ['path', function _path2uri(path) {
return path.split('/').map(encodeURIComponent).join('/')
}]
// synchronous AJAX request
, readFile:['uri', function(path) {
var request = new XMLHttpRequest();
request.open('GET', path, false);
request.send(null);
if(request.status !== 200)
throw _errorFromRequest(request);
return request.responseText;
}]
}
,
// this is the asynchronous dependency definition
{
// aynchronous AJAX request
readFile:['uri', '_callback', function(path, callback) {
var request = new XMLHttpRequest()
, result
, error
;
request.open('GET', path, true);
request.onreadystatechange = function (aEvt) {
if (request.readyState != 4 /*DONE*/)
return;
if (request.status !== 200)
error = _errorFromRequest(request);
else
result = request.responseText
callback(error, result)
}
request.send(null);
}]
}
// this are the "regular" function arguments
, ['path']
// this is the "job", a driver function that receives as first
// argument the obtain api. A method that the name of an action or
// of an argument as input and returns its result
// Note that job is potentially called multiple times during
// asynchronoys execution
, function(obtain, path){ return obtain('readFile'); }
);
})
a skeleton
var myFunction = obtain.factory(
// sync actions
{},
// async actions
{},
// arguments
[],
//job
function(obtain){}
);
action/getter definition
// To define a getter we give it a name provide a definition array.
{
// sync
sum: ['arg1', 'arg2',
// the last item in the definition array is always the action/getter itself.
// it is called when all dependencies are resolved
function(arg1, arg2) {
// function body.
var value = arg1 + arg2
return value
}]
}
// For asynchronous getters you have different options:
{
// async
// the special name "_callback" will inject a callback function
sample1: ['arg1', '_callback', function(arg1, callback) {
// callback(error, result)
}],
// you can order separate callback and errback when using both special
// names "_callback" and "_errback"
sample2: ['arg1', '_callback', '_errback', function(arg1, callback, errback) {
// errback(error)
// callback(result)
}],
// return a promise
sample3: ['arg1', function(arg1) {
var promise = new Promise(/* do what you have to*/);
return promise
}]
}
The items in the definition array before the action are the dependencies
their values are going to be injected into the call to action, when
available.
If the type of an dependency is not a string: It's injected as a value
directly. This way you can effectively do currying.
If the type of the value is a string: It's looked up in the dependency
tree for the current execution path(sync or async).
If its name is defined as an caller-argument (in the third argument of obtain.factory) the value
is taken from the invoking call.
If its name is defined as the name of another action, that action is
executed and its return value is used as a parameter. An action will
executed only once per run, later invocations will return a cached value.
If the execution path is asynchronous obtain will first look for a
asynchronous action definition. If that is not found it falls back
to a synchronous definition.
If you wish to pass a String as value to your getter you must define it as
an instance of obtain.Argument: new obtain.Argument('mystring argument is not a getter')
A more complete example
from ufoLib/glifLib/GlyphSet.js
Note that: obtainJS is aware of the host object and propagates this
correctly to all actions.
/**
* Read the glif from I/O and cache it. Return a reference to the
* cache object: [text, mtime, glifDocument(if alredy build by this.getGLIFDocument)]
*
* Has the obtainJS sync/async api.
*/
GlypSet.prototype._getGLIFcache = obtain.factory(
{ //sync
fileName: ['glyphName', function fileName(glyphName) {
var name = this.contents[glyphName];
if(!(glyphName in this.contents) || this.contents[glyphName] === undefined)
throw new KeyError(glyphName);
return this.contents[glyphName]
}]
, glyphNameInCache: ['glyphName', function(glyphName) {
return glyphName in this._glifCache;
}]
, path: ['fileName', function(fileName) {
return [this.dirName, fileName].join('/');
}]
, mtime: ['path', 'glyphName', function(path, glyphName) {
try {
return this._io.getMtime(false, path);
}
catch(error) {
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
throw error;
}
}]
, text: ['path', 'glyphName', function(path, glyphName) {
try {
return this._io.readFile(false, path);
}
catch(error) {
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
throw error;
}
}]
, refreshedCache: ['glyphName', 'text', 'mtime',
function(glyphName, text, mtime) {
return (this._glifCache[glyphName] = [text, mtime]);
}]
}
//async getters
, {
mtime: ['path', 'glyphName', '_callback',
function(path, glyphName, callback) {
var _callback = function(error, result){
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
callback(error, result)
}
this._io.getMtime({unified: _callback}, path);
}]
, text: ['path', 'glyphName', '_callback',
function(path, glyphName, callback){
var _callback = function(error, result) {
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
callback(error, result)
}
this._io.readFile({unified: _callback}, path);
}
]
}
, ['glyphName']
, function job(obtain, glyphName) {
if(obtain('glyphNameInCache')) {
if(obtain('mtime').getTime() === this._glifCache[glyphName][1].getTime()) {
// cache is fresh
return this._glifCache[glyphName];
}
}
// still here? need read!
// refreshing the cache:
obtain('refreshedCache')
return this._glifCache[glyphName];
}
)
Is there a callback for when underscore is finished it's _.each loop because if I console log immediately afterwards obviously the array I am populating with the each loop is not available. This is from a nested _.each loop.
_.each(data.recipe, function(recipeItem) {
var recipeMap = that.get('recipeMap');
recipeMap[recipeItem.id] = { id: recipeItem.id, quantity: recipeItem.quantity };
});
console.log(that.get('recipeMap')); //not ready yet.
The each function in UnderscoreJS is synchronous which wouldn't require a callback when it is finished. One it's done executing the commands immediately following the loop will execute.
If you are performing async operations in your loop, I would recommend using a library that supports async operations within the each function. One possibility is by using AsyncJS.
Here is your loop translated to AsyncJS:
async.each(data.recipe, function(recipeItem, callback) {
var recipeMap = that.get('recipeMap');
recipeMap[recipeItem.id] = { id: recipeItem.id, quantity: recipeItem.quantity };
callback(); // show that no errors happened
}, function(err) {
if(err) {
console.log("There was an error" + err);
} else {
console.log("Loop is done");
}
});
Another option is to build your callback function into the each loop on the last execution:
_.each(collection, function(model) {
if(model.collection.indexOf(model) + 1 == collection.length) {
// Callback goes here
}
});
Edit to add:
I don't know what your input/output data looks like but you might consider using _.map instead, if you're just transforming / rearranging the contents
Hi i am using async module of node.js for implementing a for loop asynchronously.
My question is: how to break the loop execution and get out of the loop? I tried giving return , return false but no luck.
Here is the sample code:
async.until(
function() { return goal; },
function(callback) {
async.each(_rules,
function(rule,callback) {
var outcome = true;
.... some code ....
if(changes){
console.log("hi");
return false;// HERE I NEED TO BREAK
}
else
callback();
},
function(err){ }
);
if (!changes || session.process)
goal = true;
callback();
},
function(err){ callback(session); }
);
async.until repeatedly calls function until the test returns true. So test must return true so that you exit the loop. This is opposite of async.whilst which runs repeatedly while test evaluates to be true.
async.each calls the functions in parallel so what it returns does not matter. It is not a loop which you can break, but an iterator looping over the array. Your condition to stop using async.each should be in test for async.until and you should iterate over the rules yourself.
There isn't really a "loop" as such to break out of. All your items in your collection are used in parallel
The only way to "break" the "loop" is to call the callback with an error argument. As there is nothing to stop you from putting other things in there you could hack it a little bit.
From the docs:
Note, that since this function applies the iterator to each item in
parallel there is no guarantee that the iterator functions will
complete in order.
Even if you return an error, you will still have several outstanding requests potentially so you really want to limit the amount of items you use in one go. To limit the amount of outstanding requests, you could use eachSeries or eachLimit.
For example:
async.each(_rules,
function(rule,callback) {
if(changes){
return callback({ data: 'hi'}); // stop
}
...
if(realerror){
return callback({ error: realerror}); // stop with an error
}
callback(); // continue
},
function(result){
if(!result){
return othercallback('no results');
}
// check if we have a real error:
if(result.error){
return othercallback(result.error);
}
return othercallback(null, result.data);
}
);
PS: if you're not doing async, use underscore
You have also async.detect
Returns the first value in coll that passes an async truth test. The iteratee is applied in parallel, meaning the first iteratee to return true will fire the detect callback with that result.
// asynchronous function that checks if a file exists
function fileExists(file, callback) {
fs.access(file, fs.constants.F_OK, (err) => {
callback(null, !err);
});
}
async.detect(['file3.txt','file2.txt','dir1/file1.txt'], fileExists,
function(err, result) {
console.log(result);
// dir1/file1.txt
// result now equals the first file in the list that exists
}
);
I'm using UnderscoreJs with nodejs and have a need for the _.times() method. times() will invoke a function X number of times
This works as expected, however I need to iterate in a series, instead of in parallel which this appears to be doing.
Any idea if there's a way to use this in series w/ callback methods?
Given something like this:
function f() {
some_async_call({ callback: function(err, results) {...})
}
_(3).times(f);
Then the three f calls will happen in series but the some_async_call calls won't necessarily happen in series because they're asynchronous.
If you want to force your calls to run in series then you need to use the callback on the async call to launch the next one in the series:
function f(times, step) {
step = step || 0;
some_async_call({
callback: function(err, results) {
// Do something with `err` and `results`...
if(step < times)
f(times, step + 1);
}
});
}
f(3);
That approach will execute the three some_async_calls in series but, alas, the initial f(3) will return immediately. One solution to that problem is, of course, another callback:
function f(from_n, upto, and_finally) {
some_async_call({
callback: function(err, results) {
// Do something with `err` and `results`...
if(from_n < upto)
f(from_n + 1, upto, and_finally);
else
and_finally();
}
});
}
f(0, 3, function() { console.log('all done') });
Where does _.times in with all this? No where really. _.times is just a for loop:
_.times = function(n, iterator, context) {
for (var i = 0; i < n; i++) iterator.call(context, i);
};
_.times exists for completeness and to allow you to add for loop when using _.chain. You could probably shoe-horn it in if you really wanted to but you would be making a big ugly mess instead of simplifying your code.
You could use 250R's async idea but you'd have to build an array of three functions but _.range and _.map would be more appropriate for that than _.times:
// Untested off the top of my head code...
function f(callback) {
some_async_call({
callback: function(err, results) {
// Deal with `err` and `results`...
callback();
}
});
}
var three_fs = _(3).range().map(function() { return f });
async.series(three_fs);
But you still have to modify f to have a callback function and if you're always calling f three times then:
async.series([f, f, f]);
might be better than dynamically building the array with _.range and _.map.
The real lesson here is that once you get into asynchronous function calls, you end up implementing all your logic as callbacks calling callbacks calling callbacks, callbacks all the way down.
This async library might get you started
https://github.com/caolan/async#series
Or if you want to do it yourself, the idea is to do recursive calls after each function callback is called, here's the source code https://github.com/caolan/async/blob/master/lib/async.js#L101