EDIT
thx to all the answers,
and finally I decide to use some tools like Step,
all I need is "flow control" and don't want any thing else which may slow down the performance (I don't know how much exactly it would effect or the effect just can be ignored).
So I just create a little tool for flow control:
line.js
/**
* Create the "next" function
*
* #param {Array} tasks
* #param {Number} index
* #param {Number} last
*/
var next = function(tasks, index, last) {
if (index == last) {
return tasks[index + 1];
}
else {
return function(data) {
var nextIndex = index + 1;
tasks[nextIndex](next(tasks, nextIndex, last), data);
};
}
};
/**
* Invoke functions in line.
*/
module.exports = function() {
var tasks = arguments,
last = tasks.length - 2;
tasks[0](next(tasks, 0, last));
};
usage:
var line = require("line.js");
line(function(next) {
someObj.find(function(err, docs) {
// codes
next(docs);
});
}, function(next, docs) {
// codes
});
Hope this helps.
EDIT END
As all know,
Node's built-in or third-part modules often provides async API,
and using "callback" function for dealing the results.
It's cool but sometimes would code like this:
//some codes
}
}
}
}
codes like this are hard to read.
I know "deferred" library can solve such problem,
Is there any good "deferred" module for Node?
And How is the performance if I code Node with "deferred"?
It is a large problem with Node-based code; you frequently grow "callback pyramids". There are several approaches to dealing with the problem:
Code style:
Use this annoyance as an opportunity to break your code into bite sized chunks. It means you're likely going to have a proliferation of tiny named funcs - that's probably just fine, though! You might also find more opportunities for reuse.
Flow-control Libraries
There are exactly 593.72 billion flow control libraries out there. Here's some of the more popular ones:
Step super basic serial & parallel flow management.
seq is a heavier but more feature-full flow control library.
There's plenty more. Search the npm registry for "flow" and "flow control" (sorry, doesn't appear to be linkable)
Language Extensions
There are several attempts to provide a more synchronous-feeling syntax on top of JavaScript (or CoffeeScript), often based on the concepts behind the tame paper.
TameJS is the OkCupid team's answer to this.
IcedCoffeeScript they've also ported TameJS over CoffeeScript as a fork.
streamline.js is very similar to TameJS.
StratifiedJS is a heavier approach to the problem.
This route is a deal-breaker for some:
It's not standard JavaScript; if you are building libraries/frameworks/etc, finding help will be more difficult.
Variable scope can behave in unexpected ways, depending on the library.
The generated code can be difficult to debug & match to the original source.
The Future:
The node core team is very aware of the problem, and are also working on lower level components to help ease the pain. It looks like they'll be introducing a basic version of domains in v0.8, which provide a way of rolling up error handling (avoiding the common return err if err pattern, primarily).
This should start to lay a great foundation for cleaner flow control libraries, and start to pave the way for a more consistent way of dealing with callback pyramids. There's too much choice out there right now, and the community isn't close to agreeing on even a handful of standards yet.
References:
Mixu's Node book has an awesome chapter on this subject.
There are tons of "deferred libraries". Have a look there http://eirikb.github.com/nipster/#promise and there http://eirikb.github.com/nipster/#deferred. To pick one, it's only a matter of style & simplicity :)
If you really don't like that, there's always the alternative of using named functions, which will reduce the indentation.
Instead of
setTimeout(function() {
fs.readFile('file', function (err, data) {
if (err) throw err;
console.log(data);
})
}, 200);
You can do this:
function dataHandler(err, data)
{
if (err) throw err;
console.log(data);
}
function getFile()
{
fs.readFile('file', dataHandler);
}
setTimeout(getFile, 200);
The same thing, no nesting.
There are some libraries that may be useful in some scenarios, but as a whole you won't be excited after using them for everything.
According to the slowness issues. Since node.js is async, the wrapped functions are not such a big performance consumer.
You could look here for deferred-like library
https://github.com/kriszyp/node-promise
Also this question is very similar
What nodejs library is most like jQuery's deferreds?
And as a final bonus I suggest you take a look at CoffeeScript. It is a language, which compiles to javascript and has more beautiful syntax, since the function braces are removed
I usually like to use the async.js library as it offers a few different options on how to execute the code
Related
I am new to databases and these MongoDB/Mongoose.js async functions have annoyed the hell out of me over the last few hours. I have written and rewritten this bit so many times:
router.get('/districts', function(req, res) {
districtNames = [];
// I'm using mongoose-simpledb, so db.District refers to the districts collection
db.District.find(function(err, found) {
found.forEach(function(element) {
findParentProv(element, districtNames);
});
res.render('districts', {title: "Districts page", district_list: districtNames});
})
});
function findParentProv(element, namesArray) {
db.Province.findById(element.parent, function(err, found) {
console.log(found.name);
namesArray.push(element.name + " " + found.name);
});
}
I want to get all items in the districts collection, follow their parent field (which contains an ObjectID), find that item from the provinces collection and push both their names as a string into districtNames.
How should I do this?
Well, you do seem to be on the right track.
The one major issue I recognize in your solution is that after kicking off all the async queries for parents, you immediately return the (most likely empty) districtNames array, without waiting for the queries to finish.
This is indeed very annoying, and not surprisingly so. MongoDB is a non-relational DB, and so join operations like what you're trying to do aren't easy to get right.
The solution that would probably require the least fundamental changes to what you're doing would be to wait on all the queries before calling res.render. The most basic way to do this would be to check the length of namesArray/districtNames after pushing each element, and once you see it's gotten to the desired size, only then calling render. There are, however, more standardized ways of doing this, and I'd suggest looking into something like Async (specifically async.parallel) or a Promise framework such as Bluebird.
Now, another approach to solving this problem is de-normalizing the data. For someone with a relational background this probably sound appalling, but in Mongo it might actually be a valid solution to just include the province names along with their IDs in the districts collection, in which case your one initial query should be sufficient.
Another approach, which might be suitable if you're dealing with relatively small collections, would be to run 2 queries, 1 for all the districts and 1 for all the provinces, and do the correlation in-app. Obviously, this isn't a very efficient solution, and should definitely be avoided if there's any chance the collections contain, or will in the future contain, more than a handful of objects.
Best bet moving forward is to use ES6 Promise patterns to help with your callback patterns..
suggested modules:
lodash [optional] has a lot of useful methods, not needed here, but you may need, for example _.flatten, or _.assign
i-promise will give you a native Promise (node 0.11.3+) or a scripted implementation
es6-promise is the fallback for i-promise to use
promisify-patch is an inline promisify for specific methods.
Install the modules required for your use (in this example).
npm install --save es6-promise i-promise promisify-patch
Use Promise pattern with your example.
require('promisify-patch').patch();
var Promise = require('i-promise')
;
//returns a promise to resolve to your list for display
function getDistricts() {
//gets all of the db.District
return db.District.find.bind(db.District).promise()
//after districts retrieved
.then(function(districts){
//resolve an array of promises, will return an array of results
return Promise.all(districts.map(getDistrictProv)); //map each district via getDistrictProv
});
}
//returns a promise to resolve a specific district/province name
function getDistrictProv(district){
return db.Provice.findById.bind(db.Province).promise(element.parent)
.then(function(province){
return district.name + ' ' + province.name;
});
}
...
//express handler
router.get('/districts', function(req, res, next) {
//get the district names
getDistricts()
//then process the rendering with the names
.then(function(names){
res.render('districts', {title: "Districts page", district_list: names});
})
//if there was an error in the promise chain
// pass it along, so it can be handled by another express plugin
.catch(next)
});
Disclosure: I made i-promise and promisify-patch to make situations like this easier to convert node-style callbacks into promise chains.
NOTE: If you are creating general purpose libraries for Node or the Browser that are not flow-control related, you should at least implement the node-style callback implementation.
Further, you may wish to look into co, koa for using generators as well.
The question seemed to be how to control the flow of data, in which promises are likely the best answer. If your issue is trying to fit non-relational data into a relational box or vice-versa, may want to re-evaluate your data structure as well...
http://blog.mongodb.org/post/88473035333/6-rules-of-thumb-for-mongodb-schema-design-part-3
You should probably have some key data for parents/children replicated to those affected documents in other collections. There are configuration options via Mongoose to support this, but that doesn't mean you should avoid the consideration.
If you do many/large join operations like this it will negatively affect your performance. This isn't meant to be a religious comment only that MongoDB/SQL or other SQL vs. NoSQL considerations should be made depending on your actual needs.
The data in question seems to be highly cacheable data that may well be better with a relational/sql database.
I'm learning Node.js through the learnyounode project. I have completed the first few assignments and they all seemed reasonably straightforward.
Then, I got to the 'Async Juggling' one, and the assignment's description went completely over my head in terms of what I need to do.
The gist of it, is I need to write a Javascript that accepts 3 URLs as arguments, but that associates the correct response to the correct server. The assignment itself notes that you cannot naively assume that things will be properly associated with the correct URL.
The (incorrect) code I came up with proved that restriction true:
var http = require('http');
var bl = require('bl');
var httpCallback = function(response) {
var pipeHandler = function (err, data) {
if(err)
return console.error(err);
console.log(data.toString());
};
response.pipe(bl(pipeHandler));
};
var juggleAsyncConnections = function(connA, connB, connC) {
http.get(connA, httpCallback);
http.get(connB, httpCallback);
http.get(connC, httpCallback);
};
juggleAsyncConnections(process.argv[2], process.argv[3], process.argv[4]);
The problem, and thus my question, is, what is the correct way to handle asynchronous connection juggling, and what are the underlying concepts I need to understand to do it correctly?
Note: I've seen other questions, like "OMG why doesn't my solution work?" I'm not asking that, I deliberately set out to see the 'naive' solution fail for myself. I don't understand the underlying principles of why it doesn't work, or what principles actually do work. Additionally, I'm not asking for someone to 'solve the problem for me.' If the general algorithm can be explained, I can probably implement it on my own.
Counting callbacks is one of the fundamental ways of managing async in Node. [...]
That's an important piece.
You know how many inputs there are (3), and, because of that, you know how many outputs there should be. Keep a running tally as responses come back, then check if you received the full set before printing to the screen. You also want to keep the original order in mind (now if there were only a datatype that had numeric indexes... :grin:).
Good luck!
I've been using the following model for Namespacing my newest Scripts. So far, it has some distinct advantages that, while I'm sure could be replicated in other ways, really help to in my coding process. Unfortunately, I've come across a significant disadvantage... When using some JS compression utilities, they mangle the code badly enough that I must avoid many advantageous options. Luckily, the code I save with this model helps mitigate the "damages".
I'm still curious to know if there is a more viable solution as the min.js only fail consistently in Chrome/IE. I know the below is a little too abstract for some. Are there any experts that might point me in the right direction. I've used YUI, Packer and JSMin. JSMin works reliably, but is not nearly as efficient...
/* Global Namspace */
(function (T) {"use strict";
/* Top.Sub1 */
(function(S1) {
// ... Some methods (public/private)
/* Top.Sub1.Mini */
(function(M) {
// ... Some methods (public/private)
}(S1.Mini = S1.Mini || function(o){}));
}
(T.Sub1 = T.Sub1 || function(o){}));
/* Top.Sub2 */
(function(S2) {
// ... Some methods (public/private)
/* Top.Sub2.Mini1 */
(function(M1) {
// ... Some methods (public/private)
}(S2.Mini1 = S2.Mini1 || function(o) {}));
/* Top.Sub2.Mini2 */
(function(M2) {
// ... Some methods (public/private)
}(S2.Mini2 = S2.Mini2 || function(o) {}));
} (T.Sub2 = T.Sub2 || function(o) {}));
} (window.Namespace = window.Namespace || function(o){}));
UPDATE: The most common error I am faced with is "unexpected token" of various sorts.. sometimes a ')' and sometimes a '}'. Every once in a while, it is a '('. I haven't yet addressed gzip as I want this out of the way.
UPDATE 2: Have checked/removed ns with a Tidied-jsHint passing file and still does not minify correctly. It definitely has to do with this model... Does anyone have a clear answer as to why? If not, further recommendations are welcome. P.S. the Github has been updated with Tidied-jsHint passing code.
I'd say read this article about what needs to be done and what needs to be avoided for good minification – http://alistapart.com/article/javascript-minification-part-II
And then choose a proper modules framework like AMD or commonjs.
UPD. My main advice will be to use a linter on your code like http://jshint.com and adhere to a coding style like http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml It mostly comes with explanations of why something will break in certain sitautions. It's also going to make your code more approachable for open source contributors.
After testing nearly every option on every minifier I could get my hands on, the code minifies quite fine. (With and without the tidy, etc...) The issue comes when any of the minifiers try to replace or obfuscate symbols. In particular, it does not handle this for loop well:
for (i = 0; i < l; i++) {
_.points[i] = new S.Point(_, pts[i]);
}
Removal of the loop allows for the optimization to occur correctly.
there are several js libraries available for flow control.
though when working with the closure compiler the ones i looked at yet do not work well with the compiler in advanced mode.
is there any closure compatible flow control library?
I am mostly interested in waiting on multiple results without complicating the code more than necessary.
What i want to archive is to reduce loading time on user actions.
For one action of the user multiple requests have to be done to the backend. To keep the code maintainable, at the moment, i do one request at a time and handle potential errors after each step.
What i want to archive is that i can just fire nondepended requests together without complicating the error handling more than necessary.
I like the syntax of flow js:
var auth = flow.define(
function(username, password) {
sys.puts("trying " + username + ", " + password);
this.password = password;
keystore.get('userId:' + username, this);
},function(err, userId) {
keystore.get('user:' + userId + ':password', this);
},function(err, passwordInDb) {
if (passwordInDb == this.password) {
sys.puts("Authenticated!");
}
else {
sys.puts("Failed Authentication!");
}
}
)
It also allows to spawn multiple async operations and collect them.
Though if you need any state between callbacks the state is stored like "this.password" above.
as the containing scope is not typed the closure compiler will not be able to rename it consistently (from my understanding) when in ADVANCED mode.
So i need an alternative which has a typed container object that is pushed as a parameter (or this) through each function.
Can you use goog.async.Deferred from Closure Library? It manage async and sync workflow.
Generally, you can use any library with Closure Compiler's advanced mode by creating an extern file for the library and loading the code separately (or concatenating the library after compilation of your code).
Ok i've found a solution.
When i create a typed container object and pass it to each of the functions with goog.bind it works.
A very tiny flow control lib for Closure Library i've written is ready.js.
As per the description:
Watches over multiple async operations and triggers listeners when all or some are complete.
It's worth a look
I'm trying to find the best way to create async calls when each call depends on the prior call to have completed. At the moment I'm chaining the methods by recursively calling a defined process function as illustrated below.
This is what I'm currently doing.
var syncProduct = (function() {
var done, log;
var IN_CAT = 1, IN_TITLES = 2, IN_BINS = 3;
var state = IN_CAT;
var processNext = function(data) {
switch(state) {
case IN_CAT:
SVC.sendJsonRequest(url("/api/lineplan/categories"), processNext);
state = IN_TITLES;
break;
case IN_TITLES:
log((data ? data.length : "No") + " categories retrieved!");
SVC.sendJsonRequest(url("/api/lineplan/titles"), processNext);
state = IN_BINS;
break;
case IN_BINS:
log((data ? data.length : "No") + " titles retrieved!");
SVC.sendJsonRequest(url("/api/lineplan/bins"), processNext);
state = IN_MAJOR;
break;
default:
log((data ? data.length : "No") + " bins retrieved!");
done();
break;
}
}
return {
start: function(doneCB, logCB) {
done = doneCB; log = logCB; state = IN_CAT;
processNext();
}
}
})();
I would then call this as follows
var log = function(message) {
// Impl removed.
}
syncProduct.start(function() {
log("Product Sync Complete!");
}, log);
While this works perfectly fine for me I can't help but think there has to be a better (simpler) way. What happens later when my recursive calls get too deep?
NOTE: I am not using javascript in the browser but natively within the Titanium framework, this is akin to Javascript for Node.js.
There are lots of libraries and tools that do async chaining and control-flow for you and they mostly come in two main flavours:
Control-flow libraries
For example, see async, seq and step (callback based) or Q and futures (promise based). The main advantage of these is that they are just plains JS libraries that ease the pain of async programming.
In my personal experience, promise-based libraries tend to lead to code that looks more like usual synchronous code, since you return values using "return" and since promise values can be passed and stored around, similarly to real values.
On the other hand, continuation-based code is more low level since it manipulates code paths explicitely. This can possibly allow for more flexible control flow and better integration with existing libraries, but it might also lead to more boilerplaty and less intuitive code.
Javascript CPS compilers
Extending the language to add native support for coroutines/generators lets you write asynchronous code in a very straightforward manner and plays nice with the rest of the language meaning you can use Javascript if statements, loops etc instead of needing to replicate them with functions. This also means that its very easy to convert previously sync code into an async version. However, there is the obvious disadvantage that not every browser will run your Javascript extension so you will need to add a compilation step in your build proccess to convert your code to regular JS with callbacks in continuation-passing-style. Anyway, one promising alternative is the generators in the Ecmascript 6 spec - while only firefox supports them natively as of now, there are projects such as regenerator and Traceur to compile them back to callbacks. There are also other projects that create their own async syntax (since es6 generators hadn't come up back then). In this category, you will find things such as tamejs and Iced Coffeescript. Finally, if you use Node.js there you could also take a look at Fibers.
My recomendation:
If you just want something simple that won't complicate your build proccess, I would recomend going with whatever control-flow library best fits your personal style and the libraries you already use.
However, if you expect to write lots of complicated and deeply-integrated asynchronous code, I would strongly recommend at least looking into a compiler-based alternative.