Develop Reference

Advice on creating asynchronous calls that depend on each other

I am attempting to create a library to make API calls to a web application (jira, if you care to know) I have my api calls working no problem, but I am looking to make the code a bit more readable and use-able. I have tried searching for my needs, but it turns out I am not sure what I need to be searching for.
I am having an issue with Asynchronous calls that depend on each other, I understand that I have to wait until the callback is ran to run my next item, but I am not sure of the best way to design this.
I really would like to make Chaining a feature of my api, which I would hope to look like this:
createProject(jsonProjectStuff)
.setLeadUser("myusername")
.createBoard("boardName")
.setBoardPermissions(boardPermissionJSONvar)
.addRole(personRoleJSONvar);
with this example, everything would have to wait on the createProject as it will return the project. createBoard doesn't rely on the project normally, but used in this context it should be "assigned" to the project made, setting the board permissions only relies on the createBoard to work. addRole is specific to the project again.
the questions I have are:
Is this possible to switch context like this and keep data in-between them without the need to run the function from the response hard coded?
If this is possible, is this a good idea? If not I am open to other schemes.
I can think of a couple ways to make it work, including registering the function calls with a dependency tree and then fulfilling promises as we go, although that is mostly conceptual for me at this point as I am trying to decide the best.
Edit 2/19/2016
So I have looked into this more and I have decided on a selective "then" only when it creating a new item doesn't relate directly to the parent.
//Numbers are ID, string is Name
copyProject(IDorName)
.setRoles(JSONItem)
.setOwner("Project.Owner")
.setDefaultEmail("noreply#fake.com")
.then(
copyBoard(IDorName)
.setName("Blah blah Name {project.key}"),
saveFilterAs(IDorName, "Board {project.key}",
"project = {project.key} ORDER BY Rank ASC")
.setFilterPermissions({shareValuesJSON})
)
I like this solution a lot, the only thing I am unsure of how to do is the string "variables", I suppose it could be "Blah blah Name " + this.project.key
either way I am unsure of how to give copyBoard or saveFilterAs access to it via the "then" function.
Any thoughts?

I've been using Nightmare (a headless browser) lately.
It has a fluent API that uses a nice design pattern.
Calling the API doesn't directly execute the actions, it only queues them and when you are ready to execute you must call the end function which returns a promise. The promise is resolved when the queue has completed its async execution.
For example, in your situation
createProject(jsonProjectStuff)
.setLeadUser("myusername")
.createBoard("boardName")
.setBoardPermissions(boardPermissionJSONvar)
.addRole(personRoleJSONvar)
.end() // Execute the queue of operations.
.then() => {
// All operations completed.
))
.catch(err => {
// An error occurred.
});
I feel like this pattern is quite elegant. It allows you to have a fluent API to build a sequence of actions. Then when you are ready to execute said operations you call end (or whatever). The sequence of operations are then completed asynchronously and you use the promise to handle completion and errors.

Basic scope issue (javascript and node)

Hi I have a very simple (i think) js question that I seem to be stuck on.
I am trying to create the route below.
What gets console.logged from within the bEtsy function is what I would like to have display on the page. however 'body' is not available outside of that scope.
app.get('/api/etsy/getListings',function(req, res){
bEtsy.getAllListings(req, res, function(err, body) {
// console.log(body);
});
res.json(req.body); // both this and res.json(body) does not work
});

Move res.json(req.body); into the callback function.
Apart from the scoping problem: It is asynchronous, so in your code it will be called long after res.json(req.body) runs.
app.get('/api/etsy/getListings', function(req, res) {
bEtsy.getAllListings(req, res, function(err, body) {
res.json(body);
//console.log(body);
});
});
A more general piece of advice (or two or three pieces), aside from the problem at hand:
What helps me with such situations and "callback thinking" is to almost never use inline callback functions: Write code only one layer deep (plus one layer for the module pattern of course), avoid callback hell! Name all callbacks and write them all on the same (top) level.
function allListingsReceived(err, body, res) {
res.json(body);
//console.log(body);
}
function getListings(req, res) {
// ASYNC
bEtsy.getAllListings(req, res, allListingsReceived);
}
//ASYNC
app.get('/api/etsy/getListings', getListings);
This allows me to get a much better overview over the actual call sequence. In this case, when getAllListings is called you know it is asynchronous - in my own code I add a clear comment (like I did above). So I know anything I were to write after that async function would not be able to access anything that async function is supposed to get me. IMHO such a comment is important - in Javascript there is no way to know if a callback function is asynchronous. Usually it is, but if it's synchronous and you expect asynchronism you may get into trouble too! So I think it's better to write it as a comment (always the exact same short string throughout the whole project), a formalized code annotation. Which by the way leads to another problem: When you write functions that accept a callback function, make sure they always call it either synchronously or asynchronously, never both ways (some functions use cached values and are able to return a result right away instead of starting an async. network request).
Basically, the written structure does not reflect the runtime situation with this style - but this is okay, since the runtime situation is completely flexible anyway (if you want to change which callback function you use, or add another one in between, do you really want to shift around tons of lines of code instead of just exchanging a name? Not to mention an increase in the ease of reusability). This is much easier to read in longer callback-style code files then several layers deep nested asynchronous functions IMHO. Avoid functions inside functions, apart from the module pattern, as much as possible.
Having named functions also is much better for debugging, stack traces are much easier to read.
A note: My example code leaves one issue open: if this is inside a module (or class), those would be internal functions, and you may have to make sure about the correct context/scope (where this points to, if you access object member variables this way from inside those functions). It works the same when those functions are on the prototype though. So this is just a general concept example that disregards this side issue.
Another note: When writing in this style variable that previously were available to an inner function via a closure - in this example res - now have to be made available as function parameters when calling the callback function. That adds some complexity - but on the other hand forces you to create clean(er) APIs in your own code. Personally I don't like excessive reliance on closures to pass arguments. I'm too stupid, I prefer to have a clean interface definition by having all parameters a function uses in its header. Apparently I'm not alone, that is one of the advantages most often touted for functional programming :) An alternative to filling the header with arguments that is "clean" too are object properties under this. My small example looks a little "procedural", but it only served to illustrate one single point. Of course this belongs into a larger context of modular programming.

Is Node's Crypto module blocking even for methods that are not defined as sync

Could someone please give me some clarity if Node's Crypto module is blocking or not? From what I thought I understood, some core modules can yield to a separate thread in c++.
I'm mainly interested in:
crypto.createHmac
crypto.createSign
crypto.createVerify
If it does process on a separate thread, then fine. If not, then I might have to think about using the cluster module. I'm attempting to avoid blocking the main event loop.
Here is a trivial example of how I would implement crypto.createHmac.
function createHmac(algo, secret, data, callback) {
var cryptoStream = crypto.createHmac(algo, secret);
// Emitters
cryptoStream
.on('error', function (err) {
return callback(err);
});
// Write the data
cryptoStream.write(data, 'utf8', function(){
cryptoStream.end();
return callback(null, cryptoHmacStream.read());
});
}

The "crypto.create*" functions you list are synchronous and blocking, however, they are quite trivial and just set up some basic data structures. They do not perform any heavy crypto computation, which happens later in asynchronous non-blocking methods. Thus, these are fine to call in the main event loop. They are no different than doing basic string manipulation, regex matching, etc in terms of overhead and do not require special treatment. You do not need the cluster module. Your snippet is already asynchronous and non-blocking because the cryptoStream is asynchronous by way of the event emitter pattern. The write method is asynchronous, which is the important part.

To contradict the previous answer, as far as I understand, the crypto.createHmac method does all computation on the main thread in C++, but this is still the same thread as the event loop.
Your snippet is not actually asynchronous and not actually non-blocking. The interface only looks asynchronous. It will still use up time on your event loop.
For truly asynchronous crypto methods and better throughput, use crypto-async.

Calling a function asynchronously in node.js?

In node.js, how do you call a function so it runs in background? Something like:
work = function(){
for (var i=0; i<1000000; ++i);
return "world!";
};
spawn(work).then(console.log);
console.log("Hello ");
Should output what you expect.
Note: not necessarily following this pattern.

Nothing in Node.JS will run "in the background". This is because JS can't multi-thread. Yet it has the ability to run code back to back, for example running 2 for loops at the same time, will cause the first for loop to iterate a set amount, then the second will iterate and they will swap processing power to make it seem as if methods can be run at the same time.
Node.JS if I am not mistaken does this with the callbacks.
"Callbacks
Callbacks are a basic idiom in node.js for asynchronous operations. When most people talk about callbacks, they mean the a function that is passed as the last parameter to an asynchronous function. The callback is then later called with any return value or error message that the function produced. For more details, see the article on callbacks"
With more example and information found here -
http://docs.nodejitsu.com/articles/getting-started/control-flow/how-to-write-asynchronous-code

Async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript
https://npmjs.org/package/async

Try looking at the child_process features.
I'm currently using child_process to fork processes and parallelize operations. The best part is that (unlike working in C or C++), node does a lot of the painful work for you.
Even more (a side note), you can pass JavaScript code back and forth between processes and build a powerful multi-CPU, multi-host and multi-process application for compute-intensive tasks. Don't let the negative voices tell you otherwise...node is great and it can do what you appear to be asking.
Check out http://nodejs.org/api/child_process.html

Variable outside of callback in NodeJS

I am fairly new to NodeJS and to JavaScript in general. Here is my script:
var data = [];
client.query(
'SELECT * FROM cds',
function selectCb(err, results, fields) {
if (err) {
throw err;
}
console.log(results);
console.log(fields);
data.push(results);
}
);
console.log(data);
How can I get access to the results (or data) var outside of the callback? I don't want to have to write a ton of callbacks inside of each other when running queries.

What you're asking for is synchronous (or blocking) execution, and that's really contrary to the design and spirit of node.js.
Node, like JavaScript, is single-threaded. If you have blocking code, then the whole process is stopped.
This means you have to use callbacks for anything that will take a long time (like querying from a database). If you're writing a command-line tool that runs through in one pass, you may be willing to live with the blocking. But if you're writing any kind of responsive application (like a web site), then blocking is murder.
So it's possible that someone could give you an answer on how to make this a blocking, synchronous call. If so, and if you implement it, you're doing it wrong. You may as well use a multi-threaded scripting language like Ruby or Python.
Writing callbacks isn't so bad, but it requires some thought about architecture and packaging in ways that are probably unfamiliar for people unaccustomed to the style.

Node.js uses the Continuation Passing Style for all of it's asynchronous interfaces. There is a lot of discussion around this and modules that have been created for easing the pain of nested callbacks. There were rumors for awhile that node.js might start reintroducing Promises into it's core in the v0.7.x branch, but I'm not sure if that's true or not.
Short of using one of the flow-control libraries from the link above (or rolling your own), you're either going to have to live with the nested callbacks, or simply provide a function reference for the callback. E.g.,
var data = [],
handler = function(err, results, fields) {
if (err) {
throw err;
}
...
};
client.query('SELECT * FROM cds', handler);