I am fairly new to NodeJS and to JavaScript in general. Here is my script:
var data = [];
client.query(
'SELECT * FROM cds',
function selectCb(err, results, fields) {
if (err) {
throw err;
}
console.log(results);
console.log(fields);
data.push(results);
}
);
console.log(data);
How can I get access to the results (or data) var outside of the callback? I don't want to have to write a ton of callbacks inside of each other when running queries.
What you're asking for is synchronous (or blocking) execution, and that's really contrary to the design and spirit of node.js.
Node, like JavaScript, is single-threaded. If you have blocking code, then the whole process is stopped.
This means you have to use callbacks for anything that will take a long time (like querying from a database). If you're writing a command-line tool that runs through in one pass, you may be willing to live with the blocking. But if you're writing any kind of responsive application (like a web site), then blocking is murder.
So it's possible that someone could give you an answer on how to make this a blocking, synchronous call. If so, and if you implement it, you're doing it wrong. You may as well use a multi-threaded scripting language like Ruby or Python.
Writing callbacks isn't so bad, but it requires some thought about architecture and packaging in ways that are probably unfamiliar for people unaccustomed to the style.
Node.js uses the Continuation Passing Style for all of it's asynchronous interfaces. There is a lot of discussion around this and modules that have been created for easing the pain of nested callbacks. There were rumors for awhile that node.js might start reintroducing Promises into it's core in the v0.7.x branch, but I'm not sure if that's true or not.
Short of using one of the flow-control libraries from the link above (or rolling your own), you're either going to have to live with the nested callbacks, or simply provide a function reference for the callback. E.g.,
var data = [],
handler = function(err, results, fields) {
if (err) {
throw err;
}
...
};
client.query('SELECT * FROM cds', handler);
Related
It's a very general question, but I don't quite understand. When would I prefer one over the other? I don't seem to understand what situations might arise, which would clearly favour one over the other. Are there strong reasons to avoid x / use x?
When would I prefer one over the other?
In a server intended to scale and serve the needs of many users, you would only use synchronous I/O during server initialization. In fact, require() itself uses synchronous I/O. In all other parts of your server that handle incoming requests once the server is already up and running, you would only use asynchronous I/O.
There are other uses for node.js besides creating a server. For example, suppose you want to create a script that will parse through a giant file and look for certain words or phrases. And, this script is designed to run by itself to process one file and it has no persistent server functionality and it has no particular reason to do I/O from multiple sources at once. In that case, it's perfectly OK to use synchronous I/O. For example, I created a node.js script that helps me age backup files (removing backup files that meet some particular age criteria) and my computer automatically runs that script once a day. There was no reason to use asynchronous I/O for that type of use so I used synchronous I/O and it made the code simpler to write.
I don't seem to understand what situations might arise, which would clearly favour one over the other. Are there strong reasons to avoid x / use x?
Avoid ever using synchronous I/O in the request handlers of a server. Because of the single threaded nature of Javascript in node.js, using synchronous I/O blocks the node.js Javascript thread so it can only do one thing at a time (which is death for a multi-user server) whereas asynchronous I/O does not block the node.js Javascript thread (allowing it to potentially serve the needs of many users).
In non-multi-user situations (code that is only doing one thing for one user), synchronous I/O may be favored because writing the code is easier and there may be no advantages to using asynchronous I/O.
I thought of an electron application with nodejs, which is simply reading a file and did not understand what difference that would make really, if my software really just has to wait for that file to load anyways.
If this is a single user application and there's nothing else for your application to be doing while waiting for the file to be read into memory (no sockets to be responding to, no screen updates, no other requests to be working on, no other file operations to be running in parallel), then there is no advantage to using asynchronous I/O so synchronous I/O will be just fine and likely a bit simpler to code.
When would I prefer one over the other?
Use the non-Sync versions (the async ones) unless there's literally nothing else you need your program to do while the I/O is pending, in which case the Sync ones are fine; see below for details...
Are there strong reasons to avoid x / use x?
Yes. NodeJS runs your JavaScript code on a single thread. If you use the Sync version of an I/O function, that thread is blocked waiting on I/O and can't do anything else. If you use the async version, the I/O can continue in the background while the JavaScript thread gets on with other work; the I/O completion will be queued as a job for the JavaScript thread to come back to later.
If you're running a foreground Node app that doesn't need to do anything else while the I/O is pending, you're probably fine using Sync calls. But if you're using Node for processing multiple things at once (like web requests), best to use the async versions.
In a comment you added under the question you've said:
I thought of an electron application with nodejs, which is simply reading a file and did not understand what difference that would make really, if my software really just has to wait for that file to load anyways.
I have virtually no knowledge of Electron, but I note that it uses a "main" process to manage windows and then a "rendering" process per window (link). That being the case, using Sync functions will block the relevant process, which may affect application or window responsiveness. But I don't have any deep knowledge of Electron (more's the pity).
Until somewhat recently, using async functions meant using lots of callback-heavy code which was hard to compose:
// (Obviously this is just an example, you wouldn't actually read and write a file this way, you'd use streaming...)
fs.readFile("file1.txt", function(err, data) {
if (err) {
// Do something about the error...
} else {
fs.writeFile("file2.txt", data, function(err) {
if (err) {
// Do something about the error...
} else {
// All good
});
}
});
Then promises came along and if you used a promisified* version of the operation (shown here with pseudonyms like fs.promisifiedXYZ), it still involved callbacks, but they were more composable:
// (See earlier caveat, just an example)
fs.promisifiedReadFile("file1.txt")
.then(function(data) {
return fs.promisifiedWriteFile("file2.txt", data);
})
.then(function() {
// All good
})
.catch(function(err) {
// Do something about the error...
});
Now, in recent versions of Node, you can use the ES2017+ async/await syntax to write synchronous-looking code that is, in fact, asynchronous:
// (See earlier caveat, just an example)
(async () => {
try {
const data = await fs.promisifiedReadFile("file1.txt");
fs.promisifiedWriteFile("file2.txt", data);
// All good
} catch (err) {
// Do something about the error...
}
})();
Node's API predates promises and has its own conventions. There are various libraries out there to help you "promisify" a Node-style callback API so that it uses promises instead. One is promisify but there are others.
Hi I have a very simple (i think) js question that I seem to be stuck on.
I am trying to create the route below.
What gets console.logged from within the bEtsy function is what I would like to have display on the page. however 'body' is not available outside of that scope.
app.get('/api/etsy/getListings',function(req, res){
bEtsy.getAllListings(req, res, function(err, body) {
// console.log(body);
});
res.json(req.body); // both this and res.json(body) does not work
});
Move res.json(req.body); into the callback function.
Apart from the scoping problem: It is asynchronous, so in your code it will be called long after res.json(req.body) runs.
app.get('/api/etsy/getListings', function(req, res) {
bEtsy.getAllListings(req, res, function(err, body) {
res.json(body);
//console.log(body);
});
});
A more general piece of advice (or two or three pieces), aside from the problem at hand:
What helps me with such situations and "callback thinking" is to almost never use inline callback functions: Write code only one layer deep (plus one layer for the module pattern of course), avoid callback hell! Name all callbacks and write them all on the same (top) level.
function allListingsReceived(err, body, res) {
res.json(body);
//console.log(body);
}
function getListings(req, res) {
// ASYNC
bEtsy.getAllListings(req, res, allListingsReceived);
}
//ASYNC
app.get('/api/etsy/getListings', getListings);
This allows me to get a much better overview over the actual call sequence. In this case, when getAllListings is called you know it is asynchronous - in my own code I add a clear comment (like I did above). So I know anything I were to write after that async function would not be able to access anything that async function is supposed to get me. IMHO such a comment is important - in Javascript there is no way to know if a callback function is asynchronous. Usually it is, but if it's synchronous and you expect asynchronism you may get into trouble too! So I think it's better to write it as a comment (always the exact same short string throughout the whole project), a formalized code annotation. Which by the way leads to another problem: When you write functions that accept a callback function, make sure they always call it either synchronously or asynchronously, never both ways (some functions use cached values and are able to return a result right away instead of starting an async. network request).
Basically, the written structure does not reflect the runtime situation with this style - but this is okay, since the runtime situation is completely flexible anyway (if you want to change which callback function you use, or add another one in between, do you really want to shift around tons of lines of code instead of just exchanging a name? Not to mention an increase in the ease of reusability). This is much easier to read in longer callback-style code files then several layers deep nested asynchronous functions IMHO. Avoid functions inside functions, apart from the module pattern, as much as possible.
Having named functions also is much better for debugging, stack traces are much easier to read.
A note: My example code leaves one issue open: if this is inside a module (or class), those would be internal functions, and you may have to make sure about the correct context/scope (where this points to, if you access object member variables this way from inside those functions). It works the same when those functions are on the prototype though. So this is just a general concept example that disregards this side issue.
Another note: When writing in this style variable that previously were available to an inner function via a closure - in this example res - now have to be made available as function parameters when calling the callback function. That adds some complexity - but on the other hand forces you to create clean(er) APIs in your own code. Personally I don't like excessive reliance on closures to pass arguments. I'm too stupid, I prefer to have a clean interface definition by having all parameters a function uses in its header. Apparently I'm not alone, that is one of the advantages most often touted for functional programming :) An alternative to filling the header with arguments that is "clean" too are object properties under this. My small example looks a little "procedural", but it only served to illustrate one single point. Of course this belongs into a larger context of modular programming.
Could someone please give me some clarity if Node's Crypto module is blocking or not? From what I thought I understood, some core modules can yield to a separate thread in c++.
I'm mainly interested in:
crypto.createHmac
crypto.createSign
crypto.createVerify
If it does process on a separate thread, then fine. If not, then I might have to think about using the cluster module. I'm attempting to avoid blocking the main event loop.
Here is a trivial example of how I would implement crypto.createHmac.
function createHmac(algo, secret, data, callback) {
var cryptoStream = crypto.createHmac(algo, secret);
// Emitters
cryptoStream
.on('error', function (err) {
return callback(err);
});
// Write the data
cryptoStream.write(data, 'utf8', function(){
cryptoStream.end();
return callback(null, cryptoHmacStream.read());
});
}
The "crypto.create*" functions you list are synchronous and blocking, however, they are quite trivial and just set up some basic data structures. They do not perform any heavy crypto computation, which happens later in asynchronous non-blocking methods. Thus, these are fine to call in the main event loop. They are no different than doing basic string manipulation, regex matching, etc in terms of overhead and do not require special treatment. You do not need the cluster module. Your snippet is already asynchronous and non-blocking because the cryptoStream is asynchronous by way of the event emitter pattern. The write method is asynchronous, which is the important part.
To contradict the previous answer, as far as I understand, the crypto.createHmac method does all computation on the main thread in C++, but this is still the same thread as the event loop.
Your snippet is not actually asynchronous and not actually non-blocking. The interface only looks asynchronous. It will still use up time on your event loop.
For truly asynchronous crypto methods and better throughput, use crypto-async.
I am developing an application using NodeJS where two queries depend on each other here is explanation of my situation.
I have to query database for some values say A then I have to query
database for some other value B only if there is an A in the database.
I know I can do this in NodeJS' natural way like execute query for A when its result is ready execute query for B in A's callback and then finally render response in B's callback.
Here is my question, is there a design issue in the solution I just mentioned perhaps about nested callbacks.
Second is there any method in which I can make NodeJs IO as Blocking from NON-Blocking?
I was made aware of a library called Async Which i wish id found when i started my NodeJs projects. It can take care of this and much more.
Async provides around 20 functions that include the usual 'functional'
suspects (map, reduce, filter, each…) as well as some common patterns
for asynchronous control flow (parallel, series, waterfall…). All
these functions assume you follow the node.js convention of providing
a single callback as the last argument of your async function.
basic example from site that would help in your scenario
async.series([
function(callback){
// do db call one
callback(null, 'one');
},
function(callback){
// do db call two
callback(null, 'two');
}
],
function(err, results){
// results is now equal to ['one', 'two']
});
Here is my question, is there a design issue in the solution I just mentioned perhaps about nested callbacks.
No, your solution is perfectly fine.
Second is there any method in which I can make NodeJs IO as Blocking from NON-Blocking?
Yes. You can write your own c++ extension to provide blocking calls.
So I started a little project in Node.js to learn a bit about it. It's a simple caching proxy for arch linux's package system as node provides most of the heavy lifting.
This has two "main" phases, server setup and serving.
Then serving has two main phases, response setup and response.
The "main" setup involves checking some files, loading some config from files. loading some json from a web address. Then launching the http server and proxy instance with this info.
setup logger/options - read config - read mirrors - read webmirror
start serving
Serving involves checking the request to see if the file exists, creating directories if needed, then providing a response.
check request - check dir - check file
proxy request or serve file
I keep referring to them as synchronisation points but searches don't lead to many results. Points where a set of async tasks have to be finished before the process can complete a next step. Perl's AnyEvent has conditional variables which I guess is what I'm trying to do, without the blocking.
To start with I found I was "cheating" and using the synchronous versions of any functions where provided but that had to stop with the web requests, so I started restructuring things. Immediately most search's led to using async or step to control the flow. To start with I was trying lots of series/parallel setups but running into issues if there were any async calls underneath the functions would "complete" straight away and the series would finish.
After much wailing and gnashing of teeth, I ended up with a "waiter" function using async.until that tests for some program state to be set by all the tasks finishing before launching the next function.
// wait for "test" to be true, execute "run",
// bail after "count" tries, waiting "sleep" ms between tries;
function waiter( test, run, count, sleep, message ) {
var i=0;
async.until(
function () {
if ( i > count ) { return true; }
logger.debug('waiting for',message, test() );
return test();
},
function (callback) {
i++;
setTimeout(callback, sleep );
},
function (err) {
if ( i > count ) {
logger.error('timeout for', message, count*sleep );
return;
}
run()
}
);
}
It struck me as being rather large and ugly and requiring a module to implement for something that I thought was standard, so I am wondering what's a better way. Am I still thinking in a non-async way? Is there something simple in Node I have overlooked? Is there a standard way of doing this?
I imagine with this setup, if the program get's complex there's going to be a lot of nesting functions to describe the flow of the program and I'm struggling to see a good way to lay it all out.
any tips would be appreciated.
You can't really make everything to be synchronous. Nodejs is designed to perform asynchronously (which may of course torment you at times). But there are a few ways techniques to make it work in a synchronous way (provided the pseudo-code is well-thought and code is designed carefully):
Using callbacks
Using events
Using promises
Callbacks and events are easy to use and understand. But with these, sometimes the code can get real messy and hard to debug.
But with promises, you can avoid all that. You can make dependency chains, called 'promises' (for instance, perform Promise B only when Promise A is complete).
Earlier versions of node.js had implementation of promises. They promised to do some work and then had separate callbacks that would be executed for success and failure as well as handling timeouts.
But in later versions, that was removed. By removing them from the core node.js, it created possibility of building up modules with different implementations of promises that can sit on top of the core. Some of these are node-promise, futures, and promises.
See these links for more info:
Framework
Promises and Futures
Deferred Promise - jQuery