I'm getting slightly annoyed now after reading many different articles on how to write proper code in node js.
I would just like some clarification on whether I'm correct or not with each of these statements:
Code is executed synchronously
A for loop or while loop etc. will be executed asynchronously
By doing the code below isn't proper asynchronous:
L
function doSomething(callback) {
// doing some code
// much much code
callback();
}
The reason that people are saying that this won't work properly is that code is executed asynchronously, so the callback won't be fired off at the end of the code it will all be executed at once.
So for example if you were filling some object by doing some stuff and you wanted to post that full object back through the callback it doesn't work because it will all be executed at the same time.
No, for and while loops are synchronous in Node.js.
Whether your doSomething function example will work or not all depends on whether you're making any calls to asynchronous functions before calling callback(). If you're making asynchronous calls, then you'd need to postpone calling callback() until those asynchronous calls have completed. If you're only making synchronous calls, then you don't need to use a callback and your function can synchronously return the result.
Code in node.js isn't standardly asynchronous. Calls that you place one after another will still be executed one after another. A for or while loop will still block the main code in this way.
However, you can pass around functions which will be executed later, through process.nextTick(callback). This method will add a function to the event queue, which node.js takes care of internally. This will only be executed after any other blocking code has been extecuted. An example:
function doWork(callback) {
//Do some work here, for instance a for loop that might take a while
callback(workReslt);
}
function workDone(workResult) {
//Process the result here
}
function startWork() {
process.nextTick(doWork(workDone));
}
This will execute doWork() when the server passes through the event loop again. This way, the startWork() function will not block the main code.
However, most modules in node.js already implement this. Database and file access is usually non-blocking, it will simply fire a callback when it is done with it's work.
If you really have to perform heavy async calculations (or perhaps some IO that doesn't already have a module for it), you can look into the async module, which has a nice tutorial here: http://justinklemm.com/node-js-async-tutorial/
As for coding conventions, think of it this way. If you know the action is going to take time, you should make it asynchronous (either through the method I mentioned first or the async module). Actions like these are usually related to I/O, which usually also means there is already an asynchronous module for it. However, most cases I have come across up to now, have never required this.
Sources:
http://howtonode.org/understanding-process-next-tick
Related
I am aiming to understand what exactly is it about certain JavaScript functions that make them asynchronous. You could have a function like this:
function someRandomFunction () {
for (let i of everything) {
iterator++
something = true
hello = that + whatever
}
}
Nothing about this function is asynchronous. It does a lot of stuff but it does it very quickly.
But then take a Node.js function like this:
fs.readFile('path/to/file', (err, data) => {
// do something
})
This function is declared to be asynchronous. But why is it? What is the reason behind it?
Is it because it takes a certain amount of time for reading a file to complete, therefore it's asynchronous? Why is that asynchronous when looping through some variables and doing some calculations is not asynchronous?
The 'idea' of something being asynchronous means "we've relinquished control to some other operational code, and we have no idea when that other code will allow us to operate again".
So in your bottom sample, you give control over to node's filesystem operations to read a file. We don't know how large that file is, so there is no way to anticipate how much time it will take to complete. So JavaScript allows us to provide a "callback" that get's fired when the async operation is complete. When the callback is executed, control is returned to our code.
The top example is synchronous because your code maintains control over the operation and Javascript executes code synchronously by nature.
A function is either explicitly synchronous or asynchronous; there's no "certain amount of time" that automatically makes it asynchronous.
The important thing to remember with Node is that it's "async I/O", which means nothing is ever async without some input/output to/from the process. You normally see this with file reading/writing, database calls, and anything that requires a network call.
This is a basic but fundamental question. Passing a function to another function doesn't make the code asynchronous automatically. It's all about attemting to work with another piece of code, mostly of unknown origins, probably but not necessarily related with operating system's IO mechanism. In another words you ask something to happen out of the realm of the currently exectuing JS context by handing a task to known or unknown resources which are possibly running on a separate thread.
As i said this can be an IO operation or a database access of which you have no idea on
What code is running
At which thread it is running
How long it will take
Whether it will succeed or not
Behind the curtains these alien codes finally trigger the event queue to invoke the callback function through a driver. In some pure functional languages like Haskell such activities are strictly kept away from the context and must be handled by monadic type functions. In JS this is basically done by asynchronous workflow by callbacks, promises or async-awiat mechanism.
Once you hand out a task to the outer world then there shall be no way for it to interfere with your synchronous code.
In terms of JavaScript, the difference between synchronous and async functions is where and when the code is executed. Synchronous functions are executed immediately one after the other in the current call stack. Async functions are passed off to the event loop, and values returned once the current call stack has completed execution.
Caveat being, nothing in JavaScript is truly asynchronous due to the fact that JS only executes on a single process thread
I've read that callbacks make JavaScript asynchronously. But I'm not sure if I understood the explanation. This is what I get
Callback functions allow us to do things asynchronously, since they ensure
that the lines prior to the callback are completely finished before loading
the next line.
Is that true? Thanks
First off, it isn't the callback that enables anything. A given operation in node.js or even browser-based javascript either is or isn't asynchronous. It really has nothing to do with the callback, though a callback is typically used to communicate results of an asynchronous operation.
For example, Javascript's array.forEach() uses a callback, but it is not asynchronous. So, async operations are async because their underlying implementation is non-blocking. You start the operation by making a function call, the operation proceeds in the background and the rest of your code continues to run. Meanwhile, when the asynchronous operation completes, it usually then needs to tell your code that it is done and perhaps communicate some results. A callback function is the chosen mechanism for communicating the completion of the async operation.
I've read that callbacks make JavaScript asynchronously.
No, that is not really true. Callbacks can be used with synchronous operations too. Just because one uses a callback does not make anything asynchronous. The underlying native code implementation of an operation must be asynchronous (such as an Ajax call or other networking operation). Callbacks are used to communicate the results of asynchronous operations. They are also have many other non-asynchronous uses too. So, a callback is just one tool used in an asynchronous operation and a callback is a tool with many other uses too. You cannot say callback === asynchronous.
Callback functions allow us to do things asynchronously, since they
ensure that the lines prior to the callback are completely finished
before loading the next line.
It is hard to tell exactly what you mean by this, but it sounds wrong to me. When using asynchronous operations, code is typically not executed in the order laid out in the file. For example, if you did this:
console.log("async start");
callSomeAsyncOperation(function(result) {
console.log("async done");
});
console.log("I'm here now");
You would see this in the log:
async start
I'm here now
async done
Callbacks and Event Queues Explained
It may also be useful to understand how an asynchronous operation works. Javascript works off an event queue. A given sequence of Javascript code runs to its completion. When that completes, the engine looks in the event queue to see if there are any more events to process. If so, the first event in the queue is pulled out and a callback that was registered for that event is called. This starts a new sequence of Javascript code running. That code continues to run until it finishes. When it finishes, the engine checks for another event. If there is one, it is then processed by calling a callback associated with that event. When there are no more events to process, the engine goes to sleep waiting for the next event. When an event occurs (outside the main Javascript thead), it is then added to the queue and that process of adding it to the queue causes the JS engine to wake up and service that event.
When writing your Javascript, you will often register and event handler for an event. The way this works in Javascript is that you say what event you're interested in (which may also include specifying some other information like what object you're looking for events on) and then you pass it is a function reference. You are, in essence, telling the Javascript engine that you want it to call your function reference when this event occurs. This type of function reference is referred to as a "callback". It's just a normal function, but the context in which it is being used is called a "callback" because some other code will "call you back" at some time in the future by executing your function. You can then place appropriate code in that function reference (inside that callback function) that can respond to that event. Depending upon the type of event, you may only get called once or it may call you every time that event occurs.
You can read more about how this event queue and callbacks work in these references:
Run Arbitrary Code While Waiting For Callback in Node?
blocking code in non-blocking http server
Hidden threads in Javascript/Node that never execute user code: is it possible, and if so could it lead to an arcane possibility for a race condition?
How does JavaScript handle AJAX responses in the background? (written about the browser, but concept is the same)
First of all, let's understand what callback is (by definition):
In computer programming, a callback is a piece of executable code that
is passed as an argument to other code, which is expected to call back
(execute) the argument at some convenient time. The invocation may be
immediate as in a synchronous callback, or it might happen at later
time as in an asynchronous callback. In all cases, the intention is to
specify a function or subroutine as an entity that is, depending on
the language, more or less similar to a variable.
Now, talking about Javascript, not always a callback is asynchronous. For example:
function syncOperation(callback) {
callback();
}
syncOperation(function() {
console.log('a');
console.log('b');
});
console.log('c');
That callback is synchronous, since it doesn't make any asynchronous operation. If you open your console and run the code above, you'll see that it will log a, then b, then c.
Now, let's see an asynchronous example:
function asyncOperation(callback) {
setTimeout(callback, 0);
}
asyncOperation(function() {
console.log('a');
console.log('b');
});
console.log('c');
You'll see firstly c, then a, and b. That's because the setTimeout function runs asynchronously.
Some built-in Javascript functions are asynchronous, and they will run, and at some time, they will call the function you passed as parameter back. That's why if you do:
var a = 'text';
setTimeout(function() { a = 'new text'; }, 0);
console.log(a);
You'll see in your console text, because it will run before the variable has been changed.
I am currently going through the nodeschool.io tutorial on Javascript. This is a sample code from one of the solutions.
module.exports = function (dir, filterStr, callback) {
fs.readdir(dir, function (err, list) {
if (err)
return callback(err)
list = list.filter(function (file) {
return path.extname(file) === '.' + filterStr
})
callback(null, list)
})
}
From what I understand, javascript is a synchronous language. However, node.js uses asynchronous functions. In this code, isn't it possible for the callback to happen before the if statement. Therefore, is there even any reason to have functions that are different from the format
function(params, callback){
// one line data manipulation
callback();
}
node is single threaded, and event based. the i/o is what is done asynchronously. when you call fs.readdir(), you're passing it a function object as an argument. think of it as a function pointer in c++. fs.readdir() is going to kick off an asynchronous operation and pass that function object to an event handler that will trigger once the i/o operation has completed.
Now, it doesn't wait for the i/o operation to finish, it just continues to execute code until it completes and then it goes back to listening for events.
When it receives the event triggered when the i/o operation completes, it will run the code in the function object that it was passed.
Because of its single threaded nature, you don't have to worry about those types of race conditions.
From what I understand, javascript is a synchronous language
I think you are misunderstanding something here. Javascript is a single-threaded language. But that does not make it synchronous. Nearly every environment in which in runs, including the current biggest ones, browsers and Node.js, provide global event queues for asynchronous processing.
If I understand your question right, I'm also guessing you're confusing levels here. The overall module that is exposed here is providing its own asynchronous behavior by accepting a callback that will only be called asynchronously. But this is built atop another asynchronous block, the call to fs.readdir. The callback to that one is the anonymous function that makes up the majority of your code sample. This callback contains the if-statement. The other one, passed in from the outside, and named callback, will happen in the same order it appears inside that inner function, that is, after the if-statement.
A proper async function will usually call the callback after a timeout expires or even an event is fired or a message is recieved, meaning that it won't keep the thread busy while waiting, leaving behind a promise. In your example the script will continue after you assign the callback, and will call it when the async function's work is done, which will most likely be after the rest of your code is done.
Is node's javascript environment single threaded, or does everything happen at the same time? Or (more likely) do neither of those statements explain what's going on with node.
I'm new to node and trying to understand how it processes callbacks. My googling on the subject hasn't proven fruitful, and it seems like there's multiple audiences using terms like "threads, locking, and single threaded" with difference contexts for each audience, and I don't have enough experience with node to correctly parse what I'm reading.
From what I've read, node's javascript execution environment is, like a browser's, single threaded. That is, despite everything being designed around asynchronous callbacks, everything happens in a deterministic order, and there's never two threads modifying the same variable or running statements at the same time. I've also read this means a node programmer-user doesn't have to worry about locking semantics.
If I'm in browser-land and use one of the popular javascript libraries to setup a non-dom-event-handler callback, something like
console.log("Here");
$.each([1,2,3],function(){
console.log("-- inside the callback --");
});
console.log("There");
My output is consistently
Here
-- inside the callback --
-- inside the callback --
-- inside the callback --
There
However, if I do something like this with a callback in node js (running it as a shell script from the command line)
var function example()
{
var fs = require('fs');
console.log("Here");
fs.readdir('/path/to/folder', function(err_read, files){
console.log('-- inside the callback --');
});
console.log("There");
for(var i=0;i<10000;i++)
{
console.log('.');
}
}
example();
console.log("Reached Top");
I consistently (seemingly — see "not much experience" above) get results like this
Here
There
.
. (repeat 10,000 times)
Reached Top
-- inside the callback --
That is, node finishes executing the function example before calling the callback.
Is this deterministic behavior in node? Or are there times where the callback will be called before the example function finishes? Or will it depend on the implementation in the library using the callback?
I understand the idea behind node is to write event based code — but I'm trying to understand what node is really doing, and what behavior can be relied on and what can't.
Is node's javascript environment single threaded, or does everything happen at the same time?
Node has single threaded program execution model, meaning that only one instruction will be executed at any time within one node process. The execution will continue until the program yields the control. This can happen at the end of the program code, or when callback reaches it's end.
In the first case:
console.log("Here");
$.each([1,2,3],function(){
console.log("-- inside the callback --");
});
console.log("There");
they key is the fact that $.each uses callbacks synchronously, so effectively it calls it's callbacks in deterministic order.
Now in the second case, fs.readdir uses callbacks asynchronously - it puts the callback waiting for the event to be triggered (that is, when reading the directory finishes). That can happen at any time. The calling function, however, doesn't yield the control so example will always finish before any of the callbacks is called.
In one sentence:
All the code of the function/global scope is executed before any callbacks defined in that function/global scope.
In this case, you are calling a function that is doing IO and is asynchronous. That is why you are seeing output the way you are.
Because the rest of your code is executing inline - on the single execution thread, it is completed before the IO events are given a chance to make their way onto the event loop.
I would say expect this behavior, but don't depend on it with absolute certainty as it does depend on the implementation of the library using the callback. An (bad, evil) implementor could be making an synchronous request to see if there is any work to be done and, if not, callback immediately. (Hopefully you'll never see this, but…).
I would like to add a bit to #dc5's answer. On it's website they describe node as
Node.js uses an event-driven, non-blocking I/O model
The event driven part is very significant. It usually clears my doubts when I am having trouble understanding the execution model of the node programs.
So taking your example what is happening is :
Here is printed to the console.
An async call is made to the files.
Node attaches a listener to the async call.
Continues down the execution line and There is printed.
Now it could be that the async call is completed before the current piece of code that is being executed but node js completes the task at hand and then returns to service the async call.
So instead of waiting for anything it keeps on executing in line. Which is why node js execution loop is never idle.
I am trying to grasp on Javascript Asynchronous functions and callbacks.
I got stuck on the concept of callback functions, where I am reading on some places: they are use to have sequential execution of code (mostly in context of jquery e.g animate)and some places specially in the context of Nodejs; they are use to have a parallel execution Asynchronous and avoid blocking of code.
So can some expert in this topic please shed light on this and clear this fuzz in my mind (examples??).
so I could make my mind for the usage of callback function
or that is solely depends on the place of where you are calling/placing a callback function in your code? .
Thanks,
P.S: I am scared that this question would be close as subjective but still I could expect concrete answer for this (perhaps some examples)
Edit: actually this is the example from internet which makes me ambigous:
function do_a(){
// simulate a time consuming function
setTimeout( function(){
console.log( '`do_a`: this takes longer than `do_b`' );
}, 1000 );
}
function do_b(){
console.log( '`do_b`: this is supposed to come out after `do_a` but it comes out before `do_a`' );
}
do_a();
do_b();
Result
`do_b`: this is supposed to come out after `do_a` but it comes out before `do_a`
`do_a`: this takes longer than `do_b`
when JS is sequential then do_b should always come after do_a according to my understanding.
The core of JavaScript is largely synchronous, in that functions complete their task fully, before completing. Prior to the advent of AJAX, it was really only setTimeout and setInterval that provided asynchronous behavior.
However, it's easy to forget that event handlers are, effectively async code. Attaching a handler does not invoke the handler code and that code isn't executed until some unknowable time in the future.
Then came AJAX, with its calls to the server. These calls could be configured to be synchronous, but developers generally preferred async calls and used callback methods to implement them.
Then, we saw the proliferation of JS libraries and toolkits. These strove to homogenize different browsers' implementations of things and built on the callback approach to async code. You also started to see a lot more synchronous callbacks for things like array iteration or CSS query result handling.
Now, we are seeing Deferreds and Promises in the mix. These are objects that represent the value of a long running operation and provide an API for handling that value when it arrives.
NodeJS leans towards an async approach to many things; that much is true. However this is more a design decision on their part, rather than any inherent async nature of JS.
Javascript is always a synchronous(blocking) single thread language but we can make Javascript act Asynchronous through programming.
Synchronous code:
console.log('a');
console.log('b');
Asynchronous code:
console.log('a');
setTimeout(function() {
console.log('b');
}, 1000);
setTimeout(function() {
console.log('c');
}, 1000);
setTimeout(function() {
console.log('d');
}, 1000);
console.log('e');
This outputs: a e b c d
In node long running processes use process.nextTick() to queue up the functions/callbacks. This is usually done in the API of node and unless your programming(outside the api) with something that is blocking or code that is long running then it doesn't really effect you much. The link below should explain it better then I can.
howtonode process.nextTick()
jQuery AJAX also takes callbacks and such as it its coded not to wait for server responses before moving on to the next block of code. It just rememebers the function to run when the server responds. This is based on XMLHTTPRequest object that the browsers expose. The XHR object will remember the function to call back when the response returns.
setTimeout(fn, 0) of javascript will run a function once the call stack is empty (next available free tick) which can be used to create async like features.setTimeout(fn, 0) question on stackoverflow
To summerise the async abilities of javascript is as much to do with the environments they are programmed in as javascript itself. You do not gain any magic by just using lots of function calls and callbacks unless your using some API/script.
Jquery Deferred Object Is another good link for async capabilities of jQuery. Googling might find you information on how jQuery Deferred works also for more insight.
In JavaScript the term "asynchronous" typically refers to code that gets executed when the call stack is empty and the engine picks a job from one of its job queues for execution.
Once code is being executed, it represents a synchronous sequence of execution, which continues until the call stack is empty again. This sequence of execution will not be interrupted by events in order to execute some other JavaScript code (when we discard Web Workers). In other words, a single JavaScript environment has no preemptive concurrency.
While synchronous execution is ongoing, events might be registered as jobs in some job queues, but the engine will not process those before first properly execution what is on the call stack. Only when the call stack is empty will the engine take notice of the job queues, pick one according to priority, and execute it (and that is called asynchronous).
Callbacks
Callbacks can be synchronous or asynchronous -- this really depends on how they are called.
For instance, here the callback is executed synchronously:
new Promise(function (resolve) { /* .... */ });
And here the callback is executed asynchronously:
setTimeout(function () { /* ... */ });
It really depends on the function that takes the callback as argument; how it deals with eventually calling that callback.
Ways to get code to execute asynchronously
The core ECMAScript language does not offer a lot of ways to do this. The well-known ones are offered via other APIs, such as the Web API, which are not part of the core language (setTimeout, setInterval, requestAnimationFrame, fetch, queueMicrotask, addEventListener, ...).
Core ECMAScript offers Promise.prototype.then and (depending on that) await. The callback passed to then is guaranteed to execute asynchronously. await will ensure that the next statements in the same function will be executed asynchronously: await makes the current function return, and this function's execution context will be restored and resumed by a job.
It also offers listening to when the garbage collector is about to garbage collect an object, with FinalizationRegistry.
Web Workers
Web Workers will execute in a separate execution environment, with their own call stack. Here preemptive concurrency is possible. When the term asynchronous is used in the JavaScript world, it typically does not refer to this kind of parallelism, although the communication with a Web Worker happens via asynchronous callback functions.
Javascript by default is "synchronous", it's the web APIs that handle "asynchronous" behaviour.
As for the setTimeout example,
console.log(...), in the global scope works straight away, while those inside functions wrapped inside setTimeout, wait inside the callback queue only to be pushed back on the call stack once ready. Thus they take time. Also, the time specified is not exact but the minimum time after which that piece of code can run anytime.
Thanks !