Develop Reference

Is it ever better to use Node's filesystem sync methods over the same async methods?

This is a question about performance more than anything else.
Node exposes three different types of methods to accomplish various filesystem tasks:
Promises API (async)
Callback API (async)
Synchronous API (sync)
I've read more articles and stackoverflow answers than I can count, all of which claiming to never need the sync methods.
I recently wrote a script which required a couple directories to be made if they didn't already exist. During this, I noticed that if I used the async/await methods (primarily fs.promises.mkdir and fs.promises.access), the event loop would simply continue to the next async bit of code, regardless of the fact that the next bits require those directories. This is expected behavior, after all, it's async.
I understand this could be solved with a nice little callback hell sesh, but that isn't the question, whereas the idea that the promises api can be used over all other methods is.
The question then becomes:
Is it ever better to use Node's filesystem sync methods over the same async methods?
Is it ever truly required in situations like this to block the process?
Or said differently:
Is it possible to completely avoid sync methods and ONLY use the promises api (NOT promises + callbacks)?
It seems like using the sync methods (given my situation above, where the directories are required to be there before any other call is made) can be EXTREMELY useful to write readable, clear code, even though it may negatively impact performance.
With that being said, there's an overwhelming level of information to say that the sync api is completely useless and never required.
Again, this purely caters to the promises api. Yes, callbacks and promises are both async, but the difference between the job and message queues makes the both api's completely different in this context.
PS: For additonal context on examples, I've provided a code sample so you don't have to imagine my example ;)
Thanks! :)
// Checks if dir exists, if not, creates it. (not the actual code, just an example)
// Sync version
if (!fs.existsSync(dirPath)) {
fs.mkdirSync(dirPath);
}
// Async version
try {
await fs.promises.access(dirPath);
} catch {
await fs.promises.mkdir(dirPath);
}

It depends on the situation. The main benefit of the sync methods is that they allow for easier consumption of their results, and the main disadvantage is that they prevent all other code from executing while working.
If you find yourself in a situation where other code not being able to respond to events is not an issue, you might consider it to be reasonable to use the sync methods - if the code in question has no chance of or reason for running in parallel with anything else.
For example, you would definitely not want to use the sync methods inside, say, a server handling a request.
If your code requires reading some configuration files (or creating some folders) when the script first runs, and there aren't enough of them such that parallelism would be a benefit, you can consider using the sync methods.
That said, even if your current implementation doesn't require parallelism, something to keep in mind is that, if the situation changes and you find that you do actually need to allow for parallel processing, you won't have to make any changes to your existing code if you had started out by using the promise-based methods in the first place - and if you understand the language, using the Promises properly should be pretty easy, so if there's a chance of that, you might consider using the Promises anyway.

Using async result (array.filter) [duplicate]

I'm curious as to whether all javascript callbacks are asynchronous, or whether that is the case only in certain situations. Also, I'm sure what makes javascript code asynchronous (or ways to use asynchronous javascript) differ between the browser and nodejs, so I'd like to know in each situation what constitutes real asynchronous javascript.
I'm under the impression that in the following scenarion, I'm not actually writing asynchronous code.
function addOne(value){
value = value + 1;
return value;
}
function simpleMap(values, callback){
for(i = 0; i < values.length; i++){
val = values[i];
val = callback(val);
values[i] = val;
}
return values;
}
newValues = simpleMap([1,2,3], addOne);
However, for example, I know that jQuery's AJAX functions are truly asynchronous (not taking in to account the promises which are now available). What is it that make jQuery's AJAX asynchronous? Is it as simple that it involves XHR requests, and in the browser, all XHR requests are asynchronous?
I have the same question for the nodejs environment. Can something in node only be asynchronous if it involves something like file i/o, process.nextTick, setTimeout, or setInterval? Why when I do something like a database call with mongodb/mongoose, is that asynchronous? What's going on behind the scenes that's making it so?
Are asynchronous "situations" predetermined by the environment? Or is there some way to make one's own function truly asynchronous without leveraging very specific functions of the environment (such as xhr, file io in node, process.nexttick, etc)?

I'm curious as to whether all javascript callbacks are asynchronous
No. For instance, the callback used by Array#sort is not asynchronous, nor is the one used by String#replace.
The only way you know whether a callback is asynchronous is from its documentation. Typically, ones involving requests for external resources (ajax calls, for instance) are asynchronous, and others may or may not be.
However, for example, I know that jQuery's AJAX functions are truly asynchronous...
Not necessarily, as currently jQuery still has the async flag which you can set false to force a synchronous request. (It's not a good idea, and they're going to remove that, but you can. jQuery passes the flag to the underlying browser object which provides the synchronous/asynchronous behavior.)
What is it that make jQuery's AJAX asynchronous?
The browser. jQuery's ajax calls use the XMLHttpRequest object (or in certain situations, a script element), which defaults to asynchronous operation provided by the browser.
Or is there some way to make one's own function truly asynchronous without leveraging very specific functions of the environment...
Until recently, no. Up through the 5th edition specification, JavaScript the language was basically silent on the entire concept of threads and asynchronicity; it was only when you got into environments that it came up. The only way to make something asynchronous was to use a host-provided function, such as nextTick (or any of the various operations that completes asynchronously) on NodeJS or setTimeout on browsers.
In the ECMAScript 6th edition specification in June 2015, they introduced promises into the language. The callbacks hooked up to an ES6 promise via then and such are always invoked asynchronously (even if the promise is already settled when the callback is attached), and so JavaScript has asynchronicity at a language level now. So if you implement your function so that it returns a promise rather than accepting a callback, you'll know that the then callbacks hooked up to it will be triggered asynchronously.

Callbacks that you call yourself are regular function calls, which are always synchronous.
Certain native APIs (eg, AJAX, geolocation, Node.js disk or network APIs) are asynchronous and will execute their callbacks later in the event loop.
If you call a callback synchronously from within an async callback, it will end up being async too.

To create your own asynchronous functions you have to make use of other asynchronous functions which may be provided by the interpreter.
This code for example defines a function "addKeyHandler" which is asynchronous. But that only works because document.onKey is called asynchronously by the JS engine. The JavaScript engine is able to provide asynchronous functionality because the operating system provides such functionality which is then used by JS. The OS in turn can only provide async functionality because the hardware provides it (called hardware interrupts).
However if the OS and hardware didn't provide any async functions it would still be possible to write a JS interpreter. But it would have to use an infinite loop and check in each iteration if any events occured and then invoke the appropriate callbacks. That would mean the CPU would always be under full load.
var keyCallbacks = [];
var addKeyHandler = function(f) {
keyCallbacks.push(f);
};
document.onkeypress = function(e) {
keyCallbacks.forEach(function(f) {
f(e);
});
};
addKeyHandler(function(e) {
console.log(String.fromCharCode(e.charCode));
});

Simply taking a callback doesn't make a function asynchronous. There are many examples of functions that take a function argument but are not asynchronous, for example, Array's forEach.
For a function to be asynchronous it needs to perform an asynchronous operation. Ways of introducing asynchronicity can be
timer functions setTimeout, setInterval
special functions nextTick, setImmediate
performing I/O (listening to network, querying a database, reading or writing from a resource)
subscribing to an event
According to article "Does taking a callback make a function asynchronous?"

Going for a simple answer:
Unless you're dealing with promises, JS callbacks are only asynchronous if they rely on an API external to JS (such as provided by the browser).
setTimeout, fetch, and so on are asynchronous because they rely on external api's. (setTimeout is part of the windowOrGlobalWorker web api, for example, and fetch is a web api in itself.)
If a callback does not rely on an external API, it's synchronous.
Promises are the only native async functionality in JS.
A great way to get your head around all this is read at least the first few articles in the MDN primer on async: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous

How to think when working with promises

I'm working for the first time with async programming, specifically with promises in Javascript and I'm really confused with it. I'm still thinking like we do in C, PHP, C# and so on where for each step the next one can be sure of it's completion. This is easy to work, because if we have a code like that (in C# for example)
IEnumerable<Page> pages = pagesRepository.getAll();
we can be sure in the next line that all pages are available for us to use in the pages object. This makes life easier, because we can already do work with that data properly.
Now, in JS with promises we would have something like
var pages = dataService.getPages();
and this would not return the data, but just a promise which is very different. Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
So how do we think when we deal with those things? Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?

Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
You don't need to.
The "somewhere else code" will use .then() and yield a new promise for its result, relying on the pages promise. You can easily map the results, or chain other asynchronous tasks. Whenever you need to know about the timing of callbacks because of multiple dependencies, you should use your library's methods for composing promises, like all().
So how do we think when we deal with those things?
Think functional!
Every task in your program should explicitly and only depend on its inputs. A function should try to only use its arguments, not any global variables that contain state.
Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
General Promise Resources in the wiki of the Q library is a very good start.

The simple answer here is that promises are similar to Task asynchronous programming in C#.
However instead of being able to use the 'await' operator, the 'then' function on a promise is analogous to using 'ContinueWith' to chain Tasks in C#.
Callbacks can get complicated to use across scopes, especially with extensive use of nested anonymous functions, so please give promises a go

What is the use of a synchronous callback function in JavaScript?

I have been doing extensive reading on asynchronous programming for the web and use of callbacks in JavaScript and jQuery. I have understood the fundamentals of AJAX. What I cannot figure out is the use of callback functions when not used in asynchronous programming.
From my understanding, simply adding a callback to a function does not make it non-blocking/asynchronous. Asynchronous capability is actually provided by the environment (browser APIs). So adding a callback to a function that I have written will not lead to any asynchronous execution.
For example:
var X;
function Test(A, B, Callback) {
X=A+B*A*B;
Callback(X);
}
Test(99999,999999,function(Data) {
alert(Data);
});
alert("This is not printed first, as it would be in Async.");
In the above I'm taking two numbers and performing algebraic operations on them. Even though I'm using a callback function, the code execution will be blocked while the operations are performed. Then the callback will be executed, displaying an alert with the result of the computation. Then the next alert follows. Had I made an XMLHttpRequest instead of the algebraic operation, I would have got the second alert first because of asynchronous implementation. The code flow would not be blocked during the request as is happening during the mathematical operation.
Thus, what is the use of a callback in non-async calls when code execution is blocked even with a callback?

Some very common examples of synchroneous callbacks are the methods on Array.prototype: forEach, map, filter, etc.
The role of the callback is to provide a partial implementation that can be easily swapped in a larger algorithm. Some design patterns like template method and strategy come to mind.

You're right.
There is usually no reason to have a sync callback.
Exceptions include callbacks that might sometimes be async, or callbacks that can be raised more than once (eg, [].map()).

I use callbacks in the way you describe whenever I am working with complicated access methods which I don't want to rewrite, and I use them when I know the component will be run in a context the component doesn't know about, but am not particularly interested in developing or using an event emitter.
I do not know if other people find them as easy to work with as I do. So on a team, I try to get a sense of whether it is productive to use callbacks in the way you describe.

Javascript callback hell with setTimeout();

Aloha. This is my first question here after eons of lurking. I apologize for the verbose but direct form of this question -- as well as the limitations on acceptable answers -- but I am here to contribute. I have a problem that other questions (for example How to avoid long nesting of asynchronous functions in Node.js or Chained callback readability in Javascript?) do not answer to my satisfaction.
I want to perform several Javascript callbacks so that I have a sequence of four actions occurring one after another. Here is the source:
var self = this;
fade(div, fadeIn, -20, function() {
load(unload, dataOut, -10, function() {
load(self, dataIn, 10, function() {
fade(div, fadeOut, 20),0);
unload = self;}
);}
);}
);
The load() and fade() functions are very similar except for internal semantic details, and they look something very similar to this:
function load(menuCategory, data, step, callback) {
menuCategory.parentNode.style.width = data+"px";
data += step;
if(dataIn <= data && data <= dataOut) {
setTimeout(function() { load(menuCategory,data,step,callback) }, 15);
} else if(callback && (typeof(callback) == 'function')) {
callback();}
}
I have several questions associated with this mess, but first let me give you the answers I am NOT looking for. Answers using jQuery are not acceptable. Answers using other timing or "synchronicity" frameworks or modules are not acceptable. Answers questioning the reason for blocking execution are not acceptable. Answers using languages other than Javascript are unacceptable. Non-abstract answers are acceptable, but the final solution will need to be as abstract as possible, meaning the answer must be able to act on a variety of similar but slightly different actions.
Here is my primary question:
Is there a function/object I could use to apply these actions sequentially, to cut down on the callback hell going on here? In other words, what would an object look like if the object/function were able to synchronously execute each action as it was individually iterated through it (i.e. passed to it)?
Secondary question:
Is anyone aware of how much this callback business looks like goto() in other languages? Meaning, this nesting of callbacks feels very awkward and inelegant to me. Is there a Javascript language construct either being developed or already developed that will reduce this crazy business? Callback nesting is some of the hardest logic flow to follow. It just feels awkward.
My goal here is a tight, elegant, attractive, homegrown Javascript object/function -- that I can call in a for() loop or equivalent -- to apply these operations on each action in sequence. If you've read this far I want to thank you for your time and consideration. :)

using setTimeout(func(),0) in order to execute synchronously
No. Either use func() for a synchronous execution, or setTimeout(func, 0) to queue the execution (which makes it asynchronous).
Is setTimeout(func[…],0) the "most elegant" way to put this action onto the execution queue for rendered webpage objects? In other words, is there a "more elegant" way to block execution (i.e. do these things sequentially)?
Yes, this is the standard method. However, this does not block execution, and has nothing to do with sequencing. If the execution is synchronous, just execute it, if not you will have to deal with the callbacks and queuing a new task does not help.
Is anyone aware of how much this callback business looks like goto() in other languages? Meaning, this nesting of callbacks feels very awkward and inelegant to me. Is there a Javascript language construct either being developed or already developed that will reduce this crazy business?
No, I'm not aware of other languages, but afaik goto is a synchronous, control-flow-structuring statement as does not deal with asynchronous actions.
But no, there are no JavaScript language constructs ("syntactic sugar") that help us around the continuation-passing style with its callback hell.
Is there a function/object I could use to apply these actions sequentially, to cut down on the callback hell going on here? In other words, what would an object look like if the object/function were able to synchronously execute each action as it was individually iterated through it (i.e. passed to it)?
My goal here is a tight, elegant, attractive, homegrown Javascript object/function -- that I can call in a for() loop or equivalent -- to apply these operations on each action in sequence.
Again you want "sequencially" instead of "synchronous" :-) Yes, there are [handcoded/homegrown] solutions which make dealing with callbacks easier. As those callback organisation libraries you mentioned do not satisfy you, I'd recommend to look into the very elegant and attractive Promise Api and its Deferred objects (see futures and promises for the concept).

I'm unsure why you're using setTimeout at all here. Its not necessary to queue callbacks. The following structure will work, assuming that fade() and load() handle callback execution correctly.
fade(div, fadeIn, -20, function() {
load(unload, dataOut, -10, function() {
load(self, dataIn, 10, function() {
fade(div, fadeOut, 20),0);
});
)};
)};
Another topic you should look into is Promises, also called deferreds and futures. Its essentially a coding pattern that specifically handles nested callbacks. At root, you create a Promise object that gets returned immediately, then when your method is done executing it fires a method on the promise that can then execute your next function. Its a different way of writing javascript, but easy to understand and very useful.

As it turns out, the practice that fits as the "best" answer to my question is events. In particular the practice I'm referring to is best described by EventEmitter on Node.js, or creating custom events and event listeners on client browsers. While the Promise API and its deference model is intriguing, it uses extensions to the language instead of core Javascript, which is what I was looking for here in particular (and thus why none of the previous answers -- despite their clarity -- support exactly what I was looking for).
This link (http://wekeroad.com/2012/04/05/cleaning-up-deep-callback-nesting-with-nodes-eventemitter) provides a very good example of what I was looking for. As well, I should mention the book "Async Javascript" by Trevor Burnham (http://www.amazon.com/Async-JavaScript-Trevor-Burnham/dp/1475247362), and "Secrets of the JavaScript Ninja" by John Resig and Bear Bibeault (http://www.amazon.com/Secrets-JavaScript-Ninja-John-Resig/dp/193398869X). These books provide extremely clear working examples of custom events and event handlers in client browsers that provide an elegant way around this "callback hell" I was referring to.