Develop Reference

Is it ever better to use Node's filesystem sync methods over the same async methods?

This is a question about performance more than anything else.
Node exposes three different types of methods to accomplish various filesystem tasks:
Promises API (async)
Callback API (async)
Synchronous API (sync)
I've read more articles and stackoverflow answers than I can count, all of which claiming to never need the sync methods.
I recently wrote a script which required a couple directories to be made if they didn't already exist. During this, I noticed that if I used the async/await methods (primarily fs.promises.mkdir and fs.promises.access), the event loop would simply continue to the next async bit of code, regardless of the fact that the next bits require those directories. This is expected behavior, after all, it's async.
I understand this could be solved with a nice little callback hell sesh, but that isn't the question, whereas the idea that the promises api can be used over all other methods is.
The question then becomes:
Is it ever better to use Node's filesystem sync methods over the same async methods?
Is it ever truly required in situations like this to block the process?
Or said differently:
Is it possible to completely avoid sync methods and ONLY use the promises api (NOT promises + callbacks)?
It seems like using the sync methods (given my situation above, where the directories are required to be there before any other call is made) can be EXTREMELY useful to write readable, clear code, even though it may negatively impact performance.
With that being said, there's an overwhelming level of information to say that the sync api is completely useless and never required.
Again, this purely caters to the promises api. Yes, callbacks and promises are both async, but the difference between the job and message queues makes the both api's completely different in this context.
PS: For additonal context on examples, I've provided a code sample so you don't have to imagine my example ;)
Thanks! :)
// Checks if dir exists, if not, creates it. (not the actual code, just an example)
// Sync version
if (!fs.existsSync(dirPath)) {
fs.mkdirSync(dirPath);
}
// Async version
try {
await fs.promises.access(dirPath);
} catch {
await fs.promises.mkdir(dirPath);
}

It depends on the situation. The main benefit of the sync methods is that they allow for easier consumption of their results, and the main disadvantage is that they prevent all other code from executing while working.
If you find yourself in a situation where other code not being able to respond to events is not an issue, you might consider it to be reasonable to use the sync methods - if the code in question has no chance of or reason for running in parallel with anything else.
For example, you would definitely not want to use the sync methods inside, say, a server handling a request.
If your code requires reading some configuration files (or creating some folders) when the script first runs, and there aren't enough of them such that parallelism would be a benefit, you can consider using the sync methods.
That said, even if your current implementation doesn't require parallelism, something to keep in mind is that, if the situation changes and you find that you do actually need to allow for parallel processing, you won't have to make any changes to your existing code if you had started out by using the promise-based methods in the first place - and if you understand the language, using the Promises properly should be pretty easy, so if there's a chance of that, you might consider using the Promises anyway.

JavaScript is synchronous then why we need callback function

since javascript is synchronous then why do we need callback function,
in the below picture both code can same thing

As #Meno-101 rightly said, you're looking at the trivial function invocation utility of callbacks, they're rather used for asynchronous calls. In asynchronous calls we wait for some data to arrive and then invoke a function, you won't see a difference when you're being presented with the result in an instant. setTimeout is one easy way to replicate such a situation;
Callbacks are generally used when the function needs to perform events before the callback is executed, or when the function does not (or cannot) have meaningful return values to act on, as is the case for Asynchronous JavaScript (based on timers) or XMLHttpRequest requests. Checkout the deets here
But if we're are bent on using it synchronously , I could only come up with this code:
This made the code reusable which isn't possible the other way, we'd have to write different definitions of same function for disp() and disp1() to function.

You’re just confusing callback’s purpose with a function invocation purpose.
Callbacks are used in the case of web requests, reading/writing to files, or maybe even just waiting for some reason the programmer only know why, take this example:
setTimeout(()=> console.log(“hello”), 3000)
There must be a reason for you to have this in your code, what ever it’s you don’t want all the page to hang up on the user.
And, I do recommend you look for function call stack if you’re just starting off.

Continuation-Passing Style And Concurrency

I find lots of blogs mention concurrent/non-blocking/asynchronous programming as a benefit of Continuation-Passing Style (CPS). I cannot figure out why CPS provides concurrency, e.g., people mention Node.js is implemented using CPS though JavaScript is a synchronous language. Would someone comment on my thoughts?
First, my naive understanding of CPS is that wrapping all subsequent code at a point into a function and pass that function explicitly as a parameter. Some blogs name the continuation function as return(), Gabriel Gonzalez calls it a hole, both of which are brilliant explanations.
My confusion mostly comes from a popular blog article Asynchronous programming and continuation-passing style in JavaScript. At the beginning of the article, Dr. Axel Rauschmayer gives two code snippets, a synchronous program and an asynchronous one in CPS (pasted here for easy reading).
The synchronous code:
function loadAvatarImage(id) {
var profile = loadProfile(id);
return loadImage(profile.avatarUrl);
}
The asynchronous code:
function loadAvatarImage(id, callback) {
loadProfile(id, function (profile) {
loadImage(profile.avatarUrl, callback);
});
}
I don't get it why the CPS one is asynchronous. After I read another article By example: Continuation-passing style in JavaScript, I think maybe there is an assumption to the code: the function loadProfile() and loadImage() are asynchronous functions by themselves. Then it is not CPS that makes it asynchronous. In the second article, the author actually shows an implementation of fetch(), which is similar to loadProfile() in the blog earlier. The fetch() function makes an explicit assumption of the underlying concurrent execution model by calling req.onreadystatechange. This leads me to think maybe it is not CPS that provides concurrency.
Assume the underlying functions are asynchronous, then I go into my second question: can we write asynchronous code without CPS? Think of the implementation of the function loadProfile(). If it is asynchronous not because of CPS, why can't we just take the same mechanism to implement loadAvatarImage() asynchronously? Assume loadProfile() uses fork() to create a new thread to send the request and wait for the response while the main thread is executing in a non-blocking manner, we can possibly do the same for loadAvatarImage().
function loadAvatarImage(id, updateDOM) {
function act () {
var profile = loadProfile(id);
var img = loadImage(profile.avatarUrl);
updateDOM (img);
}
fork (act());
}
I give it a callback function updateDOM(). Without updateDOM(), it is not fair to compare it with the CPS version -- the CPS version has extra information about what to do after the image is fetched, i.e., the callback function, but the original synchronous loadAvatarImage() does not.
Interestingly, #DarthFennec pointed out my new loadAvatarImage() is actually CPS: fork() is CPS, act() is CPS (if we explicitly give it updateDOM), and loadAvatarImage() is CPS. The chain makes loadAvatarImage() asynchronous. loadProfile() and loadImage() do not need to be asynchronous or CPS.
If the reasoning up to here is correct, can I get these two conclusions?
Given a set of synchronous APIs, someone coding following CPS will not magically create asynchronous functions.
If the underlying asynchronous/concurrent APIs are provided in the CPS style, like CPS versions of loadProfile(), loadImage(), fetch(), or fork(), then one can only code in CPS style to ensure the asynchronous APIs are used asynchronously, e.g., return loadImage(profile.avatarUrl) will nullify the concurrency of loadImage().

A Brief Overview of Javascript
Javascript's concurrency model is non-parallel and cooperative:
Javascript is non-parallel because it runs in a single thread; it achieves concurrency by interleaving multiple execution threads, rather than by actually running them at the same time.
Javascript is cooperative because the scheduler only switches to a different thread when the current thread asks it to. The alternative would be preemptive scheduling, where the scheduler decides to arbitrarily switch threads whenever it feels like it.
By doing these two things, Javascript avoids many problems that other languages don't. Parallel code, and non-parallel preemptively-scheduled code, cannot make the basic assumption that variables will not suddenly change their values in the middle of execution, since another thread might be working on the same variable at the same time, or the scheduler might decide to interleave another thread absolutely anywhere. This leads to mutual exclusion problems and confusing race condition bugs. Javascript avoids all of this because in a cooperatively-scheduled system, the programmer decides where all the interleaves happen. The main drawback of this is if the programmer decides not to create interleaves for long periods of time, other threads never have a chance to run. In a browser, even actions like polling for user input and drawing updates to the page run in the same single-threaded environment as the Javascript, so a long-running Javascript thread will cause the entire page to become unresponsive.
In the beginning, CPS was most often used in Javascript for the purpose of event-driven UI programming: if you wanted some code to run every time someone pressed a button, you would register your callback function to the button's 'click' event; when the button was clicked, the callback would run. As it turns out, this same approach could be used for other purposes as well. Say you want to wait one minute and then do a thing. The naive approach would be to stall the Javascript thread for sixty seconds, which (as stated above) would cause the page to crash for that duration. However, if the timer was exposed as a UI event, the thread could be suspended by the scheduler instead, allowing other threads to run in the meantime. Then, the timer would cause the callback to execute, in the same way a button press would. The same approach can be used to request a resource from the server, or to wait for the page to load fully, or a number of other things. The idea is that, to keep Javascript as responsive as possible, any built-in function that might take a long time to complete should be part of the event system; in other words, it should use CPS to enable concurrency.
Most languages that support cooperative scheduling (often in the form of coroutines) have special keywords and syntax that must be used to tell the language to interleave. For example, Python has the yield keyword, C# has async and await, etc. When Javascript was first designed, it had no such syntax. It did, however, have support for closures, which is a really easy way to allow CPS. I expect the intention behind this was to support the event-driven UI system, and that it was never intended to become a general-purpose concurrency model (especially once Node.js came along and removed the UI aspect entirely). I don't know for sure, though.
Why does CPS provide concurrency?
To be clear, continuation-passing style is a method that can be used to enable concurrency. Not all CPS code is concurrent. CPS isn't the only way to create concurrent code. CPS is useful for things other than enabling concurrency. Simply put, CPS does not necessarily imply concurrency, and vice versa.
In order to interleave threads, execution must be interrupted in such a way that it can be resumed later. This means the context of the thread must be preserved, and later re-instated. This context isn't generally accessible from inside of a program. Because of this, the only way to support concurrency (short of the language having special syntax for it) is to write the code in such a way that the thread context is encoded as a value. This is what CPS does: the context to be resumed is encoded as a function that can be called. This function being called is equivalent to a thread being resumed. This can happen any time: after an image is loaded, after a timer triggers, after other threads have had a chance to run for a while, or even immediately. Since the context is all encoded into the continuation closure, it doesn't matter, as long as it runs eventually.
To better understand this, we can write a simple scheduler:
var _threadqueue = []
function fork(cb) {
_threadqueue.push(cb)
}
function run(t) {
_threadqueue.push(t)
while (_threadqueue.length > 0) {
var next = _threadqueue.shift()
next()
}
}
An example of this in use:
run(function() {
fork(function() {
console.print("thread 1, first line")
fork(function() {
console.print("thread 1, second line")
})
})
fork(function() {
console.print("thread 2, first line")
fork(function() {
console.print("thread 2, second line")
})
})
})
This should print the following to the console:
thread 1, first line
thread 2, first line
thread 1, second line
thread 2, second line
The results are interleaved. While not particularly useful on its own, this logic is more or less the foundation of something like Javascript's concurrency system.
Can we write asynchronous code without CPS?
Only if you have access to the context through some other means. As previously stated, many languages do this through special keywords or other syntax. Some languages have special builtins: Scheme has the call/cc builtin, which will wrap the current context into a callable function-like object, and pass that object to its argument. Operating systems get concurrency by literally copying around the thread's callstack (the callstack contains all of the needed context to resume the thread).
If you mean in Javascript specifically, then I'm fairly certain it's impossible to reasonably write asynchronous code without CPS. Or it would be, but newer versions of Javascript also come with the async and await keywords, as well as a yield keyword, so using those is becoming an option.
Conclusion: Given a set of synchronous APIs, someone coding following CPS will not magically create asynchronous functions.
Correct. If an API is synchronous, CPS alone will not make that API asynchronous. It may introduce a level of concurrency (as in the example code earlier), but that concurrency can only exist within the thread. Asynchronous loading in Javascript works because the loading itself runs in parallel to the scheduler, so the only way to make a synchronous API asynchronous is to run it in a separate system thread (which can't be done in Javascript). But even if you did do that, it still wouldn't be asynchronous unless you also used CPS.
CPS doesn't cause asynchronicity. However, asynchronicity does require CPS, or some alternative to CPS.
Conclusion: If the underlying asynchronous/concurrent APIs are provided in the CPS style, then one can only code in CPS style
Correct. If the API is loadImage(url, callback) and you run return loadImage(profile.avatarUrl), it will return null immediately and it will not give you the image. Most likely it will throw an error because callback is undefined, since you didn't pass it. Essentially, if the API is CPS and you decide not to use CPS, you're not using the API correctly.
In general though, it is accurate to say that if you write a function that calls a CPS function, your function also needs to be CPS. This is actually a good thing. Remember what I said about the basic assumption that variables will not suddenly change their values in the middle of execution? CPS solves this issue by making it very clear to the programmer where exactly the interleave boundaries are; or rather, where values might arbitrarily change. But if you could hide CPS function calls inside of non-CPS functions, you would no longer be able to tell. This is also the reason the newer Javascript async and await keywords work the way they do: any function that uses await must be marked as async, and any call to an async function must be prefixed with the await keyword (there's more to it than that, but I don't want to get into how promises work just now). Because of this, you can always tell where your interleave boundaries are, because there will always be await keywords there.

How to think when working with promises

I'm working for the first time with async programming, specifically with promises in Javascript and I'm really confused with it. I'm still thinking like we do in C, PHP, C# and so on where for each step the next one can be sure of it's completion. This is easy to work, because if we have a code like that (in C# for example)
IEnumerable<Page> pages = pagesRepository.getAll();
we can be sure in the next line that all pages are available for us to use in the pages object. This makes life easier, because we can already do work with that data properly.
Now, in JS with promises we would have something like
var pages = dataService.getPages();
and this would not return the data, but just a promise which is very different. Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
So how do we think when we deal with those things? Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?

Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
You don't need to.
The "somewhere else code" will use .then() and yield a new promise for its result, relying on the pages promise. You can easily map the results, or chain other asynchronous tasks. Whenever you need to know about the timing of callbacks because of multiple dependencies, you should use your library's methods for composing promises, like all().
So how do we think when we deal with those things?
Think functional!
Every task in your program should explicitly and only depend on its inputs. A function should try to only use its arguments, not any global variables that contain state.
Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
General Promise Resources in the wiki of the Q library is a very good start.

The simple answer here is that promises are similar to Task asynchronous programming in C#.
However instead of being able to use the 'await' operator, the 'then' function on a promise is analogous to using 'ContinueWith' to chain Tasks in C#.
Callbacks can get complicated to use across scopes, especially with extensive use of nested anonymous functions, so please give promises a go

What is the use of a synchronous callback function in JavaScript?

I have been doing extensive reading on asynchronous programming for the web and use of callbacks in JavaScript and jQuery. I have understood the fundamentals of AJAX. What I cannot figure out is the use of callback functions when not used in asynchronous programming.
From my understanding, simply adding a callback to a function does not make it non-blocking/asynchronous. Asynchronous capability is actually provided by the environment (browser APIs). So adding a callback to a function that I have written will not lead to any asynchronous execution.
For example:
var X;
function Test(A, B, Callback) {
X=A+B*A*B;
Callback(X);
}
Test(99999,999999,function(Data) {
alert(Data);
});
alert("This is not printed first, as it would be in Async.");
In the above I'm taking two numbers and performing algebraic operations on them. Even though I'm using a callback function, the code execution will be blocked while the operations are performed. Then the callback will be executed, displaying an alert with the result of the computation. Then the next alert follows. Had I made an XMLHttpRequest instead of the algebraic operation, I would have got the second alert first because of asynchronous implementation. The code flow would not be blocked during the request as is happening during the mathematical operation.
Thus, what is the use of a callback in non-async calls when code execution is blocked even with a callback?

Some very common examples of synchroneous callbacks are the methods on Array.prototype: forEach, map, filter, etc.
The role of the callback is to provide a partial implementation that can be easily swapped in a larger algorithm. Some design patterns like template method and strategy come to mind.

You're right.
There is usually no reason to have a sync callback.
Exceptions include callbacks that might sometimes be async, or callbacks that can be raised more than once (eg, [].map()).

I use callbacks in the way you describe whenever I am working with complicated access methods which I don't want to rewrite, and I use them when I know the component will be run in a context the component doesn't know about, but am not particularly interested in developing or using an event emitter.
I do not know if other people find them as easy to work with as I do. So on a team, I try to get a sense of whether it is productive to use callbacks in the way you describe.