Develop Reference

Is it ever better to use Node's filesystem sync methods over the same async methods?

This is a question about performance more than anything else.
Node exposes three different types of methods to accomplish various filesystem tasks:
Promises API (async)
Callback API (async)
Synchronous API (sync)
I've read more articles and stackoverflow answers than I can count, all of which claiming to never need the sync methods.
I recently wrote a script which required a couple directories to be made if they didn't already exist. During this, I noticed that if I used the async/await methods (primarily fs.promises.mkdir and fs.promises.access), the event loop would simply continue to the next async bit of code, regardless of the fact that the next bits require those directories. This is expected behavior, after all, it's async.
I understand this could be solved with a nice little callback hell sesh, but that isn't the question, whereas the idea that the promises api can be used over all other methods is.
The question then becomes:
Is it ever better to use Node's filesystem sync methods over the same async methods?
Is it ever truly required in situations like this to block the process?
Or said differently:
Is it possible to completely avoid sync methods and ONLY use the promises api (NOT promises + callbacks)?
It seems like using the sync methods (given my situation above, where the directories are required to be there before any other call is made) can be EXTREMELY useful to write readable, clear code, even though it may negatively impact performance.
With that being said, there's an overwhelming level of information to say that the sync api is completely useless and never required.
Again, this purely caters to the promises api. Yes, callbacks and promises are both async, but the difference between the job and message queues makes the both api's completely different in this context.
PS: For additonal context on examples, I've provided a code sample so you don't have to imagine my example ;)
Thanks! :)
// Checks if dir exists, if not, creates it. (not the actual code, just an example)
// Sync version
if (!fs.existsSync(dirPath)) {
fs.mkdirSync(dirPath);
}
// Async version
try {
await fs.promises.access(dirPath);
} catch {
await fs.promises.mkdir(dirPath);
}

It depends on the situation. The main benefit of the sync methods is that they allow for easier consumption of their results, and the main disadvantage is that they prevent all other code from executing while working.
If you find yourself in a situation where other code not being able to respond to events is not an issue, you might consider it to be reasonable to use the sync methods - if the code in question has no chance of or reason for running in parallel with anything else.
For example, you would definitely not want to use the sync methods inside, say, a server handling a request.
If your code requires reading some configuration files (or creating some folders) when the script first runs, and there aren't enough of them such that parallelism would be a benefit, you can consider using the sync methods.
That said, even if your current implementation doesn't require parallelism, something to keep in mind is that, if the situation changes and you find that you do actually need to allow for parallel processing, you won't have to make any changes to your existing code if you had started out by using the promise-based methods in the first place - and if you understand the language, using the Promises properly should be pretty easy, so if there's a chance of that, you might consider using the Promises anyway.

XMLHttpRequest Promise, nested in some sync code

Why is a sync xmlhttprequest deprecated ?
I would appreciate if the browser waits for the data instead of
open it for further clicking without finalizing the data-query.
I would like to write a program with just one step after the other.
Moreover the code should be properly structured and I wonder how
I can realize this without the promise-hell.
Is there any sync alternative to xmlhttprequest which is not "deprecated" ?
Thanks
Thomas

Why is a sync xmlhttprequest deprecated ?
Because it blocks the Event Loop and provides bad user experience to your end users.
Basically it "freezes" the page while it's happening.
I would appreciate if the browser waits for the data
That is not how browsers work though. Fortunately with practice the way browsers do work start to make an awful lot of sense - enough for languages that have previously used mostly synchronous I/O like Python to add "promises" too.
I would like to write a program with just one step after the other.
You can return responses from asynchronous calls in JavaScript. Using async/await code looks pretty similar and readable. See the async function mdn page.
Moreover the code should be properly structured and I wonder how I can realize this without the promise-hell.
"promise-hell" refers to the fact once a function does I/O all other functions it calls that. I'm not convinced that's a bad thing or that there is anything wrong with
async function sequence() {
let one = await fetch('/your-endpoint?someParam').then(x => x.json());
// do something with one
let param = one.param;
let two = await fetch('/your-other?param=' + param).then(x => x.json());
// do something with second call
}
sequence();
That is, async/await lets you write asynchronous code synchronously.
Don't expect to understand this all at once - it's ok it takes time.

How to think when working with promises

I'm working for the first time with async programming, specifically with promises in Javascript and I'm really confused with it. I'm still thinking like we do in C, PHP, C# and so on where for each step the next one can be sure of it's completion. This is easy to work, because if we have a code like that (in C# for example)
IEnumerable<Page> pages = pagesRepository.getAll();
we can be sure in the next line that all pages are available for us to use in the pages object. This makes life easier, because we can already do work with that data properly.
Now, in JS with promises we would have something like
var pages = dataService.getPages();
and this would not return the data, but just a promise which is very different. Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
So how do we think when we deal with those things? Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?

Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
You don't need to.
The "somewhere else code" will use .then() and yield a new promise for its result, relying on the pages promise. You can easily map the results, or chain other asynchronous tasks. Whenever you need to know about the timing of callbacks because of multiple dependencies, you should use your library's methods for composing promises, like all().
So how do we think when we deal with those things?
Think functional!
Every task in your program should explicitly and only depend on its inputs. A function should try to only use its arguments, not any global variables that contain state.
Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
General Promise Resources in the wiki of the Q library is a very good start.

The simple answer here is that promises are similar to Task asynchronous programming in C#.
However instead of being able to use the 'await' operator, the 'then' function on a promise is analogous to using 'ContinueWith' to chain Tasks in C#.
Callbacks can get complicated to use across scopes, especially with extensive use of nested anonymous functions, so please give promises a go

Struggling with async synchronisation in node.js

So I started a little project in Node.js to learn a bit about it. It's a simple caching proxy for arch linux's package system as node provides most of the heavy lifting.
This has two "main" phases, server setup and serving.
Then serving has two main phases, response setup and response.
The "main" setup involves checking some files, loading some config from files. loading some json from a web address. Then launching the http server and proxy instance with this info.
setup logger/options - read config - read mirrors - read webmirror
start serving
Serving involves checking the request to see if the file exists, creating directories if needed, then providing a response.
check request - check dir - check file
proxy request or serve file
I keep referring to them as synchronisation points but searches don't lead to many results. Points where a set of async tasks have to be finished before the process can complete a next step. Perl's AnyEvent has conditional variables which I guess is what I'm trying to do, without the blocking.
To start with I found I was "cheating" and using the synchronous versions of any functions where provided but that had to stop with the web requests, so I started restructuring things. Immediately most search's led to using async or step to control the flow. To start with I was trying lots of series/parallel setups but running into issues if there were any async calls underneath the functions would "complete" straight away and the series would finish.
After much wailing and gnashing of teeth, I ended up with a "waiter" function using async.until that tests for some program state to be set by all the tasks finishing before launching the next function.
// wait for "test" to be true, execute "run",
// bail after "count" tries, waiting "sleep" ms between tries;
function waiter( test, run, count, sleep, message ) {
var i=0;
async.until(
function () {
if ( i > count ) { return true; }
logger.debug('waiting for',message, test() );
return test();
},
function (callback) {
i++;
setTimeout(callback, sleep );
},
function (err) {
if ( i > count ) {
logger.error('timeout for', message, count*sleep );
return;
}
run()
}
);
}
It struck me as being rather large and ugly and requiring a module to implement for something that I thought was standard, so I am wondering what's a better way. Am I still thinking in a non-async way? Is there something simple in Node I have overlooked? Is there a standard way of doing this?
I imagine with this setup, if the program get's complex there's going to be a lot of nesting functions to describe the flow of the program and I'm struggling to see a good way to lay it all out.
any tips would be appreciated.

You can't really make everything to be synchronous. Nodejs is designed to perform asynchronously (which may of course torment you at times). But there are a few ways techniques to make it work in a synchronous way (provided the pseudo-code is well-thought and code is designed carefully):
Using callbacks
Using events
Using promises
Callbacks and events are easy to use and understand. But with these, sometimes the code can get real messy and hard to debug.
But with promises, you can avoid all that. You can make dependency chains, called 'promises' (for instance, perform Promise B only when Promise A is complete).
Earlier versions of node.js had implementation of promises. They promised to do some work and then had separate callbacks that would be executed for success and failure as well as handling timeouts.
But in later versions, that was removed. By removing them from the core node.js, it created possibility of building up modules with different implementations of promises that can sit on top of the core. Some of these are node-promise, futures, and promises.
See these links for more info:
Framework
Promises and Futures
Deferred Promise - jQuery

How to call an asynchronous JavaScript function and block the original caller

I have an interesting situation that my usually clever mind hasn't been able to come up with a solution for :) Here's the situation...
I have a class that has a get() method... this method is called to get stored user preferences... what it does is calls on some underlying provider to actually get the data... as written now, it's calling on a provider that talks cookies... so, get() calls providerGet() let's say, providerGet() returns a value and get() passes it along to the caller. The caller expects a response before it continues its work obviously.
Here's the tricky part... I now am trying to implement a provider that is asychronous in nature (using local storage in this case)... so, providerGet() would return right away, having dispatched a call to local storage that will, some time later, call a callback function that was passed to it... but, since providerGet() already returned, and so did get() now by extension to the original called, it obviously hasn't returned the actual retrieved data.
So, the question is simply is there a way to essentially "block" the return from providerGet() until the asychronous call returns? Note that for my purposes I'm not concerned with the performance implications this might have, I'm just trying to figure out how to make it work.
I don't think there's a way, certainly I know I haven't been able to come up with it... so I wanted to toss it out and see what other people can come up with :)
edit: I'm just learning now that the core of the problem, the fact that the web sql API is asychronous, may have a solution... turns out there's a synchronous version of the API as well, something I didn't realize... I'm reading through docs now to see how to use it, but that would solve the problem nicely since the only reason providerGet() was written asychronously at all was to allow for that provider... the code that get() is a part of is my own abstraction layer above various storage providers (cookies, web sql, localStorage, etc) so the lowest common denominator has to win, which means if one is asychronous they ALL have to be asychronous... the only one that was is web sql... so if there's a way to do that synchronously my point become moot (still an interesting question generically I think though)
edit2: Ah well, no help there it seems... seems like the synchronous version of the API isn't implemented in any browser and even if it was it's specified that it can only be used from worker threads, so this doesn't seem like it'd help anyway. Although, reading some other things it sounds like there's a way to pull of this trick using recursion... I'm throwing together some test code now, I'll post it if/when I get it working, seems like a very interesting way to get around any such situation generically.
edit3: As per my comments below, there's really no way to do exactly what I wanted. The solution I'm going with to solve my immediate problem is to simply not allow usage of web SQL for data storage. It's not the ideal solution, but as that spec is in flux and not widely implemented anyway it's not the end of the world... hopefully when its properly supported the synchronous version will be available and I can plug in a new provider for it and be good to go. Generically though, there doesn't appear to be any way to pull of this miracle... confirms what I expected was the case, but wish I was wrong this one time :)

spawn a webworker thread to do the async operation for you.
pass it info it needs to do the task plus a unique id.
the trick is to have it send the result to a webserver when it finishes.
meanwhile...the function which spawned the webworker sends an ajax request to the same webserver
use the synchronous flag of the xmlhttprequest object(yes, it has a synchronous option). since it will block until the http request is complete, you can just have your webserver script poll the database for updates or whatever until the result has been sent to it.
ugly, i know. but it would block without hogging cpu :D
basically
function get(...) {
spawnWebworker(...);
var xhr = sendSynchronousXHR(...);
return xhr.responseTEXT;
}

No, you can't block until the asynch call finishes. It's that simple.
It sounds like you may already know this, but if you want to use asynchronous ajax calls, then you have to restructure the way your code is used. You cannot just have a .get() method that makes an asynchronous ajax call, blocks until it's complete and returns the result. The design pattern most commonly used in these cases (look at all of Google's javascript APIs that do networking, for example) is to have the caller pass you a completion function. The call to .get() will start the asynchronous operation and then return immediately. When the operation completes, the completion function will be called. The caller must structure their code accordingly.
You simply cannot write straight, sequential procedural javascript code when using asynchronous networking like:
var result = abc.get()
document.write(result);
The most common design pattern is like this:
abc.get(function(result) {
document.write(result);
});
If your problem is several calling layers deep, then callbacks can be passed along to different levels and invoked when needed.
FYI, newer browsers support the concept of promises which can then be used with async and await to write code that might look like this:
async function someFunc() {
let result = await abc.get()
document.write(result);
}
This is still asynchronous. It is still non-blocking. abc.get() must return a promise that resolves to the value result. This code must be inside a function that is declared async and other code outside this function will continue to run (that's what makes this non-blocking). But, you get to write code that "looks" more like blocking code when local to the specific function it's contained within.

Why not just have the original caller pass in a callback of its own to get()? This callback would contain the code that relies on the response.
The get() method will forward the callback to providerGet(), which would then invoke it when it invokes its own callback.
The result of the fetch would be passed to the original caller's callback.
function get( arg1, arg2, fn ) {
// whatever code
// call providerGet, passing along the callback
providerGet( fn );
}
function providerGet( fn ) {
// do async activity
// in the callback to the async, invoke the callback and pass it the data
// ...
fn( received_data );
// ...
}
get( 'some_arg', 'another_arg', function( data ) {
alert( data );
});

When your async method starts, I would open some sort of modal dialog (that the user cannot close) telling them that the request is in process. When the request finishes, close the modal in your callback.
One possible way to do this is with jqModal, but that would require you to load jQuery into your project. I'm not sure if that's an option for you or not.

This is ugly, but anyway I think the question is kindof implying an ugly solution is desired...
In your get function, serialize your query into a string.
Open an iframe, passing (A) this serialized query and (B) a random number in querystring to this iframe
Your iframe has some javascript code that reads the SQL query and number from its own querystring
Your iframe asynchronously begins running the query.
When your iframe query is asynchronously finished, it sends it, along with the random number to a server of yours, say to /write.php?rand=###&reslt="blahblahblah"
Write.php saves this info somewhere
Back in your main script, after creating and loading the iframe, you create a synchronous AJAX request to your server, say to /read.php?rand=####
/read.php blocks until the written info is available, then returns it to your main page
Alternately, to avoid sending the data over the network, you could instead have your iframe encode the result into a canvas-generated image that the browser caches (similar to the approach that Zombie cookie reportedly used). Then your blocking script would try to continually load this image over and over again (with some small network-generated delay on each request) until the cached version is available, which you could recognize via some flag that you've set to indicate it's done.