Develop Reference

Is it ever better to use Node's filesystem sync methods over the same async methods?

This is a question about performance more than anything else.
Node exposes three different types of methods to accomplish various filesystem tasks:
Promises API (async)
Callback API (async)
Synchronous API (sync)
I've read more articles and stackoverflow answers than I can count, all of which claiming to never need the sync methods.
I recently wrote a script which required a couple directories to be made if they didn't already exist. During this, I noticed that if I used the async/await methods (primarily fs.promises.mkdir and fs.promises.access), the event loop would simply continue to the next async bit of code, regardless of the fact that the next bits require those directories. This is expected behavior, after all, it's async.
I understand this could be solved with a nice little callback hell sesh, but that isn't the question, whereas the idea that the promises api can be used over all other methods is.
The question then becomes:
Is it ever better to use Node's filesystem sync methods over the same async methods?
Is it ever truly required in situations like this to block the process?
Or said differently:
Is it possible to completely avoid sync methods and ONLY use the promises api (NOT promises + callbacks)?
It seems like using the sync methods (given my situation above, where the directories are required to be there before any other call is made) can be EXTREMELY useful to write readable, clear code, even though it may negatively impact performance.
With that being said, there's an overwhelming level of information to say that the sync api is completely useless and never required.
Again, this purely caters to the promises api. Yes, callbacks and promises are both async, but the difference between the job and message queues makes the both api's completely different in this context.
PS: For additonal context on examples, I've provided a code sample so you don't have to imagine my example ;)
Thanks! :)
// Checks if dir exists, if not, creates it. (not the actual code, just an example)
// Sync version
if (!fs.existsSync(dirPath)) {
fs.mkdirSync(dirPath);
}
// Async version
try {
await fs.promises.access(dirPath);
} catch {
await fs.promises.mkdir(dirPath);
}

It depends on the situation. The main benefit of the sync methods is that they allow for easier consumption of their results, and the main disadvantage is that they prevent all other code from executing while working.
If you find yourself in a situation where other code not being able to respond to events is not an issue, you might consider it to be reasonable to use the sync methods - if the code in question has no chance of or reason for running in parallel with anything else.
For example, you would definitely not want to use the sync methods inside, say, a server handling a request.
If your code requires reading some configuration files (or creating some folders) when the script first runs, and there aren't enough of them such that parallelism would be a benefit, you can consider using the sync methods.
That said, even if your current implementation doesn't require parallelism, something to keep in mind is that, if the situation changes and you find that you do actually need to allow for parallel processing, you won't have to make any changes to your existing code if you had started out by using the promise-based methods in the first place - and if you understand the language, using the Promises properly should be pretty easy, so if there's a chance of that, you might consider using the Promises anyway.

Advice on creating asynchronous calls that depend on each other

I am attempting to create a library to make API calls to a web application (jira, if you care to know) I have my api calls working no problem, but I am looking to make the code a bit more readable and use-able. I have tried searching for my needs, but it turns out I am not sure what I need to be searching for.
I am having an issue with Asynchronous calls that depend on each other, I understand that I have to wait until the callback is ran to run my next item, but I am not sure of the best way to design this.
I really would like to make Chaining a feature of my api, which I would hope to look like this:
createProject(jsonProjectStuff)
.setLeadUser("myusername")
.createBoard("boardName")
.setBoardPermissions(boardPermissionJSONvar)
.addRole(personRoleJSONvar);
with this example, everything would have to wait on the createProject as it will return the project. createBoard doesn't rely on the project normally, but used in this context it should be "assigned" to the project made, setting the board permissions only relies on the createBoard to work. addRole is specific to the project again.
the questions I have are:
Is this possible to switch context like this and keep data in-between them without the need to run the function from the response hard coded?
If this is possible, is this a good idea? If not I am open to other schemes.
I can think of a couple ways to make it work, including registering the function calls with a dependency tree and then fulfilling promises as we go, although that is mostly conceptual for me at this point as I am trying to decide the best.
Edit 2/19/2016
So I have looked into this more and I have decided on a selective "then" only when it creating a new item doesn't relate directly to the parent.
//Numbers are ID, string is Name
copyProject(IDorName)
.setRoles(JSONItem)
.setOwner("Project.Owner")
.setDefaultEmail("noreply#fake.com")
.then(
copyBoard(IDorName)
.setName("Blah blah Name {project.key}"),
saveFilterAs(IDorName, "Board {project.key}",
"project = {project.key} ORDER BY Rank ASC")
.setFilterPermissions({shareValuesJSON})
)
I like this solution a lot, the only thing I am unsure of how to do is the string "variables", I suppose it could be "Blah blah Name " + this.project.key
either way I am unsure of how to give copyBoard or saveFilterAs access to it via the "then" function.
Any thoughts?

I've been using Nightmare (a headless browser) lately.
It has a fluent API that uses a nice design pattern.
Calling the API doesn't directly execute the actions, it only queues them and when you are ready to execute you must call the end function which returns a promise. The promise is resolved when the queue has completed its async execution.
For example, in your situation
createProject(jsonProjectStuff)
.setLeadUser("myusername")
.createBoard("boardName")
.setBoardPermissions(boardPermissionJSONvar)
.addRole(personRoleJSONvar)
.end() // Execute the queue of operations.
.then() => {
// All operations completed.
))
.catch(err => {
// An error occurred.
});
I feel like this pattern is quite elegant. It allows you to have a fluent API to build a sequence of actions. Then when you are ready to execute said operations you call end (or whatever). The sequence of operations are then completed asynchronously and you use the promise to handle completion and errors.

Automatically call a function when accessing an object in Javascript

I am currently on a big Javascript project with a lot of libraries.
I would like to have some part of this project to run on separate thread. There is already something inJavascript doing that : the web workers.
Though, the web workers can't access the window object, and a lot of the libraries use it. Is there a way to automatically change the call to the window object (in the libraries used for the web workers), into a message sent to the parent thread ?
Then, the parent thread would perform the action that the worker want and send back the result to the worker.
Is it possible to do that ? And id yes, do you have any idea how ?
Thank you !

I'm afraid there's no real solution to this. What you'd probably want is a special object in your worker, which, at every property access, passes the execution to the dispatching thread - which handles the request using the original window object.
To do this, you would need some sort of catch-all accessor method which would run whenever a property is referenced. Sadly, there's no such thing in Javascript, see this detailed discussion (especially T.J. Crowder's answer): Is it possible to implement dynamic getters/setters in JavaScript?
ECMAScript 6 introduces a new mechanism called Proxy (currently supported in FF and IE12 (go figure!)), which would enable you to do these dynamic property lookups, technically - but I feel there's a more fundamental problem with your idea: you're aiming to turn a local call into a message across the boundaries of single threaded environments.
Message passing from and to the worker threads must be asynchronous (as a javascript "thread" cannot be interrupted until it yields), which would mean that even if you do manage to set up a proxy like that, it'd be effectively turning a usually synchronous operation (ie. a property access) into an asynchronous one, which is a pretty big issue, especially if you're looking for a drop-in replacement in order to use some existing libraries.

Javascript callback hell with setTimeout();

Aloha. This is my first question here after eons of lurking. I apologize for the verbose but direct form of this question -- as well as the limitations on acceptable answers -- but I am here to contribute. I have a problem that other questions (for example How to avoid long nesting of asynchronous functions in Node.js or Chained callback readability in Javascript?) do not answer to my satisfaction.
I want to perform several Javascript callbacks so that I have a sequence of four actions occurring one after another. Here is the source:
var self = this;
fade(div, fadeIn, -20, function() {
load(unload, dataOut, -10, function() {
load(self, dataIn, 10, function() {
fade(div, fadeOut, 20),0);
unload = self;}
);}
);}
);
The load() and fade() functions are very similar except for internal semantic details, and they look something very similar to this:
function load(menuCategory, data, step, callback) {
menuCategory.parentNode.style.width = data+"px";
data += step;
if(dataIn <= data && data <= dataOut) {
setTimeout(function() { load(menuCategory,data,step,callback) }, 15);
} else if(callback && (typeof(callback) == 'function')) {
callback();}
}
I have several questions associated with this mess, but first let me give you the answers I am NOT looking for. Answers using jQuery are not acceptable. Answers using other timing or "synchronicity" frameworks or modules are not acceptable. Answers questioning the reason for blocking execution are not acceptable. Answers using languages other than Javascript are unacceptable. Non-abstract answers are acceptable, but the final solution will need to be as abstract as possible, meaning the answer must be able to act on a variety of similar but slightly different actions.
Here is my primary question:
Is there a function/object I could use to apply these actions sequentially, to cut down on the callback hell going on here? In other words, what would an object look like if the object/function were able to synchronously execute each action as it was individually iterated through it (i.e. passed to it)?
Secondary question:
Is anyone aware of how much this callback business looks like goto() in other languages? Meaning, this nesting of callbacks feels very awkward and inelegant to me. Is there a Javascript language construct either being developed or already developed that will reduce this crazy business? Callback nesting is some of the hardest logic flow to follow. It just feels awkward.
My goal here is a tight, elegant, attractive, homegrown Javascript object/function -- that I can call in a for() loop or equivalent -- to apply these operations on each action in sequence. If you've read this far I want to thank you for your time and consideration. :)

using setTimeout(func(),0) in order to execute synchronously
No. Either use func() for a synchronous execution, or setTimeout(func, 0) to queue the execution (which makes it asynchronous).
Is setTimeout(func[…],0) the "most elegant" way to put this action onto the execution queue for rendered webpage objects? In other words, is there a "more elegant" way to block execution (i.e. do these things sequentially)?
Yes, this is the standard method. However, this does not block execution, and has nothing to do with sequencing. If the execution is synchronous, just execute it, if not you will have to deal with the callbacks and queuing a new task does not help.
Is anyone aware of how much this callback business looks like goto() in other languages? Meaning, this nesting of callbacks feels very awkward and inelegant to me. Is there a Javascript language construct either being developed or already developed that will reduce this crazy business?
No, I'm not aware of other languages, but afaik goto is a synchronous, control-flow-structuring statement as does not deal with asynchronous actions.
But no, there are no JavaScript language constructs ("syntactic sugar") that help us around the continuation-passing style with its callback hell.
Is there a function/object I could use to apply these actions sequentially, to cut down on the callback hell going on here? In other words, what would an object look like if the object/function were able to synchronously execute each action as it was individually iterated through it (i.e. passed to it)?
My goal here is a tight, elegant, attractive, homegrown Javascript object/function -- that I can call in a for() loop or equivalent -- to apply these operations on each action in sequence.
Again you want "sequencially" instead of "synchronous" :-) Yes, there are [handcoded/homegrown] solutions which make dealing with callbacks easier. As those callback organisation libraries you mentioned do not satisfy you, I'd recommend to look into the very elegant and attractive Promise Api and its Deferred objects (see futures and promises for the concept).

I'm unsure why you're using setTimeout at all here. Its not necessary to queue callbacks. The following structure will work, assuming that fade() and load() handle callback execution correctly.
fade(div, fadeIn, -20, function() {
load(unload, dataOut, -10, function() {
load(self, dataIn, 10, function() {
fade(div, fadeOut, 20),0);
});
)};
)};
Another topic you should look into is Promises, also called deferreds and futures. Its essentially a coding pattern that specifically handles nested callbacks. At root, you create a Promise object that gets returned immediately, then when your method is done executing it fires a method on the promise that can then execute your next function. Its a different way of writing javascript, but easy to understand and very useful.

As it turns out, the practice that fits as the "best" answer to my question is events. In particular the practice I'm referring to is best described by EventEmitter on Node.js, or creating custom events and event listeners on client browsers. While the Promise API and its deference model is intriguing, it uses extensions to the language instead of core Javascript, which is what I was looking for here in particular (and thus why none of the previous answers -- despite their clarity -- support exactly what I was looking for).
This link (http://wekeroad.com/2012/04/05/cleaning-up-deep-callback-nesting-with-nodes-eventemitter) provides a very good example of what I was looking for. As well, I should mention the book "Async Javascript" by Trevor Burnham (http://www.amazon.com/Async-JavaScript-Trevor-Burnham/dp/1475247362), and "Secrets of the JavaScript Ninja" by John Resig and Bear Bibeault (http://www.amazon.com/Secrets-JavaScript-Ninja-John-Resig/dp/193398869X). These books provide extremely clear working examples of custom events and event handlers in client browsers that provide an elegant way around this "callback hell" I was referring to.

Can JavaScript talk to Selenium 2?

I know I can get Selenium 2's webdriver to run JavaScript and get return values but so much asynchronous stuff is happening I would like JavaScript to talk to Selenium instead of the other way around. I have done some searching and haven't found anything like this. Do people just generally use implicitly_wait? That seems likely to fail since it's not possible to time everything? Perfect example would be to let Selenium know when an XHR completed or an asynchronous animation with undetermined execution time.
Is this possible? We're using Selenium 2 with Python on Saucelabs.

You should look into the execute_async_script() method (JavascriptExecutor.executeAsyncScript in Java, IJavaScriptExecutor.ExecuteAsyncScript() in .NET), which allows you to wait for a callback function. The callback function is automatically appended to the arguments array in your JavaScript function. So, assuming you have a JavaScript function already on the page that waits until the condition you want, you could do something like the following (Java code below, C# and Python code should be similar):
String script = "var callback = arguments[arguments.length - 1];"
+ "callback(myJavaScriptFunctionThatWaitsUntilReady());";
driver.manage().timeouts().setScriptTimeout(15, TimeUnit.SECONDS);
((JavascriptExecutor)driver).executeAsyncScript(script);
It might be possible to be even more clever and pass the callback function directly to an event that returns the proper data. You can find more information on the executeAsyncScript() function in the project JavaDocs, and can find sample code for this in the project source tree. There's a great example of waiting for an XHR to complete in the tests in this file.
If this isn't yet available in the version of the Python bindings available for use with SauceLabs, I would expect it to be available before long. Admittedly, in a sense, this is pushing the "poll for desired state" from your test case into JavaScript, but it would make your test more readable.

Theoretically it is possible, but I would advise against it.
The solution would probably have some jQuery running on the site that sets a variable to true when the JavaScript processing has finished.
Set selenium up to loop through a getEval until this variable becomes true and then do something in Selenium.
It would meet your requirements but it's a really bad idea. If for some reason your jQuery doesn't set the trigger variable to true (or whatever state you expect) Selenium will sit there indefinetly. You could put a really long timeout on it, but then what would be the different in just getting Selenium to do a getEval and wait for a specific element to appear?
It sounds like you are trying to overengineer your solution and it will cause you more pain in the future will very few additional benefits.

Not to be overly blunt, but if you want your App to talk to your Test Runner, then you're doing it wrong.
If you need to wait for an XHR to finish, you could try displaying a spinner and then test that the spinner has disappeared to indicate a successful request.
In regards to the animation, when the animation has completed, maybe its callback could add a class indicating that the animation has finished and then you could test for the existence of that class.

Testing animation with selenium is opening a can of worms. The tests can be quite brittle and cause many false positives.
The problem is to do that the calls are asynchronous, and difficult to track the behaviour and change in state of the page.
In my experience the asynchronous call can be so quick that the spinner is never displayed, and the state of the page may skip a state entirely (that Selenium can detect).
Waiting for the state of the page to transition can make the tests less brittle, however the false positives cannot be removed entirely.
I recommend manual testing for animation.