Develop Reference

When to use 'async' built-in for ES2017 or import 'npm i async'

I recently learned that async/await is built into ES2017, but I have made several projects that I needed to require the package async in order to use async/await.
Is there an easy way to tell when I can access async normally or when I need to import it? Do I ever need to use that npm package? What is the purpose of the async package (which currently shows 47,469,002 weekly downloads) if the exact same functionality is now built into the language?
For an example project that requires async feel free to look at the Local-Library MongoDB/Express/Node tutorial project on MDN.
Since this is an Express app (as several of my own are), does this have anything to do with ExpressJS?
I have looked around for about a bit trying to find these answers but have come up empty-handed. Any help understanding this is appreciated.

The async library on NPM provides a number of utility functions for managing asynchronous operations. That is very different than the async keyword in the language that allows you to use await with promises. These are like cats and dogs. They both have something to do with asynchronous programming, but other than that, they really aren't the same thing at all.
For example, suppose you need to make 1000 separate requests to a particular host, but for a variety of reasons (memory consumption, rate limiting by the host, etc...), you cannot have more than 5 requests to target host in flight at any given time. So, you want to launch 5 requests and then each time one finishes, you'll launch another one until you've finally done all 1000.
The async keyword in ES7 could perhaps be used in some custom code to implement the algorithm above I described, but by itself, it doesn't give you that solution.
Whereas the async library from NPM has a specific function in its toolkit for doing exactly what I described. It's called parallelLimit() and it lets you specify exactly how many operations you want to be in parallel at a time (with no more than that). In fact, the async library contains all sorts of utility functions for managing asynchronous control flow and you can see a whole list of them here: https://caolan.github.io/async/v3/docs.html#parallelLimit.
Now that we have the async keyword in the language, some of those algorithms are no longer needed because it's very easy to code them in plain ES7. For example if you want to iterate over an array one at a time, calling an asynchronous function on each item, you can just use a for loop and await, all inside an async function.
async function run() {
for (let x of myArray) {
await doSomeAsyncOperation(x);
}
}
Before we had async and await in ES7, you would have had to either write a bit of custom code to do this asynchronous, serialized iteration or you would use a pre-built function from a library such as the async library.
Summary
To review, the async library contains a number of utility functions for managing asynchronous operations. Some of those are no longer necessary because of the control flow options that async and await provide in ES7 and some are still helpful/useful even with the presence of async and await.
FYI, the async library was originally developed before we even had promises and async/await in Javascript and it dealt with the older-style asynchronous operations that used plain callbacks to signal completion or error and offered utilities for managing them. Promises and async/await have replaced the need for some of the functionality in the async library, but not all.
When to use 'async' built-in for ES2017 or import 'npm i async'
Use the built-in ES7 async/await when it directly and simply solves the asynchronous control flow problem you need to solve. Use a function from the async library when you can't easily solve your problem with just async/await and the async library contains a pre-built function that solves your problem directly and simply.
So, the async library is just one of hundreds of thousands of libraries on NPM. You use it when it contains something that helps you solve your job better than what is already built-into the language, the same logic for when to use any other module from NPM.

aysnc/await that are part of ES2017 enables cleaner style there by avoiding promise chains. See the below example mentioned in mdn. You avoid promise chains like then, catch and yet have asynchronous behaviour
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Async library that you mentioned is more of a utility( they mainly designed to use with node js), It takes the array of callbacks, executes and wraps the them and gives you a promise. Iy also enables you with lot more features like in case you want invoking callback in parallel, serial, chain the callbacks, there by helping track these with a common success/error handling

How to return synchronous result to asynchronous function call

In Node 13, an external 3rd party library calls my code:
const myInput = myCode.run(somVar); // it doesn't use await
As my code then has to perform nested synchronous calls, how could I provide an appropriate return value to the 3rd party library that is not a promise, but the result of my promises? Ideally something like this:
const run = (inputVar) =>{
let result
(async ()=>{
result = await doSyncCalls(inputVar);
})(); // code should not proceed until after await
return result;
} // will return undefined, but ideally should return doSyncCalls result
deasync would be a good solution, except it has an unresolved bug that causes nested promises to not resolve.

Well, there is no way to synchronously return (from the callback) a value that you obtain asynchronously within the callback. Javascript does not currently support that. So, these are your options:
Put the asynchronous code in a child_process. Then run that child process with something like child_process.execFileSync(). That allows you to run the asynchronous code in the child process, but have the parent process block and wait for the result. This is a hack because it blocks the parent while waiting for the result. But, it can be made to work.
Before calling the library function, prefetch whatever value it is you will need when the callback is called. This allows you to use regular asynchronous programming before you call the library and then once you have the desired value in some sort of cache, you can then call the 3rd party library and when it calls you back and wants the value, you will have it synchronously in a cache somewhere. Obviously, this only works if you can figure what value or range of possible values will be required in the callback so you can pre-fetch them.
Modify the code in the library to add support for an asycnhronous callback.
Redesign code to work a different way that doesn't require this library or that can use some other library that doesn't have this problem.
Write or find some native code add-on (like deasync) that lets you block somehow during the asynchronous operation while still letting the event queue do what it needs to do to process the asynchronous completion. This would have to hook deep into the internals of the V8 engine. Or fix, deasync so it works for your case.
Write a blocking add-on in native code that could carry out your asynchronous operation in native code while blocking the V8 engine.
FYI, since everything but #2, #3 and #4 all block the main JS thread which is generally a bad thing to do in any server environment, and you've said that #2 was not practical, my preference would be #3 or #4. Since you don't share the actual code and actual detailed library and problem, we can't help you with any specifics.
Probably the most straightforward solution to implement is #1 (package the code up into a child process that you run synchronously), but it blocks the app while running which is a downside.

return deferred or deferred.promise() [duplicate]

What are the differences between Deferreds, Promises and Futures?
Is there a generally approved theory behind all these three?

These answers, including the selected answer, are good for introducing promises
conceptually, but lacking in specifics of what exactly the differences are in
the terminology that arises when using libraries implementing them (and there
are important differences).
Since it is still an evolving spec, the answer currently comes from attempting to survey both references (like wikipedia) and implementations (like jQuery):
Deferred: Never described in popular references,
1
2
3
4
but commonly used by implementations as the arbiter of promise resolution (implementing resolve and reject).
5
6
7
Sometimes deferreds are also promises (implementing then),
5
6
other times it's seen as more pure to have the Deferred only
capable of resolution, and forcing the user to access the promise for
using then.
7
Promise: The most all-encompasing word for the strategy under discussion.
A proxy object storing the result of a target function whose
synchronicity we would like to abstract, plus exposing a then function
accepting another target function and returning a new promise.
2
Example from CommonJS:
> asyncComputeTheAnswerToEverything()
.then(addTwo)
.then(printResult);
44
Always described in popular references, although never specified as to
whose responsibility resolution falls to.
1
2
3
4
Always present in popular implementations, and never given
resolution abilites.
5
6
7
Future: a seemingly deprecated term found in some popular references
1
and at least one popular implementation,
8
but seemingly being phased out of discussion in preference for the term
'promise'
3
and not always mentioned in popular introductions to the topic.
9
However, at least one library uses the term generically for abstracting
synchronicity and error handling, while not providing then functionality.
10
It's unclear if avoiding the term 'promise' was intentional, but probably a
good choice since promises are built around 'thenables.'
2
References
Wikipedia on Promises & Futures
Promises/A+ spec
DOM Standard on Promises
DOM Standard Promises Spec WIP
DOJO Toolkit Deferreds
jQuery Deferreds
Q
FutureJS
Functional Javascript section on Promises
Futures in AngularJS Integration Testing
Misc potentially confusing things
Difference between Promises/A and Promises/A+
(TL;DR, Promises/A+ mostly resolves ambiguities in Promises/A)

In light of apparent dislike for how I've attempted to answer the OP's question. The literal answer is, a promise is something shared w/ other objects, while a deferred should be kept private. Primarily, a deferred (which generally extends Promise) can resolve itself, while a promise might not be able to do so.
If you're interested in the minutiae, then examine Promises/A+.
So far as I'm aware, the overarching purpose is to improve clarity and loosen coupling through a standardized interface. See suggested reading from #jfriend00:
Rather than directly passing callbacks to functions, something which
can lead to tightly coupled interfaces, using promises allows one to
separate concerns for code that is synchronous or asynchronous.
Personally, I've found deferred especially useful when dealing with e.g. templates that are populated by asynchronous requests, loading scripts that have networks of dependencies, and providing user feedback to form data in a non-blocking manner.
Indeed, compare the pure callback form of doing something after loading CodeMirror in JS mode asynchronously (apologies, I've not used jQuery in a while):
/* assume getScript has signature like: function (path, callback, context)
and listens to onload && onreadystatechange */
$(function () {
getScript('path/to/CodeMirror', getJSMode);
// onreadystate is not reliable for callback args.
function getJSMode() {
getScript('path/to/CodeMirror/mode/javascript/javascript.js',
ourAwesomeScript);
};
function ourAwesomeScript() {
console.log("CodeMirror is awesome, but I'm too impatient.");
};
});
To the promises formulated version (again, apologies, I'm not up to date on jQuery):
/* Assume getScript returns a promise object */
$(function () {
$.when(
getScript('path/to/CodeMirror'),
getScript('path/to/CodeMirror/mode/javascript/javascript.js')
).then(function () {
console.log("CodeMirror is awesome, but I'm too impatient.");
});
});
Apologies for the semi-pseudo code, but I hope it makes the core idea somewhat clear. Basically, by returning a standardized promise, you can pass the promise around, thus allowing for more clear grouping.

What really made it all click for me was this presentation by Domenic Denicola.
In a github gist, he gave the description I like most, it's very concise:
The point of promises is to give us back functional composition and error bubbling in the async world.
In other word, promises are a way that lets us write asynchronous code that is almost as easy to write as if it was synchronous.
Consider this example, with promises:
getTweetsFor("domenic") // promise-returning async function
.then(function (tweets) {
var shortUrls = parseTweetsForUrls(tweets);
var mostRecentShortUrl = shortUrls[0];
return expandUrlUsingTwitterApi(mostRecentShortUrl); // promise-returning async function
})
.then(doHttpRequest) // promise-returning async function
.then(
function (responseBody) {
console.log("Most recent link text:", responseBody);
},
function (error) {
console.error("Error with the twitterverse:", error);
}
);
It works as if you were writing this synchronous code:
try {
var tweets = getTweetsFor("domenic"); // blocking
var shortUrls = parseTweetsForUrls(tweets);
var mostRecentShortUrl = shortUrls[0];
var responseBody = doHttpRequest(expandUrlUsingTwitterApi(mostRecentShortUrl)); // blocking x 2
console.log("Most recent link text:", responseBody);
} catch (error) {
console.error("Error with the twitterverse: ", error);
}
(If this still sounds complicated, watch that presentation!)
Regarding Deferred, it's a way to .resolve() or .reject() promises. In the Promises/B spec, it is called .defer(). In jQuery, it's $.Deferred().
Please note that, as far as I know, the Promise implementation in jQuery is broken (see that gist), at least as of jQuery 1.8.2.
It supposedly implements Promises/A thenables, but you don't get the correct error handling you should, in the sense that the whole "async try/catch" functionality won't work.
Which is a pity, because having a "try/catch" with async code is utterly cool.
If you are going to use Promises (you should try them out with your own code!), use Kris Kowal's Q. The jQuery version is just some callback aggregator for writing cleaner jQuery code, but misses the point.
Regarding Future, I have no idea, I haven't seen that in any API.
Edit: Domenic Denicola's youtube talk on Promises from #Farm's comment below.
A quote from Michael Jackson (yes, Michael Jackson) from the video:
I want you to burn this phrase in your mind:
A promise is an asynchronous value.
This is an excellent description: a promise is like a variable from the future - a first-class reference to something that, at some point, will exist (or happen).

A Promise represents a proxy for a value not necessarily known when the promise is created. It allows you to associate handlers to an asynchronous action's eventual success value or failure reason. This lets asynchronous methods return values like synchronous methods: instead of the final value, the asynchronous method returns a promise of having a value at some point in the future.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
The deferred.promise() method allows an asynchronous function to prevent other code from interfering with the progress or status of its internal request. The Promise exposes only the Deferred methods needed to attach additional handlers or determine the state (then, done, fail, always, pipe, progress, state and promise), but not ones that change the state (resolve, reject, notify, resolveWith, rejectWith, and notifyWith).
If target is provided, deferred.promise() will attach the methods onto it and then return this object rather than create a new one. This can be useful to attach the Promise behavior to an object that already exists.
If you are creating a Deferred, keep a reference to the Deferred so that it can be resolved or rejected at some point. Return only the Promise object via deferred.promise() so other code can register callbacks or inspect the current state.
Simply we can say that a Promise represents a value that is not yet known where as a Deferred represents work that is not yet finished.

A promise represents a value that is not yet known
A deferred represents work that is not yet finished
A promise is a placeholder for a result which is initially unknown while a deferred represents the computation that results in the value.
Reference
http://blog.mediumequalsmessage.com/promise-deferred-objects-in-javascript-pt1-theory-and-semantics

Using async result (array.filter) [duplicate]

I'm curious as to whether all javascript callbacks are asynchronous, or whether that is the case only in certain situations. Also, I'm sure what makes javascript code asynchronous (or ways to use asynchronous javascript) differ between the browser and nodejs, so I'd like to know in each situation what constitutes real asynchronous javascript.
I'm under the impression that in the following scenarion, I'm not actually writing asynchronous code.
function addOne(value){
value = value + 1;
return value;
}
function simpleMap(values, callback){
for(i = 0; i < values.length; i++){
val = values[i];
val = callback(val);
values[i] = val;
}
return values;
}
newValues = simpleMap([1,2,3], addOne);
However, for example, I know that jQuery's AJAX functions are truly asynchronous (not taking in to account the promises which are now available). What is it that make jQuery's AJAX asynchronous? Is it as simple that it involves XHR requests, and in the browser, all XHR requests are asynchronous?
I have the same question for the nodejs environment. Can something in node only be asynchronous if it involves something like file i/o, process.nextTick, setTimeout, or setInterval? Why when I do something like a database call with mongodb/mongoose, is that asynchronous? What's going on behind the scenes that's making it so?
Are asynchronous "situations" predetermined by the environment? Or is there some way to make one's own function truly asynchronous without leveraging very specific functions of the environment (such as xhr, file io in node, process.nexttick, etc)?

I'm curious as to whether all javascript callbacks are asynchronous
No. For instance, the callback used by Array#sort is not asynchronous, nor is the one used by String#replace.
The only way you know whether a callback is asynchronous is from its documentation. Typically, ones involving requests for external resources (ajax calls, for instance) are asynchronous, and others may or may not be.
However, for example, I know that jQuery's AJAX functions are truly asynchronous...
Not necessarily, as currently jQuery still has the async flag which you can set false to force a synchronous request. (It's not a good idea, and they're going to remove that, but you can. jQuery passes the flag to the underlying browser object which provides the synchronous/asynchronous behavior.)
What is it that make jQuery's AJAX asynchronous?
The browser. jQuery's ajax calls use the XMLHttpRequest object (or in certain situations, a script element), which defaults to asynchronous operation provided by the browser.
Or is there some way to make one's own function truly asynchronous without leveraging very specific functions of the environment...
Until recently, no. Up through the 5th edition specification, JavaScript the language was basically silent on the entire concept of threads and asynchronicity; it was only when you got into environments that it came up. The only way to make something asynchronous was to use a host-provided function, such as nextTick (or any of the various operations that completes asynchronously) on NodeJS or setTimeout on browsers.
In the ECMAScript 6th edition specification in June 2015, they introduced promises into the language. The callbacks hooked up to an ES6 promise via then and such are always invoked asynchronously (even if the promise is already settled when the callback is attached), and so JavaScript has asynchronicity at a language level now. So if you implement your function so that it returns a promise rather than accepting a callback, you'll know that the then callbacks hooked up to it will be triggered asynchronously.

Callbacks that you call yourself are regular function calls, which are always synchronous.
Certain native APIs (eg, AJAX, geolocation, Node.js disk or network APIs) are asynchronous and will execute their callbacks later in the event loop.
If you call a callback synchronously from within an async callback, it will end up being async too.

To create your own asynchronous functions you have to make use of other asynchronous functions which may be provided by the interpreter.
This code for example defines a function "addKeyHandler" which is asynchronous. But that only works because document.onKey is called asynchronously by the JS engine. The JavaScript engine is able to provide asynchronous functionality because the operating system provides such functionality which is then used by JS. The OS in turn can only provide async functionality because the hardware provides it (called hardware interrupts).
However if the OS and hardware didn't provide any async functions it would still be possible to write a JS interpreter. But it would have to use an infinite loop and check in each iteration if any events occured and then invoke the appropriate callbacks. That would mean the CPU would always be under full load.
var keyCallbacks = [];
var addKeyHandler = function(f) {
keyCallbacks.push(f);
};
document.onkeypress = function(e) {
keyCallbacks.forEach(function(f) {
f(e);
});
};
addKeyHandler(function(e) {
console.log(String.fromCharCode(e.charCode));
});

Simply taking a callback doesn't make a function asynchronous. There are many examples of functions that take a function argument but are not asynchronous, for example, Array's forEach.
For a function to be asynchronous it needs to perform an asynchronous operation. Ways of introducing asynchronicity can be
timer functions setTimeout, setInterval
special functions nextTick, setImmediate
performing I/O (listening to network, querying a database, reading or writing from a resource)
subscribing to an event
According to article "Does taking a callback make a function asynchronous?"

Going for a simple answer:
Unless you're dealing with promises, JS callbacks are only asynchronous if they rely on an API external to JS (such as provided by the browser).
setTimeout, fetch, and so on are asynchronous because they rely on external api's. (setTimeout is part of the windowOrGlobalWorker web api, for example, and fetch is a web api in itself.)
If a callback does not rely on an external API, it's synchronous.
Promises are the only native async functionality in JS.
A great way to get your head around all this is read at least the first few articles in the MDN primer on async: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous

How to think when working with promises

I'm working for the first time with async programming, specifically with promises in Javascript and I'm really confused with it. I'm still thinking like we do in C, PHP, C# and so on where for each step the next one can be sure of it's completion. This is easy to work, because if we have a code like that (in C# for example)
IEnumerable<Page> pages = pagesRepository.getAll();
we can be sure in the next line that all pages are available for us to use in the pages object. This makes life easier, because we can already do work with that data properly.
Now, in JS with promises we would have something like
var pages = dataService.getPages();
and this would not return the data, but just a promise which is very different. Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
So how do we think when we deal with those things? Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?

Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
You don't need to.
The "somewhere else code" will use .then() and yield a new promise for its result, relying on the pages promise. You can easily map the results, or chain other asynchronous tasks. Whenever you need to know about the timing of callbacks because of multiple dependencies, you should use your library's methods for composing promises, like all().
So how do we think when we deal with those things?
Think functional!
Every task in your program should explicitly and only depend on its inputs. A function should try to only use its arguments, not any global variables that contain state.
Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
General Promise Resources in the wiki of the Q library is a very good start.

The simple answer here is that promises are similar to Task asynchronous programming in C#.
However instead of being able to use the 'await' operator, the 'then' function on a promise is analogous to using 'ContinueWith' to chain Tasks in C#.
Callbacks can get complicated to use across scopes, especially with extensive use of nested anonymous functions, so please give promises a go