What I need:
asyncFunc(..., function cb() {
console.log("async function finished");
});
console.log("AFTER async function finished");
I am aware that the title and this example may trigger some strong responses regarding proper async programming in JS, but if you don't believe me that this is at least an ok solution, I added a description why I need it this way, but I would also be quite happy with something "better".
My project uses the newest version of typescript, therefore a possible solution could use async/await, generators or anything else from the newer concepts.
Why I need it:
I know that this is wrong in most contexts and I may have overlooked some options in the specifics, but I think that's the best solution.
I am developing a library that runs before the actual project (Angular, React, ...) and is based on #Decorators. This library is only used in development, therefore the added startup time - which is introduced by a blocking solution - is acceptable. I have a class decorator that sets up a data structure which then is used inside other decorators. The data structure setup is asynchronous (asyncFunc from above) and I need to halt further execution until the data structure is set up, otherwise it is not ready when the decorators that use this data are called. Since decorators are decoupled and have a predefined signature and behavior, I am not aware of any way to make them asynchronous or dependent on another function.
thanks
Related
This is a question about performance more than anything else.
Node exposes three different types of methods to accomplish various filesystem tasks:
Promises API (async)
Callback API (async)
Synchronous API (sync)
I've read more articles and stackoverflow answers than I can count, all of which claiming to never need the sync methods.
I recently wrote a script which required a couple directories to be made if they didn't already exist. During this, I noticed that if I used the async/await methods (primarily fs.promises.mkdir and fs.promises.access), the event loop would simply continue to the next async bit of code, regardless of the fact that the next bits require those directories. This is expected behavior, after all, it's async.
I understand this could be solved with a nice little callback hell sesh, but that isn't the question, whereas the idea that the promises api can be used over all other methods is.
The question then becomes:
Is it ever better to use Node's filesystem sync methods over the same async methods?
Is it ever truly required in situations like this to block the process?
Or said differently:
Is it possible to completely avoid sync methods and ONLY use the promises api (NOT promises + callbacks)?
It seems like using the sync methods (given my situation above, where the directories are required to be there before any other call is made) can be EXTREMELY useful to write readable, clear code, even though it may negatively impact performance.
With that being said, there's an overwhelming level of information to say that the sync api is completely useless and never required.
Again, this purely caters to the promises api. Yes, callbacks and promises are both async, but the difference between the job and message queues makes the both api's completely different in this context.
PS: For additonal context on examples, I've provided a code sample so you don't have to imagine my example ;)
Thanks! :)
// Checks if dir exists, if not, creates it. (not the actual code, just an example)
// Sync version
if (!fs.existsSync(dirPath)) {
fs.mkdirSync(dirPath);
}
// Async version
try {
await fs.promises.access(dirPath);
} catch {
await fs.promises.mkdir(dirPath);
}
It depends on the situation. The main benefit of the sync methods is that they allow for easier consumption of their results, and the main disadvantage is that they prevent all other code from executing while working.
If you find yourself in a situation where other code not being able to respond to events is not an issue, you might consider it to be reasonable to use the sync methods - if the code in question has no chance of or reason for running in parallel with anything else.
For example, you would definitely not want to use the sync methods inside, say, a server handling a request.
If your code requires reading some configuration files (or creating some folders) when the script first runs, and there aren't enough of them such that parallelism would be a benefit, you can consider using the sync methods.
That said, even if your current implementation doesn't require parallelism, something to keep in mind is that, if the situation changes and you find that you do actually need to allow for parallel processing, you won't have to make any changes to your existing code if you had started out by using the promise-based methods in the first place - and if you understand the language, using the Promises properly should be pretty easy, so if there's a chance of that, you might consider using the Promises anyway.
I recently learned that async/await is built into ES2017, but I have made several projects that I needed to require the package async in order to use async/await.
Is there an easy way to tell when I can access async normally or when I need to import it? Do I ever need to use that npm package? What is the purpose of the async package (which currently shows 47,469,002 weekly downloads) if the exact same functionality is now built into the language?
For an example project that requires async feel free to look at the Local-Library MongoDB/Express/Node tutorial project on MDN.
Since this is an Express app (as several of my own are), does this have anything to do with ExpressJS?
I have looked around for about a bit trying to find these answers but have come up empty-handed. Any help understanding this is appreciated.
The async library on NPM provides a number of utility functions for managing asynchronous operations. That is very different than the async keyword in the language that allows you to use await with promises. These are like cats and dogs. They both have something to do with asynchronous programming, but other than that, they really aren't the same thing at all.
For example, suppose you need to make 1000 separate requests to a particular host, but for a variety of reasons (memory consumption, rate limiting by the host, etc...), you cannot have more than 5 requests to target host in flight at any given time. So, you want to launch 5 requests and then each time one finishes, you'll launch another one until you've finally done all 1000.
The async keyword in ES7 could perhaps be used in some custom code to implement the algorithm above I described, but by itself, it doesn't give you that solution.
Whereas the async library from NPM has a specific function in its toolkit for doing exactly what I described. It's called parallelLimit() and it lets you specify exactly how many operations you want to be in parallel at a time (with no more than that). In fact, the async library contains all sorts of utility functions for managing asynchronous control flow and you can see a whole list of them here: https://caolan.github.io/async/v3/docs.html#parallelLimit.
Now that we have the async keyword in the language, some of those algorithms are no longer needed because it's very easy to code them in plain ES7. For example if you want to iterate over an array one at a time, calling an asynchronous function on each item, you can just use a for loop and await, all inside an async function.
async function run() {
for (let x of myArray) {
await doSomeAsyncOperation(x);
}
}
Before we had async and await in ES7, you would have had to either write a bit of custom code to do this asynchronous, serialized iteration or you would use a pre-built function from a library such as the async library.
Summary
To review, the async library contains a number of utility functions for managing asynchronous operations. Some of those are no longer necessary because of the control flow options that async and await provide in ES7 and some are still helpful/useful even with the presence of async and await.
FYI, the async library was originally developed before we even had promises and async/await in Javascript and it dealt with the older-style asynchronous operations that used plain callbacks to signal completion or error and offered utilities for managing them. Promises and async/await have replaced the need for some of the functionality in the async library, but not all.
When to use 'async' built-in for ES2017 or import 'npm i async'
Use the built-in ES7 async/await when it directly and simply solves the asynchronous control flow problem you need to solve. Use a function from the async library when you can't easily solve your problem with just async/await and the async library contains a pre-built function that solves your problem directly and simply.
So, the async library is just one of hundreds of thousands of libraries on NPM. You use it when it contains something that helps you solve your job better than what is already built-into the language, the same logic for when to use any other module from NPM.
aysnc/await that are part of ES2017 enables cleaner style there by avoiding promise chains. See the below example mentioned in mdn. You avoid promise chains like then, catch and yet have asynchronous behaviour
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Async library that you mentioned is more of a utility( they mainly designed to use with node js), It takes the array of callbacks, executes and wraps the them and gives you a promise. Iy also enables you with lot more features like in case you want invoking callback in parallel, serial, chain the callbacks, there by helping track these with a common success/error handling
Hi I have a very simple (i think) js question that I seem to be stuck on.
I am trying to create the route below.
What gets console.logged from within the bEtsy function is what I would like to have display on the page. however 'body' is not available outside of that scope.
app.get('/api/etsy/getListings',function(req, res){
bEtsy.getAllListings(req, res, function(err, body) {
// console.log(body);
});
res.json(req.body); // both this and res.json(body) does not work
});
Move res.json(req.body); into the callback function.
Apart from the scoping problem: It is asynchronous, so in your code it will be called long after res.json(req.body) runs.
app.get('/api/etsy/getListings', function(req, res) {
bEtsy.getAllListings(req, res, function(err, body) {
res.json(body);
//console.log(body);
});
});
A more general piece of advice (or two or three pieces), aside from the problem at hand:
What helps me with such situations and "callback thinking" is to almost never use inline callback functions: Write code only one layer deep (plus one layer for the module pattern of course), avoid callback hell! Name all callbacks and write them all on the same (top) level.
function allListingsReceived(err, body, res) {
res.json(body);
//console.log(body);
}
function getListings(req, res) {
// ASYNC
bEtsy.getAllListings(req, res, allListingsReceived);
}
//ASYNC
app.get('/api/etsy/getListings', getListings);
This allows me to get a much better overview over the actual call sequence. In this case, when getAllListings is called you know it is asynchronous - in my own code I add a clear comment (like I did above). So I know anything I were to write after that async function would not be able to access anything that async function is supposed to get me. IMHO such a comment is important - in Javascript there is no way to know if a callback function is asynchronous. Usually it is, but if it's synchronous and you expect asynchronism you may get into trouble too! So I think it's better to write it as a comment (always the exact same short string throughout the whole project), a formalized code annotation. Which by the way leads to another problem: When you write functions that accept a callback function, make sure they always call it either synchronously or asynchronously, never both ways (some functions use cached values and are able to return a result right away instead of starting an async. network request).
Basically, the written structure does not reflect the runtime situation with this style - but this is okay, since the runtime situation is completely flexible anyway (if you want to change which callback function you use, or add another one in between, do you really want to shift around tons of lines of code instead of just exchanging a name? Not to mention an increase in the ease of reusability). This is much easier to read in longer callback-style code files then several layers deep nested asynchronous functions IMHO. Avoid functions inside functions, apart from the module pattern, as much as possible.
Having named functions also is much better for debugging, stack traces are much easier to read.
A note: My example code leaves one issue open: if this is inside a module (or class), those would be internal functions, and you may have to make sure about the correct context/scope (where this points to, if you access object member variables this way from inside those functions). It works the same when those functions are on the prototype though. So this is just a general concept example that disregards this side issue.
Another note: When writing in this style variable that previously were available to an inner function via a closure - in this example res - now have to be made available as function parameters when calling the callback function. That adds some complexity - but on the other hand forces you to create clean(er) APIs in your own code. Personally I don't like excessive reliance on closures to pass arguments. I'm too stupid, I prefer to have a clean interface definition by having all parameters a function uses in its header. Apparently I'm not alone, that is one of the advantages most often touted for functional programming :) An alternative to filling the header with arguments that is "clean" too are object properties under this. My small example looks a little "procedural", but it only served to illustrate one single point. Of course this belongs into a larger context of modular programming.
I'm working for the first time with async programming, specifically with promises in Javascript and I'm really confused with it. I'm still thinking like we do in C, PHP, C# and so on where for each step the next one can be sure of it's completion. This is easy to work, because if we have a code like that (in C# for example)
IEnumerable<Page> pages = pagesRepository.getAll();
we can be sure in the next line that all pages are available for us to use in the pages object. This makes life easier, because we can already do work with that data properly.
Now, in JS with promises we would have something like
var pages = dataService.getPages();
and this would not return the data, but just a promise which is very different. Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
So how do we think when we deal with those things? Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
Of course we can add a callback with then but I'm getting confused, because somewhere else in my code I might need to reference this variable, and I cannot be sure when the callback will be called.
You don't need to.
The "somewhere else code" will use .then() and yield a new promise for its result, relying on the pages promise. You can easily map the results, or chain other asynchronous tasks. Whenever you need to know about the timing of callbacks because of multiple dependencies, you should use your library's methods for composing promises, like all().
So how do we think when we deal with those things?
Think functional!
Every task in your program should explicitly and only depend on its inputs. A function should try to only use its arguments, not any global variables that contain state.
Is there some tutorial or some videos out there that show in detail how to work with async programming with promises in JS?
General Promise Resources in the wiki of the Q library is a very good start.
The simple answer here is that promises are similar to Task asynchronous programming in C#.
However instead of being able to use the 'await' operator, the 'then' function on a promise is analogous to using 'ContinueWith' to chain Tasks in C#.
Callbacks can get complicated to use across scopes, especially with extensive use of nested anonymous functions, so please give promises a go
I'm a java developer. I really like to learn javascript. I'm finding it very difficult to pick-up a library and just learn/use it for two reasons:
1) There is no decent auto-complete. I've tried, eclipse, vjet, nodeclipse and webstorm...each has its own frustrating set of issues. Maybe the language is such that, autocomplete is super-difficult.
2) The API documentation is extremely confusing. I guess it is because I'm new to JavaScript.
For example, I wanted to figure what the callback function in mongoose.connect method does and how to declare it. So I checked the api doc. All it says is that the callback is a function...it doesn't say how many params it takes, what the values of the params are under various invocation scenarios...etc.
I feel like I'm missing something...
How does one go about reading these docs?
It's not you. I often find myself scratching my head about what the callback arguments should be. It's a problem with many JavaScript libraries.
However, in Node at least there is a convention that most libraries follow:
In node.js, it is considered standard practice to handle errors in asynchronous functions by returning them as the first argument to the current function's callback. If there is an error, the first parameter is passed an Error object with all the details. Otherwise, the first parameter is null.
For what it's worth, I haven't yet found an IDE that offers JavaScript autocomplete at anything nearly approaching the level of what's available for Java.
For the connect function, the callback passes an error argument given failure:
mongoose.connect('mongodb://localhost/dbname', function(err) {
if (err) {
// handle error
}
});
Generally, JavaScript tools are behind those for Java.
I feel like I'm missing something...
Me too. But, I think situation will change in 1-2 ears.
You can just wait for things to change or improve that you need by small step in a time.
Welcome to Nodeclipse.
It is time inefficient to solve problem only for one library (e.g. Mongoose),
but if there is web service like one for Node.js there is big chance for things change. Especially if you care enough to contribute, e.g. with ideas and materials or just raising an issue.