Develop Reference

Node and async: Why is await not being honored in a sub-function?

I'm a Node.js student, and I'm writing a brief example. The get on /org works as expected.
With the get on /orgg I tried to segregate the server "fetch the data" logic from the controller. However, the getRowsAsync() immediately returns a Promise.
This "put the await in express.get()" isn't good for hiding logic. If my biz logic needed some sequential Promises then the biz logic would have to bubble into the controller.
How can I do the equivalent to calling "doBizLogic()" in the controller, hiding my awaits, and having the controller wait for the logic to complete? Must I pass a callback function to the biz logic to make this scheme work?
Here is my index.js. I omit database.js, but I borrowed it from https://mhagemann.medium.com/create-a-mysql-database-middleware-with-node-js-8-and-async-await-6984a09d49f4
const express = require("express");
const urlPort = 3000;
let app = express();
let mysql = require("mysql2");
let pool = require("./database");
app.listen(urlPort, () => {
console.log("Server is running at port " + urlPort);
});
app.get("/", (req, res) => {
res.send("This app responds to /org and /org/:id only.");
});
app.get("/org", async (req, res) => {
let rows = await getRows("select * from org");
// the log always prints a JSON array.
console.log("in app.get for /org, rows: ", rows);
res.send(rows);
});
app.get("/orgg", (req, res) => {
let rows = getRowsAsync();
// the log() prints immediately prints a Promise.
console.log("in app.get for /orgg, rows: ", rows);
res.send(rows);
});
function getRows(sql, params = []) {
let rows = pool.query(sql, params);
// the log() prints a Promise.
console.log("in getRows, rows: ", rows); // returns
return rows;
}
async function getRowsAsync() {
let rows = await getRows("select * from org");
// the log() prints a JSON array, once it is finally called.
console.log("in getRowsAsync, rows: ", rows);
return rows;
}

Your specific questions
How can I do the equivalent to calling "doBizLogic()" in the controller, hiding my awaits, and having the controller wait for the logic to complete?
You can't hide that logic. Any API that "might" be asynchronous must be designed as an asynchronous API and the caller needs to know how to use the asynchronous API properly.
You cannot hide asynchronous-ness because there's no way in Javascript to make an asynchronous operation into a synchronous one. If any part of your operation is asynchronous, then it will need an asynchronous interface to communicate the result and completion (via a promise, a callback or an event).
Must I pass a callback function to the biz logic to make this scheme work?
You could use a callback, but I would suggest just returning a promise that the caller can use await on as that's a more modern design than a callback and is somewhat easier to program with, particularly if there are other asynchronous operations in the same function. The caller will have to make use of that returned promise for things to work properly.
So, just have your caller use await. The caller has to "know" it's an asynchronous interface anyway because you program with asynchronous interfaces differently than synchronous interfaces.
This "put the await in express.get()" isn't good for hiding logic. If my biz logic needed some sequential Promises then the biz logic would have to bubble into the controller.
Yes, that's the way it is in Javascript. A caller has to know if they are calling a synchronous or asynchronous API so they know how to get the result/err and completion. There is no way around that unless you design all interfaces as asynchronous and treat all calls to those APIs as asynchronous (which is not efficient for entirely synchronous APIs).
Why is await not being honored in a sub-function?
In this piece of code:
function getRows(sql, params = []) {
let rows = pool.query(sql, params);
// the log() prints a Promise.
console.log("in getRows, rows: ", rows); // returns
return rows;
}
The code is not properly handling the asynchronous pool.query(). That function apparently returns a promise and you aren't waiting for that promise to resolve either with await or .then(). Therefore, when you try to log the value, you just get the promise. This function will be returning the promise so a caller can get the value by using .then() or await on that promise.
Await requires proper promise implementation
I will also mention that in order for await to work properly for your callers, your function must return a promise that resolves or rejects when the asynchronous operation is complete and any resolved value or rejected error is in that promise also. So, just slapping an await on a function that isn't returning a promise that is properly wired into the asynchronous operations will not work. The promise must be correctly implemented for await to do its job.
Use a promise-supporting version of mysql
Since it appears you're using mysql, I'd strong suggest you get the module that implements a promise interface for the database so you can do all your asynchronous coding with promises, not plain callbacks.
The link you had in your question to a 2018 article shows lots of mysql examples using plain callbacks. I would not recommend starting a new project using plain callbacks for asynchronous operations. You will quickly learn to appreciate the advantages of using promise-based interfaces once you start doing that and promises are the modern way of handling asynchronous operations in Javascript.
You must catch all promise rejections
P.S. Any promise you use MUST have a .catch() or an await with a try/catch somewhere in the call chain that will catch a potential rejected promise. So, all your express request handlers that are calling your asynchronous APIs are missing that proper error handling. Without that, you will get an uncaught promise rejection. At best, the request handler will just never send a response. At worst, your server will shut-down. All possible rejected promises must be caught somewhere in your code.
So, something like this:
app.get("/org", async (req, res) => {
let rows = await getRows("select * from org");
// the log always prints a JSON array.
console.log("in app.get for /org, rows: ", rows);
res.send(rows);
});
should be something like this:
app.get("/org", async (req, res) => {
try {
let rows = await getRows("select * from org");
// the log always prints a JSON array.
console.log("in app.get for /org, rows: ", rows);
res.send(rows);
} catch(e) {
console.log(e);
res.sendStatus(500);
}
});
Or, if you want to handle all errors centrally:
app.get("/org", async (req, res) => {
try {
let rows = await getRows("select * from org");
// the log always prints a JSON array.
console.log("in app.get for /org, rows: ", rows);
res.send(rows);
} catch(e) {
console.log(e);
next(e); // forward to central Express error handler
}
});
If you want to catch the error in your business logic so you can do something with it, but still eventually return an error back to your controller functions, then you can catch and rethrow:
async function getRows(params) {
try {
let rows = await pool.query("select * from org", params);
console.log("in getRows, rows: ", rows);
return rows;
} catch(e) {
// catch the error, log it and do whatever else you want with it
// either return a value or rethrow some error
// usually, you would throw some error here
// so a failed operation rejects the promise
// this function returns
console.log(e);
// you can throw a different error if you want to invent
// your own specific errors, rather than pass back db errors
throw e;
}
}

How to get value from db.any() promise in pg-promise?

I use pg-promise and i tried to make a request to database in user model to get an array of all users like this:
exports.getAllUsers = function () {
let users = db.any("SELECT * FROM users;").then(function (data) {
return data; // in debug mode data is my expected array
});
return users;
};
But when i make get request from controller, it returns promise Promise { <pending> }, not an array of json objects that I expected.
There is a controller:
exports.getUsers = function (req, res) {
let users = userModel.getAllUsers();
console.log(users); // here it outputs Promise { <pending> }
res.json(users);
};
How to get array instead of promise?

Let's break down your code. You are exporting a function of your design, getUsers, which is supposed to return a sequence of user objects. But it does not -- db.any(...) returns a Promise. How else do you think you can call then on the returned object? then is a member function of the Promise class, and you pass another function to then to work with the actual data (passed as parameter to the function you pass to then). Meaning that indeed data, as passed to the then call invoked on the return value of db.any(...) call, is the sequence of users you're after.
The mistake you are making is to assume that if you return data from the callback passed to then, it will become the return value of the then(...) call. It will not. then always returns a Promise -- whatever you return from the callback to then will become the resolved value of the returned promise, but it is the promise that is returned and that is why your users variable is a Promise.
You need to read more about promises, and how they are resolved asynchronously (between running your scripts), and how to synchronise your code on them resolving.
Hint: Use the await keyword to wait for and use the value given a promise, like the value you return from the callback that you pass to your then(...) call, or accept the design of your getUsers function returning a promise and adapt the rest of your code to work with that, without use of await.
Your getAllUsers function can be reduced to:
exports.getAllUsers = function () {
return db.any("SELECT * FROM users;");
}
...and with await you can then use it like:
let users = await getUsers();
The statement above has to be part of a function that is tagged as async though -- ECMAScript requires marking these explicitly in order to allow use of await expressions:
async function whatever() {
let users = await getUsers();
/// ...
}
Execution of a script that invokes an async function such as one containing the statement above, will be interrupted by the JavaScript interpreter at each await expression, and resumed when the promise expressed after the await keyword, resolves to a value, with the value being assigned to the users variable.
Using async and await does require a version of Node.js that can support them, but current release does (as of the time of writing this).
Otherwise you can still keep the getUsers function, but you need to use it differently since you can't use await -- having to use promises as one did prior to introduction of await and async, like:
getUsers().then(users => {
/// Do something with the `users` array.
});
At any rate, it looks like you have some holes in your understanding of how promises work, and I recommend you fill these holes by reading about them this time and not just going straight through to the pg-promise API, which builds on them.

Like you already found out userModel.getAllUsers() will return a promise and not an array. You need to wait for this promise to be resolved. This can be done using the an async function.
exports.getUsers = async function (req, res) {
let users = await userModel.getAllUsers();
console.log(users);
res.json(users);
};
Note that an async function will always return a promise.
Or you can make use of then method of the returned promise.
exports.getUsers = function (req, res) {
userModel.getAllUsers().then(users => {
console.log(users);
res.json(users);
});
};

Async function not returning value, but console.log() does: how to do? [duplicate]

This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 5 years ago.
I have an es6 class, with an init() method responsible for fetching data, transforming it, then update the class's property this.data with newly transformed data.
So far so good.
The class itself has another getPostById() method, to just do what it sounds like. Here is the code for the class:
class Posts {
constructor(url) {
this.ready = false
this.data = {}
this.url = url
}
async init() {
try {
let res = await fetch( this.url )
if (res.ok) {
let data = await res.json()
// Do bunch of transformation stuff here
this.data = data
this.ready = true
return data
}
}
catch (e) {
console.log(e)
}
}
getPostById(id){
return this.data.find( p => p.id === id )
}
}
Straightforward, except I have an async/await mechanism in the init() method.
Now, this code will work correctly:
let allPosts = new Posts('https://jsonplaceholder.typicode.com/posts')
allPosts.init()
.then( d => console.log(allPosts.getPostById(4)) )
// resulting Object correctly logged in console
but it only gets printed into the console:
How could I use allPosts.getPostById(4) as a return of a function ?
Like:
let myFunc = async () => {
const postId = 4
await allPosts.init() // I need to wait for this to finish before returning
// This is logging correct value
console.log( 'logging: ' + JSON.stringify(allPosts.getPostById( postId ), null, 4) )
// How can I return the RESULT of allPosts.getPostById( postId ) ???
return allPosts.getPostById( postId )
}
myFunc() returns a Promise but not the final value. I have read several related posts on the subject but they all give example of logging, never returning.
Here is a fiddle that includes two ways of handling init(): using Promise and using async/await. No matter what I try, I can't manage to USE the FINAL VALUE of getPostById(id).
The question of this post is: how can I create a function that will RETURN the VALUE of getPostById(id) ?
EDIT:
A lot of good answers trying to explain what Promises are in regards to the main execution loop.
After a lot of videos and other good reads, here is what I understand now:
my function init() correctly returns. However, within the main event loop: it returns a Promise, then it is my job to catch the result of this Promise from within a kinda parallel loop (not a new real thread). In order to catch the result from the parallel loop there are two ways:
use .then( value => doSomethingWithMy(value) )
use let value = await myAsyncFn(). Now here is the foolish hiccup:
await can only be used within an async function :p
thus itself returning a Promise, usable with await which should be embed in an async function, which will be usable with await etc...
This means we cannot really WAIT for a Promise: instead we should catch parallel loop indefinitely: using .then() or async/await.
Thanks for the help !

As for your comment; I'll add it as answer.
The code you write in JavaScript is run on one thread, that means that if your code could actually wait for something it will block any of your other code from getting executed. The event loop of JavaScript is explained very well in this video and if you like to read in this page.
A good example of blocking code in the browser is alert("cannot do anything until you click ok");. Alert blocks everything, the user can't even scroll or click on anything in the page and your code also blocks from executing.
Promise.resolve(22)
.then(x=>alert("blocking")||"Hello World")
.then(
x=>console.log(
"does not resolve untill you click ok on the alert:",
x
)
);
Run that in a console and you see what I mean by blocking.
This creates a problem when you want to do something that takes time. In other frameworks you'd use a thread or processes but there is no such thing in JavaScript (technically there is with web worker and fork in node but that's another story and usually far more complicated than using async api's).
So when you want to make a http request you can use fetch but fetch takes some time to finish and your function should not block (has to return something as fast as possible). This is why fetch returns a promise.
Note that fetch is implemented by browser/node and does run in another thread, only code you write runs in one thread so starting a lot of promises that only run code you write will not speed up anything but calling native async api's in parallel will.
Before promises async code used callbacks or would return an observable object (like XmlHttpRequest) but let's cover promises since you can convert the more traditional code to a promise anyway.
A promise is an object that has a then function (and a bunch of stuff that is sugar for then but does the same), this function takes 2 parameters.
Resolve handler: A function that will be called by the promise when the promise resolves (has no errors and is finished). The function will be passed one argument with the resolve value (for http requests this usually is the response).
Reject handler: A function that will be called by the promise when the promise rejects (has an error). This function will be passed one argument, this is usually the error or reason for rejection (can be a string, number or anything).
Converting callback to promise.
The traditional api's (especially nodejs api's) use callbacks:
traditionalApi(
arg
,function callback(err,value){
err ? handleFail(err) : processValue(value);
}
);
This makes it difficult for the programmer to catch errors or handle the return value in a linear way (from top to bottom). It gets even more impossible to try and do things parallel or throttled parallel with error handling (impossible to read).
You can convert traditional api's to promises with new Promise
const apiAsPromise = arg =>
new Promise(
(resolve,reject)=>
traditionalApi(
arg,
(err,val) => (err) ? reject(err) : resolve(val)
)
)
async await
This is what's called syntax sugar for promises. It makes promise consuming functions look more traditional and easier to read. That is if you like to write traditional code, I would argue that composing small functions is much easier to read. For example, can you guess what this does?:
const handleSearch = search =>
compose([
showLoading,
makeSearchRequest,
processRespose,
hideLoading
])(search)
.then(
undefined,//don't care about the resolve
compose([
showError,
hideLoading
])
);
Anayway; enough ranting. The important part is to understand that async await doesn't actually start another thread, async functions always return a promise and await doesn't actually block or wait. It's syntax sugar for someFn().then(result=>...,error=>...) and looks like:
async someMethod = () =>
//syntax sugar for:
//return someFn().then(result=>...,error=>...)
try{
const result = await someFn();
...
}catch(error){
...
}
}
The examples allways show try catch but you don't need to do that, for example:
var alwaysReject = async () => { throw "Always returns rejected promise"; };
alwaysReject()
.then(
x=>console.log("never happens, doesn't resolve")
,err=>console.warn("got rejected:",err)
);
Any error thrown or await returning a rejected promise will cause the async function to return a rejected promise (unless you try and catch it). Many times it is desirable to just let it fail and have the caller handle errors.
Catching errors could be needed when you want the promise to succeed with a special value for rejected promises so you can handle it later but the promise does not technically reject so will always resolve.
An example is Promise.all, this takes an array of promises and returns a new promise that resolves to an array of resolved values or reject when any one of them rejects. You may just want to get the results of all promises back and filter out the rejected ones:
const Fail = function(details){this.details=details;},
isFail = item => (item && item.constructor)===Fail;
Promise.all(
urls.map(//map array of urls to array of promises that don't reject
url =>
fetch(url)
.then(
undefined,//do not handle resolve yet
//when you handle the reject this ".then" will return
// a promise that RESOLVES to the value returned below (new Fail([url,err]))
err=>new Fail([url,err])
)
)
)
.then(
responses => {
console.log("failed requests:");
console.log(
responses.filter(//only Fail type
isFail
)
);
console.log("resolved requests:");
console.log(
responses.filter(//anything not Fail type
response=>!isFail(response)
)
);
}
);

Your question and the comments suggest you could use a little intuition nudge about the way the event loop works. It really is confusing at first, but after a while it becomes second nature.
Rather than thinking about the FINAL VALUE, think about the fact that you have a single thread and you can't stop it — so you want the FUTURE VALUE -- the value on the next or some future event loop. Everything you write that is not asynchronous is going to happen almost immediately — functions return with some value or undefined immediately. There's nothing you can do about. When you need something asynchronously, you need to setup a system that is ready to deal with the async values when they return sometime in the future. This is what events, callbacks, promises (and async/await) all try to help with. If some data is asynchronous, you simply can not use it in the same event loop.
So what do you do?
If you want a pattern where you create an instance, call init() and then some function that further process it, you simply need to setup a system that does the processing when the data arrives. There are a lot of ways to do this. Here's one way that's a variation on your class:
function someAsync() {
console.log("someAsync called")
return new Promise(resolve => {
setTimeout(() => resolve(Math.random()), 1000)
})
}
class Posts {
constructor(url) {
this.ready = false
this.data = "uninitilized"
this.url = url
}
init() {
this.data = someAsync()
}
time100() {
// it's important to return the promise here
return this.data.then(d => d * 100)
}
}
let p = new Posts()
p.init()
processData(p)
// called twice to illustrate point
processData(p)
async function processData(posts) {
let p = await posts.time100()
console.log("randomin * 100:", p)
}
init() saves the promise returned from someAsync(). someAsync() could be anything that returns a promise. It saves the promise in an instance property. Now you can call then() or use async/await to get the value. It will either immediately return the value if the promise has already resolved or it will deal with it when it has resolved. I called processData(p) twice just to illustrate that it doesn't calle the someAsync() twice.
That's just one pattern. There are a lot more — using events, observables, just using then() directly, or even callbacks which are unfashionable, but still can be useful.

NOTE: Wherever you use await it has to be inside an async function.
Check out the UPDATED FIDDLE
You need to use await myFunc() to get the value you expect from getPostById because an async function always returns a promise.
This sometimes is very frustrating as the whole chain needs to be converted into async functions but that's the price you pay for converting it to a synchronous code, I guess. I am not sure if that can be avoided but am interested in hearing from people who have more experience on this.
Try out the below code in your console by copying over the functions and then accessing final and await final.
NOTE:
An async function CAN contain an await expression.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
There is no rule that is must have await in order to even declare an async function.
The example below uses an async function without await just to show that an async function always returns a promise.
const sample = async () => {
return 100;
}
// sample() WILL RETURN A PROMISE AND NOT 100
// await sample() WILL RETURN 100
const init = async (num) => {
return new Promise((resolve, reject) => {
resolve(num);
});
}
const myFunc = async (num) => {
const k = await init(num);
return k;
}
// const final = myFunc();
// final; This returns a promise
// await final; This returns the number you provided to myFunc

What is the correct syntax for a method declared as async?

I stumbled upon the following ES6 method during a rewiew:
async getUsers(res) {
User.findAll().then(users => res.json(users));
}
This seems buggy to me. I would have expected the following:
async getUsers(res) {
return User.findAll().then(users => res.json(users));
}
But then, I know that async has many tricks such as auto-wrapping promises, so is my intuition correct?

The code as it is written (without the return) is likely fine as long as the caller does not need access to the asynchronously retrieved value.
First off, async is needed as part of your function definition ONLY if you plan on using await inside the function. If you aren't using await inside (which your code does not show), then the async part of the definition is not needed. So, it could probably be just this:
getUsers(res) {
User.findAll().then(users => res.json(users));
}
Second, if a caller wants to use await on the getUsers() function, then it needs to return a promise (as you proposed). If the caller will not use await with it or otherwise need any access to a return value, then the return is not necessary. So, if you wanted to use await on this function, then it should probably be this:
getUsers(res) {
// return promise so caller can get async value
return User.findAll().then(users => {
res.json(users);
return users; // return value so caller can get access to the value
});
}
So, the original code may be just fine as long as the caller is never expecting an asynchronous return value from that function (which is possible here because the result of the async operation is being sent with res.json() and that may be the only result that is needed. As far as await/async are concerned, you only have to return a promise from functions that you plan to use await on.
To recap:
async is only needed if you plan on using await inside the function. Your function probably doesn't even need the async keyword on it at all.
A returned promise is only needed if the caller wants to be able to get access to the asynchronous response either via a traditional f().then() or via let x = await f().
A caller can await a function that is not marked as async. The await will get the return value from the function and, if the return value is not a promise it will wrap it in a resolved promise. If it is a promise, it will wait for it to resolve.
FYI, this is a very good, short and sweet review of async/await and how they interact with promises.

When using the await?

I'm using sequelize with typescript. I know that the code is asynchronous, and
Here, I am using promise, and the code works..
I would want to know when I must to use await keyword ?
const promises = []
let tabIdDoc = requestedListIdDoc.toString().split(",")
for (let thisIdDoc of tabIdDoc) {
promises.push(sequelize.models['Document'].findById(thisIdDoc))
}
q.all(promises).then((resultReq) => {
const lstDocs = []
for (let MyDoc of resultReq) {
if (MyDoc.matriculeDoc != "") {
lstDocs.push(sequelize.models['FolderDocContent'].findOrCreate({
where: {
}
}))
}
}
q.all(lstDocs).then(() => {
return response.status(201)
})
}
Is await keyword necessary here ?

You don't ever have to use await as other programming using .then() can always get the job done, but there are numerous times when using await makes your code simpler. This is particularly true when you are trying to sequence a number of asynchronous operations and even more so when you need to use previous results in more than one operation that follows.
Example #1: Serializing operations in a for loop
Suppose you want to save a bunch of items to your database, but for various reasons, you need to save them one by one and you need to save them in order (in other words, you need to sequence them):
async function saveItems(shoppingList) {
for (let item of shoppingList) {
// for loop will pause until this promise resolves
await db.save(item);
}
}
saveItems(myList).then(() => {
// all done
}).catch(err => {
// error here
});
Without using await, you'd have to use a significantly more verbose design pattern using .reduce() or perhaps a recursive function that you call on the completion of the prior operation. This is a lot simpler way to sequence iteration of a loop.
Example #2: Sequencing different operations in a function
Suppose, you need to contact three different outside services. You need to get some data from one, then use that data when you make a second API call, then use both of those pieces of data in a third API call:
const rp = require('request-promise');
async function getPrice(productName) {
// look up productID
const productID = await rp(`http://service1.com/api/getID?name=${productName}`);
// use productID to lookup price
const productPrice = await rp(`http://service1.com/api/getPrice?id=${productID}`);
// put both productID and price into the shopping cart
return rp({uri: 'http://service1.com/api/addToCart', body: {name: productName, id: productID}, json: true);
}
getPrice("Nikon_d750").then(total => {
// all done here, total is new cart total
}).catch(err => {
// error here
});
Both of these examples would require more code in order to properly sequence the asynchronous operations and you'd have to nest logic following inside a .then() handler. Using await instructs the JS interpreter to do that nesting for you automatically.
Some other examples here on MDN.
Several rules to keep in mind with await:
You can only use it inside a function prefixed with the async keyword as in my examples above.
All async functions return a promise. If you have an explicit return value in the function as in return x, then that value becomes the resolved value of the promise when all the asynchronous operations are done. Otherwise, it just returns a promise that has an undefined resolved value. So, to use a result from an async function or to know when it's done, you either have to await the result of that function (inside another async function) or you have to use .then() on it.
await only suspends execution within the containing function. It is as if the rest of the code in the function is placed inside an invisible .then() handler and the function will still return its promise immediately when it hits the first await. It does not block the event loop or block other execution outside the async function. This confuses many people when they first encounter await.
If the promise that you await rejects, then it throws an exception within your function. That exception can be caught with try/catch. If it is not caught, then the async function automatically catches it and rejects the promise that the function returns where the thrown value is the reject reason. It is easy when first using await to completely forget about the error cases when promises you are awaiting can reject.

You never necessarily must use await. You however should use await in every place where you'd otherwise have then with a callback, to simplify your code.
In your example:
const promises = requestedListIdDoc.toString().split(",").map(thisIdDoc =>
sequelize.models['Document'].findById(thisIdDoc)
);
const resultReq = await q.all(promises);
const lstDocs = resultReq.filter(myDoc => MyDoc.matriculeDoc != "").map(myDoc =>
sequelize.models['FolderDocContent'].findOrCreate({
where: {}
})
);
await q.all(lstDocs);
return response.status(201)