I have a very simple code I'm trying to debug. I have an async function:
async function updateResult(event){
let result = db.fetchResult(event.ProcessId);
return result;
}
And I'm calling this from another simple async function:
exports.processEvent = async (event) => {
try {
let promises = [];
const controls = await db.fetchControls(event.EventCode);
if (controls) {
promises = controls.map((control) => {
console.log(control);
control.map(x => this.updateResult(event));
});
}
return Promise.allSettled(promises);
}
catch (err) {
console.error(err);
}
};
The problem is that in exports.processEvent, the content of db.fetchControls(event.EventCode) is executed to completion (a few logics and a db getItem call).
But the call to db.fetchResult(event.ProcessId) via this.updateResult(x.Id, x.Version, event) does NOT complete executing the fetchResult tasks (a few other logics a db GET call).
I feel like fetchResult returns prematurely. What am I doing wrong?
.map() is NOT promise-aware. It does not wait for any of the promises in your callback to complete. So, in this piece of code:
control.map(x => this.updateResult(x.Id, x.Version, event));
It just runs through the entire array ignoring all the promises that are returned and thus NOTHING waits for those promises.
You can fix that by returning those promises to the higher level .map() which you are then awaiting with Promise.all():
promises = controls.map((control) => {
console.log(control);
return Promise.all(control.map(x => this.updateResult(x.Id, x.Version, event)));
});
Related
in an async function i, want to main function to return only in a specific point and not to return void at the end of the function
const fun = () => {
const list = [];
let streamFinished = 0;
let streamCount = files.length;
await fs.readdir(JSON_DIR, async(err, files) => {
await files.forEach((filename) => {
const readStream = fs.createReadStream(path.join("directory", filename));
const parseStream = json.createParseStream();
await parseStream.on('data', async(hostlist: Host[]) => {
hostlist.forEach(async host => {
list.push(host);
});
});
parseStream.on('end', () => {
streamFinished++;
if (streamFinished === streamCount) {
// End of all streams...
return list; //this should be the only point when the function return something
}
})
readStream.pipe(parseStream);
})
});
};
There are lots of things wrong with this code. It's a big mix of event-driven streams, callback-driven functions and promises. The first order of business is to make the main business logic just be promise-driven (as you can see in the new parseData() function and switch all control flow outside of that to just use promises. Here's one way to do it:
const fsp = fs.promises;
function parseData(filename) {
return new Promise((resolve, reject) => {
const readStream = fs.createReadStream(path.join("directory", filename));
const parseStream = json.createParseStream();
const list = [];
parseStream.on('data', (hostlist: Host[]) => {
list.push(...hostlist);
}).on('end', () => {
resolve(list);
}).on('error', reject);
readStream.pipe(parseStream);
});
}
const fun = async () => {
const list = [];
const files = await fsp.readdir(JSON_DIR);
for (let filename of files) {
const listData = await parseData(filename);
list.push(...listData);
}
return list;
};
fun().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
A few thoughts here:
The easiest way to "wait" for a stream operation to complete is to encapsulate it in a promise which you can then use await or .then() with. That's the purpose of the parseData() function. In addition, this also enables error handling by hooking stream errors to the promise rejection.
For processing a loop of asynchronous operations, you can either do them one at a time, using await on each asynchronous operation in the loop or you can run them in parallel by collecting an array of promises and using let data = await Promise.all(arrayOfPromises); on that array of promises.
It is only useful to use await if the thing you're awaiting is a promise that is connected to your asynchronous operation. So, things like await parseStream.on('data', ...) and await files.forEach(...) are pointless because neither of those return promises. Do NOT just stick await in places unless you KNOW you are awaiting a promise.
You will generally NOT want to use .forEach() with an asynchronous operation in the loop. Because .forEach() has no return value and no loop control, you can't really control much. Use a regular for loop instead which gives you full control. I consider .forEach() pretty much obsolete for asynchronous programming.
Don't mix promises and callbacks and don't mix promises and streams. If you have those, then "promisify" the stream or callback so all your main logic/control flow can be promises. This will vastly simplify your code and make error handling possible/practical.
In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.
So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.
Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);
I have config like JSON where we can define any JavaScript functions inside. Now I have execution function which would take that array of functions and execute. How can I do that?
const listOfFuncs = [
{
"func1": (...args) => console.log(...args)
},
{
"func2": async (...args) => {
return await fetch('some_url');
}
}
]
function execute() {
// how to execute the above array of functions now ?
}
// Should this be called as await execute()?
execute();
As you can see one function sync & another function as async & await. Defining everything function as async & await seems bad ( creating a lot of new promises ) + I can't define all function as synchronous also.
Thanks for your answer in advance.
You can use Promise.all() to resolve an array of promises.
Values other than promises will be returned as-is
const listOfFuncs = [
() => 45,
async () => new Promise(resolve => {
setTimeout(() => resolve(54), 100);
}),
() => Promise.resolve(34)
];
// Convert to list of promises and values
const listOfMixedPromisesAndValues = listOfFuncs.map(func => func());
// Promise.all returns a Promise which resolves to an array
// containing all results in the same order.
Promise.all(listOfMixedPromisesAndValues).then(result => {
// Logs: [45, 54, 34] after 100ms
console.log(result)
});
There is no native function to resolve an object containing promises.
However some libraries implement alternative Promise API, to make it easier to use more complex pattern (cancellation, races, ...). The most well known is Bluebird.
It implements a Promise.props methods which almost do what you want: http://bluebirdjs.com/docs/api/promise.props.html
var Promise = require("bluebird");
Promise.props({
pictures: getPictures(),
comments: getComments(),
tweets: getTweets()
}).then(function(result) {
console.log(result.tweets, result.pictures, result.comments);
});
You should simply treat all your functions as potentially returning a promise. No need to distinguish them, just await the result. The execution will carry on immediately if it was not a thenable (read: non-promise) value. So just write
async function execute() {
for (const func of listOfFuncs) {
await func();
}
}
If you want the asynchronous tasks to run concurrently, just collect all values into an array and pass them to Promise.all. It deals with non-promise elements just fine as well.
async function execute() {
await Promise.all(listOfFuncs.map(func => func()));
}
Solution without promises.
Use process.nextTick(callback) or setImmediate
const listOfFuncs = [
{
"func3": setImmediate(execute, "setImmadiate executes after process.nextTick")
},
{
"func2": process.nextTick(execute, "process.nextTick executes after sync function but before setImmediate")
},
{
"func1": execute
}
]
function execute() {
console.log("executing...", arguments[0]);
}
execute(listOfFuncs);
// results:
// executing...[...]
// executing...process.tick
// executing...setImmediate...
In this example, you create an array of the executed functions and use the Promise.all() with a map to get all promisses if the function results. (in a case of an async function the function returns a promise-value, which you can await)
function isPromise(promise) {
return !!promise && typeof promise.then === 'function'
}
let functions = [
(() => {})(),
(async () => {})()
];
await Promise.all(functions.map(function_result => (isPromise(function_result) ? function_result : undefined))
Maybe this is a solution for you:
await Promise.all([
(async () => {
//Your Function here
})(),
]);
Ideally having everything async would be cleaner. But if not, this would work :
async function execute(...args) {
for (let i = 0; i < listOfFuncs.length; i++) {
if (listOfFuncs[i].constructor.name === "AsyncFunction") {
await listOfFuncs[i](...args);
} else {
listOfFuncs[i](...args);
}
}
}
Imagine the following hypothetical, minimal implementation of a function that does a simple HTTP GET request using axios. This uses await/async as depicted in the post.
const axios = require('axios')
exports.getStatus = async id => {
const resp = await axios.get(
`https://api.example.com/status/${id}`
)
return resp
}
Is the promise not resolved using await? Is it required that the client uses await as depicted below? Is it safe to assume that anytime a client consumes an async function, that it also needs to use await when calling the function?
// client
const { getStatus } = require('./status')
const response = await getStatus(4)
Short answer, no.
Labelling a function async means 2 things:
1) you're allowed to use await inside the function.
2) The function returns a promise; i.e. you can use await on its return value, or you can use .then, or Promise.all, or even ignore the return value, or anything else you can do with promises.
E.g. you could do the following which (depending on your use case) could be more performant because it needs not wait for the response to continue.
// client
const { getStatus } = require('./status')
const response4 = getStatus(4);
const response5 = getStatus(5);
// do stuff that don't rely on the responses.
response4.then(response => myOutput.setFirstOutput(response));
response5.then(response => myOutput.setSecondOutput(response));
Btw, your first snippet is redundant with its usage of await; it's equivalent to
const axios = require('axios')
exports.getStatus = id =>
axios.get(`https://api.example.com/status/${id}`);
This is because return await promise is equivalent to return promise, both return the same promise. There is one exception regarding rejected promises. Awaiting a rejected promise will throw an exception (which you can catch with a try/catch block), whereas directly returning a rejected promise will not throw an exception but you should still handle the rejection (with a .catch clause). To illustrate:
let promise = new Promise(resolve => setTimeout(() => resolve('resolved after 2 seconds'), 2000));
let returnPromise = () => promise;
let returnAwaitPromise = async () => await promise;
returnPromise().then(value => console.log('returnPromise,', value));
returnAwaitPromise().then(value => console.log('returnAwaitPromise,', value));
No, you don't need to use await. An async function returns a Promise that should be resolved. Instead of using await, you can simply call .then() like so:
// client
const { getStatus } = require('./status');
getStatus(4).then((response) => {
//Do something with the response
}).catch((error) => {
//Handle possible exceptions
});
Below I'll address your question from the comments. The getStatus function could be rewritten like this:
exports.getStatus = (id) => {
return new Promise((resolve, reject) => {
axios.get(`https://api.example.com/status/${id}`).then((resp) => {
resolve(resp);
}).catch((error) => {
reject(error);
})
});
}
It would act exactly the same as in your version. When you call asynchronous functions in series, you have to resolve each of the returned Promises. In your case, there are two of them - the one returned by axios.get and the one returned by getStatus.
No you don’t have to. The async function executes and waits until it returns error or completes execution and return a response!
This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 3 years ago.
In the code below, I expect mark-1 to fire before mark-2. I'm guessing that I'm not using await and async correctly some where.
I'm pretty sure that this line:
const things = await fs.promises.readdir(folder);
and this line
const stats = await fs.promises.stat(path);
are correct as we are awaiting the file system to respond.
I'm not conerned with error checking or cross-platform code as of yet, just getting the promises to work correctly
// Libraries
const fs = require('fs');
// API
getThings('_t/').then(() => {
console.log('mark-2')
})
// Accesses the file system and returns an array of files / folders
async function getThings (folder) {
const things = await fs.promises.readdir(folder);
things.forEach(async (thing)=>{
await getStats(thing, folder);
});
}
// Gets statistics for each file/folder
async function getStats (thing, folder) {
const path = folder + thing;
const stats = await fs.promises.stat(path);
console.log('mark-1');
}
The problem is that you are using async and await in the forEach call, and this doesn't work as you would expect.
The forEach method doesn't really care about the return value of the callback function (in this case the promise that getStats returns).
You should map the things array to promises, and use Promise.all:
async function getThings (folder) {
const things = await fs.promises.readdir(folder);
const promises = things.map(thing => getStats(thing, folder));
return await Promise.all(promises);
}
Note that this will execute the promises "in parallel", and not sequentially.
If you want to execute the promises secuentially, one by one, you can either, "reduce" the promise array, or use a conventional loop (for, for-of).
EDIT:
Let me try to clarify why using an async callback function with forEach doesn't work.
Attempt #1: De-sugarizing it:
In the original example we have something like this:
// ...
things.forEach(async (thing)=>{
await getStats(thing, folder);
});
// ...
If we separate the callback from the forEach, we have:
const callback = async (thing)=>{
await getStats(thing, folder);
};
things.forEach(callback);
If we "desugarize" the async function:
const callback = function (thing) {
return new Promise((resolve, reject) => {
try {
getStats(thing, folder).then(stat => /*do nothing*/);
} catch (err) {
reject(err);
}
resolve();
});
};
things.forEach(callback);
Marking a function with async ensures the function will return always return a promise, regardless its completion, if the function reaches its execution without a explicit return value, the promise will be resolved with undefined, if it returns some value, the promise will resolve to it, and finally if something within the function throws, the promise will be rejected.
As you can see the problem is that the promises are just not being awaited by anything, and also they are not resolving to any value. The await placed in the callback does actually nothing with the value, just as in the above I'm doing .then and doing nothing with the value.
Attempt 2: Implementing a simple forEach function:
function forEach(array, callback) {
for(const value of array) {
callback(value);
}
}
const values = [ 'a', 'b', 'c' ];
forEach(values, (item) => {
console.log(item)
});
The above forEach is an oversimplification of the Array.prototype.forEach method, just to show its structure, in reality the callback function is called passing the array as the this value, and passing three arguments, the current element, the current index, and again the array instance, but we get the idea.
If we would like to implement an async forEach function, we would have to await the callback call:
const sleep = (time, value) => new Promise(resolve => setTimeout(resolve(value), time));
const values = [ { time: 300, value: 'a'}, { time: 200, value: 'b' }, {time: 100, value: 'c' } ];
async function forEachAsync(array, callback) {
for(const value of array) {
await callback(value);
}
}
(async () => {
await forEachAsync(values, async (item) => {
console.log(await sleep(item.time, item.value))
});
console.log('done');
})()
The above forEachAsync function will iterate and await item by item, sequentially, normally you don't want that, if the async functions are independent, they can be done in parallel, just as I suggested in first place.
const sleep = (time, value) => new Promise(resolve => setTimeout(resolve(value), time));
const values = [ { time: 300, value: 'a'}, { time: 200, value: 'b' }, {time: 100, value: 'c' } ];
(async () => {
const promises = values.map(item => sleep(item.time, item.value));
const result = await Promise.all(promises);
console.log(result);
})()
And as you can see, even if the promises are executed in parallel, we get the results in the same order as the promises where in the array.
But the difference between this example and the first one is that this one takes only 300ms (the longest promise to resolve), and the first one takes 600ms (300ms + 200ms + 100ms).
Hope it makes it clearer.