This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 3 years ago.
In the code below, I expect mark-1 to fire before mark-2. I'm guessing that I'm not using await and async correctly some where.
I'm pretty sure that this line:
const things = await fs.promises.readdir(folder);
and this line
const stats = await fs.promises.stat(path);
are correct as we are awaiting the file system to respond.
I'm not conerned with error checking or cross-platform code as of yet, just getting the promises to work correctly
// Libraries
const fs = require('fs');
// API
getThings('_t/').then(() => {
console.log('mark-2')
})
// Accesses the file system and returns an array of files / folders
async function getThings (folder) {
const things = await fs.promises.readdir(folder);
things.forEach(async (thing)=>{
await getStats(thing, folder);
});
}
// Gets statistics for each file/folder
async function getStats (thing, folder) {
const path = folder + thing;
const stats = await fs.promises.stat(path);
console.log('mark-1');
}
The problem is that you are using async and await in the forEach call, and this doesn't work as you would expect.
The forEach method doesn't really care about the return value of the callback function (in this case the promise that getStats returns).
You should map the things array to promises, and use Promise.all:
async function getThings (folder) {
const things = await fs.promises.readdir(folder);
const promises = things.map(thing => getStats(thing, folder));
return await Promise.all(promises);
}
Note that this will execute the promises "in parallel", and not sequentially.
If you want to execute the promises secuentially, one by one, you can either, "reduce" the promise array, or use a conventional loop (for, for-of).
EDIT:
Let me try to clarify why using an async callback function with forEach doesn't work.
Attempt #1: De-sugarizing it:
In the original example we have something like this:
// ...
things.forEach(async (thing)=>{
await getStats(thing, folder);
});
// ...
If we separate the callback from the forEach, we have:
const callback = async (thing)=>{
await getStats(thing, folder);
};
things.forEach(callback);
If we "desugarize" the async function:
const callback = function (thing) {
return new Promise((resolve, reject) => {
try {
getStats(thing, folder).then(stat => /*do nothing*/);
} catch (err) {
reject(err);
}
resolve();
});
};
things.forEach(callback);
Marking a function with async ensures the function will return always return a promise, regardless its completion, if the function reaches its execution without a explicit return value, the promise will be resolved with undefined, if it returns some value, the promise will resolve to it, and finally if something within the function throws, the promise will be rejected.
As you can see the problem is that the promises are just not being awaited by anything, and also they are not resolving to any value. The await placed in the callback does actually nothing with the value, just as in the above I'm doing .then and doing nothing with the value.
Attempt 2: Implementing a simple forEach function:
function forEach(array, callback) {
for(const value of array) {
callback(value);
}
}
const values = [ 'a', 'b', 'c' ];
forEach(values, (item) => {
console.log(item)
});
The above forEach is an oversimplification of the Array.prototype.forEach method, just to show its structure, in reality the callback function is called passing the array as the this value, and passing three arguments, the current element, the current index, and again the array instance, but we get the idea.
If we would like to implement an async forEach function, we would have to await the callback call:
const sleep = (time, value) => new Promise(resolve => setTimeout(resolve(value), time));
const values = [ { time: 300, value: 'a'}, { time: 200, value: 'b' }, {time: 100, value: 'c' } ];
async function forEachAsync(array, callback) {
for(const value of array) {
await callback(value);
}
}
(async () => {
await forEachAsync(values, async (item) => {
console.log(await sleep(item.time, item.value))
});
console.log('done');
})()
The above forEachAsync function will iterate and await item by item, sequentially, normally you don't want that, if the async functions are independent, they can be done in parallel, just as I suggested in first place.
const sleep = (time, value) => new Promise(resolve => setTimeout(resolve(value), time));
const values = [ { time: 300, value: 'a'}, { time: 200, value: 'b' }, {time: 100, value: 'c' } ];
(async () => {
const promises = values.map(item => sleep(item.time, item.value));
const result = await Promise.all(promises);
console.log(result);
})()
And as you can see, even if the promises are executed in parallel, we get the results in the same order as the promises where in the array.
But the difference between this example and the first one is that this one takes only 300ms (the longest promise to resolve), and the first one takes 600ms (300ms + 200ms + 100ms).
Hope it makes it clearer.
Related
In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.
So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.
Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);
It seems I'm unable to use an async function as the first argument to Array.find(). I can't see why this code would not work what is happening under the hood?
function returnsPromise() {
return new Promise(resolve => resolve("done"));
}
async function findThing() {
const promiseReturn = await returnsPromise();
return promiseReturn;
}
async function run() {
const arr = [1, 2];
const found = await arr.find(async thing => {
const ret = await findThing();
console.log("runs once", thing);
return false;
});
console.log("doesn't wait");
}
run();
https://codesandbox.io/s/zk8ny3ol03
Simply put, find does not expect a promise to be returned, because it is not intended for asynchronous things. It loops through the array until one of the elements results in a truthy value being returned. An object, including a promise object, is truthy, and so the find stops on the first element.
If you want an asynchronous equivalent of find, you'll need to write it yourself. One consideration you'll want to have is whether you want to run things in parallel, or if you want to run them sequentially, blocking before you move on to the next index.
For example, here's a version that runs them all in parallel, and then once the promises are all resolved, it finds the first that yielded a truthy value.
async function findAsync(arr, asyncCallback) {
const promises = arr.map(asyncCallback);
const results = await Promise.all(promises);
const index = results.findIndex(result => result);
return arr[index];
}
//... to be used like:
findAsync(arr, async (thing) => {
const ret = await findThing();
return false;
})
Here is a TypeScript version that runs sequentially:
async function findAsyncSequential<T>(
array: T[],
predicate: (t: T) => Promise<boolean>,
): Promise<T | undefined> {
for (const t of array) {
if (await predicate(t)) {
return t;
}
}
return undefined;
}
It might help you to note that Array.prototype.filter is synchronous so it doesn't support async behaviour. I think that the same applies to the "find" property. You can always define your own async property :) Hope this helps!
The other answers provide two solutions to the problem:
Running the promises in parallel and awaiting all before returning the index
Running the promises sequencially, so awaiting every single promise before moving on to the next one.
Imagine you have five promises that finish at different times: The first after one second, the second after two seconds, etc... and the fifth after five seconds.
If I'm looking for the one that finished after three seconds:
The first solution will wait 5 seconds, until all promises are resolved. Then it looks for the one that matches.
The second one will evaluate the first three matches (1 + 2 + 3 = 6 seconds), before
returning.
Here's a third option that should usually be faster: Running the promises in parallel, but only waiting until the first match ("racing them"): 3 seconds.
function asyncFind(array, findFunction) {
return new Promise(resolve => {
let i = 0;
array.forEach(async item => {
if (await findFunction(await item)) {
resolve(item);
return;
}
i++;
if (array.length == i) {
resolve(undefined);
}
});
});
}
//can be used either when the array contains promises
var arr = [asyncFunction(), asyncFunction2()];
await asyncFind(arr, item => item == 3);
//or when the find function is async (or both)
var arr = [1, 2, 3];
await asyncFind(arr, async item => {
return await doSomething(item);
}
When looking for a non-existant item, solutions 1 and 3 will take the same amount of time (until all promises are evaluated, here 5 seconds). The sequencial approach (solution 2) would take 1+2+3+4+5 = 15 seconds.
Demo: https://jsfiddle.net/Bjoeni/w4ayh0bp
I came up with a solution that doesn't appear to be covered here, so I figured I'd share. I had the following requirements.
Accepts an async function (or a function that returns a promise)
Supplies item, index, and array to the function
Returns the item with the fastest resolved promise (all are evaluated in parallel)
Does not exit early if the async function rejects (throws an error)
Supports generic types (in TypeScript)
With those requirements, I came up with the following solution using Promise.any. It will resolve with the first fulfilled value or reject with an AggregateError if none of the promises are fulfilled.
type Predicate<T> = (item: T, index: number, items: T[]) => Promise<boolean>
export const any = async <T>(array: T[], predicate: Predicate<T>): Promise<T> => {
return Promise.any(
array.map(async (item, index, items) => {
if (await predicate(item, index, items)) {
return item
}
throw new Error()
})
)
}
Now we can search an array in parallel with async functions and return the fastest resolved result.
const things = [{ id: 0, ... }, { id: 1, ... }]
const found = await any(things, async (thing) => {
const otherThing = await getOtherThing()
return thing.id === otherThing.id
})
Enumerate the array sequentially
If we want to enumerate the array in sequence, as Array.find does, then we can do that with some modifications.
export const first = async <T>(array: T[], predicate: Predicate<T>): Promise<T> => {
for (const [index, item] of array.entries()) {
try {
if (await predicate(item, index, array)) {
return item
}
} catch {
// If we encounter an error, keep searching.
}
}
// If we do not find any matches, "reject" by raising an error.
throw new Error()
}
Now, we can search an array in sequence and return the first resolved result.
const things = [{ id: 0, ... }, { id: 1, ... }]
const found = await first(things, async (thing) => {
const otherThing = await getOtherThing()
return thing.id === otherThing.id
})
I have config like JSON where we can define any JavaScript functions inside. Now I have execution function which would take that array of functions and execute. How can I do that?
const listOfFuncs = [
{
"func1": (...args) => console.log(...args)
},
{
"func2": async (...args) => {
return await fetch('some_url');
}
}
]
function execute() {
// how to execute the above array of functions now ?
}
// Should this be called as await execute()?
execute();
As you can see one function sync & another function as async & await. Defining everything function as async & await seems bad ( creating a lot of new promises ) + I can't define all function as synchronous also.
Thanks for your answer in advance.
You can use Promise.all() to resolve an array of promises.
Values other than promises will be returned as-is
const listOfFuncs = [
() => 45,
async () => new Promise(resolve => {
setTimeout(() => resolve(54), 100);
}),
() => Promise.resolve(34)
];
// Convert to list of promises and values
const listOfMixedPromisesAndValues = listOfFuncs.map(func => func());
// Promise.all returns a Promise which resolves to an array
// containing all results in the same order.
Promise.all(listOfMixedPromisesAndValues).then(result => {
// Logs: [45, 54, 34] after 100ms
console.log(result)
});
There is no native function to resolve an object containing promises.
However some libraries implement alternative Promise API, to make it easier to use more complex pattern (cancellation, races, ...). The most well known is Bluebird.
It implements a Promise.props methods which almost do what you want: http://bluebirdjs.com/docs/api/promise.props.html
var Promise = require("bluebird");
Promise.props({
pictures: getPictures(),
comments: getComments(),
tweets: getTweets()
}).then(function(result) {
console.log(result.tweets, result.pictures, result.comments);
});
You should simply treat all your functions as potentially returning a promise. No need to distinguish them, just await the result. The execution will carry on immediately if it was not a thenable (read: non-promise) value. So just write
async function execute() {
for (const func of listOfFuncs) {
await func();
}
}
If you want the asynchronous tasks to run concurrently, just collect all values into an array and pass them to Promise.all. It deals with non-promise elements just fine as well.
async function execute() {
await Promise.all(listOfFuncs.map(func => func()));
}
Solution without promises.
Use process.nextTick(callback) or setImmediate
const listOfFuncs = [
{
"func3": setImmediate(execute, "setImmadiate executes after process.nextTick")
},
{
"func2": process.nextTick(execute, "process.nextTick executes after sync function but before setImmediate")
},
{
"func1": execute
}
]
function execute() {
console.log("executing...", arguments[0]);
}
execute(listOfFuncs);
// results:
// executing...[...]
// executing...process.tick
// executing...setImmediate...
In this example, you create an array of the executed functions and use the Promise.all() with a map to get all promisses if the function results. (in a case of an async function the function returns a promise-value, which you can await)
function isPromise(promise) {
return !!promise && typeof promise.then === 'function'
}
let functions = [
(() => {})(),
(async () => {})()
];
await Promise.all(functions.map(function_result => (isPromise(function_result) ? function_result : undefined))
Maybe this is a solution for you:
await Promise.all([
(async () => {
//Your Function here
})(),
]);
Ideally having everything async would be cleaner. But if not, this would work :
async function execute(...args) {
for (let i = 0; i < listOfFuncs.length; i++) {
if (listOfFuncs[i].constructor.name === "AsyncFunction") {
await listOfFuncs[i](...args);
} else {
listOfFuncs[i](...args);
}
}
}
I'm trying to figure out why promises seem to work differently inside of for...of vs. map().
data is an array of objects of the form {app: MY_APP, link: MY_LINK}. I'm trying to convert it to the form {app: MY_APP, status: true OR false}}. The function checkLink() is async because it uses node-fetch and basically returns whether the given link is 200. Using for...of, I get the desired object:
let objToSend = [];
for (obj of data) {
objToSend.push({ app: obj.app, status: await checkLink(obj.link) });
}
// returns [{app: MY_APP, status: true}, ...]
But using map, as follows, I get an array of pending promises instead. Any help would be appreciated:
let objToSend = data.map(async obj => {
return {
app: obj.app,
status: await checkLink(obj.link)
};
});
// returns [ Promise { <pending> }, ...]
I also tried doing Promise.all(objToSend) after the map code, but it instead returns Promise { <pending> }
I'll leave the explanation up to the other answers, but just want to point out that there is also an performance difference.
You first solution waits on every iteration for the promise to resolve. Only calling checkLink if the previous one has resolved. This is an sequential solution.
check link1 => wait => check link2 => wait => check link3 => wait
The second solution iterates over every elements and sends out requests, without waiting for promises to resolve (for this reason an array of promises is returned). If you wait for all promises to be resolved you find this solution is a lot faster, since the requests are send out in parallel.
check link1 => check link2 => check link3 => wait for all
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
async function checkLink(link) {
await sleep(Math.random() * 1000 + 500); // sleep between 500 and 1500 ms
console.log(link);
return `status #${link}`;
}
(async function () {
const data = new Array(5).fill().map((_, index) => ({ app: "app", link: index }));
let objToSend;
console.log("solution #1");
objToSend = [];
for (let obj of data) {
objToSend.push({ app: obj.app, status: await checkLink(obj.link) });
}
console.log(objToSend);
console.log("==============================================");
console.log("solution #2");
objToSend = await Promise.all(data.map(async obj => {
return {
app: obj.app,
status: await checkLink(obj.link)
};
}));
console.log(objToSend);
})();
In the snippet the first solution takes 500/1500 * 5 = 2500/7500 between 2500 and 7500 ms. While the second solution takes between 500 and 1500 ms (depending on the slowest value to resolve).
Async functions always returns Promises. In your map function, as the callback returns promises, an array of promises is being created.
To convert it to your format, use it with Promise.all():
(async()=>{
let objToSend = await Promise.all(data.map(async obj => {
return {
app: obj.app,
status: await checkLink(obj.link)
};
}));
console.log(objToSend) //[{app: MY_APP, status: true}, ...]
})()
The await always halts the execution of the async function it is in, it doesn't work differently in both cases. In the later example however, you do call some async functions, and those calls will evaluate to promises, whereas in your first example, all those awaits are in the same async function.
I also tried doing Promise.all(objToSend) after the map code, but it instead returns Promise { <pending> }
Yup, await that and you get an array of results.
I know that promise.all() fails when even 1 of the promise is failed. I only want to try for failed promises and don't want to run promise.all() again.
Any recommendations on how I can achieve this in minimal way?
Promises are eager construct and model a value obtained asynchronously,
a Promise is produced using some kind of producer, like fetch for instance.
If you retain a reference to this producer then you can replay the nmechanism
that produced the Promise in the first place.
// producer function
function getData (arg) {
const result = new Promise();
return result.then(value => {
return { ok:true, value };
}, error => {
return {
ok: false,
value: error,
// retry is a function which calls the producer with the same arguments
retry: () => getData(arg)
};
})
}
Then if you have something like:
const data = [];
// Promise<{ok: boolean, value: any, retry?: function}>
// No promises will fail in this array
const asyncResults = data.map(getResults);
Promise.all(asyncResults)
.then((results) => {
const successes = results.filter(res => res.ok);
const retrys = results.filter(res => !res.ok).map(res => res.retry()); // retry all failed promises
})
Memory leaks, stack overflow: because I retain a reference to original arguments in order to retry and the algorithm is recursive there could be a memory leak. However the algorithm cannot "stack overflow":
getData calls do not get "deeper" over time (see retry definition)
the asyncrhonicity of the algorithm prevent this behaviour if a promise was never resolved
old data is properly discarded when accessing the results as const resultData = results.filter(res => res.ok).map(res => res.value);
However the algorithm could take a long time to settle if a promise keep on not getting resolved and prevent access to the rest of the values.
In an alternative I suggest you take a look at another async primitive, not yet part of the language (maybe some day) : Observables which are designed for this kind of tasks: lazy, retry-able async operations.
You may use async package and wrap all promise calls with closure with done argument.
Then simply resolve results.
const async = require('async');
const promiseAll = promises => {
return new Promise((resolve) => {
// preparing array of functions which has done method as callback
const parallelCalls = promises.map(promise => {
return done => {
promise
.then(result => done(null, result)
.catch(error => {
console.error(error.message);
done();
});
}
});
// calling array of functions in parallel
async.parallel(
parallelCalls,
(_, results) => resolve(results.filter(Boolean))
);
});
};
router.get('/something', async (req, res) => {
...
const results = await promiseAll(promises);
...
});
Or we can simply do Promise.all without using async package:
router.get('/something', async (req, res) => {
...
const results = (
await Promise.all(
promises.map(promise => {
return new Promise(resolve => {
promise.then(resolve).catch(e => resolve());
});
});
)
).filter(Boolean);
...
});