How to map over async generators? - javascript

Let's say we have an async generator:
exports.asyncGen = async function* (items) {
for (const item of items) {
const result = await someAsyncFunc(item)
yield result;
}
}
is it possible to map over this generator? Essentially I want to do this:
const { asyncGen } = require('./asyncGen.js')
exports.process = async function (items) {
return asyncGen(items).map(item => {
//... do something
})
}
As of now .map fails to recognize async iterator.
The alternative is to use for await ... of but that's nowhere near elegant as with .map

The iterator methods proposal that would provide this method is still at stage 2 only. You can use some polyfill, or write your own map helper function though:
async function* map(asyncIterable, callback) {
let i = 0;
for await (const val of asyncIterable)
yield callback(val, i++);
}
exports.process = function(items) {
return map(asyncGen(items), item => {
//... do something
});
};

The alternative is to use for await ... of, but that's nowhere near elegant as with .map
For an elegant and efficient solution, here's one using iter-ops library:
import {pipe, map} from 'iter-ops';
const i = pipe(
asyncGen(), // your async generator result
map(value => /*map logic*/)
); //=> AsyncIterable
It is elegant, because the syntax is clean, simple, and applicable to any iterable or iterators, not just asynchronous generators.
It is more flexible and reusable, as you can add lots of other operators to the same pipeline.
Since it produces a standard JavaScript AsyncIterable, you can do:
for await(const a of i) {
console.log(a); //=> print values
}
P.S. I'm the author of iter-ops.

TL;DR - If the mapping function is async:
To make asyncIter not wait for each mapping before producing the next value, do
async function asyncIterMap(asyncIter, asyncFunc) {
const promises = [];
for await (const value of asyncIter) {
promises.push(asyncFunc(value))
}
return await Promise.all(promises)
}
// example - how to use:
const results = await asyncIterMap(myAsyncIter(), async (str) => {
await sleep(3000)
return str.toUpperCase()
});
More Demoing:
// dummy asyncIter for demonstration
const sleep = (ms) => new Promise(res => setTimeout(res, ms))
async function* myAsyncIter() {
await sleep(1000)
yield 'first thing'
await sleep(1000)
yield 'second thing'
await sleep(1000)
yield 'third thing'
}
Then
// THIS IS BAD! our asyncIter waits for each mapping.
for await (const thing of myAsyncIter()) {
console.log('starting with', thing)
await sleep(3000)
console.log('finished with', thing)
}
// total run time: ~12 seconds
Better version:
// this is better.
const promises = [];
for await (const thing of myAsyncIter()) {
const task = async () => {
console.log('starting with', thing)
await sleep(3000)
console.log('finished with', thing)
};
promises.push(task())
}
await Promise.all(promises)
// total run time: ~6 seconds

Related

For-await loop inside Promise

Imagine we have an async generator function:
async f * (connection) {
while (true) {
...
await doStuff()
yield value
}
}
Suppose that this function is virtually endless and gives us results of some async actions. We want to iterate these results:
for await (const result of f(connection)) {
...
}
Now imagine we want to break out of this for-await loop when some timeout ends and clean things up:
async outerFunc() {
setTimeout(() => connection.destroy(), TIMEOUT_MS)
for await (const result of f(connection)) {
...
if (something) {
return 'end naturally'
}
}
}
Assume that connection.destroy() ends the execution of f and ends the for-await loop. Now it would be great to return some value from the outerFunc when we end by timeout. The first thought is wrapping in a Promise:
async outerFunc() {
return await new Promise((resolve, reject) => {
setTimeout(() => {
connection.destroy()
resolve('end by timeout')
}, TIMEOUT_MS)
for await (const result of f(connection)) { // nope
...
if (something) {
resolve('end naturally')
}
}
})
}
But we cannot use awaits inside Promise and we cannot make the function async due to this antipattern
The question is: how do we return by timeout the right way?
It gets much easier, if you use an existing library that can handle asynchronous generators and timeouts automatically. The example below is using library iter-ops for that:
import {pipe, timeout} from 'iter-ops';
// test async endless generator:
async function* gen() {
let count = 0;
while (true) {
yield count++; // endless increment generator
}
}
const i = pipe(
gen(), // your generator
timeout(5, () => {
// 5ms has timed out, do disconnect or whatever
})
); //=> AsyncIterable<number>
// test:
(async function () {
for await(const a of i) {
console.log(a); // display result
}
})();
Assume that connection.destroy() ends the execution of f and ends the for-await loop.
In that case, just place your return statement so that it is executed when the loop ends:
async outerFunc() {
setTimeout(() => {
connection.destroy()
}, TIMEOUT_MS)
for await (const result of f(connection)) {
...
if (something) {
return 'end naturally'
}
}
return 'end by timeout'
}

Detect the end of multiple async loops [duplicate]

As far as I understand, in ES7/ES2016 putting multiple await's in code will work similar to chaining .then() with promises, meaning that they will execute one after the other rather than in parallel. So, for example, we have this code:
await someCall();
await anotherCall();
Do I understand it correctly that anotherCall() will be called only when someCall() is completed? What is the most elegant way of calling them in parallel?
I want to use it in Node, so maybe there's a solution with async library?
EDIT: I'm not satisfied with the solution provided in this question: Slowdown due to non-parallel awaiting of promises in async generators, because it uses generators and I'm asking about a more general use case.
You can await on Promise.all():
await Promise.all([someCall(), anotherCall()]);
To store the results:
let [someResult, anotherResult] = await Promise.all([someCall(), anotherCall()]);
Note that Promise.all fails fast, which means that as soon as one of the promises supplied to it rejects, then the entire thing rejects.
const happy = (v, ms) => new Promise((resolve) => setTimeout(() => resolve(v), ms))
const sad = (v, ms) => new Promise((_, reject) => setTimeout(() => reject(v), ms))
Promise.all([happy('happy', 100), sad('sad', 50)])
.then(console.log).catch(console.log) // 'sad'
If, instead, you want to wait for all the promises to either fulfill or reject, then you can use Promise.allSettled. Note that Internet Explorer does not natively support this method.
const happy = (v, ms) => new Promise((resolve) => setTimeout(() => resolve(v), ms))
const sad = (v, ms) => new Promise((_, reject) => setTimeout(() => reject(v), ms))
Promise.allSettled([happy('happy', 100), sad('sad', 50)])
.then(console.log) // [{ "status":"fulfilled", "value":"happy" }, { "status":"rejected", "reason":"sad" }]
Note: If you use Promise.all actions that managed to finish before rejection happen are not rolled back, so you may need to take care of such situation. For example
if you have 5 actions, 4 quick, 1 slow and slow rejects. Those 4
actions may be already executed so you may need to roll back. In such situation consider using Promise.allSettled while it will provide exact detail which action failed and which not.
TL;DR
Use Promise.all for the parallel function calls, the answer behaviors not correctly when the error occurs.
First, execute all the asynchronous calls at once and obtain all the Promise objects. Second, use await on the Promise objects. This way, while you wait for the first Promise to resolve the other asynchronous calls are still progressing. Overall, you will only wait for as long as the slowest asynchronous call. For example:
// Begin first call and store promise without waiting
const someResult = someCall();
// Begin second call and store promise without waiting
const anotherResult = anotherCall();
// Now we await for both results, whose async processes have already been started
const finalResult = [await someResult, await anotherResult];
// At this point all calls have been resolved
// Now when accessing someResult| anotherResult,
// you will have a value instead of a promise
JSbin example: http://jsbin.com/xerifanima/edit?js,console
Caveat: It doesn't matter if the await calls are on the same line or on different lines, so long as the first await call happens after all of the asynchronous calls. See JohnnyHK's comment.
Update: this answer has a different timing in error handling according to the #bergi's answer, it does NOT throw out the error as the error occurs but after all the promises are executed.
I compare the result with #jonny's tip: [result1, result2] = Promise.all([async1(), async2()]), check the following code snippet
const correctAsync500ms = () => {
return new Promise(resolve => {
setTimeout(resolve, 500, 'correct500msResult');
});
};
const correctAsync100ms = () => {
return new Promise(resolve => {
setTimeout(resolve, 100, 'correct100msResult');
});
};
const rejectAsync100ms = () => {
return new Promise((resolve, reject) => {
setTimeout(reject, 100, 'reject100msError');
});
};
const asyncInArray = async (fun1, fun2) => {
const label = 'test async functions in array';
try {
console.time(label);
const p1 = fun1();
const p2 = fun2();
const result = [await p1, await p2];
console.timeEnd(label);
} catch (e) {
console.error('error is', e);
console.timeEnd(label);
}
};
const asyncInPromiseAll = async (fun1, fun2) => {
const label = 'test async functions with Promise.all';
try {
console.time(label);
let [value1, value2] = await Promise.all([fun1(), fun2()]);
console.timeEnd(label);
} catch (e) {
console.error('error is', e);
console.timeEnd(label);
}
};
(async () => {
console.group('async functions without error');
console.log('async functions without error: start')
await asyncInArray(correctAsync500ms, correctAsync100ms);
await asyncInPromiseAll(correctAsync500ms, correctAsync100ms);
console.groupEnd();
console.group('async functions with error');
console.log('async functions with error: start')
await asyncInArray(correctAsync500ms, rejectAsync100ms);
await asyncInPromiseAll(correctAsync500ms, rejectAsync100ms);
console.groupEnd();
})();
Update:
The original answer makes it difficult (and in some cases impossible) to correctly handle promise rejections. The correct solution is to use Promise.all:
const [someResult, anotherResult] = await Promise.all([someCall(), anotherCall()]);
Original answer:
Just make sure you call both functions before you await either one:
// Call both functions
const somePromise = someCall();
const anotherPromise = anotherCall();
// Await both promises
const someResult = await somePromise;
const anotherResult = await anotherPromise;
There is another way without Promise.all() to do it in parallel:
First, we have 2 functions to print numbers:
function printNumber1() {
return new Promise((resolve,reject) => {
setTimeout(() => {
console.log("Number1 is done");
resolve(10);
},1000);
});
}
function printNumber2() {
return new Promise((resolve,reject) => {
setTimeout(() => {
console.log("Number2 is done");
resolve(20);
},500);
});
}
This is sequential:
async function oneByOne() {
const number1 = await printNumber1();
const number2 = await printNumber2();
}
//Output: Number1 is done, Number2 is done
This is parallel:
async function inParallel() {
const promise1 = printNumber1();
const promise2 = printNumber2();
const number1 = await promise1;
const number2 = await promise2;
}
//Output: Number2 is done, Number1 is done
I've created a gist testing some different ways of resolving promises, with results. It may be helpful to see the options that work.
Edit: Gist content as per Jin Lee's comment
// Simple gist to test parallel promise resolution when using async / await
function promiseWait(time) {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve(true);
}, time);
});
}
async function test() {
return [
await promiseWait(1000),
await promiseWait(5000),
await promiseWait(9000),
await promiseWait(3000),
]
}
async function test2() {
return {
'aa': await promiseWait(1000),
'bb': await promiseWait(5000),
'cc': await promiseWait(9000),
'dd': await promiseWait(3000),
}
}
async function test3() {
return await {
'aa': promiseWait(1000),
'bb': promiseWait(5000),
'cc': promiseWait(9000),
'dd': promiseWait(3000),
}
}
async function test4() {
const p1 = promiseWait(1000);
const p2 = promiseWait(5000);
const p3 = promiseWait(9000);
const p4 = promiseWait(3000);
return {
'aa': await p1,
'bb': await p2,
'cc': await p3,
'dd': await p4,
};
}
async function test5() {
return await Promise.all([
await promiseWait(1000),
await promiseWait(5000),
await promiseWait(9000),
await promiseWait(3000),
]);
}
async function test6() {
return await Promise.all([
promiseWait(1000),
promiseWait(5000),
promiseWait(9000),
promiseWait(3000),
]);
}
async function test7() {
const p1 = promiseWait(1000);
const p2 = promiseWait(5000);
const p3 = promiseWait(9000);
return {
'aa': await p1,
'bb': await p2,
'cc': await p3,
'dd': await promiseWait(3000),
};
}
let start = Date.now();
test().then((res) => {
console.log('Test Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test2().then((res) => {
console.log('Test2 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test3().then((res) => {
console.log('Test3 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test4().then((res) => {
console.log('Test4 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test5().then((res) => {
console.log('Test5 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test6().then((res) => {
console.log('Test6 Done, elapsed', (Date.now() - start) / 1000, res);
});
start = Date.now();
test7().then((res) => {
console.log('Test7 Done, elapsed', (Date.now() - start) / 1000, res);
});
});
});
});
});
});
/*
Test Done, elapsed 18.006 [ true, true, true, true ]
Test2 Done, elapsed 18.009 { aa: true, bb: true, cc: true, dd: true }
Test3 Done, elapsed 0 { aa: Promise { <pending> },
bb: Promise { <pending> },
cc: Promise { <pending> },
dd: Promise { <pending> } }
Test4 Done, elapsed 9 { aa: true, bb: true, cc: true, dd: true }
Test5 Done, elapsed 18.008 [ true, true, true, true ]
Test6 Done, elapsed 9.003 [ true, true, true, true ]
Test7 Done, elapsed 12.007 { aa: true, bb: true, cc: true, dd: true }
*/
In my case, I have several tasks I want to execute in parallel, but I need to do something different with the result of those tasks.
function wait(ms, data) {
console.log('Starting task:', data, ms);
return new Promise(resolve => setTimeout(resolve, ms, data));
}
var tasks = [
async () => {
var result = await wait(1000, 'moose');
// do something with result
console.log(result);
},
async () => {
var result = await wait(500, 'taco');
// do something with result
console.log(result);
},
async () => {
var result = await wait(5000, 'burp');
// do something with result
console.log(result);
}
]
await Promise.all(tasks.map(p => p()));
console.log('done');
And the output:
Starting task: moose 1000
Starting task: taco 500
Starting task: burp 5000
taco
moose
burp
done
(async function(){
function wait(ms, data) {
console.log('Starting task:', data, ms);
return new Promise(resolve => setTimeout(resolve, ms, data));
}
var tasks = [
async () => {
var result = await wait(1000, 'moose');
// do something with result
console.log(result);
},
async () => {
var result = await wait(500, 'taco');
// do something with result
console.log(result);
},
async () => {
var result = await wait(5000, 'burp');
// do something with result
console.log(result);
}
]
await Promise.all(tasks.map(p => p()));
console.log('done');
})();
await Promise.all([someCall(), anotherCall()]); as already mention will act as a thread fence (very common in parallel code as CUDA), hence it will allow all the promises in it to run without blocking each other, but will prevent the execution to continue until ALL are resolved.
another approach that is worth to share is the Node.js async that will also allow you to easily control the amount of concurrency that is usually desirable if the task is directly linked to the use of limited resources as API call, I/O operations, etc.
// create a queue object with concurrency 2
var q = async.queue(function(task, callback) {
console.log('Hello ' + task.name);
callback();
}, 2);
// assign a callback
q.drain = function() {
console.log('All items have been processed');
};
// add some items to the queue
q.push({name: 'foo'}, function(err) {
console.log('Finished processing foo');
});
q.push({name: 'bar'}, function (err) {
console.log('Finished processing bar');
});
// add some items to the queue (batch-wise)
q.push([{name: 'baz'},{name: 'bay'},{name: 'bax'}], function(err) {
console.log('Finished processing item');
});
// add some items to the front of the queue
q.unshift({name: 'bar'}, function (err) {
console.log('Finished processing bar');
});
Credits to the Medium article autor (read more)
You can call multiple asynchronous functions without awaiting them. This will execute them in parallel. While doing so, save the returned promises in variables, and await them at some point either individually or using Promise.all() and process the results.
You can also wrap the function calls with try...catch to handle failures of individual asynchronous actions and provide fallback logic.
Here's an example:
Observe the logs, the logs printed at the beginning of execution of the individual asynchronous functions get printed immediately even though the first function takes 5 seconds to resolve.
function someLongFunc () {
return new Promise((resolve, reject)=> {
console.log('Executing function 1')
setTimeout(resolve, 5000)
})
}
function anotherLongFunc () {
return new Promise((resolve, reject)=> {
console.log('Executing function 2')
setTimeout(resolve, 5000)
})
}
async function main () {
let someLongFuncPromise, anotherLongFuncPromise
const start = Date.now()
try {
someLongFuncPromise = someLongFunc()
}
catch (ex) {
console.error('something went wrong during func 1')
}
try {
anotherLongFuncPromise = anotherLongFunc()
}
catch (ex) {
console.error('something went wrong during func 2')
}
await someLongFuncPromise
await anotherLongFuncPromise
const totalTime = Date.now() - start
console.log('Execution completed in ', totalTime)
}
main()
// A generic test function that can be configured
// with an arbitrary delay and to either resolve or reject
const test = (delay, resolveSuccessfully) => new Promise((resolve, reject) => setTimeout(() => {
console.log(`Done ${ delay }`);
resolveSuccessfully ? resolve(`Resolved ${ delay }`) : reject(`Reject ${ delay }`)
}, delay));
// Our async handler function
const handler = async () => {
// Promise 1 runs first, but resolves last
const p1 = test(10000, true);
// Promise 2 run second, and also resolves
const p2 = test(5000, true);
// Promise 3 runs last, but completes first (with a rejection)
// Note the catch to trap the error immediately
const p3 = test(1000, false).catch(e => console.log(e));
// Await all in parallel
const r = await Promise.all([p1, p2, p3]);
// Display the results
console.log(r);
};
// Run the handler
handler();
/*
Done 1000
Reject 1000
Done 5000
Done 10000
*/
Whilst setting p1, p2 and p3 is not strictly running them in parallel, they do not hold up any execution and you can trap contextual errors with a catch.
This can be accomplished with Promise.allSettled(), which is similar to Promise.all() but without the fail-fast behavior.
async function Promise1() {
throw "Failure!";
}
async function Promise2() {
return "Success!";
}
const [Promise1Result, Promise2Result] = await Promise.allSettled([Promise1(), Promise2()]);
console.log(Promise1Result); // {status: "rejected", reason: "Failure!"}
console.log(Promise2Result); // {status: "fulfilled", value: "Success!"}
Note: This is a bleeding edge feature with limited browser support, so I strongly recommend including a polyfill for this function.
I create a helper function waitAll, may be it can make it sweeter.
It only works in nodejs for now, not in browser chrome.
//const parallel = async (...items) => {
const waitAll = async (...items) => {
//this function does start execution the functions
//the execution has been started before running this code here
//instead it collects of the result of execution of the functions
const temp = [];
for (const item of items) {
//this is not
//temp.push(await item())
//it does wait for the result in series (not in parallel), but
//it doesn't affect the parallel execution of those functions
//because they haven started earlier
temp.push(await item);
}
return temp;
};
//the async functions are executed in parallel before passed
//in the waitAll function
//const finalResult = await waitAll(someResult(), anotherResult());
//const finalResult = await parallel(someResult(), anotherResult());
//or
const [result1, result2] = await waitAll(someResult(), anotherResult());
//const [result1, result2] = await parallel(someResult(), anotherResult());
I vote for:
await Promise.all([someCall(), anotherCall()]);
Be aware of the moment you call functions, it may cause unexpected result:
// Supposing anotherCall() will trigger a request to create a new User
if (callFirst) {
await someCall();
} else {
await Promise.all([someCall(), anotherCall()]); // --> create new User here
}
But following always triggers request to create new User
// Supposing anotherCall() will trigger a request to create a new User
const someResult = someCall();
const anotherResult = anotherCall(); // ->> This always creates new User
if (callFirst) {
await someCall();
} else {
const finalResult = [await someResult, await anotherResult]
}

Async Delay in sending data [duplicate]

As far as I understand, in ES7/ES2016 putting multiple await's in code will work similar to chaining .then() with promises, meaning that they will execute one after the other rather than in parallel. So, for example, we have this code:
await someCall();
await anotherCall();
Do I understand it correctly that anotherCall() will be called only when someCall() is completed? What is the most elegant way of calling them in parallel?
I want to use it in Node, so maybe there's a solution with async library?
EDIT: I'm not satisfied with the solution provided in this question: Slowdown due to non-parallel awaiting of promises in async generators, because it uses generators and I'm asking about a more general use case.
You can await on Promise.all():
await Promise.all([someCall(), anotherCall()]);
To store the results:
let [someResult, anotherResult] = await Promise.all([someCall(), anotherCall()]);
Note that Promise.all fails fast, which means that as soon as one of the promises supplied to it rejects, then the entire thing rejects.
const happy = (v, ms) => new Promise((resolve) => setTimeout(() => resolve(v), ms))
const sad = (v, ms) => new Promise((_, reject) => setTimeout(() => reject(v), ms))
Promise.all([happy('happy', 100), sad('sad', 50)])
.then(console.log).catch(console.log) // 'sad'
If, instead, you want to wait for all the promises to either fulfill or reject, then you can use Promise.allSettled. Note that Internet Explorer does not natively support this method.
const happy = (v, ms) => new Promise((resolve) => setTimeout(() => resolve(v), ms))
const sad = (v, ms) => new Promise((_, reject) => setTimeout(() => reject(v), ms))
Promise.allSettled([happy('happy', 100), sad('sad', 50)])
.then(console.log) // [{ "status":"fulfilled", "value":"happy" }, { "status":"rejected", "reason":"sad" }]
Note: If you use Promise.all actions that managed to finish before rejection happen are not rolled back, so you may need to take care of such situation. For example
if you have 5 actions, 4 quick, 1 slow and slow rejects. Those 4
actions may be already executed so you may need to roll back. In such situation consider using Promise.allSettled while it will provide exact detail which action failed and which not.
TL;DR
Use Promise.all for the parallel function calls, the answer behaviors not correctly when the error occurs.
First, execute all the asynchronous calls at once and obtain all the Promise objects. Second, use await on the Promise objects. This way, while you wait for the first Promise to resolve the other asynchronous calls are still progressing. Overall, you will only wait for as long as the slowest asynchronous call. For example:
// Begin first call and store promise without waiting
const someResult = someCall();
// Begin second call and store promise without waiting
const anotherResult = anotherCall();
// Now we await for both results, whose async processes have already been started
const finalResult = [await someResult, await anotherResult];
// At this point all calls have been resolved
// Now when accessing someResult| anotherResult,
// you will have a value instead of a promise
JSbin example: http://jsbin.com/xerifanima/edit?js,console
Caveat: It doesn't matter if the await calls are on the same line or on different lines, so long as the first await call happens after all of the asynchronous calls. See JohnnyHK's comment.
Update: this answer has a different timing in error handling according to the #bergi's answer, it does NOT throw out the error as the error occurs but after all the promises are executed.
I compare the result with #jonny's tip: [result1, result2] = Promise.all([async1(), async2()]), check the following code snippet
const correctAsync500ms = () => {
return new Promise(resolve => {
setTimeout(resolve, 500, 'correct500msResult');
});
};
const correctAsync100ms = () => {
return new Promise(resolve => {
setTimeout(resolve, 100, 'correct100msResult');
});
};
const rejectAsync100ms = () => {
return new Promise((resolve, reject) => {
setTimeout(reject, 100, 'reject100msError');
});
};
const asyncInArray = async (fun1, fun2) => {
const label = 'test async functions in array';
try {
console.time(label);
const p1 = fun1();
const p2 = fun2();
const result = [await p1, await p2];
console.timeEnd(label);
} catch (e) {
console.error('error is', e);
console.timeEnd(label);
}
};
const asyncInPromiseAll = async (fun1, fun2) => {
const label = 'test async functions with Promise.all';
try {
console.time(label);
let [value1, value2] = await Promise.all([fun1(), fun2()]);
console.timeEnd(label);
} catch (e) {
console.error('error is', e);
console.timeEnd(label);
}
};
(async () => {
console.group('async functions without error');
console.log('async functions without error: start')
await asyncInArray(correctAsync500ms, correctAsync100ms);
await asyncInPromiseAll(correctAsync500ms, correctAsync100ms);
console.groupEnd();
console.group('async functions with error');
console.log('async functions with error: start')
await asyncInArray(correctAsync500ms, rejectAsync100ms);
await asyncInPromiseAll(correctAsync500ms, rejectAsync100ms);
console.groupEnd();
})();
Update:
The original answer makes it difficult (and in some cases impossible) to correctly handle promise rejections. The correct solution is to use Promise.all:
const [someResult, anotherResult] = await Promise.all([someCall(), anotherCall()]);
Original answer:
Just make sure you call both functions before you await either one:
// Call both functions
const somePromise = someCall();
const anotherPromise = anotherCall();
// Await both promises
const someResult = await somePromise;
const anotherResult = await anotherPromise;
There is another way without Promise.all() to do it in parallel:
First, we have 2 functions to print numbers:
function printNumber1() {
return new Promise((resolve,reject) => {
setTimeout(() => {
console.log("Number1 is done");
resolve(10);
},1000);
});
}
function printNumber2() {
return new Promise((resolve,reject) => {
setTimeout(() => {
console.log("Number2 is done");
resolve(20);
},500);
});
}
This is sequential:
async function oneByOne() {
const number1 = await printNumber1();
const number2 = await printNumber2();
}
//Output: Number1 is done, Number2 is done
This is parallel:
async function inParallel() {
const promise1 = printNumber1();
const promise2 = printNumber2();
const number1 = await promise1;
const number2 = await promise2;
}
//Output: Number2 is done, Number1 is done
I've created a gist testing some different ways of resolving promises, with results. It may be helpful to see the options that work.
Edit: Gist content as per Jin Lee's comment
// Simple gist to test parallel promise resolution when using async / await
function promiseWait(time) {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve(true);
}, time);
});
}
async function test() {
return [
await promiseWait(1000),
await promiseWait(5000),
await promiseWait(9000),
await promiseWait(3000),
]
}
async function test2() {
return {
'aa': await promiseWait(1000),
'bb': await promiseWait(5000),
'cc': await promiseWait(9000),
'dd': await promiseWait(3000),
}
}
async function test3() {
return await {
'aa': promiseWait(1000),
'bb': promiseWait(5000),
'cc': promiseWait(9000),
'dd': promiseWait(3000),
}
}
async function test4() {
const p1 = promiseWait(1000);
const p2 = promiseWait(5000);
const p3 = promiseWait(9000);
const p4 = promiseWait(3000);
return {
'aa': await p1,
'bb': await p2,
'cc': await p3,
'dd': await p4,
};
}
async function test5() {
return await Promise.all([
await promiseWait(1000),
await promiseWait(5000),
await promiseWait(9000),
await promiseWait(3000),
]);
}
async function test6() {
return await Promise.all([
promiseWait(1000),
promiseWait(5000),
promiseWait(9000),
promiseWait(3000),
]);
}
async function test7() {
const p1 = promiseWait(1000);
const p2 = promiseWait(5000);
const p3 = promiseWait(9000);
return {
'aa': await p1,
'bb': await p2,
'cc': await p3,
'dd': await promiseWait(3000),
};
}
let start = Date.now();
test().then((res) => {
console.log('Test Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test2().then((res) => {
console.log('Test2 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test3().then((res) => {
console.log('Test3 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test4().then((res) => {
console.log('Test4 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test5().then((res) => {
console.log('Test5 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test6().then((res) => {
console.log('Test6 Done, elapsed', (Date.now() - start) / 1000, res);
});
start = Date.now();
test7().then((res) => {
console.log('Test7 Done, elapsed', (Date.now() - start) / 1000, res);
});
});
});
});
});
});
/*
Test Done, elapsed 18.006 [ true, true, true, true ]
Test2 Done, elapsed 18.009 { aa: true, bb: true, cc: true, dd: true }
Test3 Done, elapsed 0 { aa: Promise { <pending> },
bb: Promise { <pending> },
cc: Promise { <pending> },
dd: Promise { <pending> } }
Test4 Done, elapsed 9 { aa: true, bb: true, cc: true, dd: true }
Test5 Done, elapsed 18.008 [ true, true, true, true ]
Test6 Done, elapsed 9.003 [ true, true, true, true ]
Test7 Done, elapsed 12.007 { aa: true, bb: true, cc: true, dd: true }
*/
In my case, I have several tasks I want to execute in parallel, but I need to do something different with the result of those tasks.
function wait(ms, data) {
console.log('Starting task:', data, ms);
return new Promise(resolve => setTimeout(resolve, ms, data));
}
var tasks = [
async () => {
var result = await wait(1000, 'moose');
// do something with result
console.log(result);
},
async () => {
var result = await wait(500, 'taco');
// do something with result
console.log(result);
},
async () => {
var result = await wait(5000, 'burp');
// do something with result
console.log(result);
}
]
await Promise.all(tasks.map(p => p()));
console.log('done');
And the output:
Starting task: moose 1000
Starting task: taco 500
Starting task: burp 5000
taco
moose
burp
done
(async function(){
function wait(ms, data) {
console.log('Starting task:', data, ms);
return new Promise(resolve => setTimeout(resolve, ms, data));
}
var tasks = [
async () => {
var result = await wait(1000, 'moose');
// do something with result
console.log(result);
},
async () => {
var result = await wait(500, 'taco');
// do something with result
console.log(result);
},
async () => {
var result = await wait(5000, 'burp');
// do something with result
console.log(result);
}
]
await Promise.all(tasks.map(p => p()));
console.log('done');
})();
await Promise.all([someCall(), anotherCall()]); as already mention will act as a thread fence (very common in parallel code as CUDA), hence it will allow all the promises in it to run without blocking each other, but will prevent the execution to continue until ALL are resolved.
another approach that is worth to share is the Node.js async that will also allow you to easily control the amount of concurrency that is usually desirable if the task is directly linked to the use of limited resources as API call, I/O operations, etc.
// create a queue object with concurrency 2
var q = async.queue(function(task, callback) {
console.log('Hello ' + task.name);
callback();
}, 2);
// assign a callback
q.drain = function() {
console.log('All items have been processed');
};
// add some items to the queue
q.push({name: 'foo'}, function(err) {
console.log('Finished processing foo');
});
q.push({name: 'bar'}, function (err) {
console.log('Finished processing bar');
});
// add some items to the queue (batch-wise)
q.push([{name: 'baz'},{name: 'bay'},{name: 'bax'}], function(err) {
console.log('Finished processing item');
});
// add some items to the front of the queue
q.unshift({name: 'bar'}, function (err) {
console.log('Finished processing bar');
});
Credits to the Medium article autor (read more)
You can call multiple asynchronous functions without awaiting them. This will execute them in parallel. While doing so, save the returned promises in variables, and await them at some point either individually or using Promise.all() and process the results.
You can also wrap the function calls with try...catch to handle failures of individual asynchronous actions and provide fallback logic.
Here's an example:
Observe the logs, the logs printed at the beginning of execution of the individual asynchronous functions get printed immediately even though the first function takes 5 seconds to resolve.
function someLongFunc () {
return new Promise((resolve, reject)=> {
console.log('Executing function 1')
setTimeout(resolve, 5000)
})
}
function anotherLongFunc () {
return new Promise((resolve, reject)=> {
console.log('Executing function 2')
setTimeout(resolve, 5000)
})
}
async function main () {
let someLongFuncPromise, anotherLongFuncPromise
const start = Date.now()
try {
someLongFuncPromise = someLongFunc()
}
catch (ex) {
console.error('something went wrong during func 1')
}
try {
anotherLongFuncPromise = anotherLongFunc()
}
catch (ex) {
console.error('something went wrong during func 2')
}
await someLongFuncPromise
await anotherLongFuncPromise
const totalTime = Date.now() - start
console.log('Execution completed in ', totalTime)
}
main()
// A generic test function that can be configured
// with an arbitrary delay and to either resolve or reject
const test = (delay, resolveSuccessfully) => new Promise((resolve, reject) => setTimeout(() => {
console.log(`Done ${ delay }`);
resolveSuccessfully ? resolve(`Resolved ${ delay }`) : reject(`Reject ${ delay }`)
}, delay));
// Our async handler function
const handler = async () => {
// Promise 1 runs first, but resolves last
const p1 = test(10000, true);
// Promise 2 run second, and also resolves
const p2 = test(5000, true);
// Promise 3 runs last, but completes first (with a rejection)
// Note the catch to trap the error immediately
const p3 = test(1000, false).catch(e => console.log(e));
// Await all in parallel
const r = await Promise.all([p1, p2, p3]);
// Display the results
console.log(r);
};
// Run the handler
handler();
/*
Done 1000
Reject 1000
Done 5000
Done 10000
*/
Whilst setting p1, p2 and p3 is not strictly running them in parallel, they do not hold up any execution and you can trap contextual errors with a catch.
This can be accomplished with Promise.allSettled(), which is similar to Promise.all() but without the fail-fast behavior.
async function Promise1() {
throw "Failure!";
}
async function Promise2() {
return "Success!";
}
const [Promise1Result, Promise2Result] = await Promise.allSettled([Promise1(), Promise2()]);
console.log(Promise1Result); // {status: "rejected", reason: "Failure!"}
console.log(Promise2Result); // {status: "fulfilled", value: "Success!"}
Note: This is a bleeding edge feature with limited browser support, so I strongly recommend including a polyfill for this function.
I create a helper function waitAll, may be it can make it sweeter.
It only works in nodejs for now, not in browser chrome.
//const parallel = async (...items) => {
const waitAll = async (...items) => {
//this function does start execution the functions
//the execution has been started before running this code here
//instead it collects of the result of execution of the functions
const temp = [];
for (const item of items) {
//this is not
//temp.push(await item())
//it does wait for the result in series (not in parallel), but
//it doesn't affect the parallel execution of those functions
//because they haven started earlier
temp.push(await item);
}
return temp;
};
//the async functions are executed in parallel before passed
//in the waitAll function
//const finalResult = await waitAll(someResult(), anotherResult());
//const finalResult = await parallel(someResult(), anotherResult());
//or
const [result1, result2] = await waitAll(someResult(), anotherResult());
//const [result1, result2] = await parallel(someResult(), anotherResult());
I vote for:
await Promise.all([someCall(), anotherCall()]);
Be aware of the moment you call functions, it may cause unexpected result:
// Supposing anotherCall() will trigger a request to create a new User
if (callFirst) {
await someCall();
} else {
await Promise.all([someCall(), anotherCall()]); // --> create new User here
}
But following always triggers request to create new User
// Supposing anotherCall() will trigger a request to create a new User
const someResult = someCall();
const anotherResult = anotherCall(); // ->> This always creates new User
if (callFirst) {
await someCall();
} else {
const finalResult = [await someResult, await anotherResult]
}

async await, process one item of For loop at a time

I can't seem to wrap my head around async await, let's say I want to process one item of an array at a time, with a 1 second delay:
myfunction() {
clearTimeout(this.timer);
this.timer = setTimeout(() => {
for (const item of this.myarray) {
this.dosomething(item).then((res: string) => {
setTimeout(() => {
await this.final_thing_before_the_next_item.push(res);
}, 1000);
});
}
}, 200);
}
Where would I put the "async"?
Stackblitz demo
You can put an async inside the setTimeout:
myfunction() {
setTimeout(async () => {
for (const item of this.myarray) {
await this.wait(1000);
console.log(item);
}
});
}
wait(timer: number): Promise<void> {
return new Promise((resolve, reject) => {
setTimeout(() => resolve(), timer);
});
}
In a simplified form, await is syntax sugar for consuming promises. The following snippets are pretty much equivalent:
doSomethingAsync().then(value => {
console.log('it finished', value);
});
const value = await doSomethingAsync();
The caveat is that if you want to use await, you need to mark the function as async. I will assume you want to have a 1s delay after full completion. Your code might look something like this:
function wait() {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
async function processItems(items) {
for (const item of items) {
const res = await this.doSomething(item);
// This isn't processed until doSomething completes.
const other = await this.doFinalThing(res);
// This isn't run until doFinalThing completes.
await wait(1000);
// The loop now continues after waiting 1s.
}
}
you marked this as angular, so I'm going to say, consider rxjs...
// imports
import {timer, from, concat} from 'rxjs'
import {switchMap, delay} from 'rxjs/operators'
// wait 200 ms
timer(200).pipe(
// switch
switchMap(() => {
// map array into streams
const obs$ = this.myArray.map(i => {
// from promise, do whatever
return from(this.dosomething(i)).pipe(
// switch again, to do final thing, again from promise
switchMap(res => from(this.finalThing(res))),
// delay 1s
delay(1000)
)
});
return concat(...obs$); // execute one after the other
})
).subscribe(v => console.log('values 1 by 1, spaced 1s apart', v))
If you want to process one asynchronous operation at a time, you could do the following:
// "wait" for a 1000ms before resolving the Promise
const wait = (ms = 1000) => {
return new Promise(resolve => {
setTimeout(() => resolve(), ms);
});
};
// array of asynchronous operations
const promises = Array.from({ length: 4 }).map((_, idx) =>
Promise.resolve(idx)
);
async function main() {
const data = [];
for (let item of promises) {
const result = await item;
await wait(1000);
data.push(result);
console.log("result", result);
}
console.log("final data", data);
}
main();
If I understand your question (and code sample) correctly, you basically want to
Sleep 200 ms
Iterate over a list of items. For each item you want to:
Call a function, passing the current item to it and getting a response in return.
Pause for 1 second.
Call a second function, passing it the response
To do that you need a sleep() function that returns a promise that will resolve when the specified time has elapsed:
function sleep(ms = 1000) {
const p = new Promise( (resolve, reject) => {
setTimeout(() => resolve());
});
return p;
}
Having that, you need an async function to do the actual work. It has to be async because that is what lets it await other things:
async function process_items( items, delayInMs = 1000 ) {
for ( item of items ) {
const res = await doSomething( item, delayInMs );
await sleep(delay);
await doSomethingElse( res );
}
}
As you can see from the above bit of code, the advantage of using async/await over callbacks or promise chains is that it gives you a rather more succinct and declarative syntax.
Then you can wrap it all up in another async function:
async function myfunction() {
await sleep(200);
await process_items( this.myarray, 1000 );
}
Marking a function as async does two things: it
Enables the use of await within that function, and
Converts the function into a function returning a promise, regardless of what its ostensible return value is.
If you take this function:
function foo() {
return 1;
}
and mark it as async:
async function foo() {
return 1;
}
it is (more or less) as if you had changed to to read:
function foo() {
return Promise.resolve(1);
}
The easiest way, without looking at the specifics of your code is before myFunction ie async myFunction(). I see 'this' being used so I imagine this is a method of some object/class in which case that would be one way to go about it.

How to use Array.prototype.filter with async?

Background
I am trying to filter an array of objects. Before I filter, I need to convert them to some format, and this operation is asynchronous.
const convert = () => new Promise( resolve => {
setTimeout( resolve, 1000 );
});
So, my first try was to do something like the following using async/await:
const objs = [ { id: 1, data: "hello" }, { id: 2, data: "world"} ];
objs.filter( async ( obj ) => {
await convert();
return obj.data === "hello";
});
Now, as some of you may know, Array.protoype.filter is a function which callback must return either true or false. filter is synchronous. In the previous example, I am returning none of them, I return a Promise ( all async functions are Promises ).
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
So as one can assume, the code before doesn't really work... That assumption is correct.
Problem
To make filter work with an async function, I checked stackoverflow and found this topic:
Filtering an array with a function that returns a promise
Unfortunately, the chosen answer is overly complex and uses classes. This won't do for me. I am instead looking for a more simple solution, using simple functions with a functional approach.
There is one solution at the very end, using a map with a callback to simulate a filter:
https://stackoverflow.com/a/46842181/1337392
But I was hoping to fix my filter function, not to replace it.
Questions
Is there a way to have an async function inside a filter?
If not, what is the simplest replacement I can do?
There is no way to use filter with an async function (at least that I know of).
The simplest way that you have to use filter with a collection of promises is to use Promise.all and then apply the function to your collection of results.
It would look something like this:
const results = await Promise.all(your_promises)
const filtered_results = results.filter(res => //do your filtering here)
Hope it helps.
Adapted from the article How to use async functions with Array.filter in Javascript by Tamás Sallai, you basically have 2 steps:
One that creates the conditions for an object to pass
One that receives the objects and returns true or false according to conditions
Here's an example
const arr = [1, 2, 3, 4, 5];
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
const asyncFilter = async (arr, predicate) => {
const results = await Promise.all(arr.map(predicate));
return arr.filter((_v, index) => results[index]);
}
const asyncRes = await asyncFilter(arr, async (i) => {
await sleep(10);
return i % 2 === 0;
});
console.log(asyncRes);
// 2,4
Use Scramjet fromArray/toArray methods...
const result = await scramjet.fromArray(arr)
.filter(async (item) => somePromiseReturningMethod(item))
.toArray();
as simple as that - here's a ready example to copy/paste:
const scramjet = require('../../');
async function myAsyncFilterFunc(data) {
return new Promise(res => {
process.nextTick(res.bind(null, data % 2));
});
}
async function x() {
const x = await scramjet.fromArray([1,2,3,4,5])
.filter(async (item) => myAsyncFilterFunc(item))
.toArray();
return x;
}
x().then(
(out) => console.log(out),
(err) => (console.error(err), process.exit(3)) // eslint-disable-line
);
Disclamer: I am the author of scramjet. :)
Build a parallel array to your array which you want to call filter on.
Await all of the promises from your filter func, in my eg, isValid.
In the callback in filter, use the 2nd arg, index, to index into your parallel array to determine if it should be filtered.
// ===============================================
// common
// ===============================================
const isValid = async (value) => value >= 0.5;
const values = [0.2, 0.3, 0.4, 0.5, 0.6];
// ===============================================
// won't filter anything
// ===============================================
const filtered = values.filter(async v => await isValid(v));
console.log(JSON.stringify(filtered));
// ===============================================
// filters
// ===============================================
(async () => {
const shouldFilter = await Promise.all(values.map(isValid));
const filtered2 = values.filter((value, index) => shouldFilter[index]);
console.log(JSON.stringify(filtered2));
})();
This behavior makes sense since any Promise instance has a truthy value, but it's not intuitive at a glance.
This answer uses library iter-ops, which handles iterable objects, and supports async filtering:
import {pipeAsync, filter, toAsync} from 'iter-ops';
// your input data:
const objs = [{id: 1, data: 'hello'}, {id: 2, data: 'world'}];
const i = pipeAsync(
objs,
filter(async value => {
await convert(); // any async function
return value.data === 'hello'; // filtering logic
})
);
for await(const a of i) {
console.log(a); // filtered data
}
P.S. I'm the author of iter-ops.
Reduce method can mimic filter and can operate with promises.
const isPositiveNumberAsync = async (number) => number >= 0;
const filterPositiveNumbersAsync = async (numbers) => numbers?.reduce(async (accumulatorPromise, number) => {
const accumulator = await accumulatorPromise;
if (await isPositiveNumberAsync(number)) {
return [...accumulator, number];
}
return accumulator;
}, Promise.resolve([])) || [];
(async () => {
// no numbers argument provided
console.log(await filterPositiveNumbersAsync());
// an empty argument list provided
console.log(await filterPositiveNumbersAsync([]));
// ok, but no positive numbers provided
console.log(await filterPositiveNumbersAsync([-1,-2,-3]));
// ok, positive numbers filtered.
console.log(await filterPositiveNumbersAsync([0,1,-1,-3,2,-2]));
})();
Array.prototype.asyncFilter =function( filterFn) {
const arr = this;
return new Promise(function(resolve){
const booleanArr = [];
arr.forEach(function (e) {
booleanArr.push(filterFn(e))
})
Promise.all(booleanArr).then(function (booleanArr) {
const arr2 = arr.filter(function (e, i) {
return booleanArr[i]
})
resolve(arr2)
})
})
}
/** use it like this**/
const arr=[1,2,3]
arr.asyncFilter(async e=>{}).then(...)
You can use Promise.filter from Bluebird that works similarly to Array.filter but it supports async & await.
Add asyncFilter as an extension to Array:
#available(macOS 10.15.0, *)
extension Array where Element: Any {
public func asyncFilter(closure: (Element) async -> Bool) async -> Array {
var result = [Element]()
for item in self {
if await closure(item) {
result.append(item)
}
}
return result
}
}
Usage:
result = await result.asyncFilter { item in
if <item match> {
return true
}
}

Categories

Resources