Develop Reference

Retry failed API call in for...of loop

I am trying to build a wrapper over Notion JS SDK's iteratePaginatedAPI that handles errors as well. I feel particularly lost on how do I catch API errors in such a way that I can actually retry them (aka retry the iteration that failed). Here's my attempt:
async function* queryNotion(listFn, firstPageArgs) {
try {
for await (const result of iteratePaginatedAPI(listFn, firstPageArgs)) {
yield* result
}
} catch (error) {
if (error.code === APIErrorCode.RateLimited) {
console.log('rate_limited');
console.log(error);
sleep(1);
// How would I retry the last iteration?
}
}
}
Coming from the Ruby world, there is a retry in a rescue block. Any help would be appreciated!

Very interesting problem. The issue is that the exception comes from the for await itself, not from its body, so you cannot catch it there. When the exception hits, loops are over.
Note that the iterator might be done after a rejection/exception, in which case there is nothing you can do except starting a new one.
That said, you can always call Iterator.next() yourself and process the result manually. The next() call of an async iterator will return an object like {value: Promise<any>, done: boolean}, and when running it in a loop, you can await the promise in a try..catch and only exit the loop when done becomes true:
async function* queryNotion(listFn, firstPageArgs) {
const asyncGenerator = mockIteratePaginatedAPI(listFn, firstPageArgs)
while (true) {
const current = asyncGenerator.next()
if (current.done) {
break
}
try {
yield* await current.value
} catch (e) {
console.log(`got exception: "${e}" - trying again`)
continue
}
}
}
function* mockIteratePaginatedAPI(a, b) {
for (let i = 0; i < 8; i++) {
yield new Promise((resolve, reject) => setTimeout(() => [3, 5].includes(i) ? reject(`le error at ${i}`) : resolve([i]), 500))
}
}
(async function() {
for await (const n of queryNotion('foo', 'bar')) {
console.log(n)
}
})()
If we keep a reference to the generator, we can also put it back into a for async. This might be easier to read, however a for await ...of will call the iterator's return() when exiting the loop early, likely finishing it, in which case, this will not work:
async function* queryNotion(listFn, firstPageArgs) {
const asyncGenerator = mockIteratePaginatedAPI(listFn, firstPageArgs)
while (true) {
try {
for await (const result of asyncGenerator) {
yield* result
}
break
} catch (e) {
console.log('got exception:', e, 'trying again')
}
}
}
function* mockIteratePaginatedAPI(a, b) {
for (let i = 0; i < 8; i++) {
yield new Promise((resolve, reject) => setTimeout(() => [3, 5].includes(i) ? reject(`le error at ${i}`) : resolve([i]), 500))
}
}
(async function () {
for await (const n of queryNotion('foo', 'bar')) {
console.log(n)
}
})()

Simply add a continue statement inside your if
async function* queryNotion(listFn, firstPageArgs) {
try {
for await (const result of iteratePaginatedAPI(listFn, firstPageArgs)) {
yield* result
}
} catch (error) {
if (error.code === APIErrorCode.RateLimited) {
console.log('rate_limited');
console.log(error);
await sleep(1);
continue; // retry the last iteration
}
}
}

How to map over async generators?

Let's say we have an async generator:
exports.asyncGen = async function* (items) {
for (const item of items) {
const result = await someAsyncFunc(item)
yield result;
}
}
is it possible to map over this generator? Essentially I want to do this:
const { asyncGen } = require('./asyncGen.js')
exports.process = async function (items) {
return asyncGen(items).map(item => {
//... do something
})
}
As of now .map fails to recognize async iterator.
The alternative is to use for await ... of but that's nowhere near elegant as with .map

The iterator methods proposal that would provide this method is still at stage 2 only. You can use some polyfill, or write your own map helper function though:
async function* map(asyncIterable, callback) {
let i = 0;
for await (const val of asyncIterable)
yield callback(val, i++);
}
exports.process = function(items) {
return map(asyncGen(items), item => {
//... do something
});
};

The alternative is to use for await ... of, but that's nowhere near elegant as with .map
For an elegant and efficient solution, here's one using iter-ops library:
import {pipe, map} from 'iter-ops';
const i = pipe(
asyncGen(), // your async generator result
map(value => /*map logic*/)
); //=> AsyncIterable
It is elegant, because the syntax is clean, simple, and applicable to any iterable or iterators, not just asynchronous generators.
It is more flexible and reusable, as you can add lots of other operators to the same pipeline.
Since it produces a standard JavaScript AsyncIterable, you can do:
for await(const a of i) {
console.log(a); //=> print values
}
P.S. I'm the author of iter-ops.

TL;DR - If the mapping function is async:
To make asyncIter not wait for each mapping before producing the next value, do
async function asyncIterMap(asyncIter, asyncFunc) {
const promises = [];
for await (const value of asyncIter) {
promises.push(asyncFunc(value))
}
return await Promise.all(promises)
}
// example - how to use:
const results = await asyncIterMap(myAsyncIter(), async (str) => {
await sleep(3000)
return str.toUpperCase()
});
More Demoing:
// dummy asyncIter for demonstration
const sleep = (ms) => new Promise(res => setTimeout(res, ms))
async function* myAsyncIter() {
await sleep(1000)
yield 'first thing'
await sleep(1000)
yield 'second thing'
await sleep(1000)
yield 'third thing'
}
Then
// THIS IS BAD! our asyncIter waits for each mapping.
for await (const thing of myAsyncIter()) {
console.log('starting with', thing)
await sleep(3000)
console.log('finished with', thing)
}
// total run time: ~12 seconds
Better version:
// this is better.
const promises = [];
for await (const thing of myAsyncIter()) {
const task = async () => {
console.log('starting with', thing)
await sleep(3000)
console.log('finished with', thing)
};
promises.push(task())
}
await Promise.all(promises)
// total run time: ~6 seconds

async await, process one item of For loop at a time

I can't seem to wrap my head around async await, let's say I want to process one item of an array at a time, with a 1 second delay:
myfunction() {
clearTimeout(this.timer);
this.timer = setTimeout(() => {
for (const item of this.myarray) {
this.dosomething(item).then((res: string) => {
setTimeout(() => {
await this.final_thing_before_the_next_item.push(res);
}, 1000);
});
}
}, 200);
}
Where would I put the "async"?

Stackblitz demo
You can put an async inside the setTimeout:
myfunction() {
setTimeout(async () => {
for (const item of this.myarray) {
await this.wait(1000);
console.log(item);
}
});
}
wait(timer: number): Promise<void> {
return new Promise((resolve, reject) => {
setTimeout(() => resolve(), timer);
});
}

In a simplified form, await is syntax sugar for consuming promises. The following snippets are pretty much equivalent:
doSomethingAsync().then(value => {
console.log('it finished', value);
});
const value = await doSomethingAsync();
The caveat is that if you want to use await, you need to mark the function as async. I will assume you want to have a 1s delay after full completion. Your code might look something like this:
function wait() {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
async function processItems(items) {
for (const item of items) {
const res = await this.doSomething(item);
// This isn't processed until doSomething completes.
const other = await this.doFinalThing(res);
// This isn't run until doFinalThing completes.
await wait(1000);
// The loop now continues after waiting 1s.
}
}

you marked this as angular, so I'm going to say, consider rxjs...
// imports
import {timer, from, concat} from 'rxjs'
import {switchMap, delay} from 'rxjs/operators'
// wait 200 ms
timer(200).pipe(
// switch
switchMap(() => {
// map array into streams
const obs$ = this.myArray.map(i => {
// from promise, do whatever
return from(this.dosomething(i)).pipe(
// switch again, to do final thing, again from promise
switchMap(res => from(this.finalThing(res))),
// delay 1s
delay(1000)
)
});
return concat(...obs$); // execute one after the other
})
).subscribe(v => console.log('values 1 by 1, spaced 1s apart', v))

If you want to process one asynchronous operation at a time, you could do the following:
// "wait" for a 1000ms before resolving the Promise
const wait = (ms = 1000) => {
return new Promise(resolve => {
setTimeout(() => resolve(), ms);
});
};
// array of asynchronous operations
const promises = Array.from({ length: 4 }).map((_, idx) =>
Promise.resolve(idx)
);
async function main() {
const data = [];
for (let item of promises) {
const result = await item;
await wait(1000);
data.push(result);
console.log("result", result);
}
console.log("final data", data);
}
main();

If I understand your question (and code sample) correctly, you basically want to
Sleep 200 ms
Iterate over a list of items. For each item you want to:
Call a function, passing the current item to it and getting a response in return.
Pause for 1 second.
Call a second function, passing it the response
To do that you need a sleep() function that returns a promise that will resolve when the specified time has elapsed:
function sleep(ms = 1000) {
const p = new Promise( (resolve, reject) => {
setTimeout(() => resolve());
});
return p;
}
Having that, you need an async function to do the actual work. It has to be async because that is what lets it await other things:
async function process_items( items, delayInMs = 1000 ) {
for ( item of items ) {
const res = await doSomething( item, delayInMs );
await sleep(delay);
await doSomethingElse( res );
}
}
As you can see from the above bit of code, the advantage of using async/await over callbacks or promise chains is that it gives you a rather more succinct and declarative syntax.
Then you can wrap it all up in another async function:
async function myfunction() {
await sleep(200);
await process_items( this.myarray, 1000 );
}
Marking a function as async does two things: it
Enables the use of await within that function, and
Converts the function into a function returning a promise, regardless of what its ostensible return value is.
If you take this function:
function foo() {
return 1;
}
and mark it as async:
async function foo() {
return 1;
}
it is (more or less) as if you had changed to to read:
function foo() {
return Promise.resolve(1);
}

The easiest way, without looking at the specifics of your code is before myFunction ie async myFunction(). I see 'this' being used so I imagine this is a method of some object/class in which case that would be one way to go about it.

Await for an async queue to be proceeded (async module)

I'm trying to build a queue with nodejs and async module but it's not working as desired.
Here is my code:
const async = require('async');
const queueSize = 10;
const taskHandler = function (task, done) {
task(done);
};
const myQueue = async.queue(taskHandler, queueSize);
myQueue.drain = function () {
console.log('The queue is now empty.');
};
function delay() {
return new Promise(resolve => setTimeout(resolve, 1000));
}
async function delayedLog(item) {
await delay();
console.log(item);
}
const run = async () => {
for (let item = 0; item < 30; item++) {
myQueue.push(async function (done) {
await delayedLog(item)
done();
});
}
}
(async () => {
console.log('START');
await run();
console.log('END);
})()
What I want:
START
// logs from delayedLog
END
Output:
START
END
// logs from delayedLog
As you can see await isn't working. I tried to promisify the module but the problem is still here. I tried with d3-queue and I had exactly the same problem.
Any suggestion ?

As #Chiến Nghê said you are using some async/await where there is no need to but this is another problem.
Your await isn't working at the end because your drain function is called at the end of your queue and it's not returning a promise.
You have to promisify your function, here is a sample example to do that:
function end() {
return new Promise((resolve, reject) => {
myQueue.drain = function () {
console.log('The queue is now empty.');
resolve();
};
})
}
And then you can use your async await on your end function:
(async () => {
console.log('START');
run();
await end();
console.log('END');
})()
Output:
START
// a lot of logs
END

Your program is working as it is. Actually you dont need to declare async for function run(). This function is simply to put 30 async tasks to the queue for later execution then it ends after that.
I don't know exact what you want, but if you want the console.log('END') logic is executed only every async tasks in queue are finished. Please move it to queue's drain() function.

Limit concurrency of pending promises

I'm looking for a promise function wrapper that can limit / throttle when a given promise is running so that only a set number of that promise is running at a given time.
In the case below delayPromise should never run concurrently, they should all run one at a time in a first-come-first-serve order.
import Promise from 'bluebird'
function _delayPromise (seconds, str) {
console.log(str)
return Promise.delay(seconds)
}
let delayPromise = limitConcurrency(_delayPromise, 1)
async function a() {
await delayPromise(100, "a:a")
await delayPromise(100, "a:b")
await delayPromise(100, "a:c")
}
async function b() {
await delayPromise(100, "b:a")
await delayPromise(100, "b:b")
await delayPromise(100, "b:c")
}
a().then(() => console.log('done'))
b().then(() => console.log('done'))
Any ideas on how to get a queue like this set up?
I have a "debounce" function from the wonderful Benjamin Gruenbaum. I need to modify this to throttle a promise based on it's own execution and not the delay.
export function promiseDebounce (fn, delay, count) {
let working = 0
let queue = []
function work () {
if ((queue.length === 0) || (working === count)) return
working++
Promise.delay(delay).tap(function () { working-- }).then(work)
var next = queue.shift()
next[2](fn.apply(next[0], next[1]))
}
return function debounced () {
var args = arguments
return new Promise(function (resolve) {
queue.push([this, args, resolve])
if (working < count) work()
}.bind(this))
}
}

I don't think there are any libraries to do this, but it's actually quite simple to implement yourself:
function sequential(fn) { // limitConcurrency(fn, 1)
let q = Promise.resolve();
return function(x) {
const p = q.then(() => fn(x));
q = p.reflect();
return p;
};
}
For multiple concurrent requests it gets a little trickier, but can be done as well.
function limitConcurrency(fn, n) {
if (n == 1) return sequential(fn); // optimisation
let q = Promise.resolve();
const active = new Set();
const fst = t => t[0];
const snd = t => t[1];
return function(x) {
function put() {
const p = fn(x);
const a = p.reflect().then(() => {
active.delete(a);
});
active.add(a);
return [Promise.race(active), p];
}
if (active.size < n) {
const r = put()
q = fst(t);
return snd(t);
} else {
const r = q.then(put);
q = r.then(fst);
return r.then(snd)
}
};
}
Btw, you might want to have a look at the actors model and CSP. They can simplify dealing with such things, there are a few JS libraries for them out there as well.
Example
import Promise from 'bluebird'
function sequential(fn) {
var q = Promise.resolve();
return (...args) => {
const p = q.then(() => fn(...args))
q = p.reflect()
return p
}
}
async function _delayPromise (seconds, str) {
console.log(`${str} started`)
await Promise.delay(seconds)
console.log(`${str} ended`)
return str
}
let delayPromise = sequential(_delayPromise)
async function a() {
await delayPromise(100, "a:a")
await delayPromise(200, "a:b")
await delayPromise(300, "a:c")
}
async function b() {
await delayPromise(400, "b:a")
await delayPromise(500, "b:b")
await delayPromise(600, "b:c")
}
a().then(() => console.log('done'))
b().then(() => console.log('done'))
// --> with sequential()
// $ babel-node test/t.js
// a:a started
// a:a ended
// b:a started
// b:a ended
// a:b started
// a:b ended
// b:b started
// b:b ended
// a:c started
// a:c ended
// b:c started
// done
// b:c ended
// done
// --> without calling sequential()
// $ babel-node test/t.js
// a:a started
// b:a started
// a:a ended
// a:b started
// a:b ended
// a:c started
// b:a ended
// b:b started
// a:c ended
// done
// b:b ended
// b:c started
// b:c ended
// done

Use the throttled-promise module:
https://www.npmjs.com/package/throttled-promise
var ThrottledPromise = require('throttled-promise'),
promises = [
new ThrottledPromise(function(resolve, reject) { ... }),
new ThrottledPromise(function(resolve, reject) { ... }),
new ThrottledPromise(function(resolve, reject) { ... })
];
// Run promises, but only 2 parallel
ThrottledPromise.all(promises, 2)
.then( ... )
.catch( ... );

I have the same problem. I wrote a library to implement it. Code is here. I created a queue to save all the promises. When you push some promises to the queue, the first several promises at the head of the queue would be popped and running. Once one promise is done, the next promise in the queue would also be popped and running. Again and again, until the queue has no Task. You can check the code for details. Hope this library would help you.

Advantages
you can define the amount of concurrent promises (near simultaneous requests)
consistent flow: once one promise resolve, another request start no need to guess the server capability
robust against data choke, if the server stop for a moment, it will just wait, and next tasks will not start just because the
clock allowed
do not rely on a 3rd party module it is Vanila node.js
1st thing is to make https a promise, so we can use wait to retrieve data (removed from the example)
2nd create a promise scheduler that submit another request as any promise get resolved.
3rd make the calls
Limiting requests taking by limiting the amount of concurrent promises
const https = require('https')
function httpRequest(method, path, body = null) {
const reqOpt = {
method: method,
path: path,
hostname: 'dbase.ez-mn.net',
headers: {
"Content-Type": "application/json",
"Cache-Control": "no-cache"
}
}
if (method == 'GET') reqOpt.path = path + '&max=20000'
if (body) reqOpt.headers['Content-Length'] = Buffer.byteLength(body);
return new Promise((resolve, reject) => {
const clientRequest = https.request(reqOpt, incomingMessage => {
let response = {
statusCode: incomingMessage.statusCode,
headers: incomingMessage.headers,
body: []
};
let chunks = ""
incomingMessage.on('data', chunk => { chunks += chunk; });
incomingMessage.on('end', () => {
if (chunks) {
try {
response.body = JSON.parse(chunks);
} catch (error) {
reject(error)
}
}
console.log(response)
resolve(response);
});
});
clientRequest.on('error', error => { reject(error); });
if (body) { clientRequest.write(body) }
clientRequest.end();
});
}
const asyncLimit = (fn, n) => {
const pendingPromises = new Set();
return async function(...args) {
while (pendingPromises.size >= n) {
await Promise.race(pendingPromises);
}
const p = fn.apply(this, args);
const r = p.catch(() => {});
pendingPromises.add(r);
await r;
pendingPromises.delete(r);
return p;
};
};
// httpRequest is the function that we want to rate the amount of requests
// in this case, we set 8 requests running while not blocking other tasks (concurrency)
let ratedhttpRequest = asyncLimit(httpRequest, 8);
// this is our datase and caller
let process = async () => {
patchData=[
{path: '/rest/slots/80973975078587', body:{score:3}},
{path: '/rest/slots/809739750DFA95', body:{score:5}},
{path: '/rest/slots/AE0973750DFA96', body:{score:5}}]
for (let i = 0; i < patchData.length; i++) {
ratedhttpRequest('PATCH', patchData[i].path, patchData[i].body)
}
console.log('completed')
}
process()

The classic way of running async processes in series is to use async.js and use async.series(). If you prefer promise based code then there is a promise version of async.js: async-q
With async-q you can once again use series:
async.series([
function(){return delayPromise(100, "a:a")},
function(){return delayPromise(100, "a:b")},
function(){return delayPromise(100, "a:c")}
])
.then(function(){
console.log(done);
});
Running two of them at the same time will run a and b concurrently but within each they will be sequential:
// these two will run concurrently but each will run
// their array of functions sequentially:
async.series(a_array).then(()=>console.log('a done'));
async.series(b_array).then(()=>console.log('b done'));
If you want to run b after a then put it in the .then():
async.series(a_array)
.then(()=>{
console.log('a done');
return async.series(b_array);
})
.then(()=>{
console.log('b done');
});
If instead of running each sequentially you want to limit each to run a set number of processes concurrently then you can use parallelLimit():
// Run two promises at a time:
async.parallelLimit(a_array,2)
.then(()=>console.log('done'));
Read up the async-q docs: https://github.com/dbushong/async-q/blob/master/READJSME.md