Lets say I have a generator:
function* source() {
yield "hello"; yield "world";
}
I create the iterable, iterate with a for-loop, and then break out of the loop before the iterator fully completes (returns done).
function run() {
for (let item of source()) {
console.log(item);
break;
}
}
Question: How can I find out, from the iterable-side, that the iterator terminated early?
There doesn't seem to be any feedback if you try to do this directly in the generator itself:
function* source2() {
try {
let result = yield "hello";
console.log("foo");
} catch (err) {
console.log("bar");
}
}
... neither "foo" nor "bar" is logged.
Edit: See Newer accepted answer. I will keep this as it does/did work, and I was pretty happy at the time I was able to hack a solution. However, as you can see in the accepted answer, the finally solution is so simple now it has been identified.
I noticed that typescript defines Iterator (lib.es2015) as:
interface Iterator<T> {
next(value?: any): IteratorResult<T>;
return?(value?: any): IteratorResult<T>;
throw?(e?: any): IteratorResult<T>;
}
I intercepted these methods and logged calls and it does appear that if an iterator is terminated early --at least via a for-loop-- then the return method is called. It will also be called if the consumer throws an error. If the loop is allowed to fully iterate the iterator return is not called.
Return hack
So, I made a bit of a hack to allow capturing another iterable - so I don't have to re-implement the iterator.
function terminated(iterable, cb) {
return {
[Symbol.iterator]() {
const it = iterable[Symbol.iterator]();
it.return = function (value) {
cb(value);
return { done: true, value: undefined };
}
return it;
}
}
}
function* source() {
yield "hello"; yield "world";
}
function source2(){
return terminated(source(), () => { console.log("foo") });
}
for (let item of source2()) {
console.log(item);
break;
}
and it works!
hello
foo
remove the break and you get:
hello
world
Check after each yield
While typing this answer, I realised a better problem/solution is to find out in the original generator method.
The only way I can see to pass information back to the original iterable is to use next(value). So if we pick some unique value (say Symbol.for("terminated")) to signal the termination, and we alter the above return-hack to call it.next(Symbol.for("terminated")):
function* source() {
let terminated = yield "hello";
if (terminated == Symbol.for("terminated")) {
console.log("FooBar!");
return;
}
yield "world";
}
function terminator(iterable) {
return {
[Symbol.iterator]() {
const it = iterable[Symbol.iterator]();
const $return = it.return;
it.return = function (value) {
it.next(Symbol.for("terminated"));
return $return.call(it)
}
return it;
}
}
}
for (let item of terminator(source())) {
console.log(item);
break;
}
Success!
hello
FooBar!
Chaining Cascades Return
If you chain some extra transform iterators, then the return call cascades through them all:
function* chain(source) {
for (let item of source) { yield item; }
}
for (let item of chain(chain(terminator(source())))) {
console.log(item);
break
}
hello
FooBar!
Package
I've wrapped the above solution as a package. It supports both [Symbol.iterator] and [Symbol.asyncIterator]. The async iterator case is of particular interest to me, especially when some resource needs to be disposed of correctly.
There's a much simpler way to do this: use a finally block.
function *source() {
let i;
try {
for(i = 0; i < 5; i++)
yield i;
}
finally {
if(i !== 5)
console.log(' terminated early');
}
}
console.log('First:')
for(const val of source()) {
console.log(` ${val}`);
}
console.log('Second:')
for(const val of source()) {
console.log(` ${val}`);
if(val > 2)
break;
}
...yields:
First:
0
1
2
3
4
Second:
0
1
2
3
terminated early
I've run into a similar need to figure out when an iterator terminates early. The accepted answer is really clever and probably the best way to solve the problem generically, but I think this solution could also be helpful for other uses cases.
Say, for example, you have an infinite iterable, such as the fibonacci sequence described in MDN's Iterators and Generators docs.
In any sort of loop there needs to be a condition set to break out of the loop early, like in the solution already given. But what if you want to destructure the iterable in order to create an array of values? In that case, you'd want to limit the number of iterations, essentially setting a maximum length on the iterable.
To do this, I wrote a function called limitIterable, which takes as arguments an iterable, an iteration limit, and an optional callback function to be executed in the event that the iterator terminates early. The return value is a Generator object (which is both an iterator and an iterable) created using an Immediately Invoked (Generator) Function Expression.
When the generator is executed, whether in a for..of loop, with destructuring, or by calling the next() method, it will check to see if iterator.next().done === true or iterationCount < iterationLimit. In the case of an infinite iterable like the fibonacci sequence, the latter will always cause the while loop to be exited. However, note that one could also set an iterationLimit that is greater than the length of some finite iterable, and everything will still work.
In either case, once the while loop is exited the most recent result will be checked to see if the iterator is done. If so, the original iterable's return value will be used. If not, the optional callback function is executed and used as a return value.
Note that this code also allows the user to pass in values to next(), which will in turn be passed to the original iterable (see the example using MDN's fibonacci sequence inside the code snippet attached). It also allows for additional calls to next() beyond the set iterationLimit within the callback function.
Run the code snippet to see the results of a few possible use cases! Here's the limitIterable function code by itself:
function limitIterable(iterable, iterationLimit, callback = (itCount, result, it) => undefined) {
// callback will be executed if iterator terminates early
if (!(Symbol.iterator in Object(iterable))) {
throw new Error('First argument must be iterable');
}
if (iterationLimit < 1 || !Number.isInteger(iterationLimit)) {
throw new Error('Second argument must be an integer greater than or equal to 1');
}
if (!(callback instanceof Function)) {
throw new Error('Third argument must be a function');
}
return (function* () {
const iterator = iterable[Symbol.iterator]();
// value passed to the first invocation of next() is always ignored, so no need to pass argument to next() outside of while loop
let result = iterator.next();
let iterationCount = 0;
while (!result.done && iterationCount < iterationLimit) {
const nextArg = yield result.value;
result = iterator.next(nextArg);
iterationCount++;
}
if (result.done) {
// iterator has been fully consumed, so result.value will be the iterator's return value (the value present alongside done: true)
return result.value;
} else {
// iteration was terminated before completion (note that iterator will still accept calls to next() inside the callback function)
return callback(iterationCount, result, iterator);
}
})();
}
function limitIterable(iterable, iterationLimit, callback = (itCount, result, it) => undefined) {
// callback will be executed if iterator terminates early
if (!(Symbol.iterator in Object(iterable))) {
throw new Error('First argument must be iterable');
}
if (iterationLimit < 1 || !Number.isInteger(iterationLimit)) {
throw new Error('Second argument must be an integer greater than or equal to 1');
}
if (!(callback instanceof Function)) {
throw new Error('Third argument must be a function');
}
return (function* () {
const iterator = iterable[Symbol.iterator]();
// value passed to the first invocation of next() is always ignored, so no need to pass argument to next() outside of while loop
let result = iterator.next();
let iterationCount = 0;
while (!result.done && iterationCount < iterationLimit) {
const nextArg = yield result.value;
result = iterator.next(nextArg);
iterationCount++;
}
if (result.done) {
// iterator has been fully consumed, so result.value will be the iterator's return value (the value present alongside done: true)
return result.value;
} else {
// iteration was terminated before completion (note that iterator will still accept calls to next() inside the callback function)
return callback(iterationCount, result, iterator);
}
})();
}
// EXAMPLE USAGE //
// fibonacci function from:
//https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators#Advanced_generators
function* fibonacci() {
let fn1 = 0;
let fn2 = 1;
while (true) {
let current = fn1;
fn1 = fn2;
fn2 = current + fn1;
let reset = yield current;
if (reset) {
fn1 = 0;
fn2 = 1;
}
}
}
console.log('String iterable with 26 characters terminated early after 10 iterations, destructured into an array. Callback reached.');
const itString = limitIterable('abcdefghijklmnopqrstuvwxyz', 10, () => console.log('callback: string terminated early'));
console.log([...itString]);
console.log('Array iterable with length 3 terminates before limit of 4 is reached. Callback not reached.');
const itArray = limitIterable([1,2,3], 4, () => console.log('callback: array terminated early?'));
for (const val of itArray) {
console.log(val);
}
const fib = fibonacci();
const fibLimited = limitIterable(fibonacci(), 9, (itCount) => console.warn(`Iteration terminated early at fibLimited. ${itCount} iterations completed.`));
console.log('Fibonacci sequences are equivalent up to 9 iterations, as shown in MDN docs linked above.');
console.log('Limited fibonacci: 11 calls to next() but limited to 9 iterations; reset on 8th call')
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next(true).value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log('Original (infinite) fibonacci: 11 calls to next(); reset on 8th call')
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next(true).value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
Related
I have an array of functions, as in:
funcArray = [func1, func2, func3];
When in a given function, I want to execute the next function in the array. How do I do this? Here is my basic skeleton:
function func1() {
// I get current function caller
var currentFunc = func1.caller;
// I want to execute the next function. Happens to be func2 in the example.
}
I cannot use indexOf function, as one would for an array of strings or numbers.
NOTE: This question appears to be similar to this and the one it refers to. However, it is a different question.
I want to alter the sequence of processing by merely modifying the array. That's the goal. A possibly more efficient approach would be appreciated.
Clarification: Based upon some of the comments:
funcArray is global.
The goal is to implement middleware for a Node.js HTTP module in as simple and efficient a manner as possible without using any third-party modules.
Unless func1 closes over funcArray, you cannot have it reach out and find func2 and execute it, nor should you. Even if func1 does close over funcArray, it would be poor separation of concerns for func1 to reach out and find itself in funcArray and then execute func2.
Instead, have other code that's in charge of running the functions.
If they're synchronous
If the functions complete their work synchronously, then it's simply:
funcArray.forEach(fn => fn());
or
for (const fn of funcArray) {
fn();
}
or if the result of one function should be passed to the next, you can use reduce:
const finalResult = funcArray.reduce((previousResult, fn) => fn(previousResult), undefined);
...where undefined is the value to pass to func1.
If they're asynchronous
If they don't do their work synchronously, you'll need to provide them a way to notify their caller that they've completed their work. Promises are a good, standard way to do that, but you could use simple callbacks instead.
If you make them return promises, for instance, you can use the old promise reduce trick:
funcArray.reduce((p, fn) => {
return p.then(() => {
fn();
});
}, Promise.resolve());
or if the result of one function should be passed to the next:
funcArray.reduce((p, fn) => {
return p.then(fn);
}, Promise.resolve());
You can provide an argument to Promise.resolve to set the value to pass to func1 (without one, it'll receive undefined).
You can bind to the function the index where it is in the array so you can use this index to get and call the next function:
var funcArray = [func1, func2];
var boundFuncArray = funcArray.map((f, i) => f.bind(null, i));
boundFuncArray[0]();
function func1(nextFunctionIndex) {
console.log('func1 called');
// Execute next function:
var nextFunc = boundFuncArray[nextFunctionIndex + 1];
nextFunc && nextFunc();
}
function func2(nextFunctionIndex) {
console.log('func2 called');
// Execute next function:
var nextFunc = boundFuncArray[nextFunctionIndex + 1];
nextFunc && nextFunc();
}
As T.J Crowder stated in the comment below, you can also bind the next function to the current one:
var funcArray = [func1, func2];
var boundFuncArray= funcArray.map((f, i, arr) => f.bind(null, arr[i + 1]));
boundFuncArray[0]();
function func1(nextFunc) {
console.log('func1 called');
// Execute next function:
nextFunc && nextFunc();
}
function func2(nextFunc ) {
console.log('func2 called');
// Execute next function:
nextFunc && nextFunc();
}
You can get the current function's name with arguments.callee.name, loop through the array of functions, and call the next function:
funcArray = [func1, func2, func3];
// Only func1() and func2() will be documented since the others have repeating code
function func1() {
// show the current function name
console.log(arguments.callee.name);
// loop the array of functions
for(var i = 0; i < funcArray.length; ++i)
{
// when the current array item is our current function name and
// another function exists after this then call it and break
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
function func2() {
console.log(arguments.callee.name);
// some logic which switches our next function to be func4()
funcArray[2] = func4;
for(var i = 0; i < funcArray.length; ++i)
{
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
function func3() {
console.log(arguments.callee.name);
for(var i = 0; i < funcArray.length; ++i)
{
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
function func4() {
console.log(arguments.callee.name);
for(var i = 0; i < funcArray.length; ++i)
{
if(funcArray[i] === arguments.callee && funcArray[i+1])
{
funcArray[i+1]();
break;
}
}
}
// call the first function
funcArray[0]();
Output:
func1
func2
func4
I have solved it this way:
// Adding next options to array
function addNext(array) {
array.last = 1
Object.defineProperty(array, 'next', {get:
function() {
if(this.last < this.length) {
this.last++
return this[this.last-1]
} else {
this.last = 1
return () => {}
}
}
});
}
// The functions for array (has to be function and not arrow function)
function first(param) {
console.log('first',param)
return this.next(param)
}
function second(param) {
console.log('second',param)
return this.next(param)
}
function third(param) {
console.log('third',param)
return this.next(param)
}
// The array
let fns = [first,second,third]
// Adding next option to array
addNext(fns)
// Run first function from array
fns[0]('test')
I dont know if your functions require certain parameters but this is the first thing that came to my mind.
var functArray = [
function() {
console.log("function1 executed");
},
function() {
console.log("function2 executed");
},
function() {
console.log("function3 executed");
},
function() {
console.log("function4 executed");
}];
functArray.forEach(function(x){
x();
});
The accepted answer and other comments did help me, but the way I implemented it is as follows:
//The functions are defined as variables.
//They do not get hoisted, so must be defined first.
func1 = function (arg1, arg2) {
//Code to do whatever...
...
//Execute the next function.
//The name of the function is returned by executing nextFunc()
global[nextFunc()](arg1, arg2, arg3);
}
func2 = function (arg1) { //Note different type of args
...
}
//Note that this is an array of strings representing function names.
funcArray = ["func1", "func2", "func3",...]
//Start the execution...
func1(arg1, arg2);
function nextFunc() {
var currentFuncName = nextFunc.caller.name;
var index = funcArray.indexOf(currentFuncName);
if (index < funcArray.length)
return funcArray[index+1];
}
The sequence of functions to be executed is easily managed through the array funcArray. The number or type of arguments is not fixed for each function. Additionally, the functions control if they should stop the chain or continue with the next function.
It is very simple to understand requiring basic Javascript skills. No overheads of using Promises.
"global" gets replaced by "window" for browser. This is a Node.js implementation. The use of function names in the array will, however, break if you minify the JS code. As I am going to use it on the server, I do not expect to minify it.
You can do it in this way with promise.all if your functions to be executed in parallel.
let toBeExecutedList = [];
toBeExecutedList.push(() => this.addTwoNumber(2, 3));
toBeExecutedList.push(()=>this.square(2));
And Then wherever you want to use them, do it like this:
const resultArr = await Promise.all([
toBeExecutedList.map(func => func()),
]);
I am having trouble finding out what happens if you use a return statement instead of yield.
function *gen(){
const val = yield someAsyncFn();
assert.equal(val,4);
return val;
}
how does the return behave differently from the yield? I assume the return acts as a normal return statement, but the context of a generator function, does it also call gen.return() as well? Sort of confusing.
Perhaps the above is merely identical to this?
function *gen(){
const val = yield someAsyncFn();
assert.equal(val,4);
yield val;
}
return deliveres a return value for an iterators last iteration (when done equals true).
I've simplified your example a bit, since the async operation doesn't seem to be relevant to the question:
function *gen(){
const val = yield 4;
return val * 2;
}
var it = gen();
var val = it.next(); // { value: 4, done: false }
console.log(val.value); // 4
var res = it.next(val.value); // { value: 8, done: true }
console.log(res.value); // 8
Whereas without a return value, on the last iteration you will return a value of undefined:
function *gen2(){
const val = yield 4;
yield val * 2;
}
var it2 = gen2();
var val2 = it2.next(); // { value: 4, done: false }
console.log(val2.value); // 4
var res2 = it2.next(val2.value); // { value: 8, done: false }
console.log(res2.value); // 8
it2.next(); // { value: undefined, done: true }
Sidenote: As a rule of thumb, there is always one more next call then there are yield statements, which is why there is one more next call in the second example.
Let's say you're using a generator-runner like co, then the value you get after finishing the generator would be the value you return:
co(function* () {
var result = yield Promise.resolve(true);
return result;
}).then(function (value) {
console.log(value); // value equals result
}, function (err) {
console.error(err.stack); // err equals result
});
Important: If you are iterating through an iterator, using a for ... of loop or something like Array.from, the return value is going to be ignored (Since you are doing async operations, this is probably not the case anyway):
function *gen(){
const val = yield 4;
return val * 2;
}
for (let value of gen()) {
console.log(value);
}
// 4
In the end, calling a generator just creates an iterator. Whether the final value that the iterator returns is relevant, depends entirely on how you use it.
In addition to the thorough answer by #nils, there is one additional way to capture the return value of a generator function, namely as the value of yield* (necessarily inside another generator function):
function* arrayGenerator(arr) {
for (const element of arr)
yield element
return arr.length
}
function* elementsFollowedByLength(arr) {
const len = yield* arrayGenerator(arr);
yield len;
}
Note the first generator function which returns a value, after it is done yielding the array elements.
The second generator function, through yield*, causes the first generator function to yield all its values. When the first generator function is done and returns, that return value becomes the value of the yield* expression.
I found some code online. I've squashed the original code down into this little excerpt, that when ran, will print 1-20 to the console.
var NumbersFromOne = {
*[Symbol.iterator] () {
for (let i = 1;; ++i) yield i;
}
};
var take = function* (numberToTake, iterable) {
let remaining = numberToTake;
for (let value of NumbersFromOne) {
if (remaining-- <= 0) break;
yield value;
}
}
var printToTwenty = take(20, NumbersFromOne)
console.log(...printToTwenty);
Now, I understand that take() is a GeneratorFunction.
When take() is called, it is given an iterator.
The code "...printToTwenty" uses the spread operator to iterate through that function.
I understand that NumbersFromOne is an object.
I've come here looking for an explanation of what this part means:
*[Symbol.iterator] () {}
Declaring generator functions is done like this: function* () {}
So I'm assuming this isn't declaring a generator function.
* also doesn't represent the function name
* also can't be replaced with another operator (/, -, +)
What is the deal with that syntax, and why is the * before [Symbol.iterator]
If placed after, it will not run.
I had considered that *[Symbol.iterator] () is a way to overwrite the existing iterator property, but then wouldn't it say this[Symbol.iterator].
Thanks!
There are a few things that might make this code look complicated:
It uses the object property shorthand notation. What you're seeing here is actually the following:
var NumbersFromOne = {
[Symbol.iterator]: function* () {
for (let i = 1;; ++i) yield i;
}
};
Symbol.iterator creates a custom iterator for your NumbersFromOne object.
So your code basically means that the iterator of NumbersFromOne is defined as a generator. Instead of manually having to define a function which returns a next and other properties:
var NumbersFromOne = {
[Symbol.iterator]: function () {
var i = 1;
return {
next: function() {
return { value: i++, done: false };
}
};
}
};
Returning the generator creates the next function automatically for. This allows you to yield when you need to.
It can then be called as:
const it = NumbersFromOne[Symbol.iterator]();
it.next(); // 1
it.next(); // 2
it.next(); // 3
// ...
Note: Written this way, this iterator never ends! So if you were to call it in a for ... of loop without an end-condition, it would freeze your program.
Thats how I do it:
function processArray(array, index, callback) {
processItem(array[index], function(){
if(++index === array.length) {
callback();
return;
}
processArray(array, index, callback);
});
};
function processItem(item, callback) {
// do some ajax (browser) or request (node) stuff here
// when done
callback();
}
var arr = ["url1", "url2", "url3"];
processArray(arr, 0, function(){
console.log("done");
});
Is it any good? How to avoid those spaghetti'ish code?
Checkout the async library, it's made for control flow (async stuff) and it has a lot of methods for array stuff: each, filter, map. Check the documentation on github. Here's what you probably need:
each(arr, iterator, callback)
Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the each function is immediately called with the error.
eachSeries(arr, iterator, callback)
The same as each only the iterator is applied to each item in the array in series. The next iterator is only called once the current one has completed processing. This means the iterator functions will complete in order.
As pointed in some answer one can use "async" library. But sometimes you just don't want to introduce new dependency in your code. And below is another way how you can loop and wait for completion of some asynchronous functions.
var items = ["one", "two", "three"];
// This is your async function, which may perform call to your database or
// whatever...
function someAsyncFunc(arg, cb) {
setTimeout(function () {
cb(arg.toUpperCase());
}, 3000);
}
// cb will be called when each item from arr has been processed and all
// results are available.
function eachAsync(arr, func, cb) {
var doneCounter = 0,
results = [];
arr.forEach(function (item) {
func(item, function (res) {
doneCounter += 1;
results.push(res);
if (doneCounter === arr.length) {
cb(results);
}
});
});
}
eachAsync(items, someAsyncFunc, console.log);
Now, running node iterasync.js will wait for about three seconds and then print [ 'ONE', 'TWO', 'THREE' ]. This is a simple example, but it can be extended to handle many situations.
As correctly pointed out, you have to use setTimeout, for example:
each_async = function(ary, fn) {
var i = 0;
-function() {
fn(ary[i]);
if (++i < ary.length)
setTimeout(arguments.callee, 0)
}()
}
each_async([1,2,3,4], function(p) { console.log(p) })
The easiest way to handle async iteration of arrays (or any other iterable) is with the await operator (only in async functions) and for of loop.
(async function() {
for(let value of [ 0, 1 ]) {
value += await(Promise.resolve(1))
console.log(value)
}
})()
You can use a library to convert any functions you may need which accept callback to return promises.
In modern JavaScript there are interesting ways to extend an Array into an async itarable object.
Here I would like to demonstrate a skeleton of a totally new type AsyncArray which extends the Array type by inheriting it's goodness just to become an async iterable array.
This is only available in the modern engines. The code below uses the latest gimmicks like the private instance fields and for await...of.
If you are not familiar with them then I would advise you to have a look at the above linked topics in advance.
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
So an Async Iterable Array must contain promises. Only then it can return an iterator object which with every next() call returns a promise to eventually resolve into an object like {value : "whatever", done: false} or {done: true}. So basically everything returned is a promise here. The await abstraction unpacks the value within and gives it to us.
Now as I mentioned before, this AsyncArray type, since extended from Array, allows us to use those Array methods we are familiar with. That should simplify our job.
Let's see what happens;
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
var aa = AsyncArray.from({length:10}, (_,i) => new Promise(resolve => setTimeout(resolve,i*1000,[i,~~(Math.random()*100)])));
async function getAsycRandoms(){
for await (let random of aa){
console.log(`The Promise at index # ${random[0]} gets resolved with a random value of ${random[1]}`);
};
};
getAsycRandoms();
For modern Node.js:
To iterate through a collection truly asynchronously, you can try my tiny package with zero dependencies, compatible with ESM and CJS modules with .d.ts typings. Check the code it's really tiny.
https://www.npmjs.com/package/array-to-async-iterable
You can use it just like this:
for await(const el of new AsyncTimeIterator(arrayOfObjects)){
...
}
You can't just use for await of loop because of the JavaScript engines' microtasks and macrotasks nature.
In a brief, you won't get new HTTP requests and let other timers' callbacks to be executed with this code:
for await(const el of array){
...
}
You force V8 or the other engine to execute all the microtasks (your loop iteration) and when the loop completes you'll unblock the event loop and be ready to receive HTTP connections. So this code is completely useless.
Thats how I do it:
function processArray(array, index, callback) {
processItem(array[index], function(){
if(++index === array.length) {
callback();
return;
}
processArray(array, index, callback);
});
};
function processItem(item, callback) {
// do some ajax (browser) or request (node) stuff here
// when done
callback();
}
var arr = ["url1", "url2", "url3"];
processArray(arr, 0, function(){
console.log("done");
});
Is it any good? How to avoid those spaghetti'ish code?
Checkout the async library, it's made for control flow (async stuff) and it has a lot of methods for array stuff: each, filter, map. Check the documentation on github. Here's what you probably need:
each(arr, iterator, callback)
Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the each function is immediately called with the error.
eachSeries(arr, iterator, callback)
The same as each only the iterator is applied to each item in the array in series. The next iterator is only called once the current one has completed processing. This means the iterator functions will complete in order.
As pointed in some answer one can use "async" library. But sometimes you just don't want to introduce new dependency in your code. And below is another way how you can loop and wait for completion of some asynchronous functions.
var items = ["one", "two", "three"];
// This is your async function, which may perform call to your database or
// whatever...
function someAsyncFunc(arg, cb) {
setTimeout(function () {
cb(arg.toUpperCase());
}, 3000);
}
// cb will be called when each item from arr has been processed and all
// results are available.
function eachAsync(arr, func, cb) {
var doneCounter = 0,
results = [];
arr.forEach(function (item) {
func(item, function (res) {
doneCounter += 1;
results.push(res);
if (doneCounter === arr.length) {
cb(results);
}
});
});
}
eachAsync(items, someAsyncFunc, console.log);
Now, running node iterasync.js will wait for about three seconds and then print [ 'ONE', 'TWO', 'THREE' ]. This is a simple example, but it can be extended to handle many situations.
As correctly pointed out, you have to use setTimeout, for example:
each_async = function(ary, fn) {
var i = 0;
-function() {
fn(ary[i]);
if (++i < ary.length)
setTimeout(arguments.callee, 0)
}()
}
each_async([1,2,3,4], function(p) { console.log(p) })
The easiest way to handle async iteration of arrays (or any other iterable) is with the await operator (only in async functions) and for of loop.
(async function() {
for(let value of [ 0, 1 ]) {
value += await(Promise.resolve(1))
console.log(value)
}
})()
You can use a library to convert any functions you may need which accept callback to return promises.
In modern JavaScript there are interesting ways to extend an Array into an async itarable object.
Here I would like to demonstrate a skeleton of a totally new type AsyncArray which extends the Array type by inheriting it's goodness just to become an async iterable array.
This is only available in the modern engines. The code below uses the latest gimmicks like the private instance fields and for await...of.
If you are not familiar with them then I would advise you to have a look at the above linked topics in advance.
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
So an Async Iterable Array must contain promises. Only then it can return an iterator object which with every next() call returns a promise to eventually resolve into an object like {value : "whatever", done: false} or {done: true}. So basically everything returned is a promise here. The await abstraction unpacks the value within and gives it to us.
Now as I mentioned before, this AsyncArray type, since extended from Array, allows us to use those Array methods we are familiar with. That should simplify our job.
Let's see what happens;
class AsyncArray extends Array {
#INDEX;
constructor(...ps){
super(...ps);
if (this.some(p => p.constructor !== Promise)) {
throw "All AsyncArray items must be a Promise";
}
}
[Symbol.asyncIterator]() {
this.#INDEX = 0;
return this;
};
next() {
return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
: Promise.resolve({done: true});
};
};
var aa = AsyncArray.from({length:10}, (_,i) => new Promise(resolve => setTimeout(resolve,i*1000,[i,~~(Math.random()*100)])));
async function getAsycRandoms(){
for await (let random of aa){
console.log(`The Promise at index # ${random[0]} gets resolved with a random value of ${random[1]}`);
};
};
getAsycRandoms();
For modern Node.js:
To iterate through a collection truly asynchronously, you can try my tiny package with zero dependencies, compatible with ESM and CJS modules with .d.ts typings. Check the code it's really tiny.
https://www.npmjs.com/package/array-to-async-iterable
You can use it just like this:
for await(const el of new AsyncTimeIterator(arrayOfObjects)){
...
}
You can't just use for await of loop because of the JavaScript engines' microtasks and macrotasks nature.
In a brief, you won't get new HTTP requests and let other timers' callbacks to be executed with this code:
for await(const el of array){
...
}
You force V8 or the other engine to execute all the microtasks (your loop iteration) and when the loop completes you'll unblock the event loop and be ready to receive HTTP connections. So this code is completely useless.