Develop Reference

How can I find out if a javascript iterator terminates early?

Lets say I have a generator:
function* source() {
yield "hello"; yield "world";
}
I create the iterable, iterate with a for-loop, and then break out of the loop before the iterator fully completes (returns done).
function run() {
for (let item of source()) {
console.log(item);
break;
}
}
Question: How can I find out, from the iterable-side, that the iterator terminated early?
There doesn't seem to be any feedback if you try to do this directly in the generator itself:
function* source2() {
try {
let result = yield "hello";
console.log("foo");
} catch (err) {
console.log("bar");
}
}
... neither "foo" nor "bar" is logged.

Edit: See Newer accepted answer. I will keep this as it does/did work, and I was pretty happy at the time I was able to hack a solution. However, as you can see in the accepted answer, the finally solution is so simple now it has been identified.
I noticed that typescript defines Iterator (lib.es2015) as:
interface Iterator<T> {
next(value?: any): IteratorResult<T>;
return?(value?: any): IteratorResult<T>;
throw?(e?: any): IteratorResult<T>;
}
I intercepted these methods and logged calls and it does appear that if an iterator is terminated early --at least via a for-loop-- then the return method is called. It will also be called if the consumer throws an error. If the loop is allowed to fully iterate the iterator return is not called.
Return hack
So, I made a bit of a hack to allow capturing another iterable - so I don't have to re-implement the iterator.
function terminated(iterable, cb) {
return {
[Symbol.iterator]() {
const it = iterable[Symbol.iterator]();
it.return = function (value) {
cb(value);
return { done: true, value: undefined };
}
return it;
}
}
}
function* source() {
yield "hello"; yield "world";
}
function source2(){
return terminated(source(), () => { console.log("foo") });
}
for (let item of source2()) {
console.log(item);
break;
}
and it works!
hello
foo
remove the break and you get:
hello
world
Check after each yield
While typing this answer, I realised a better problem/solution is to find out in the original generator method.
The only way I can see to pass information back to the original iterable is to use next(value). So if we pick some unique value (say Symbol.for("terminated")) to signal the termination, and we alter the above return-hack to call it.next(Symbol.for("terminated")):
function* source() {
let terminated = yield "hello";
if (terminated == Symbol.for("terminated")) {
console.log("FooBar!");
return;
}
yield "world";
}
function terminator(iterable) {
return {
[Symbol.iterator]() {
const it = iterable[Symbol.iterator]();
const $return = it.return;
it.return = function (value) {
it.next(Symbol.for("terminated"));
return $return.call(it)
}
return it;
}
}
}
for (let item of terminator(source())) {
console.log(item);
break;
}
Success!
hello
FooBar!
Chaining Cascades Return
If you chain some extra transform iterators, then the return call cascades through them all:
function* chain(source) {
for (let item of source) { yield item; }
}
for (let item of chain(chain(terminator(source())))) {
console.log(item);
break
}
hello
FooBar!
Package
I've wrapped the above solution as a package. It supports both [Symbol.iterator] and [Symbol.asyncIterator]. The async iterator case is of particular interest to me, especially when some resource needs to be disposed of correctly.

There's a much simpler way to do this: use a finally block.
function *source() {
let i;
try {
for(i = 0; i < 5; i++)
yield i;
}
finally {
if(i !== 5)
console.log(' terminated early');
}
}
console.log('First:')
for(const val of source()) {
console.log(` ${val}`);
}
console.log('Second:')
for(const val of source()) {
console.log(` ${val}`);
if(val > 2)
break;
}
...yields:
First:
0
1
2
3
4
Second:
0
1
2
3
terminated early

I've run into a similar need to figure out when an iterator terminates early. The accepted answer is really clever and probably the best way to solve the problem generically, but I think this solution could also be helpful for other uses cases.
Say, for example, you have an infinite iterable, such as the fibonacci sequence described in MDN's Iterators and Generators docs.
In any sort of loop there needs to be a condition set to break out of the loop early, like in the solution already given. But what if you want to destructure the iterable in order to create an array of values? In that case, you'd want to limit the number of iterations, essentially setting a maximum length on the iterable.
To do this, I wrote a function called limitIterable, which takes as arguments an iterable, an iteration limit, and an optional callback function to be executed in the event that the iterator terminates early. The return value is a Generator object (which is both an iterator and an iterable) created using an Immediately Invoked (Generator) Function Expression.
When the generator is executed, whether in a for..of loop, with destructuring, or by calling the next() method, it will check to see if iterator.next().done === true or iterationCount < iterationLimit. In the case of an infinite iterable like the fibonacci sequence, the latter will always cause the while loop to be exited. However, note that one could also set an iterationLimit that is greater than the length of some finite iterable, and everything will still work.
In either case, once the while loop is exited the most recent result will be checked to see if the iterator is done. If so, the original iterable's return value will be used. If not, the optional callback function is executed and used as a return value.
Note that this code also allows the user to pass in values to next(), which will in turn be passed to the original iterable (see the example using MDN's fibonacci sequence inside the code snippet attached). It also allows for additional calls to next() beyond the set iterationLimit within the callback function.
Run the code snippet to see the results of a few possible use cases! Here's the limitIterable function code by itself:
function limitIterable(iterable, iterationLimit, callback = (itCount, result, it) => undefined) {
// callback will be executed if iterator terminates early
if (!(Symbol.iterator in Object(iterable))) {
throw new Error('First argument must be iterable');
}
if (iterationLimit < 1 || !Number.isInteger(iterationLimit)) {
throw new Error('Second argument must be an integer greater than or equal to 1');
}
if (!(callback instanceof Function)) {
throw new Error('Third argument must be a function');
}
return (function* () {
const iterator = iterable[Symbol.iterator]();
// value passed to the first invocation of next() is always ignored, so no need to pass argument to next() outside of while loop
let result = iterator.next();
let iterationCount = 0;
while (!result.done && iterationCount < iterationLimit) {
const nextArg = yield result.value;
result = iterator.next(nextArg);
iterationCount++;
}
if (result.done) {
// iterator has been fully consumed, so result.value will be the iterator's return value (the value present alongside done: true)
return result.value;
} else {
// iteration was terminated before completion (note that iterator will still accept calls to next() inside the callback function)
return callback(iterationCount, result, iterator);
}
})();
}
function limitIterable(iterable, iterationLimit, callback = (itCount, result, it) => undefined) {
// callback will be executed if iterator terminates early
if (!(Symbol.iterator in Object(iterable))) {
throw new Error('First argument must be iterable');
}
if (iterationLimit < 1 || !Number.isInteger(iterationLimit)) {
throw new Error('Second argument must be an integer greater than or equal to 1');
}
if (!(callback instanceof Function)) {
throw new Error('Third argument must be a function');
}
return (function* () {
const iterator = iterable[Symbol.iterator]();
// value passed to the first invocation of next() is always ignored, so no need to pass argument to next() outside of while loop
let result = iterator.next();
let iterationCount = 0;
while (!result.done && iterationCount < iterationLimit) {
const nextArg = yield result.value;
result = iterator.next(nextArg);
iterationCount++;
}
if (result.done) {
// iterator has been fully consumed, so result.value will be the iterator's return value (the value present alongside done: true)
return result.value;
} else {
// iteration was terminated before completion (note that iterator will still accept calls to next() inside the callback function)
return callback(iterationCount, result, iterator);
}
})();
}
// EXAMPLE USAGE //
// fibonacci function from:
//https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators#Advanced_generators
function* fibonacci() {
let fn1 = 0;
let fn2 = 1;
while (true) {
let current = fn1;
fn1 = fn2;
fn2 = current + fn1;
let reset = yield current;
if (reset) {
fn1 = 0;
fn2 = 1;
}
}
}
console.log('String iterable with 26 characters terminated early after 10 iterations, destructured into an array. Callback reached.');
const itString = limitIterable('abcdefghijklmnopqrstuvwxyz', 10, () => console.log('callback: string terminated early'));
console.log([...itString]);
console.log('Array iterable with length 3 terminates before limit of 4 is reached. Callback not reached.');
const itArray = limitIterable([1,2,3], 4, () => console.log('callback: array terminated early?'));
for (const val of itArray) {
console.log(val);
}
const fib = fibonacci();
const fibLimited = limitIterable(fibonacci(), 9, (itCount) => console.warn(`Iteration terminated early at fibLimited. ${itCount} iterations completed.`));
console.log('Fibonacci sequences are equivalent up to 9 iterations, as shown in MDN docs linked above.');
console.log('Limited fibonacci: 11 calls to next() but limited to 9 iterations; reset on 8th call')
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next(true).value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log(fibLimited.next().value);
console.log('Original (infinite) fibonacci: 11 calls to next(); reset on 8th call')
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next(true).value);
console.log(fib.next().value);
console.log(fib.next().value);
console.log(fib.next().value);

How does Generator.next() processes its parameter?

The documentation says that "You can also provide a parameter to the next method to send a value to the generator." Where does it sends it to?
For example, take these 3 generators:
function* one() {
while(true) {
var value = yield null;
}
}
var g1 = one(); g1.next();
g1.next(1000); //yields null
function* two() {
var i = 0;
while (true) {
i += yield i;
}
}
var g2 = two(); g2.next();
g2.next(1000) // yields 1000
function* three(){
var index = 0;
while (true)
yield index++;
}
g3 = three();
g3.next();
g3.next(1000); // yields 1
In generators 3 and 1, the argument passed has no effect on next. Why is that? How does generator 2 calculates its return value? Why it is affected by the given argument?

The key to understanding this is knowing how the next function retrieves the argument passed to next(), which is as the return value of the yield operator:
[rv] = yield [expression];
Independently of the value of [expression], yield will assign to rv the value passed to next().
But, here comes the tricky part: yield will only assign the value passed to next() when resuming execution from a previous iteration. As a consequence, on the first iteration, yield does not assign anything to rv.
For example, if I have this generator:
function* gen() {
// On the first iteration, yield does not return anything.
//because it returns something ONLY when execution is resumed
returnedFromYield = yield 'foo';
yield returnedFromYield;
}
returnedFromYield is undefined on the first iteration. When execution is resumed on the second iteration, yield assigns the passed value to the returnedFromYield variable, which is then returned:
g.next(1); // 'foo'
g.next(2); // 2
Let's review another example:
function* gen() {
yield yield yield 5;
}
On the first iteration, (g.next()), yield will return 5, on the second iteration, (g.next(10)) yield is going to pass 10 to the second yield. That is, yield yield yield 5; on the second iteration is equivalent to yield yield 10;, and, on the third iteration, it's equivalent to yield valuePassedToNext.

Here's an annotated example showing the logical flow and scope for the values passed to a generator's next method. The affected area of the next method call is highlighted in the corresponding color.
a: There is no yield, so argument 'one' is ignored. The generator pauses on line 5, returning the value 1.
b: Execution resumes and 'two' replaces yield on line 5, and is assigned to the yieldValue variable. Line 6 is executed. The generator pauses on line 8, returning the value 2.
c: Execution resumes and 'three' replaces yield on line 8, and is assigned to the yieldValue variable. Line 9 is executed. The generator pauses on line 11, returning the value 3.
d: Execution resumes and 'four' replaces yield on line 11, and is assigned to the yieldValue variable. Line 12 is executed. Line 14 is executed. Finally the generator is now done, returning the value undefined.
See the logs on line 28-36

Using return in ES6 generator function

I am having trouble finding out what happens if you use a return statement instead of yield.
function *gen(){
const val = yield someAsyncFn();
assert.equal(val,4);
return val;
}
how does the return behave differently from the yield? I assume the return acts as a normal return statement, but the context of a generator function, does it also call gen.return() as well? Sort of confusing.
Perhaps the above is merely identical to this?
function *gen(){
const val = yield someAsyncFn();
assert.equal(val,4);
yield val;
}

return deliveres a return value for an iterators last iteration (when done equals true).
I've simplified your example a bit, since the async operation doesn't seem to be relevant to the question:
function *gen(){
const val = yield 4;
return val * 2;
}
var it = gen();
var val = it.next(); // { value: 4, done: false }
console.log(val.value); // 4
var res = it.next(val.value); // { value: 8, done: true }
console.log(res.value); // 8
Whereas without a return value, on the last iteration you will return a value of undefined:
function *gen2(){
const val = yield 4;
yield val * 2;
}
var it2 = gen2();
var val2 = it2.next(); // { value: 4, done: false }
console.log(val2.value); // 4
var res2 = it2.next(val2.value); // { value: 8, done: false }
console.log(res2.value); // 8
it2.next(); // { value: undefined, done: true }
Sidenote: As a rule of thumb, there is always one more next call then there are yield statements, which is why there is one more next call in the second example.
Let's say you're using a generator-runner like co, then the value you get after finishing the generator would be the value you return:
co(function* () {
var result = yield Promise.resolve(true);
return result;
}).then(function (value) {
console.log(value); // value equals result
}, function (err) {
console.error(err.stack); // err equals result
});
Important: If you are iterating through an iterator, using a for ... of loop or something like Array.from, the return value is going to be ignored (Since you are doing async operations, this is probably not the case anyway):
function *gen(){
const val = yield 4;
return val * 2;
}
for (let value of gen()) {
console.log(value);
}
// 4
In the end, calling a generator just creates an iterator. Whether the final value that the iterator returns is relevant, depends entirely on how you use it.

In addition to the thorough answer by #nils, there is one additional way to capture the return value of a generator function, namely as the value of yield* (necessarily inside another generator function):
function* arrayGenerator(arr) {
for (const element of arr)
yield element
return arr.length
}
function* elementsFollowedByLength(arr) {
const len = yield* arrayGenerator(arr);
yield len;
}
Note the first generator function which returns a value, after it is done yielding the array elements.
The second generator function, through yield*, causes the first generator function to yield all its values. When the first generator function is done and returns, that return value becomes the value of the yield* expression.

Javascript strange generator yield sub function behavior

I'm using MySQL (mysql-co) and ASQ(asynquence) in a simple project to get a better understanding of ES6 generators and yield functions, and I'm stumped on an odd behavior.
Short explanation of asynquence
asynquence (https://github.com/getify/asynquence) provides an easy way for me to run generators in sequence. It can also do pseudo-parallel execution but that's not what I need for now. The structure of function *x(token) is from there. token holds a connection object at [0]. yield token passes control on to the next generator function in sequence.
Code Sample 1 (works)
function *test1(token) {
var conn = token.messages[0];
var values = {id:1, dev:1, description:'This is it!'};
yield conn.query("INSERT INTO version SET ?", values);
yield token;
}
This works fine. The row described above gets inserted. I didn't know the MySQL driver allowed such a simple looking insert function but it does.
Code Sample 2 (doesn't work)
function *test1(token) {
var conn = token.messages[0];
var values = {id:1, dev:1, description:'This is it!'};
yield subtest1(conn, values);
yield token;
}
function *subtest1(conn, values) {
yield conn.query("INSERT INTO version SET ?", values);
}
This doesn't work. The actual code in question for subtest1 is in a model class, so I would prefer not to have it merged in with the controller.
I've tried a bunch of different things around with or without yield on the subtest function.
What's going on?

subtest1(conn, values) is a generator. yielding a generator object does not execute its body. That is, the yielded generator remains suspended, and it would require a call to the next() method for the first yield to be reached. There are no explicit or implicit calls to next() in Code Sample 2, and this is the reason conn.query(...) isn't executed.
How about yield* subtest1(conn, values)? From the linked page:
The yield* expression iterates over the operand and yields each value
returned by it.
It will still execute subtest lazily.
An alternative solution is to turn subtest into a regular function and return the result of conn.query(...) (assuming you only need to perform one query):
function subtest1(conn, values) {
return conn.query("INSERT INTO version SET ?", values);
}

yield subtest1(conn, values) calls subtest1(conn, values), which returns an iterator object, and then yields that from test1 as the value for that iteration of test1. It doesn't iterate the values returned by the subtest1 iterator.
You could have test1 iterate the values from subtest1's iterator by adding a * after the yield:
yield* subtest1(conn, values);
but looking at your code, I don't think you want to. If you want to split the conn.query line out into a function, it would look like this:
function *test1(token) {
var conn = token.messages[0];
var values = {id:1, dev:1, description:'This is it!'};
yield subtest1(conn, values);
yield token;
}
function subtest1(conn, values) {
return conn.query("INSERT INTO version SET ?", values);
}
This simpler version may help with the difference between yield and yield*:
function* foo() {
console.log("f");
yield bar();
console.log("f done");
}
function* bar() {
console.log("b1");
let i = 0;
while (i < 3) {
console.log("b2");
yield i;
++i;
}
console.log("b done");
}
for (let v of foo()) {
console.log(v);
}
The output of that is: (live copy on Babel's REPL)
f
{}
f done
Note how we don't see any of the "b" logging at all; that's because the {} you see after f1 is the iterator object returned by calling bar.
If we just add * on the yield bar(); line, turning it into yield* bar(), the output changes: (live copy)
f
b1
b2
0
b2
1
b2
2
b done
f done
If the operand to yield* is an iterator, yield* iterates it, yielding each of its values. It's basically:
for (value of bar()) {
yield value;
}

Are generators really intrusive

As I understand the current spec for Javascript generators, you have to mark functions containing yields explicitly.
I wonder what the rationate behind this is.
If this is true, it would force people to write:
let thirdfunc = function*() {
let value = 5;
let other = yield 6;
return value;
};
let secondfunc = function*() {
yield thirdfunc();
};
let firstfunc = function*() {
yield secondfunc();
};
let gen = function*() {
// some code
// more code
yield firstfunc();
// and code
};
let it = gen();
while( !it.done() ) {
it.next();
};
Which means, generators would spread like cancer in a codebase.
While in the end, to the developer only yielding and handling the iterator is really interesting.
I would find it much more practical, to just define, where I want to handle the iteration.
let thirdfunc = function() {
let value = 5;
let other = yield 6; // Change 1: incorporate yield
return value;
};
let secondfunc = function() {
thirdfunc();
};
let firstfunc = function() {
secondfunc();
};
let gen = function*() { // Change 2: at this level I want to deal with descendant yields
// some code
// more code
firstfunc();
// and code
};
// Change 3: Handle iterator
let it = gen();
while( !it.done() ) {
it.next();
}
If the browser then has to turn everything between the yield call and the generator handler (firstfunc, secondfunc, thirdfunc) into promise / future form, that should work automagically and not be the business of Javascript developers.
Or are there really good arguments for not doing this?

I described the rationale for this aspect of the design at http://calculist.org/blog/2011/12/14/why-coroutines-wont-work-on-the-web/ -- in short, full coroutines (which is what you describe) interfere with the spirit of JavaScript's run-to-completion model, making it harder to predict when your code can be "pre-empted" in a similar sense to multithreaded languages like Java, C#, and C++. The blog post goes into more detail and some other reasons as well.