Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have 2 very related questions. One practical, relating to a real problem I have, and one more theoretical. Let me start with the latter:
Q1: Does async/await + generators make sense?
In general, it doesn't make too much sense to have yield inside of callbacks or promises. The problem (among others) is there could be races between multiple yields, it's impossible to tell when the generator ends, etc.
function* this_is_nonsense() {
setTimeout(() => { yield 'a'; }, 500);
setTimeout(() => { yield 'b'; }, 500);
return 'c';
}
When using async/await these problems seem to actually mostly go away
function timeout(ns) {
return new Promise((resolve) => setTimeout(resolve, ns));
}
async function* this_could_maybe_make_sense() {
// now stuff is sequenced instead of racing
await timeout(500);
yield 'a';
await timeout(500);
yield 'b';
return 'c';
}
I assume nothing like this is currently possible (please correct me if wrong!). So my first question is, is there a conceptual issue with using generators with async/await? If not, is it on any kind of roadmap? It seems like implementation would require more than just mashing together the two features.
Q2: How to write lazy wiki crawler
This is a contrived version of a real code problem I have.
Imagine writing a function that does traversal of Wikipedia, starting from some arbitrary page and repeatedly following links (doesn't matter how - can be depth first, breadth first or something else). Assume we can asynchronously crawl Wikipedia pages to get linked pages. Something like:
async function traverseWikipedia(startLink) {
const visited = {};
const result = [];
function async visit(link) {
if (visited[link]) { return; }
visited[link] = true;
result.push(link);
const neighboringLinks = await getNeighboringWikipediaLinks(link);
for (const i = 0; i < neighboringLinks.length; i++) {
await visit(neighboringLinks[i]);
}
// or to parallelize:
/*
await Promise.all(
neighboringLinks.map(
async (neighbor) => await visit(neighbor)
)
);
*/
);
await visit(startLink);
return result;
}
This is fine... except that sometimes I only want the first 10 links, and this is gonna crawl over tons of links. Sometimes I'm searching for the first 10 that contains the substring 'Francis'. I want a lazy version of this.
Conceptually, instead of a return type of Promise<Array<T>>, I want Generator<T>, or at least Generator<Promise<T>>. And I want the consumer to be able to do arbitrary logic, including asynchronous stopping condition. Here's a fake, totally wrong (i assume) version:
function* async traverseWikipedia(startLink) {
const visited = {};
function* async visit(link) {
if (visited[link]) { return; }
visited[link] = true;
yield link;
const neighboringLinks = await getNeighboringWikipediaLinks(link);
for (const i = 0; i < neighboringLinks.length; i++) {
yield* await visit(neighboringLinks[i]);
}
);
yield* await visit(startLink);
}
This is the same nonsense from Q1, combining async/await + generators as if it did what I want. But if I did have it, I could consume like this:
function find10LinksWithFrancis() {
const results = []
for (const link of traverseWikipedia()) {
if (link.indexOf('Francis') > -1) {
results.push(link);
}
if (results.length === 10) {
break;
}
}
return results;
}
But I could also search for different number of results and strings, have asynchronous stopping conditions, etc. For example, it could show 10 results, and when the user presses a button, keeps crawling and shows the next 10.
I'm okay with just using promises/callbacks, without async/await, as long as I still get the same API for the consumer.
The caller of the generator can do asynchronous things. So I think it's possible to implement this, where the generator function yields the promise of neighbors, and it's up to the caller to call gen.next with the resulting actual neighbors. Something like:
function* traverseWikipedia(startLink) {
const visited = {};
function* visit(link) {
if (visited[link]) { return; }
visited[link] = true;
yield { value: link };
const neighboringLinks = yield { fetch: getNeighboringWikipediaLinks(link) };
for (const i = 0; i < neighboringLinks.length; i++) {
yield* visit(neighboringLinks[i]);
}
);
yield* visit(startLink);
}
But that's not very nice to use... you have to detect the case of an actual result vs. the generator asking you to give it back an answer, it's less typesafe, etc. I just want something that looks like Generator<Promise<T>> and a relatively straightforward way to use it. Is this possible?
Edited to add: it occurred to me you could implement this if the body of the generator function could access the Generator it returned somehow... not sure how to do that though
Edited again to add example of consumer, and note: I would be okay with the consumer being a generator also, if that helps. I'm thinking it might be possible with some kind of engine layer underneath, reminiscent of redux-saga. Not sure if anyone has written such a thing
Related
I am thinking about a scenario of building up a promise queue:
//Let's assume that promises is an array of promises
var promiseQueue = [];
for (var promise of promises) {
if (promiseQueue.length) promiseQueue[promiseQueue.length - 1].then(promise);
promiseQueue.push(promise);
}
I am thinking about implementing a function called resolver:
function *resolve() {
var promise;
while (promise = yield) Promise.resolve(promise);
}
and then iterating it:
var promiseGenerator = resolve();
The problem is the for..of here which would be responsible for the actual iteration:
for (var r of promiseGenerator) {
}
At the code above the generator will be successfully iterated, but unfortunately I am not aware of a way to successfully pass a parameter to this generator at the iteration of for..of.
I would like to clarify that I do not need an alternative, I am perfectly aware that we can do something like this:
for (var p in promiseQueue) promiseGenerator.next(promiseQueue[p]);
I am specifically interested to know whether I can pass parameters to the generator when I execute a for..of cycle.
EDIT
The problem raised by amn is that in the example he/she was focusing on would always get undefined. That's true if we pass undefined to next(), but not true if we pass something else. The problem I was raising is that a for..of loop does not allow us to pass anything to yield, which is this specific question is all about, the example is a mere illustration of the problem, showing that the promises we would create will never be created in a for..of loop. However, there is life for Iterable objects outside the realm of for..of loops and we can pass defined values into the yield. An example with the criticized code chunk can look like:
function *resolve() {
var promise;
while (promise = yield) Promise.resolve(promise);
}
var responses = [];
var f = resolve();
var temp;
for (var i = 10; !(temp = f.next(i)).done; i--) responses.push(temp);
As we can see above, the yield above cannot be assumed ab ovo to be undefined. And of course we can pass some custom thenables, like
Promise.resolve({
then: function(onFulfill, onReject) { onFulfill('fulfilled!'); }
});
or even promises which were not resolved yet. The point of the example was to show that we cannot pass values to the yield using the for..of loop, which is quite a feature gap in my opinion.
No, it is not possible to pass arguments to next.
function* generateItems() { /* ... */ }
for (var item of generateItems()) {
console.log(item);
}
is mostly short for
function* generateItems() { /* ... */ }
var iterator = generateItems()[Symbol.iterator]();
do {
const result = iterator.next();
if (result.done) break;
const item = result.value;
console.log(item);
} while (true);
barring a few missing try/catch wrappers. You can see in the spec here that it calls .next with no arguments:
Let nextResult be ? Call(iteratorRecord.[[NextMethod]], iteratorRecord.[[Iterator]], « »).
e.g.
iterator.next.apply(iterator, []);
calling next() with an empty array of arguments.
To give you an idea what I'm trying to do:
As someone who teaches sorting algorithms, I'd like to enable students to easily visualize how their sorting algorithms work and found generators to be realy useful for that, sice I can interrupt the execution at any point: The following code that a student could write can be turned into an animation by my library:
function* bubbleSort(arr){
let done = false;
while(!done){
done = true;
for(let i=1;i<arr.length;i++){
yield {type:"comparison",indexes:[i-1,i]};
if(arr[i-1]>arr[i]){
yield {type:"swap",indexes:[i-1,i]};
swap(arr,i-1,i);
done = false;
}
}
}
return "sorted";
}
This works quite well, but it would be a lot better, if I could write functions compare(i,j) and swap(i,j) that handle the yield internally (and in case of compare also return a boolean value). So I'd like to be able to express the above as:
function* bubbleSort(arr){
let done = false;
while(!done){
done = true;
for(let i=1;i<arr.length;i++){
if(compare(i-1,i)){
swap(arr,i-1,i);
done = false;
}
}
}
return "sorted";
}
Is there a way to do this?
You could just do
if(yield* compare(i-1,i))
Which will pass the yield calls inside of compare to the outside.
What I can do
I figured out that the asynchronous generator pattern is fairly new to JavaScript and only available in Node.js as of version 10. So after I did all that, I can now use the following function to yield multiple similar elements in a web page using Selenium, which uses async/await for many of its functions:
formElements = async function*(containerCss, childCss, defaultCount) {
let countCSS = containerCss + " " + childCss;
let numElements = await countCss(countCSS) || defaultCount;
for (let cIndex = 1; cIndex <= numElements; cIndex++) {
let elementCss = containerCss + ":nth-child(" + cIndex + ") " + childCss;
yield await elmCSS(elementCss);
}
}
This function works fine as an asynchronous generator using this call:
for await (const button of formElements("button-container-css", "button-css", 6))
{
await clickOn(button);
//other async stuff...
}
What I can't figure out
But what I want to do is take those specific parameters out of the main function and have a helper function (formButtons()) provide them instead, which I think should look like this (?):
formButtons = async function*() {
yield await formElements("button-container-css", "button-css", 6);
}
so my main function can be a LITTLE cleaner:
for await (const button of formButtons())
{
await clickOn(button);
//other async stuff...
}
But it literally skips everything once it hits the yield in formButtons(), returning null and causing errors. I think the error must either be in my formButtons() function or else it's some unfortunate artifact of not using the stable release of Node, but I'm hopeful it's something I can fix
Is there an error or a workaround for this?
With the caveat that I've used generators only a few times in toy applications, the situation you've got is basically a delegation from one generator to another one. Since what you want to do is "pass through" control to the sub-generator completely, you can use yield* instead of yield. It basically means, "yield everything you can get from this generator". So:
formButtons = async function*() {
yield* await formElements("button-container-css", "button-css", 6);
}
I understand how to use generators to make async code look nice. I have a simple generator *all, that takes a page, will return a single value.
Then I have another generator *allDo, that will use *all for pages 1 to 30 and for each result, do some async task.
Then I have another generator *allBatchDo, that will batch 3 pages, and do some async task.
function mockPromise(value) {
return Promise(function(resolve, reject) {
resolve(value);
});
}
function *all(page) {
var ls = yield mockPromise("page " + page);
// do all kinds of promises
return yield ls;
};
function *allDo(task) {
var page = 1;
while (true) {
var res = yield * all(page);
res = yield task(res);
if (page == 30) {
break;
}
page++;
}
}
function *allBatchDo(task) {
var page = 1;
var arr = [];
while (true) {
var res = yield * all(author, page);
arr.push(res);
if (arr.length >= 3) {
yield task(arr);
arr = [];
}
if (page == 30) {
break;
}
page++;
}
}
function logTask(res) {
return mockPromise(res).then(function(v) {
console.log(v);
});
}
Example use of these generators would be:
// return a single page promise
async(all(1)).then(function(value) { console.log(value); });
// do `logTask` for all pages 1 thru 30
async(allDo(logTask));
// do `logTask` for all pages with batches of 10
async(allBatchDo(logTask));
The question is, is this a legitimate use of es6 async features, or is there an abstract built-in solution for my use case?
If you want use generators to make async then you code is valid. ES6 contains only promises to async operations. ES7 will have async/await. You can also use a good library: https://github.com/kriskowal/q or use only native promises Promise.All without generators.
I would say this code might be quite slow, since you are using yield* all the task will run sequentially potentially taking much more time than necessary (assuming mockPromise does some io) you might be better of yielding Promise.all or just using promises
also your usage of while(true) is very weird..
Below are some links that can help you for asynquence's runner
http://davidwalsh.name/concurrent-generators And
http://spion.github.io/posts/analysis-generators-and-other-async-patterns-node.html
From what I understand, the future style to write async code in JS is to use generators instead of callbacks. At least, or esp. in the V8 / Nodejs community. Is that right? (But that might be debatable and is not my main question here.)
To write async code with generators, I have found a few libraries:
gen-run (What I'm currently using.)
co
task.js
Galaxy
They all look kind of similar and I'm not that sure which of them to use (or if that even matters). (However, that might again be debatable and is also not my main question here -- but I still would be very happy about any advice.)
(I'm anyway only using pure V8 - if that matters. I'm not using Nodejs but I use pure V8 in my custom C++ app. However, I already have a few node-style elements in my code, including my custom require().)
Now I have some function X written in callback-style, which itself calls other async functions with callback arguments, e.g.:
function X(v, callback) {
return Y(onGotY);
function onGotY(err, res) {
if(err) return callback(err);
return Z(onGotZ);
}
function onGotZ(err, res, resExtended) {
if(err) return callback(err);
return callback(null, v + res + resExtended);
}
}
And I want to turn X into a generator, e.g. I guess function* X(v) { ... }. How would that look like?
I went with my very simple own lib which works quite well for my small V8 environment and also makes it easy to debug because the JS callstack stays intact. To make it work with Nodejs or on the web, it would need some modifications, though.
Rationales here:
For debugging, we don't really want async code - we want to have nice understandable call stacks in debuggers, esp. in the node-inspector debugger.
Also note that so far, all async code is completely artificial and we execute everything completely in sync, somewhat emulated via V8 Microtasks.
Callback-style async code is already hard to understand in call stacks.
Generator-style async code looses the call stack information completely in conventional debuggers - that includes the current Chrome Beta Developer Tools V8 debugger used with node-inspector. That is very much the nature of generators (coroutines in general). Later versions of the debugger might handle that, but that is not the case today. We even need some special C++ code to get the info. Example code can be found here:
https://github.com/bjouhier/galaxy-stack/blob/master/src/galaxy-stack.cc
So if we want to have useful debuggers, today, we cannot use generators -- at least not as it is usually done.
We still want to use generator-style async code because it makes the code much more readable, compared to callback-style code.
We introduce a new function async to overcome this. Imagine the following callback-style async code:
function doSthX(a, b, callback) { ... }
function doSthY(c, callback) {
doSthX(c/2, 42, function(err, res) {
if(err) return callback(err);
callback(null, c + res);
})
}
Now, the same code with the async function and generator-style code:
function* doSthX(a, b) { ... }
function* doSthY(c) {
var res = yield async(doSthX(c/2, 42));
return c + res;
}
We can provide two versions of async(iter):
Run the iterator iter on top of the stack. That will keep a sane call stack and everything is run serially.
Yield the iterator down to some lower handler and make it really async.
Note that there are a few notable libraries which can be used for the second approach:
https://github.com/visionmedia/co
http://taskjs.org/
https://github.com/creationix/gen-run
https://github.com/bjouhier/galaxy
For now, we just implement the first approach - to make debugging easier.
If we once want to have both, we can introduce a debug flag to switch via both implementations.
global.async = async;
function async(iter) {
// must be an iterator
assert(iter.next);
var gotValue;
var sendValue;
while(true) {
var next = iter.next(sendValue);
gotValue = next.value;
if(!next.done) {
// We expect gotValue as a value returned from this function `async`.
assert(gotValue.getResult);
var res = gotValue.getResult();
sendValue = res;
}
if(next.done) break;
}
return {
getResult: function() {
return gotValue;
}
};
}
// Like `async`, but wraps a callback-style function.
global.async_call_cb = async_call_cb;
function async_call_cb(f, thisArg /* , ... */) {
assert(f.apply && f.call);
var args = Array.prototype.slice.call(arguments, 2);
return async((function*() {
var gotCalled = false;
var res;
// This expects that the callback is run on top of the stack.
// We will get this if we always use the wrapped enqueueMicrotask().
// If we have to force this somehow else at some point, we could
// call runMicrotasks() here - or some other waiter function,
// to wait for our callback.
args.push(callback);
f.apply(thisArg, args);
function callback(err, _res) {
assert(!gotCalled);
if(err) throw err;
gotCalled = true;
res = _res;
}
assert(gotCalled);
return res;
})());
}
// get the result synchronously from async
global.sync_from_async = sync_from_async;
function sync_from_async(s) {
assert(s.getResult); // see async()
return s.getResult();
}
// creates a node.js callback-style function from async
global.callback_from_async = callback_from_async;
function callback_from_async(s) {
return function(callback) {
var res;
try { res = sync_from_async(s); }
catch(err) {
return callback(err);
}
return callback(null, res);
};
}
global.sync_get = sync_get;
function sync_get(iter) {
return sync_from_async(async(iter));
}
// this is like in gen-run.
// it's supposed to run the main-function which is expected to be a generator.
// f must be a generator
// returns the result.
global.run = run;
function run(f) {
return sync_get(f());
}
I assume that by "turning X into a generator" you mean "turning X into something you can yield", because that's how these libs work.
If you are using co (which I recommend), you need to make your function return thunks, that is a function, which accepts only a callback. Quite simple:
function coX (v) {
return function (cb) {
X(v, cb);
}
}
And then just:
co(function * () {
yield coX('v');
})();