Develop Reference

why is my async code inside for loop not running sequentially?

Why do all my function calls get executed in parallel here when using await? What needs to be changed in order to start a function call once the previous is done?
for (let collection of collections) {
await this.updateListings(collection)
}
private async updateListings(collection: Collection) {
try {
const startTime = Date.now();
const supply = collection.stats.total_supply;
const contract_address = collection.primary_asset_contracts[0].address;
const token_ids = [];
for (let i = 1; i <= supply; i++) {
token_ids.push(i);
}
let offset = 0;
let limit = 30;
while (offset + limit <= supply) {
this.apiKey = this.apiKeys.find(key => key !== this.apiKey); //rotate api key on each request
const ids = token_ids.splice(0, 30);
const params = new URLSearchParams();
params.set("offset", offset.toString());
params.set("limit", limit.toString());
params.set("side", "1");
params.set("sale_kind", "0");
params.set("order_by", "eth_price");
params.set("order_direction", "asc");
params.set("asset_contract_address", contract_address);
for (let i = 0; i < ids.length; i++) {
params.append("token_ids", ids[i])
}
await this.delayRequest(700);
const response = await axios.get(process.env.OPENSEA_API_BASE_URL + "/orders", {
headers: {
"X-API-KEY": this.apiKey,
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
},
params: params
})
const { orders } = response.data;
if (orders.length > 0) await getRepository(Listing).save(orders);
offset += limit;
console.log(`update listings for ${collection.slug} status: ${offset} / ${supply}`)
}
const endTime = Date.now() - startTime;
console.log("fetched listings for " + collection.slug + " in " + endTime + " ms")
} catch (error) {
console.log("updating listings for " + collection.slug + " failed", error.message);
}
}
constructor() {
setTimeout(() => {
this.update();
}, 6000)
}
What I expect is that the first iteration finishes before it proceeds but its calling each iteration "updateListings(coll)" at the same time.

In fact. your code is executed sequentially. How did you check that your code was running in parallel?
You can use Array.reduce to make it sequentially:
return collections.reduce((currentPromiseChain, collection) => {
return currentPromiseChain.then(() => this.updateListings(collection));
}, Promise.resolve());

Just add an await before the condition. This will ensure each loop finishes before continuing to the next one.
Note: not fully sure on the compatibility for this, you may need check for compatibility.
for await (let collection of collections) { //notice the await in this line.
await this.updateListings(collection)
}

This is to demostrate there isn't anything wrong with the way you are calling for loop, and also how you have a while loop inside your updateListing function.
There's probably something you didn't show?
The only reason I can see, affecting your codes, is if there are errors, and your try/catch block returns the promise with error immediately, making it appears it is running in parallel.
const whileLoopFunction = async(value) => {
let limit = 0;
let supply = 10
while (limit < supply) {
await new Promise(resolve => setTimeout(resolve,100))
limit++
}
console.log('While loop complete for value ', value)
};
(async() => {
const collections = [ 1, 2, 3, 4, 5 ]
console.log("For loop works with await")
for (let collection of collections) {
await whileLoopFunction(collection)
}
})()

How to use multiple promises in recursion?

I am trying to solve the problem where the script enters a website, takes the first 10 links from it and then goes on those 10 links and then goes on to the next 10 links found on each of these 10 previous pages. Until the number of visited pages will be 1000.
This is what it looks like:
I was trying to get this by using for loop inside promise and recursion, this is my code:
const rp = require('request-promise');
const url = 'http://somewebsite.com/';
const websites = []
const promises = []
const getOnSite = (url, count = 0) => {
console.log(count, websites.length)
promises.push(new Promise((resolve, reject) => {
rp(url)
.then(async function (html) {
let links = html.match(/https?:\/\/(www\.)?[-a-zA-Z0-9#:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()#:%_\+.~#?&//=]*)/g)
if (links !== null) {
links = links.splice(0, 10)
}
websites.push({
url,
links,
emails: emails === null ? [] : emails
})
if (links !== null) {
for (let i = 0; i < links.length; i++) {
if (count < 3) {
resolve(getOnSite(links[i], count + 1))
} else {
resolve()
}
}
} else {
resolve()
}
}).catch(err => {
resolve()
})
}))
}
getOnSite(url)

I think you might want a recursive function that takes three arguments:
an array of urls to extract links from
an array of the accumulated links
a limit for when to stop crawling
You'd kick it off by calling it with just the root url, and await all of the returned promises:
const allLinks = await Promise.all(crawl([rootUrl]));
On the initial call the second and third arguments could assume default values:
async function crawl (urls, accumulated = [], limit = 1000) {
...
}
The function would fetch each url, extract its links, and recurse until it hit the limit. I haven't tested any of this, but I'm thinking something along these lines:
// limit the number of links per page to 10
const perPageLimit = 10;
async function crawl (urls, accumulated = [], limit = 1000) {
// if limit has been depleted or if we don't have any urls,
// return the accumulated result
if (limit === 0 || urls.length === 0) {
return accumulated;
}
// process this set of links
const links = await Promise.all(
urls
.splice(0, perPageLimit) // limit to 10
.map(url => fetchHtml(url) // fetch the url
.then(extractUrls)); // and extract its links
);
// then recurse
return crawl(
links, // newly extracted array of links from this call
[...accumulated, links], // pushed onto the accumulated list
limit - links.length // reduce the limit and recurse
);
}
async fetchHtml (url) {
//
}
const extractUrls = (html) => html.match( ... )

JavaScript promise won't resolve or reject, it just blocks for some reason

I'm writing a node script, which is supposed to order images by a score value, calculated by a function called getImageScore(). This score takes quite some time to be calculated, therefore I created a promise (promiseImageScore) that would return the score value. After getting the promised value by the promiseImageScore promise, the program would log the name of the image and its score. Despite creating a promise for the score values, they still come back as undefined. I observed the promises would never finish, they just always remain in pending state. I tried tracing what's actually happening in the getImageScore() function, by logging some messages through the function and, as expected, those logs stop at some point, but I cannot get why this happens. Here is the full program:
const fs = require('fs');
const { resolve } = require('path');
const { reject } = require('q');
const { Console } = require('console');
const gm = require('gm').subClass({imageMagick: true});
const PNG = require("pngjs").PNG;
let pathToFolder = '/home/eugen/Pictures/wallpapers1';
let pathToImage = '';
let promiseImageScore = new Promise((resolve, reject) => {
resolve(getImageScore());
});
function getImageScore() {
console.log('entered this promise....');
let img = gm(pathToImage);
// Get the PNG buffer
img.toBuffer("PNG", (err, buff) => {
if (err) return new Error(err);
console.log('got buffer...');
// Get the image size
img.size((err, size) => {
if (err) {
console.log(err);
return new Error(err);
}
console.log('got image size...');
// Parse the PNG buffer
let str = new PNG();
str.end(buff);
// After it's parsed...
str.on("parsed", buffer => {
// Get the pixels from the image
let idx, score = 0, rgb = {r: 0, g: 0, b: 0};
for (let y = 0; y < size.height; y++)
for (let x = 0; x < size.width; x++) {
idx = (size.width * y + x) << 2;
rgb.r = buffer[idx];
rgb.g = buffer[idx + 1];
rgb.b = buffer[idx + 2];
score += (rgb.r + rgb.g + rgb.b) / 765;
}
console.log('one promised finished...');
return score / (size.height * size.width);
});
str.on("error", e => {
return new Error(e);
});
});
});
}
// see which images are to be found in the specificd directory
fs.readdir(pathToFolder, function (err, files) {
if (err) return console.log('Unable to scan directory: ' + err);
console.log('files in directory:\n');
files.forEach(function (file) {
pathToImage = pathToFolder + '/' + file;
//showImageScore();
promiseImageScore
.then(imageScore => {
console.log(file + ' has a score of ' + imageScore);
})
.catch(e => {
throw e;
})
});
});
Here is the output of the code:
entered this promise....
files in directory:
Boats_Thailand_Sea_Crag_Nature_8000x5224.jpg has a score of undefined
Wallpaper_8K_0_7680x4320.jpg has a score of undefined
Water_mountains_landscapes_nature_snow_valley_rocks_switzerland_rivers_3840x2400.jpg has a score of undefined
Waterfalls_USA_Crag_Trees_Hocking_Hills_State_Park_Ohio_Nature_10929x5553.jpg has a score of undefined
cats_blue_eyes_animals_pets_4288x2848.jpg has a score of undefined
cats_blue_eyes_animals_pets_4288x2848.png has a score of undefined
city_night_panorama_117682_3840x2160.jpg has a score of undefined
starry_sky_tree_night_sky_119989_1920x1080.jpg has a score of undefined
got buffer...
After the got buffer... log, the program keeps running, never stopping, apparently doing nothing.

Your code is not working with promises correctly. You need to make several changes:
Pass the resolve and reject to getImageScore:
const promiseImageScore = new Promise((resolve, reject) => {
// You need to pass resolve and reject to getImageScore
getImageScore(resolve, reject);
});
Use resolve and reject in getImageScore:
// For example
img.toBuffer("PNG", (err, buff) => {
// If you get an error you need to call reject and return from the function
// You will need to do the same for img.size((err, size) => {})
// and for str.on("error", e => {})
if (err) return reject(err);
// Then when you have your result and there were no errors,
// you will need to call resolve with the result
resolve(score / (size.height * size.width));
});

Bluebird promise: why it doesn't timeout?

So, I'm trying to model some long computation. for this purpose I'm computing the fibonacci number. In case when computation takes to much time I need to reject it.
The question: why TimeoutErrror handler doesn't work? How to fix the code?
const expect = require('chai').expect
const Promise = require('bluebird')
function profib(n, prev = '0', cur = '1') {
return new Promise.resolve(n < 2)
.then(function(isTerm) {
if(isTerm) {
return cur
} else {
n = n - 2
return profib(n, cur, strAdd(cur, prev));
}
})
}
const TIMEOUT = 10000
const N = 20000
describe('recursion', function() {
it.only('cancelation', function() {
this.timeout(2 * TIMEOUT)
let prom = profib(N).timeout(1000)
.catch(Promise.TimeoutError, function(e) {
console.log('timeout', e)
return '-1'
})
return prom.then((num) => {
expect(num).equal('-1')
})
})
})
const strAdd = function(lnum, rnum) {
lnum = lnum.split('').reverse();
rnum = rnum.split('').reverse();
var len = Math.max(lnum.length, rnum.length),
acc = 0;
res = [];
for(var i = 0; i < len; i++) {
var subres = Number(lnum[i] || 0) + Number(rnum[i] || 0) + acc;
acc = ~~(subres / 10); // integer division
res.push(subres % 10);
}
if (acc !== 0) {
res.push(acc);
}
return res.reverse().join('');
};
Some info about environment:
➜ node -v
v6.3.1
➜ npm list --depth=0
├── bluebird#3.4.6
├── chai#3.5.0
└── mocha#3.2.0

If I'm reading your code correctly profib does not exit until it's finished.
Timeouts are not interrupts. They are just events added to the list of events for the browser/node to run. The browser/node runs the next event when the code for the current event finishes.
Example:
setTimeout(function() {
console.log("timeout");
}, 1);
for(var i = 0; i < 100000; ++i) {
console.log(i);
}
Even though the timeout is set for 1 millisecond it doesn't appear until after the loop finishes (Which takes about 5 seconds on my machine)
You can see the same problem with a simple forever loop
const TIMEOUT = 10000
describe('forever', function() {
it.only('cancelation', function() {
this.timeout(2 * TIMEOUT)
while(true) { } // loop forever
})
})
Run in with your environment and you'll see it never times out. JavaScript does not support interrupts, it only supports events.
As for fixing the code you need to insert a call to setTimeout. For example, let's change forever loop so it exits (and therefore allows other events)
const TIMEOUT = 100
function alongtime(n) {
return new Promise(function(resolve, reject) {
function loopTillDone() {
if (n) {
--n;
setTimeout(loopTillDone);
} else {
resolve();
}
}
loopTillDone();
});
}
describe('forever', function() {
it.only('cancelation', function(done) {
this.timeout(2 * TIMEOUT)
alongtime(100000000).then(done);
})
})
Unfortunately using setTimeout is really a slow operation and arguably shouldn't be used in a function like profib. I don't really know what to suggest.

The problem appears because promises work in a "greedy" manner(it's my own explanation). For this reason function profib doesn't release event loop. To fix this issue I need to release event loop. The easiest way to do that with Promise.delay():
function profib(n, prev = '0', cur = '1') {
return new Promise.resolve(n < 2)
.then(function(isTerm) {
if(isTerm) {
return cur
} else {
n = n - 2
return Promise.delay(0).then(() => profib(n, cur, strAdd(cur, prev));
}
})
}

gman has already explained why your idea doesn't work. The simple and efficient solution would be to add a condition in your loop that checks the time and breaks, like thus :
var deadline = Date.now() + TIMEOUT
function profib(n, prev = '0', cur = '1') {
if (Date.now() >= deadline) throw new Error("timed out")
// your regular fib recursion here
}
Calling profib will either eventually return the result, or throw an error. However, it will block any other JavaScript from running while doing the calculation. Asynchronous execution isn't the solution here. Or at least, not all of it. What you need for such CPU-intensive tasks is a WebWorker to run it in another JavaScript context. Then you can wrap your WebWorker's communication channel in a Promise to get the API you envisioned originally.

When babel processes async / await code does it bundle unrelated calls?

Is babels async / await code smart enough to see the code below:
async function alpha () {
let resultOne = await processNumber(5)
let resultTwo = await processNumber(5 + 8)
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
As something like the following, where the first two promises within this function can happen together because the values needed to perform those operations does not need to wait for anything.
import Promise from 'bluebird'
async function beta () {
let {resultOne, resultTwo} = await Promise.props({
resultOne: processNumber(5),
resultTwo: processNumber(5 + 8)
})
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
I would understand the alpha function as waiting for each async function call before it moves on to the next, where in beta resultOne and resultTwo are happening simultaneously, this is only possible because they don't need to wait on any other calls. I was wondering if this was truly the case or if babel does something behind the scenes to bundle these together.

I setup a benchmark between the two and it seems it does not account for this on it's own.
Here's the test:
import Promise from 'bluebird'
async function processNumber (int) {
await Promise.delay(500)
return {number: int + 3}
}
async function alpha () {
let resultOne = await processNumber(5)
let resultTwo = await processNumber(5 + 8)
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
async function beta () {
let {resultOne, resultTwo} = await Promise.props({
resultOne: processNumber(5),
resultTwo: processNumber(5 + 8)
})
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
async function main () {
const TEST_ALPHA = 'test alpha'
const TEST_BETA = 'test beta'
console.time(TEST_ALPHA)
let resultAlpha = await alpha()
console.log(resultAlpha)
console.timeEnd(TEST_ALPHA)
console.time(TEST_BETA)
let resultBeta = await beta()
console.log(resultBeta)
console.timeEnd(TEST_BETA)
return true
}
main()
.then(console.log)
.catch(console.error)
Here's the results:
thomasreggi#zx:PAS-api$ babel-node test.js
{ number: 22 }
test alpha: 2025ms
{ number: 22 }
test beta: 1508ms
true
thomasreggi#zx:PAS-api$ babel-node test.js
{ number: 22 }
test alpha: 2033ms
{ number: 22 }
test beta: 1511ms
true

In JS it is nearly impossible to make any strict statements whether a given expression is "unrelated" to another arbitrary expression (especially statically).
It happens because due to its highly-dynamic nature almost every expression may cause hidden (or not so hidden) side effects that may break the expected flow of the program.
For your very code it is really easy to break the code in case if both "unrelated" calls are triggered "simultaneously":
let isFirst = true;
async function processNumber(v) {
await Promise.delay(2000 - v * 100);
if (v < 10) {
if (!isFirst) {
throw new Error();
}
}
isFirst = false;
return { number: v + 3 };
}
This would work for alpha, but throw for beta.
If you know it would be fine and want to run them "in parallel" just use awaits correspondingly:
async function alpha () {
let one = processNumber(5)
let two = processNumber(5 + 8)
const resultOne = await one;
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
Also note, that the resultTwo is not used anywhere in your code.