Develop Reference

Best way to to use async/promise/callbacks in for loop [duplicate]

This question already has answers here:
How do I convert an existing callback API to promises?
(24 answers)
Closed 2 years ago.
I'm building a trading bot that needs to get stock names from separate files. But even I have used async function and await in my code, that doesn't work.
My index file init method.
const init = async () => {
const symbols = await getDownTrendingStock();
console.log("All stocks that are down: " + symbols);
const doOrder = async () => {
//do stuff
}
doOrder();
}
my getDownTrendeingStock file
const downStocks = []
function getDownTrendingStock () {
for(i = 0; i < data.USDTPairs.length; i++){
const USDTPair = data.USDTPairs[i] + "USDT";
binance.prevDay(USDTPair, (error, prevDay, symbol) => {
if(prevDay.priceChangePercent < -2){
downStocks.push(symbol)
}
});
}
return downStocks;
}
I have tried to also use async in for loop because the getDownTrendinStock function returns an empty array before for loop is finished. I didn't find the right way to do that because I was confused with all async, promise and callback stuff. What is the right statement to use in this situation?
Output:
All stocks that are down:
Wanted output:
All stocks that are down: [BTCUSDT, TRXUSDT, ATOMUSDT...]

I think the main issue in the code you posted is that you are mixing callbacks and promises.
This is what's happening in your getDownTrendingStock function:
You start iterating over the data.USDTPairs array, picking the first element
You call binance.prevDay. This does nothing yet, because its an asynchronous function that takes a bit of time. Notably, no data is added to downStocks yet.
You continue doing 1-2, still, no data is added
You return downStocks, which is still empty.
Your function is complete, you print the empty array
Now, at some point, the nodejs event loop continues and starts working on those asynchronous tasks you created earlier by calling binance.prevDay. Internally, it probably calls an API, which takes time; once that call is completed, it calls the function you provided, which pushes data to the downStocks array.
In summary, you didn't wait for the async code to complete. You can achieve this in multiple ways.
One is to wrap this in a promise and then await that promise:
const result= await new Promise((resolve, reject) => {
binance.prevDay(USDTPair, (error, prevDay, symbol) => {
if (error) {
reject(error);
} else {
resolve({prevDay, symbol});
}
});
});
if(result.prevDay.priceChangePercent < -2){
downStocks.push(result.symbol)
}
Note that you can probably also use promisify for this. Also, this means that you will wait for one request to finish before starting the next, which may slow down your code considerably, depending on how many calls you need; you may want to look into Promise.all as well.

Generally speaking, I use two technics:
const asyncFunc = () => {smthAsync};
const arrayToProcess = [];
// 1
const result = await arrayToProcess.reduce((acc, value) => acc.then(() => asyncFunc(value)), Promise.resolve(someInitialValue));
// 2
// here will be eslint errors
for(let i = 0 i < arrayToProcess.length; i+=1) {
const processResult = await asyncFunc(value);
// do with processResult what you want
};

Array of objects in javascript not returning as expected

I have a function which returns a list of objects in Javascript, and I'm calling this function from another and attempting to use some of the values from it, but whenever I try to access said values, they come back undefined.
This is my function which generates the list - the idea is that it creates a sqlite3 database if it does not exist, and returns an array containing every event.
function listAllEvents() {
const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('schedule.db');
const selectionArray = [];
db.serialize(() => {
db.run(`
CREATE TABLE IF NOT EXISTS todo (
name text,
date text,
id text primary key
)
`);
db.all('SELECT * FROM todo ORDER BY date', [], (err, rows) => {
if (err) {
throw err;
}
rows.forEach((row) => {
selectionArray.push(row);
});
});
});
return selectionArray;
}
I call this function from another, but when I try to access values from the array, they don't seem to be working and I can't quite figure it out.
function displayUpcomingEvents() {
const events = listAllEvents();
// console.log(events); <-- This line here! In the console, it correctly states the length of the array
// console.log(events.length) <-- This line, however, returns 0. Why?
// console.log(events[0]) <-- This doesn't work either, it just returns "undefined".
for (let i = 0; i < events.length; i += 1) {
$('#upcomingEvents').after('<li>asdf</li>');
}
}
For example, if I were to create two events in the database, through the console,
events is an Array(2) with indices
- 0: {name: "my_event", date: "2019-06-04", id: "c017c392d446d4b2"}
- 1: {name: "my_event_2", date: "2019-06-04", id: "6d655ac8dd02e3fd"},
events.length returns 0,
and events[0] returns undefined.
Why is this, and what can I do to fix it?

The possible reason why this is happening, is because of the async nature of JS, that means all the console.log statements are getting executed before the successful execution of the listAllEvents() function,
So my suggestion is to try using the promises, and perform all the actions mentioned after the listAllEvents() function only when that function returns a promise.
You can also try making the function async and using await to wait for its successful execution. (Much Smarter Choice will be using async)
Link to ASYNC Functions and Usage
Link to Promises
Also you can check the validity of answer by doing console.log(row) where you are pushing rows to the array. You will observer that the console.log(row) will be executed at the last, after printing events and other log statements.

The problem is that your function is returning the variable before a value is set. The db.serialize function will run asynchronously (outside the normal flow of the program) and the return statement will run immediately after. One thing you can do is use async/await in conjunction with Promise. In this case the the variable results will wait for the promise to be resolved before continuing to the next line.
async function listAllEvents() {
const selectionArray = [];
let promise = new Promise( function (resolve, reject) {
db.serialize(() => {
db.run(
CREATE TABLE IF NOT EXISTS todo (
name text,
date text,
id text primary key
)
);
db.all('SELECT * FROM todo ORDER BY date', [], (err, rows) => {
if (err) {
// add code here to reject promise
throw err;
}
rows.forEach((row) => {
selectionArray.push(row);
});
resolve(selectionArray);// resolve the promise
});
});
});
let results = await promise;
return results;
};
async function displayUpcomingEvents() {
const events = await listAllEvents();
// console.log(events); <-- This line here! In the console, it correctly states the length of the array
// console.log(events.length) <-- This line, however, returns 0. Why?
// console.log(events[0]) <-- This doesn't work either, it just returns "undefined".
for (let i = 0; i < events.length; i += 1) {
$('#upcomingEvents').after('<li>asdf</li>');
}
}
Note here that the displayUpcomingEvents function will also need to be async or you cannot use the await keyword.
Additional reading for Promise keyword MDN: Promise
Additional reading for Async/Await MDN: Asyn/Await

wait on a promise for each iteration in nodejs

I am developing a nodejs application that needs to get settings from an array(in a settings object), call a rest api based on the settings and write the response to mongodb and repeat this for the next setting in the array.
Here is a simplified version of the application
var setting //global
process(){ //top level function
for(let s of config.settings){
setting = s;
getData();
}
}
function getData(){
init()
.then(makeRequest) // constructs and makes the rest api call
.then(insert) // writes the response to the db
.catch(function(err){
// logs err
}
}
Running it, only the data for the last setting (in the array) is written to the db and this happens for each iteration. Basically the same data is written on the db for as many iterations.
The problem I can see from this is that the for loop finishes executing, before the promises return with the value.
I have seen some examples of async.for
Any suggestions on fixing this. How do you go about designing this kind of a flow?

You can bind the settings to each function call to preserve the value. looks like you'd have to refactor though as the value would be passed in as an argument though i'm not sure if your code is pseudo code or actual code.
async await would work as well but would take longer as it would pause execution at each api call.

You should return an object or array that you can use to store an internal state for your request. Please see the example for how it works.
Also never set a global variable to store your state, with your function being asynchronous the value may not be what you expect it to be.
With this approach you are passing { init } for the first promise, then { init, request } for the next so you have the response from each part of your promise chain that you can use to make further requests.
// return an object to store the state on init
const init = () =>
new Promise((res, rej) => res({
init: 'initted'
}))
// pass init and the request to the next function in the chain
const makeRequest = ({ init }) =>
new Promise((res, rej) => res({
init,
request: {
msg: 'this is the response',
id: 33
}
}))
// insert stuff from the request
// then return the data to the next query
const insert = ({ init, request }) =>
new Promise((res, rej) => res({
request,
init,
created_at: Date.now()
}))
const trace = name => x => (console.log(name, x), x)
function getData(){
return init() // return your promise so you can chain it further
.then(trace('after init'))
.then(makeRequest)
.then(trace('after request'))
.then(insert)
.then(trace('after insert'))
.catch(console.error)
}
// call you function
getData()
// since the promise is returned we can continue the chain
.then(state => console.log({ state }))
<script src="https://codepen.io/synthet1c/pen/KyQQmL.js"></script>

All of your loop will have executed by the time the callbacks are coming in. So settings will be the last value.
Instead of relying on globals, pass setting into getData, for example.

JavaScript: Sequentially iterate promise function

This is the scenario:
Using AWS Kinesis >
To get records from Kinesis I need a shard iterator >
Kinesis won't return a new shard iterator until a
request is complete >
The call to "getRecords" is asynchronous >
Attempt to iterate this process fails because the request hasn't resolved before the shard iterator is needed
const getRecordsInShard = (shard, batchSize, streamName) => {
const records = [];
const loop = (response) => {
if (_.isEmpty(response.NextShardIterator)) {
return Promise.resolve(records);
}
records.push(response.Records);
return getRecordsByShardIterator(response.NextShardIterator, batchSize).then(loop);
};
return getStreamIterator(shard.ShardId, shard.SequenceNumberRange.StartingSequenceNumber, (streamName || process.env.KINESIS_STREAM))
.then(response => getRecordsByShardIterator(response.ShardIterator, batchSize))
.then(loop);
};
The above code fails because the promise returned from loop doesn't resolve before the return of the super function. How can I iterate a promise function sequentially, using the return value from the previous iteration as the next's input?
Caveats:
Each iteration relies on the information returned by the previous, the result needs to collect records from each iteration altogether

I don't know why it went so complicated when the library provider give perfect examples to consume kinesis stream.
https://github.com/awslabs/amazon-kinesis-client-nodejs/tree/master/samples
It has basic samples and clickstream sample.
Hope it helps.

Best es6 way to get name based results with Promise.all

By default the Promise.All([]) function returns a number based index array that contains the results of each promise.
var promises = [];
promises.push(myFuncAsync1()); //returns 1
promises.push(myFuncAsync1()); //returns 2
Promise.all(promises).then((results)=>{
//results = [0,1]
}
What is the best vanilla way to return a named index of results with Promise.all()?
I tried with a Map, but it returns results in an array this way:
[key1, value1, key2, value2]
UPDATE:
My questions seems unclear, here is why i don't like ordered based index:
it's crappy to maintain: if you add a promise in your code you may have to rewrite the whole results function because the index may have change.
it's awful to read: results[42] (can be fixed with jib's answer below)
Not really usable in a dynamic context:
var promises = [];
if(...)
promises.push(...);
else{
[...].forEach(... => {
if(...)
promises.push(...);
else
[...].forEach(... => {
promises.push(...);
});
});
}
Promise.all(promises).then((resultsArr)=>{
/*Here i am basically fucked without clear named results
that dont rely on promises' ordering in the array */
});

ES6 supports destructuring, so if you just want to name the results you can write:
var myFuncAsync1 = () => Promise.resolve(1);
var myFuncAsync2 = () => Promise.resolve(2);
Promise.all([myFuncAsync1(), myFuncAsync2()])
.then(([result1, result2]) => console.log(result1 +" and "+ result2)) //1 and 2
.catch(e => console.error(e));
Works in Firefox and Chrome now.

Is this the kind of thing?
var promises = [];
promises.push(myFuncAsync1().then(r => ({name : "func1", result : r})));
promises.push(myFuncAsync1().then(r => ({name : "func2", result : r})));
Promise.all(promises).then(results => {
var lookup = results.reduce((prev, curr) => {
prev[curr.name] = curr.result;
return prev;
}, {});
var firstResult = lookup["func1"];
var secondResult = lookup["func2"];
}

If you don't want to modify the format of result objects, here is a helper function that allows assigning a name to each entry to access it later.
const allNamed = (nameToPromise) => {
const entries = Object.entries(nameToPromise);
return Promise.all(entries.map(e => e[1]))
.then(results => {
const nameToResult = {};
for (let i = 0; i < results.length; ++i) {
const name = entries[i][0];
nameToResult[name] = results[i];
}
return nameToResult;
});
};
Usage:
var lookup = await allNamed({
rootStatus: fetch('https://stackoverflow.com/').then(rs => rs.status),
badRouteStatus: fetch('https://stackoverflow.com/badRoute').then(rs => rs.status),
});
var firstResult = lookup.rootStatus; // = 200
var secondResult = lookup.badRouteStatus; // = 404
If you are using typescript you can even specify relationship between input keys and results using keyof construct:
type ThenArg<T> = T extends PromiseLike<infer U> ? U : T;
export const allNamed = <
T extends Record<string, Promise<any>>,
TResolved extends {[P in keyof T]: ThenArg<T[P]>}
>(nameToPromise: T): Promise<TResolved> => {
const entries = Object.entries(nameToPromise);
return Promise.all(entries.map(e => e[1]))
.then(results => {
const nameToResult: TResolved = <any>{};
for (let i = 0; i < results.length; ++i) {
const name: keyof T = entries[i][0];
nameToResult[name] = results[i];
}
return nameToResult;
});
};

A great solution for this is to use async await. Not exactly ES6 like you asked, but ES8! But since Babel supports it fully, here we go:
You can avoid using only the array index by using async/await as follows.
This async function allows you to literally halt your code inside of it by allowing you to use the await keyword inside of the function, placing it before a promise. As as an async function encounters await on a promise that hasn't yet been resolved, the function immediately returns a pending promise. This returned promise resolves as soon as the function actually finishes later on. The function will only resume when the previously awaited promise is resolved, during which it will resolve the entire await Promise statement to the return value of that Promise, allowing you to put it inside of a variable. This effectively allows you to halt your code without blocking the thread. It's a great way to handle asynchronous stuff in JavaScript in general, because it makes your code more chronological and therefore easier to reason about:
async function resolvePromiseObject(promiseObject) {
await Promise.all(Object.values(promiseObject));
const ret = {};
for ([key, value] of Object.entries(promiseObject)) {
// All these resolve instantly due to the previous await
ret[key] = await value;
};
return ret;
}
As with anything above ES5: Please make sure that Babel is configured correctly so that users on older browsers can run your code without issue. You can make async await work flawlessly on even IE11, as long as your babel configuration is right.

in regards to #kragovip's answer, the reason you want to avoid that is shown here:
https://medium.com/front-end-weekly/async-await-is-not-about-making-asynchronous-code-synchronous-ba5937a0c11e
"...it’s really easy to get used to await all of your network and I/O calls.
However, you should be careful when using it multiple times in a row as the await keyword stops execution of all the code after it. (Exactly as it would be in synchronous code)"
Bad Example (DONT FOLLOW)
async function processData() {
const data1 = await downloadFromService1();
const data2 = await downloadFromService2();
const data3 = await downloadFromService3();
...
}
"There is also absolutely no need to wait for the completion of first request as none of other requests depend on its result.
We would like to have requests sent in parallel and wait for all of them to finish simultaneously. This is where the power of asynchronous event-driven programming lies.
To fix this we can use Promise.all() method. We save Promises from async function calls to variables, combine them to an array and await them all at once."
Instead
async function processData() {
const promise1 = downloadFromService1();
const promise2 = downloadFromService2();
const promise3 = downloadFromService3();
const allResults = await Promise.all([promise1, promise2, promise3]);