Develop Reference

Best way to to use async/promise/callbacks in for loop [duplicate]

This question already has answers here:
How do I convert an existing callback API to promises?
(24 answers)
Closed 2 years ago.
I'm building a trading bot that needs to get stock names from separate files. But even I have used async function and await in my code, that doesn't work.
My index file init method.
const init = async () => {
const symbols = await getDownTrendingStock();
console.log("All stocks that are down: " + symbols);
const doOrder = async () => {
//do stuff
}
doOrder();
}
my getDownTrendeingStock file
const downStocks = []
function getDownTrendingStock () {
for(i = 0; i < data.USDTPairs.length; i++){
const USDTPair = data.USDTPairs[i] + "USDT";
binance.prevDay(USDTPair, (error, prevDay, symbol) => {
if(prevDay.priceChangePercent < -2){
downStocks.push(symbol)
}
});
}
return downStocks;
}
I have tried to also use async in for loop because the getDownTrendinStock function returns an empty array before for loop is finished. I didn't find the right way to do that because I was confused with all async, promise and callback stuff. What is the right statement to use in this situation?
Output:
All stocks that are down:
Wanted output:
All stocks that are down: [BTCUSDT, TRXUSDT, ATOMUSDT...]

I think the main issue in the code you posted is that you are mixing callbacks and promises.
This is what's happening in your getDownTrendingStock function:
You start iterating over the data.USDTPairs array, picking the first element
You call binance.prevDay. This does nothing yet, because its an asynchronous function that takes a bit of time. Notably, no data is added to downStocks yet.
You continue doing 1-2, still, no data is added
You return downStocks, which is still empty.
Your function is complete, you print the empty array
Now, at some point, the nodejs event loop continues and starts working on those asynchronous tasks you created earlier by calling binance.prevDay. Internally, it probably calls an API, which takes time; once that call is completed, it calls the function you provided, which pushes data to the downStocks array.
In summary, you didn't wait for the async code to complete. You can achieve this in multiple ways.
One is to wrap this in a promise and then await that promise:
const result= await new Promise((resolve, reject) => {
binance.prevDay(USDTPair, (error, prevDay, symbol) => {
if (error) {
reject(error);
} else {
resolve({prevDay, symbol});
}
});
});
if(result.prevDay.priceChangePercent < -2){
downStocks.push(result.symbol)
}
Note that you can probably also use promisify for this. Also, this means that you will wait for one request to finish before starting the next, which may slow down your code considerably, depending on how many calls you need; you may want to look into Promise.all as well.

Generally speaking, I use two technics:
const asyncFunc = () => {smthAsync};
const arrayToProcess = [];
// 1
const result = await arrayToProcess.reduce((acc, value) => acc.then(() => asyncFunc(value)), Promise.resolve(someInitialValue));
// 2
// here will be eslint errors
for(let i = 0 i < arrayToProcess.length; i+=1) {
const processResult = await asyncFunc(value);
// do with processResult what you want
};

How to wait for this function to finish?

I need to wait for mapping function to finish before I send the data to the console. I know it has something to do with Promise. I've been trying for hours and I couldn't get it to work even after reading so much about promises and async functions...
async function inactiveMemberWarner() {
var msg = "```javascript\nI have sent warnings to members that have been inactive for 2 weeks.\n\n"
var inactiveMembers = '';
var count = 0;
var guildMembers = client.guilds.find(g => g.name === mainGuild).members;
const keyPromises = await guildMembers.map(async (member) => {
if (isMod(member)) {
connection.query(`SELECT * from users WHERE userID='${member.id}'`, (err, data) => {
if (data[0]) {
if (!data[0].warnedForInactivity && moment().isSameOrAfter(moment(data[0].lastMSGDate).add('2', 'week'))) {
count++;
var updateWarning = {warnedForInactivity: 1}
connection.query(`UPDATE users SET ? WHERE userID='${data[0].userID}'`, updateWarning);
member.send(`**[*]** WARNING: You've been inactive on \`\`${mainGuild}\`\` for 2 weeks. Members that have been inactive for at least a month will be kicked.`);
inactiveMembers += `${count}. ${member.user.tag}\n`;
return inactiveMembers;
}
}
});
}
});
await Promise.all(keyPromises).then(inactiveMembersData => console.log(inactiveMembers)); // RETURNS AN EMPTY STRING
setTimeout(() => console.log(inactiveMembers), 5000); // RETURNS THE INACTIVE MEMBERS AFTER WAITING FOR 5 SECONDS (PRMITIVE WAY)
}
inactiveMemberWarner();
Thank you in advance!

You're close, but not quite there.
First, some notes:
await can be used on any value, but it is entirely pointless to use it on anything that isn't a Promise. Your guildMembers.map(...); returns an array, not a Promise.
Mixing await and .then(...) works, but is kinda messy. You're already using await - why bother dealing with callbacks?
Using guildMembers.map(async ...) like this will ensure that all the requests are fired more or less instantaneously, and they could finish in any order. This is fine, but it is kind of a race condition and results in a more or less random order of results.
This is not a good approach even just conceptually! Any time you ever have to loop queries, try and investigate ways to do it in only one query. SQL is quite powerful.
The reason your current code doesn't work is because your connection.query function escapes the async control flow. What I mean by this is that the whole point of using async/await and Promises is basically to keep track of the callbacks locally, and to make use of promise chaining to dynamically add callbacks. If you call an async function which returns a Promise, you can now carry that Promise anywhere else in your code and attach a success handler to it dynamically: either with .then() or with the sugar await.
But the connection.query function doesn't return a Promise, it just has you pass another naked callback - this one is not being tracked by a Promise! The Promise doesn't have a reference to that callback, it can't know when that callback is getting called, and thus your async/await control flow is escaped and your promises resolve long before the queries have ran.
You can resolve this by making a new Promise in the async function:
async function inactiveMemberWarner() {
var msg = "```javascript\nI have sent warnings to members that have been inactive for 2 weeks.\n\n"
var inactiveMembers = '';
var count = 0;
var guildMembers = client.guilds.find(g => g.name === mainGuild).members;
const keyPromises = guildMembers.map(async (member) => {
if (isMod(member)) {
return new Promise((resolve, reject) => {
connection.query(`SELECT * from users WHERE userID='${member.id}'`, (err, data) => {
if (err) reject(err); //make errors bubble up so they can be handled
if (data[0]) {
if (!data[0].warnedForInactivity && moment().isSameOrAfter(moment(data[0].lastMSGDate).add('2', 'week'))) {
count++;
var updateWarning = {warnedForInactivity: 1}
connection.query(`UPDATE users SET ? WHERE userID='${data[0].userID}'`, updateWarning);
member.send(`**[*]** WARNING: You've been inactive on \`\`${mainGuild}\`\` for 2 weeks. Members that have been inactive for at least a month will be kicked.`);
resolve(`${count}. ${member.user.tag}\n`;);
}
} else resolve(""); //make sure to always resolve or the promise may hang
});
});
}
});
let inactiveMembersData = await Promise.all(keyPromises); // Returns an array of inactive member snippets.
inactiveMembers = inactiveMembersData.join(""); //join array of snippets into one string
}
inactiveMemberWarner();
This will work, but there is a much much much better way. SQL supports the IN operator, which allows you to have conditions like WHERE userID IN (list_of_ids). In other words, you can do this in one query. You can even specify more conditions, such as warnedForInactivity = 0 and lastMSGDate BETWEEN (NOW() - INTERVAL 14 DAY) AND NOW(). This way you can offload all of your current processing logic onto the SQL server - something that you should try to do virtually every single time you can. It would simplify this code a lot too. I won't go any further as it's out of scope for this question but feel free to ask another if you can't figure it out.

I can't test this, but this is what normally works for me is when wanting to wait on smt:
async function inactiveMemberWarner() {
new Promise(function(cb,rj){
var msg = "```javascript\nI have sent warnings to members that have been inactive for 2 weeks.\n\n"
var inactiveMembers = '';
var count = 0;
var guildMembers = client.guilds.find(g => g.name === mainGuild).members;
const keyPromises = await guildMembers.map(async (member) => {
if (isMod(member)) {
connection.query(`SELECT * from users WHERE userID='${member.id}'`, (err, data) => {
if (data[0]) {
if (!data[0].warnedForInactivity && moment().isSameOrAfter(moment(data[0].lastMSGDate).add('2', 'week'))) {
count++;
var updateWarning = {warnedForInactivity: 1}
connection.query(`UPDATE users SET ? WHERE userID='${data[0].userID}'`, updateWarning);
member.send(`**[*]** WARNING: You've been inactive on \`\`${mainGuild}\`\` for 2 weeks. Members that have been inactive for at least a month will be kicked.`);
inactiveMembers += `${count}. ${member.user.tag}\n`;
cb(inactiveMembers);
}
}
});
}
});
cb('No Members');
}).then(inactiveMembersData => console.log(inactiveMembers)); // SHOULD RETURNS THE INACTIVE MEMBERS
}
inactiveMemberWarner();

AngularJS $q.all - wait between http calls

So I have a situation where I need to perform a bunch of http calls, then once they are complete, continue on to the next step in the process.
Below is the code which does this and works fine.
However, I now need to wait a few seconds between each of the http calls. Is there a way to pass in a timeout with my current set up, or will it involve a good bit of refactoring?
Can post more code if needs be. I have tried passing in a timeout config varable into the http call, however, they still get fired at the same time.
Any advice would be great.
Code
var allThings = array.map(function(object) {
var singleThingPromise = getFile(object.id);
return singleThingPromise;
});
$q.all(allThings).then(function() {
deferred.resolve('Finished');
}, function(error) {
deferred.reject(error);
});

Instead of using $q.all, you might want to perform sequential calls one on success of previous and probably with use of $timeout. Maybe you could build a recursive function.
Something like this..
function performSequentialCalls (index) {
if(angular.isUndefined(array[index])) {
return;
}
getFile(array[index].id).then(function() {
$timeout(function() {
performSequentialCalls(index + 1)
}, 1000) // waiting 1 sec after each call
})
}
Inject required stuff properly. This assumes array to contain objects with ids using which you perform API calls. Also assumes that you are using $http. If using $resource, add $promise accordingly.
Hope that helps a bit!

function getItemsWithDelay(index) {
getFile(object[index].id).then(()=>{
setTimeout(()=>{
if(index+1 > object.length) { return }
getItemsWithDelay(index+1)
}, 5000)
})
}
You can make sequential calls

This is a awesome trick question to be asked in an interview, anyways I had a similar requirement and did some research on the internet and thanks to reference https://codehandbook.org/understanding-settimeout-inside-for-loop-in-javascript
I was able to delay all promise call in angularjs and the same can be applied in normal JS syntax as well.
I need to send tasks to a TTP API, and they requested to add a delay in each call
_sendTasks: function(taskMeta) {
var defer = $q.defer();
var promiseArray = [];
const delayIncrement = 1000 * 5;
let delay = 0;
for (i = 0; i < taskMeta.length; i++) {
// using 'let' keyword is VERY IMPORTANT else 'var' will send the same task in all http calls
let requestTask = {
"action": "SOME_ACTION",
"userId": '',
"sessionId": '',
};
// new Promise can be replaced with $q - you can try that, I haven't test it although.
promiseArray.push(new Promise(() => setTimeout(() => $http.post(config.API_ROOT_URL + '/' + requestTask.action, requestTask), delay)));
delay += delayIncrement;
}
$q.all(promiseArray).
then(function(results) {
// handle the results and resolve it at the end
defer.resolve(allResponses);
})
.catch(error => {
console.log(error);
defer.reject("failed to execute");
});
return defer.promise;
}
Note:: using 'let' keyword in FOR loop is VERY IMPORTANT else 'var' will send the same task in all http calls - due to closure/context getting switched

Async request into for loop angular.js

I have an array and i need to send values of array to webservice through http post request one by one . For the node.js , i'm using "async" package to do that for ex: async.eachSeries doing it well , how can i do that same thing for angular.js , my normal async code;
//this code sends all queries of array (maybe 5.000 request at same time , it is hard to process for webservice :=) ) at same time and wait for all responses.
//it works but actually for me , responses should wait others at end of loop should work one by one
//like async.eachSeries module!
for (var i = 0; i < myArr.lenght; i++) {
(function (i) {
var data = {
"myQuery": myArr[i].query
};
$http.post("/myServiceUrl", data).success(function (result) {
console.log(result);
});
})(i);
}
Both Matt Way and Chris L answers Correct , you can investigate Chris's answer for understanding about async to sync functions in for loops.

You can use $q to create a similar requirement by chaining promises together. For example:
var chain = $q.when();
angular.forEach(myArr, function(item){
chain = chain.then(function(){
var data = {
myQuery: item.query
};
return $http.post('/myServiceUrl', data).success(function(result){
console.log(result);
});
});
});
// the final chain object will resolve once all the posts have completed.
chain.then(function(){
console.log('all done!');
});
Essentially you are just running the next promise once the previous one has completed. Emphasis here on the fact that each request will wait until the previous one has completed, as per your question.

function logResultFromWebService(value)
{
$http.post("/myServiceUrl", value).success(console.log);
}
angular.forEach(myArray, logResultFromWebService);

If I understand your question correctly. You want to run a for loop in a synchronized manner such that the next iteration only occurs once the previous iteration is completed. For that, you can use a synchronized loop/callbacks. Especially if the order matters.
var syncLoop = function (iterations, process, exit) {
var index = 0,
done = false,
shouldExit = false;
var loop = {
next: function () {
if (done) {
if (shouldExit && exit) {
return exit(); // Exit if we're done
}
}
// If we're not finished
if (index < iterations) {
index++; // Increment our index
process(loop); // Run our process, pass in the loop
// Otherwise we're done
} else {
done = true; // Make sure we say we're done
if (exit) exit(); // Call the callback on exit
}
},
iteration: function () {
return index - 1; // Return the loop number we're on
},
break: function (end) {
done = true; // End the loop
shouldExit = end; // Passing end as true means we still call the exit callback
}
};
console.log('running first time');
loop.next();
return loop;
}
For your particular implementation:
syncLoop(myArray.length, function (loop) {
var index = loop.iteration();
var data = {
"myQuery": myArray[index].query
};
$http.post("/myServiceUrl", data).success(function (result) {
console.log(result);
loop.next();
});
}, function () {
console.log('done');
});
If you intend on doing something with the data once returned (such as perform calculations) you can do so with this method because you will return the data in a specified order.
I implemented something similar in a statistical calculation web app I built.
EDIT:
To illustrate the problem I had when using $q.when I have set up a fiddle. Hopefully this will help illustrate why I did this the way I did.
https://jsfiddle.net/chrislewispac/6atp3w8o/
Using the following code from Matt's answer:
var chain = $q.when(promise.getResult());
angular.forEach(myArr, function (item) {
chain = chain.then(function () {
$rootScope.status = item;
console.log(item);
});
});
// the final chain object will resolve once all the posts have completed.
chain.then(function () {
console.log('all done!');
});
And this fiddle is an example of my solution:
https://jsfiddle.net/chrislewispac/Lgwteone/3/
Compare the $q version to my version. View the console and imagine those being delivered to the user interface for user intervention in the process and/or performing statistical operations on the sequential returns.
You will see that it does not sequentially give the numbers 1,2,3,4 etc. either in the console or in the view in Matt's answer. It 'batches' the responses and then returns them. Therefore, if step 3 is not to be run depending on the response in step 2 there is not, at least in the answer provided, a way to break out or explicitly control the synchronous operation here. This presents a significant problem when attempting to perform sequential calculations and/or allow the user to control break points, etc.
Now, I am digging through both the $q libraries and the Q library to see if there is a more elegant solution for this problem. However, my solution does work as requested and is very explicit which allows me to place the function in a service and manipulate for certain use cases at my will because I completely understand what it is doing. For me, that is more important than using a library (at least at this stage in my development as a programmer and I am sure there are lots of other people at the same stage on StackOverflow as well).

If the order doesn't matter in which they are sent
var items = [/* your array */];
var promises = [];
angular.forEach(items, function(value, key){
var promise = $http.post("/myServiceUrl", { "myQuery": value.query });
promises.push(promise);
});
return $q.all(promises);

node-mysql timing

i have a recursive query like this (note: this is just an example):
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
database.query('select * from users where owner='+data.id,function(err,result,fields)
{
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
this don't work because the timmings are all wrong, the code don't wait for the query.
i've tried to do this:
var MyQuery = function(QueryString){
var Data;
var Done = false;
database.query(QueryString, function(err, result, fields) {
Data = result;
Done = true;
});
while(Done != true){};
return Data;
}
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
result= MyQuery('select * from users where owner='+data.id);
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
but he just freezes on the while, am i missing something?

The first hurdle to solving your problem is understanding that I/O in Node.js is asynchronous. Once you know how this applies to your problem the recursive part will be much easier (especially if you use a flow control library like Async or Step).
Here is an example that does some of what you're trying to do (minus the recursion). Personally, I would avoid recursively loading a possibly unknown number/depth of records like that; Instead load them on demand, like in this example:
var User = function(data) {
this.data = data
this.minions;
};
User.prototype.getMinions = function(primaryCallback) {
var that = this; // scope handle
if(this.minions) { // bypass the db query if results cached
return primaryCallback(null, this.minions);
}
// Callback invoked by database.query when it has the records
var aCallback = function(error, results, fields) {
if(error) {
return primaryCallback(error);
}
// This is where you would put your recursive minion initialization
// The problem you are going to have is callback counting, using a library
// like async or step would make this party much much easier
that.minions = results; // bypass the db query after this
primaryCallback(null, results);
}
database.query('SELECT * FROM users WHERE owner = ' + data.id, aCallback);
};
var user = new User(someData);
user.getMinions(function(error, minions) {
if(error) {
throw error;
}
// Inside the function invoked by primaryCallback(...)
minions.forEach(function(minion) {
console.log('found this minion:', minion);
});
});
The biggest thing to note in this example are the callbacks. The database.query(...) is asynchronous and you don't want to tie up the event loop waiting for it to finish. This is solved by providing a callback, aCallback, to the query, which is executed when the results are ready. Once that callback fires and after you perform whatever processing you want to do on the records you can fire the primaryCallback with the final results.

Each Node.js process is single-threaded, so the line
while(Done != true){};
takes over the thread, and the callback that would have set Done to true never gets run because the thead is blocked on an infinite loop.
You need to refactor your program so that code that depends on the results of the query is included within the callback itself. For example, make MyQuery take a callback argument:
MyQuery = function(QueryString, callback){
Then call the callback at the end of your database.query callback -- or even supply it as the database.query callback.

The freezing is unfortunately correct behaviour, as Node is single-threaded.
You need a scheduler package to fix this. Personally, I have been using Fibers-promise for this kind of issue. You might want to look at this or another promise library or at async