I am patching the exec() function to allow subpopulating in Mongoose, which is why I am not able to use async/await here -- my function will be chained off a db call, so there is no opportunity to call await on it, and within the submodule itself, there I can't add async/await outside of an async function itself.
With that out of the way, let's look at what I'm trying to do. I have two separate arrays (matchingMealPlanFoods and matchingMealPlanRecipeFoods) full of IDs that I need to populate. Both of them reside on the same array, foods. They each require a db call with aggregation, and the problem in my current scenario is that only one of the arrays populates because they are happening asynchronously.
What I am trying to do now is use the reduce function to return the updated foods array to the next run of reduce so that when the final result is returned, I can replace the entire foods array once on my doc. The problem of course is that my aggregate/exec has not yet returned a value by the time the reduce function goes into its next run. Is there a way I can achieve this without async/await here? I'm including the high-level structure here so you can see what needs to happen, and why using .then() is probably not viable.
EDIT: Updating code with async suggestion
function execute(model, docs, options, lean, cb) {
options = formatOptions(options);
let resolvedCount = 0;
let error = false;
(async () => {
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (let option of options) {
const path = option.path.split(".");
// ... various things happen here to prep the data
const aggregationOptions = [
// // $match, then $unwind, then $replaceRoot
];
await rootRefModel
.aggregate(aggregationOptions)
.exec((err, refSubDocuments) => {
// more stuff happens
console.log('newFoodsArray', newFoodsArray); // this is to check whether the second iteration is using the updated newFoods Array
const arrToReturn = newFoodsArray.map((food) => {
const newMatchingArray = food[nests[1]].map((matchingFood) => {
//more stuff
return matchingFood;
});
const updatedFood = food;
updatedFood[`${nests[1]}`] = newMatchingArray;
return updatedFood;
});
console.log('arrToReturn', arrToReturn);
newFoodsArray = [...arrToReturn];
});
}
};
console.log('finalNewFoods', newFoodsArray); // this should log after the other two, but it is logging first.
const document = doc.toObject();
document.foods = newFoodsArray;
if (resolvedCount === options.length) cb(null, [document]);
}
})()
EDIT: Since it seems it will help, here is the what is calling the execute function I have excerpted above.
/**
* This will populate sub refs
* #param {import('mongoose').ModelPopulateOptions[]|
* import('mongoose').ModelPopulateOptions|String[]|String} options
* #returns {Promise}
*/
schema.methods.subPopulate = function (options = null) {
const model = this.constructor;
if (options) {
return new Promise((resolve, reject) => execute(model, [this], options, false, (err, docs) => {
if (err) return reject(err);
return resolve(docs[0]);
}));
}
Promise.resolve();
};
};
We can use async/await just fine here, as long as we remember that async is the same as "returning a Promise" and await is the same as "resolving a Promise's .then or .catch".
So let's turn all those "synchronous but callback-based" calls into awaitables: your outer code has to keep obeying the API contract, but since it's not meant to a return a value, we can safely mark our own version of it as async, and then we can use await in combination with promises around any other callback based function calls in our own code just fine:
async function execute(model, docs, options, lean, andThenContinueToThis) {
options = formatOptions(options);
let option, resolvedCount = 0;
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (option of options) {
// ...things happen here...
const aggregationOptions = [/*...data...*/];
try {
const refSubDocuments = await new Promise((resolve, reject) => rootRefModel
.aggregate(aggregationOptions)
.exec((err, result) => err ? reject(err) : resolve(result));
// ...do some work based on refSubDocuments...
}
// remember to forward errors and then stop:
catch (err) {
return andThenContinueToThis(err);
}
}
// remember: bind newFoodsArray somewhere so it doesn't get lost next iteration
}
// As our absolutely last action, when all went well, we trigger the call forwarding:
andThenContinueToThis(null, dataToForward);
}
Related
What is the correct way to implement array.push so that it "array_of_results" is returned after the forEach iteration if finished?
const postgres = require("./postgres");
function get_array(value) {
var array_of_results = []
value.forEach( item => {
postgres.query(item["id"],function(res){
console.log(res) //gives proper res after empty array
array_of_results.push(res);
})
});
console.log(array_of_results)// prints empty array
return array_of_results;
}
Edit:
and postgres.js looks like :
const { Pool } = require("pg");
const pool = new Pool();
var query_string = "select...."
function query(id, call) {
pool.query(query_string, [id], (err, res) => {
if (err) {
console.log(err.stack)
} else {
call(res.rows[0])
}
})
}
module.exports = {
query
}
There are a few ways to do this, but first you need to understand what is actually happening.
In postgres.query(item["id"],function(res){ you are calling postgres.query with (1) an item ID and (2) a callback function. That call happens and then immediately continues in your calling code. So now you've just sent a bunch of requests to your database, and then immediately return an empty array. Those callbacks (2) have not been called yet.
To get the data back to your calling function, you'll need to either pass a callback instead of using return, or change to async/await.
Using async/await in every iteration of your loop is not as efficient, as you're waiting for each call to return sequentially. For the most efficient method, you will need to fire the requests and wait for them all to complete. You can do this by using promises.
You can modify your code to push a promise into an array for each iteration of the loop, then call (and await) Promise.all on the array of promises.
Here's a basic rewrite for you:
postgres.js:
function query(id) {
return new Promise((resolve, reject) => {
pool.query(query_string, [id], (err, res) => {
if (err) {
console.log(err.stack)
reject(err)
} else {
resolve(res.rows[0])
}
})
})
}
module.exports = {
query
}
get_array implementation :
async function get_array(value) {
var array_of_promises = [], array_of_results = []
value.forEach( item => {
array_of_promises.push(postgres.query(item["id"]));
});
array_of_results = await Promise.all(array_of_promises);
console.log(array_of_results)// prints populated array
return array_of_results;
}
Note that when you call get_array you'll have to use await before the call, e.g. change var array = get_array(items) to var array = await get_array(items) and using await in a function requires it to be declared as an async function.
If you can't declare it as an async function, you may change the calling code to consume the promise:
var arrayPromise = get_array(items);
arrayPromise.then((results) => {
// do something with results
// but remember you cannot _return_ from within a callback, as discussed above
});
I'm working with fetching information from a github repository. I want to get the list of pull requests within that repo, get the list of commits associated with each pull request, then for each commit I want to get information such as the author of the commit, the number of files associated with each commit and the number of additions and deletions made to each file. I'm using axios and the github API to accomplish this. I know how to work with the API, but the promises and async functions are keeping me from accomplishing my task. I have the following code:
const axios = require('axios');
var mapOfInformationObjects = new Map();
var listOfCommits = [];
var listOfSHAs = [];
var gitApiPrefix = link I'll use to start fetching data;
var listOfPullRequestDataObjects = [];
var listOfPullRequestNumbers = [];
var mapOfPullNumberToCommits = new Map();
function getAllPullRequests(gitPullRequestApiLink) {
return new Promise((resolve, reject) => {
axios.get(gitPullRequestApiLink).then((response) =>{
listOfPullRequestDataObjects = response['data'];
var k;
for (k = 0; k < listOfPullRequestDataObjects.length; k++){
listOfPullRequestNumbers.push(listOfPullRequestDataObjects[k]['number']);
}
resolve(listOfPullRequestNumbers);
}).catch((error) => {
reject(error);
})
})
}
function getCommitsForEachPullRequestNumber(listOfPRNumbers) {
var j;
for (j = 0; j < listOfPRNumbers.length; j++) {
currPromise = new Promise((resolve, reject) => {
currentGitApiLink = gitApiPrefix + listOfPRNumbers[j] + "/commits";
axios.get(currentGitApiLink).then((response) => {
mapOfPullNumberToCommits.set(listOfPRNumbers[j], response['data']);
resolve("Done with Pull Request Number: " + listOfPRNumbers[j]);
}).catch((error) => {
reject(error);
})
})
}
}
function getListOfCommits(gitCommitApiLink){
return new Promise((resolve, reject) => {
axios.get(gitCommitApiLink).then((response) => {
resolve(response);
}).catch((error) => {
reject(error);
})
})
}
So far, I made some functions that I would like to call sequentially.
First I'd like to call getAllPullRequestNumbers(someLink)
Then I'd like to call getCommitsForEachPullRequestNumber(listofprnumbers)
Then getListOfCommits(anotherLink)
So it would look something like
getAllPullRequestNumbers(someLink)
getCommitsForEachPullRequestNumber(listofprnumbers)
getListOfCommits(anotherlink)
But two problems arise:
1) I'm not sure if this is how you would call the functions so that the first function in the sequence completes before the other.
2) Because I'm not familiar with Javascript, I'm not sure, especially with the getCommitsForEachPullRequestNumber function since you run a loop and call axios.get() on each iteration of the loop, if this is how you work with promises within the functions.
Would this be how you would go about accomplishing these two tasks? Any help is much appreciated. Thanks!
When you a number of asynchronous operations (represented by promises) that you can run all together and you want to know when they are all done, you use Promise.all(). You collect an array of promises and pass it to Promise.all() and it will tell you when they have all completed or when one of them triggers an error. If all completed, Promise.all() will return a promise that resolves to an array of results (one for each asynchronous operation).
When you're iterating an array to do your set of asynchronous operations, it then works best to use .map() because that helps you create a parallel array of promises that you can feed to Promise.all(). Here's how you do that in getCommitsForEachPullRequestNumber():
function getCommitsForEachPullRequestNumber(listOfPRNumbers) {
let mapOfPullNumberToCommits = new Map();
return Promise.all(listOfPRNumbers.map(item => {
let currentGitApiLink = gitApiPrefix + item + "/commits";
return axios.get(currentGitApiLink).then(response => {
// put data into the map
mapOfPullNumberToCommits.set(item, response.data);
});
})).then(() => {
// make resolved value be the map we created, now that everything is done
return mapOfPullNumberToCommits;
});
}
// usage:
getCommitsForEachPullRequestNumber(list).then(results => {
console.log(results);
}).catch(err => {
console.log(err);
});
Then, in getListOfCommits(), since axios already returns a promise, there is no reason to wrap it in a manually created promise. That is, in fact, consider a promise anti-pattern. Instead, just return the promise that axios already returns. In fact, there's probably not even a reason to have this as a function since one can just use axios.get() directly to achieve the same result:
function getListOfCommits(gitCommitApiLink){
return axios.get(gitCommitApiLink);
}
Then, in getAllPullRequests() it appears you are just doing one axios.get() call and then processing the results. That can be done like this:
function getAllPullRequests(gitPullRequestApiLink) {
return axios.get(gitPullRequestApiLink).then(response => {
let listOfPullRequestDataObjects = response.data;
return listOfPullRequestDataObjects.map(item => {
return item.number;
});
});
}
Now, if you're trying to execute these three operations sequentially in this order:
getAllPullRequests(someLink)
getCommitsForEachPullRequestNumber(listofprnumbers)
getListOfCommits(anotherlink)
You can chain the promises from those three operations together to sequence them:
getAllPullRequests(someLink)
.then(getCommitsForEachPullRequestNumber)
.then(mapOfPullNumberToCommits => {
// not entirely sure what you want to do here, perhaps
// call getListOfCommits on each item in the map?
}).catch(err => {
console.log(err);
});
Or, if you put this code in an async function, then you can use async/awit:
async function getAllCommits(someLink) {
let pullRequests = await getAllPullRequests(someLink);
let mapOfPullNumberToCommits = await getCommitsForEachPullRequestNumber(pullRequests);
// then use getlistOfCommits() somehow to process mapOfPullNumberToCommits
return finalResults;
}
getAllCommits.then(finalResults => {
console.log(finalResults);
}).catch(err => {
console.log(err);
});
not as clean as jfriend00 solution,
but I played with your code and it finally worked
https://repl.it/#gui3/githubApiPromises
you get the list of commits in the variable listOfCommits
I don't understand the purpose of your last function, so I dropped it
I'm fairly new to NodeJS and am trying to understand async, await, and promises. The likes variable is undefined if I take resolve() outside of the callback scope of 'con.query' (still inside the new Promise scope). It is only populated if resolve() is inside the callback. Why does this happen? Examples:
//WORKS
async function findLikes(q, req, res){
var likes_q = `SELECT * FROM PublicStoryLike WHERE authorId =
${req.body.token_id}`;
var likes;
var lookup = {};
//const query = util.promisify(con.query).bind(con);
const query = new Promise((resolve, reject) => {
con.query(likes_q, (err, result) => {
likes = JSON.stringify(result);
resolve(); //INSIDE con.query
});
})
//DOESN'T WORK
async function findLikes(q, req, res){
var likes_q = `SELECT * FROM PublicStoryLike WHERE authorId =
${req.body.token_id}`;
var likes;
var lookup = {};
//const query = util.promisify(con.query).bind(con);
const query = new Promise((resolve, reject) => {
con.query(likes_q, (err, result) => {
likes = JSON.stringify(result);
});
resolve(); //OUTSIDE con.query
})
Let's forget about promises. The missunderstanding is rather about callbacks here: Your case can be illustrated as:
function works(callback) {
setTimeout(function() {
console.log("inside timeout");
callback("result");
}, 1000);
}
function doesnt(callback) {
setTimeout(function() {
console.log("inside timeout");
}, 1000);
callback();
}
works(function(result) { console.log("called back"); });
In the working version, you call back the outer callback when the inner callback was called, meaning the result arrived from the db in your case.
In the non working version you call back the callback directly, yet the timer (or the database call) did not finish, therefore the results are not there yet.
You should usually not use global variables, especially when asynchrony is involved. Your likes variable will cause you (and does already) cause you headaches, just remove it completely. Instead, resolve the promise (or call the callback back) with the value needed. That way you can easily detect errors:
getResult(function(result) {
callback(result); // works, result is ready
});
callback(result); // does not work, throws an error
In your case, that'll be:
async function findLikes(q, req, res){
const query = `SELECT * FROM PublicStoryLike WHERE authorId = ${req.body.token_id}`;
return new Promise((resolve, reject) => {
con.query(query, (err, result) => {
if(err) reject(err) else resolve(result);
});
});
}
findLikes(/*...*/)
.then(likes => {
// Work with likes here!
});
The problem in the second code block is that you are resolving the promise immediately. You are not waiting for the con.query method to finish.
Since you want to use async/await and you seem to be working with util.promisify I would recommend the following:
async function findLikes(q, req, res) {
const likes_q = `SELECT * FROM PublicStoryLike WHERE authorId = ${
req.body.token_id
}`;
const query = util.promisify(con.query).bind(con);
const likes = await query(likes_q);
// ....
}
You can see you don't need to manually create a Promise. That's what util.promisify will do for you. You can simply just await for it.
EDIT: By further looking at your code, I see that findLikes looks like an express route. Generally, I don't recommend coupling your web-request logic with your data access layer.
I would suggest you separate them, by making findLikes a standalone async function:
async function findLikes(authorId) {
const likes_q = `SELECT * FROM PublicStoryLike WHERE authorId = ${
authorId
}`;
const query = util.promisify(con.query).bind(con);
return await query(likes_q);
}
And later on in your route:
async function findLikesRoute(q, req, res){
const likes = await findLikes(req.body.token_id);
//... work with the data
}
This will give you the benefits of separation of concerns. For example, what would you do if let's say you would want to change the way you expose data?. Instead of exposing it through this express route, you now want now to expose it through GraphQL?
If you have your data access logic separated, it would be really easy.
I have a function which returns a list of objects in Javascript, and I'm calling this function from another and attempting to use some of the values from it, but whenever I try to access said values, they come back undefined.
This is my function which generates the list - the idea is that it creates a sqlite3 database if it does not exist, and returns an array containing every event.
function listAllEvents() {
const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('schedule.db');
const selectionArray = [];
db.serialize(() => {
db.run(`
CREATE TABLE IF NOT EXISTS todo (
name text,
date text,
id text primary key
)
`);
db.all('SELECT * FROM todo ORDER BY date', [], (err, rows) => {
if (err) {
throw err;
}
rows.forEach((row) => {
selectionArray.push(row);
});
});
});
return selectionArray;
}
I call this function from another, but when I try to access values from the array, they don't seem to be working and I can't quite figure it out.
function displayUpcomingEvents() {
const events = listAllEvents();
// console.log(events); <-- This line here! In the console, it correctly states the length of the array
// console.log(events.length) <-- This line, however, returns 0. Why?
// console.log(events[0]) <-- This doesn't work either, it just returns "undefined".
for (let i = 0; i < events.length; i += 1) {
$('#upcomingEvents').after('<li>asdf</li>');
}
}
For example, if I were to create two events in the database, through the console,
events is an Array(2) with indices
- 0: {name: "my_event", date: "2019-06-04", id: "c017c392d446d4b2"}
- 1: {name: "my_event_2", date: "2019-06-04", id: "6d655ac8dd02e3fd"},
events.length returns 0,
and events[0] returns undefined.
Why is this, and what can I do to fix it?
The possible reason why this is happening, is because of the async nature of JS, that means all the console.log statements are getting executed before the successful execution of the listAllEvents() function,
So my suggestion is to try using the promises, and perform all the actions mentioned after the listAllEvents() function only when that function returns a promise.
You can also try making the function async and using await to wait for its successful execution. (Much Smarter Choice will be using async)
Link to ASYNC Functions and Usage
Link to Promises
Also you can check the validity of answer by doing console.log(row) where you are pushing rows to the array. You will observer that the console.log(row) will be executed at the last, after printing events and other log statements.
The problem is that your function is returning the variable before a value is set. The db.serialize function will run asynchronously (outside the normal flow of the program) and the return statement will run immediately after. One thing you can do is use async/await in conjunction with Promise. In this case the the variable results will wait for the promise to be resolved before continuing to the next line.
async function listAllEvents() {
const selectionArray = [];
let promise = new Promise( function (resolve, reject) {
db.serialize(() => {
db.run(
CREATE TABLE IF NOT EXISTS todo (
name text,
date text,
id text primary key
)
);
db.all('SELECT * FROM todo ORDER BY date', [], (err, rows) => {
if (err) {
// add code here to reject promise
throw err;
}
rows.forEach((row) => {
selectionArray.push(row);
});
resolve(selectionArray);// resolve the promise
});
});
});
let results = await promise;
return results;
};
async function displayUpcomingEvents() {
const events = await listAllEvents();
// console.log(events); <-- This line here! In the console, it correctly states the length of the array
// console.log(events.length) <-- This line, however, returns 0. Why?
// console.log(events[0]) <-- This doesn't work either, it just returns "undefined".
for (let i = 0; i < events.length; i += 1) {
$('#upcomingEvents').after('<li>asdf</li>');
}
}
Note here that the displayUpcomingEvents function will also need to be async or you cannot use the await keyword.
Additional reading for Promise keyword MDN: Promise
Additional reading for Async/Await MDN: Asyn/Await
Hi I need to execute promises one after the other how do I achieve this using promise.all any help would be awesome. Below is the sample of my code I am currently using but it executes parallel so the search will not work properly
public testData: any = (req, res) => {
// This method is called first via API and then promise is triggerd
var body = req.body;
// set up data eg 2 is repeated twice so insert 2, 5 only once into DB
// Assuming we cant control the data and also maybe 3 maybe inside the DB
let arrayOfData = [1,2,3,2,4,5,5];
const promises = arrayOfData.map(this.searchAndInsert.bind(this));
Promise.all(promises)
.then((results) => {
// we only get here if ALL promises fulfill
console.log('Success', results);
res.status(200).json({ "status": 1, "message": "Success data" });
})
.catch((err) => {
// Will catch failure of first failed promise
console.log('Failed:', err);
res.status(200).json({ "status": 0, "message": "Failed data" });
});
}
public searchAndInsert: any = (data) => {
// There are database operations happening here like searching for other
// entries in the JSON and inserting to DB
console.log('Searching and updating', data);
return new Promise((resolve, reject) => {
// This is not an other function its just written her to make code readable
if(dataExistsInDB(data) == true){
resolve(data);
} else {
// This is not an other function its just written her to make code readable
insertIntoDB(data).then() => resolve(data);
}
});
}
I looked up in google and saw the reduce will help I would appreciate any help on how to convert this to reduce or any method you suggest (Concurrency in .map did not work)
the Promises unfortunatelly does not allow any control of their flow. It means -> once you create new Promise, it will be doing its asynchronous parts as they like.
The Promise.all does not change it, its only purpose is that it checks all promises that you put into it and it is resolved once all of them are finished (or one of them fail).
To be able to create and control asynchronous flow, the easiest way is to wrap the creation of Promise into function and create some kind of factory method. Then instead of creating all promises upfront, you just create only one promise when you need it, wait until it is resolved and after it continue in same behaviour.
async function doAllSequentually(fnPromiseArr) {
for (let i=0; i < fnPromiseArr.length; i++) {
const val = await fnPromiseArr[i]();
console.log(val);
}
}
function createFnPromise(val) {
return () => new Promise(resolve => resolve(val));
}
const arr = [];
for (let j=0; j < 10; j++) {
arr.push(createFnPromise(Math.random()));
}
doAllSequentually(arr).then(() => console.log('finished'));
PS: It is also possible without async/await using standard promise-chains, but it requires to be implemented with recursion.
If anyone else cares about ESLint complaining about the use of "for" and the "no await in loop" here is a typescript ESLint friendly version of the above answer:
async function runPromisesSequentially<T>(promises: Array<Promise<T>>):Promise<Array<T>> {
if (promises.length === 0) return [];
const [firstElement, ...rest] = promises;
return [await firstElement, ...(await runPromisesSequentially(rest))];
}
You can then just replace Promise.all by runPromisesSequentially.
#lmX2015's answer is close but it's taking in promises that have already started executing.
A slight modification fixes it
export async function runPromisesSequentially<T>(functions: (() => Promise<T>)[]): Promise<T[]> {
if (functions.length === 0) {
return [];
}
const [first, ...rest] = functions;
return [await first(), ...(await runPromisesSequentially(rest))];
}