I have a scenario where I have three tables in a PostgreSQL database, users, orgs, and users_orgs, which links the first two. I use Express and pg to handle the DB calls. Now, if I want to, say, attach a list of org records to a user record, I want to perform one query for the user record, one for all of the users_orgs associated with that user, and then one for each of those records to get the orgs.
When I do this, I end up with something like
const user_query = `SELECT * FROM users WHERE id=${id}`;
pg.query(user_query)
.then((user_result) => {
const users = user_result.rows;
users.map((user) => {
user.orgs = [];
const users_orgs_query = `SELECT org_id FROM users_orgs WHERE user_id = '${user.id}'`;
pg.query(users_orgs_query)
.then((users_orgs_result) => {
return new Promise((res, rej) => {
const users_orgs = users_orgs_result.rows;
let c = 0;
users_orgs.map((user_org) => {
const orgs_query = `SELECT * FROM orgs WHERE id = '${user_org.org_id}'`;
pg.query(orgs_query)
.then((r) => {
const orgs = r.rows;
user.orgs.push(orgs[0]);
c += 1;
if (c >= users_orgs.length) {
res(user);
}
});
});
});
})
.then((u) => {
res.status(200).json(u);
});
});
Now, this works, but I feel confident that counting into my map is not a good idea. Clearly, I could replace the inner map with a for loop and just count and resolve the promise that way as well, but that feels like something I would do in C (which is where I normally work) and that feels like cheating.
However, I need the resolve to happen after the last element maps because otherwise, I will respond to the request before adding all the orgs to the user. Surely this is not uncommon, but I feel like I am not seeing anything related when I search, which leads me to believe that I'm searching poorly and/or thinking about this all wrong.
There may even be a SQL query-based solution to this type of thing, and that's great, but I would still be curious if there is a way to wait for a loop of nested promises to resolve in an elegant manner.
To make it clear, the actual question is, is there a way to know that all of the inner promises have been resolved without having to actually count them all, as I do here?
You can wait for all the promises to finish by using Promise.all. This function accepts an array of promises and returns a Promise by itself, which will resolve when all given promises have successfully been resolved.
Use this is in combination map method of the array to return an array of promises.
async function getUsersWithOrgs(id) {
const user_query = `SELECT * FROM users WHERE id=${id}`;
const { rows: users } = await pg.query(user_query)
return Promise.all(users.map(async user => {
const users_orgs_query = `SELECT org_id FROM users_orgs WHERE user_id = '${user.id}'`;
const { rows: users_orgs } = await pg.query(users_orgs_query);
const orgs = await Promise.all(users_orgs.map(async user_org => {
const orgs_query = `SELECT * FROM orgs WHERE id = '${user_org.org_id}'`;
const { rows: orgs } = await pg.query(orgs_query);
return orgs[0];
}));
return {
...user,
orgs
};
}));
}
getUsersWithOrgs(id).then(users => {
res.status(200).json(users);
});
You need to use async, await and for the inner promises you can use Promise.all
awaiting individually query will be un-optimal.
You can structure your code like (taking the innermost query as an example):
const orgs_query = `SELECT * FROM orgs WHERE id = '${user_org.org_id}'`;
await Promise.all(user_orgs.map((user_org) => {
pg.query(orgs_query)
.then((r) => {
const orgs = r.rows;
user.orgs.push(orgs[0]);
});
}));
// Return the response as you like from the users object
The pq.query returns a Promise from the mapped function. The outer Promise.all will collect these promises and await till all of them return.
This way you don't need to nest.
Async version, btw your code is a little complex, so maybe not work.
If not working, please comment:
const user_query = `SELECT * FROM users WHERE id=${id}`;
const user_result = await pg.query(user_query);
const users = result.rows;
for (let i = 0; i < users.length; i++) {
const user = users[i];
user.orgs = [];
const users_orgs_query = `SELECT org_id FROM users_orgs WHERE user_id = '${user.id}'`;
const users_orgs_result = await pg.query(users_orgs_query);
const users_orgs = users_orgs_result.rows;
for (let j = 0; j < users_orgs.length; j++) {
const user_org = users_orgs[j];
const orgs_query = `SELECT * FROM orgs WHERE id = '${user_org.org_id}'`;
const r = await pg.query(orgs_query);
const orgs = r.rows;
user.orgs.push(orgs[0]);
if (c == users_orgs.length - 1) {
res.status(200).json(u);
}
}
}
Related
When I run this code it give me a random order of Pokémons. I don't know where is the problem.
Thank you so much.
for (var i = 1; i <= 20; i++) {
apiPokemon("https://pokeapi.co/api/v2/pokemon/"+i);
async function apiPokemon(urlPokemon) {
const response = await fetch(urlPokemon);
const dataPokemon = await response.json();
var id = dataPokemon.id;
var name = dataPokemon.name;
console.log(id, name);
}
}
First thing's first: "Why are they coming back in random order?" - Because you are not awaiting each response. Instead you are firing off all 20 async calls which can come back in any order, so that's why they are logging in a random order.
In order to fix this, there are a few changes I'd recommend:
Extract your apiPokemon function out of your loop so it doesn't get recreated for each loop iteration
Return the entire data object from your apiPokemon function
Add all of the apiPokemon requests to an array and await them with Promise.all()
Log the output of the Promise.all() and you'll see that they will now always be in correct order
async function apiPokemon(urlPokemon) {
const response = await fetch(urlPokemon);
const dataPokemon = await response.json();
return dataPokemon;
}
async function getPokemon(startIndex, stopIndex) {
let requests = [];
for (let i = startIndex; i <= stopIndex; i++) {
requests.push(apiPokemon("https://pokeapi.co/api/v2/pokemon/"+i));
}
let pokemonList = await Promise.all(requests);
for (let pokemon of pokemonList) {
console.log(pokemon.id, pokemon.name);
}
}
getPokemon(1, 20)
Essentially, I have an object with string keys and values (ex. {"michigan":"minnesota"}). I'm trying to loop through all of these key value pairs and make a query from my database, and add the result to a list, which will then be what is returned to the front end.
var return_list = []
Object.keys(obj).forEach(function(key){
const state1 = key;
const state2 = obj[key];
const sql_select = 'SELECT column1,column2 from database WHERE state = ? OR state=?';
db.query(sql_select,[state1,state2], (err,result) => {
return_list.push(result);
});
})
This is what I have in simplest terms, and would like to send return_list back to the front end. The problem I'm running into is I can console.log the result within db.query call, but I can't push the result to the list or call it anywhere outside of the query. I'm fairly new to both front end and back end development, so any possible ideas would definitely be helpful!
The problem is that the forEach returns void.
So you can't wait for the asynchronous code to run before you return it.
When we're dealing with an array of promises such as db queries ( like in your case ) or API calls, we should wait for every one of them to be executed.
That's when we use the Promise.all
Try doing it like this:
const queryResults = await Promise.all(
Object.keys(obj).map(async (key) => {
const state1 = key;
const state2 = obj[key];
const sql_select = 'SELECT column1,column2 from database WHERE state = ? OR state=?';
return new Promise((resolve, reject) =>
db.query(sql_select,[state1,state2], (err, result) => {
if (err)
return reject(err)
else
return resolve(result)
})
)
})
)
console.log('queryResults', queryResults)
// now you give this queryResults back to your FE
Small tips for your fresh start:
never use var, try always use const or if needed, let.
try always use arrow functions ( () => {...} ) instead of regular functions ( function () {...} ), It's hard to predict which scope this last one is using
The issue is because the database transaction is not instant, so you need to use either promises or async-await.
Using async await would be something like this (untested),
async function get_return_list () {
var return_list = []
Object.keys(obj).forEach(function(key){
const state1 = key;
const state2 = obj[key];
const sql_select = 'SELECT column1,column2 from database WHERE state = ? OR state=?';
await db.query(sql_select,[state1,state2], (err,result) => {
return_list.push(result);
});
})
return return_list
}
see for more detail: https://eloquentjavascript.net/11_async.html
First, make sure you are working with mysql2 from npm. Which provides async method of mysql.
Second, Note that when you query SELECT, you don't get the "real" result in first. Suppose you get result, then, the "real" results are held in result[0].
(async () => {
const promiseArr = [];
for (const key of Object.keys(yourOBJ)) {
const state1 = key;
const state2 = yourOBJ[key];
const sql_select = 'SELECT column1,column2 from database WHERE state = ? OR state=?';
promiseArr.push(db.query(sql_select, [state1, state2]));
}
let results;
try {
results = await Promise.all(promiseArr);
} catch (e) {
throw '...';
}
const return_list = results.reduce((finalArray, item) => {
finalArray = [
...finalArray,
...item[0],
]
}, []);
})();
I record favorite offers that a user 'hearts' in my app . These records include the owner and offer IDs. I want to collect the top 25 favorited Offers for a particular user. All firestore commands are asynchronous and I need to collect all the offer objects before I render the page.
Its my first time using async / await and what started as one has quickly grown into nested async / awaits. There must be a simpler way to collect the IDs from the fav objects and then lookup the Offers with those IDs?
async getItems() {
const collectfavs = async () => {
favsRef = firestore.collection('favs').where('owner','==',getUserID()).orderBy('created', 'desc').limit(25);
let allFavsSnapshot = await favsRef.get();
allFavsSnapshot.forEach(doc => {
let data = doc.data();
favsList.push(data.offer);
});
console.log('favs:',favsList);
}
const collectoffers = async () => {
favsList.forEach(async (fav) => {
let doc = await firestore.collection('offers').doc(fav).get()
console.log('doc:', doc);
let data = doc.data();
data.id = doc.id;
offerList.push(data);
});
console.log('offers:', offerList);
}
await collectfavs();
await collectoffers();
}
I'm not sure why you're defining two local functions just to call them each once. That seems like more code than necessary to get the job done. Other than that, what you're doing doesn't seem very complex to me. But if you want to reduce the lines of code:
async getItems() {
favsRef = firestore.collection('favs').where('owner','==',getUserID()).orderBy('created', 'desc').limit(25);
let allFavsSnapshot = await favsRef.get();
allFavsSnapshot.forEach(doc => {
let data = doc.data();
favsList.push(data.offer);
});
console.log('favs:',favsList);
favsList.forEach(async (fav) => {
let doc = await firestore.collection('offers').doc(fav).get()
console.log('doc:', doc);
let data = doc.data();
data.id = doc.id;
offerList.push(data);
});
console.log('offers:', offerList);
}
Bear in mind that I have no idea where you defined favsList and offerList, so I'm just blindly using it the same way you showed.
As I understand from your comment to Doug Stevenson above you want to ensure that you lists will be filled before using them. In order to do that you could you Promises. What Promises do is give us a way to deal with asynchrony in the code in a sequential manner.
All you need to do is create a promise that will ensure that you first fill the Lists you want and then you use them.
Let me know if this helps.
I ended up needing to create my own asyncForEach routine to accommodate the async calls to firebase inside the loop. Thanks to an article by Sebastien Chopin asyncForEach
async function asyncForEach(array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array);
}
}
and then in my getItems:
await asyncForEach(favsList, async (fav) => {
let doc = await firestore.collection('offers').doc(fav).get()
let data = doc.data();
data.id = doc.id;
offerList.push(data);
});
I am developing a small web app to track points for a local board game group. The scoreboard will track a players total games played, total points earned as well as a breakdown of games played and points earned per faction available in the game.
I am currently hitting my head against a wall trying to create the node/express endpoint for /players/withFactions/, which will return aggregate player data as well as the per faction breakdown.
As you will see in the code attached, the logic works correctly and the data is correct right at the location of the first console.log. That data is simply returned and I am logging the return at the second console log, where it seems the data has changed. The mutated data seems to always take the form of the last returned element from the location of the first console.log. I hope this will make more sense upon review of the code.
I am sure my problem lies in how I am handling all of the async, but I cannot figure out the solution.
I have attacked this issue a multitude of different ways: callbacks, promises, async/await. All with the same outcome. The data seems to change on me right before being returned to the front end.
router.get('/withFactions', async (req, res) => {
const { rows: players } = await db.query('SELECT * FROM players');
const { rows: factions } = await db.query('SELECT * FROM factions');
const playerResults = await Promise.all(players.map( async (player) => {
await db.query('SELECT * FROM gameentry WHERE player_id = $1', [player.id]).then((result) => {
const gameEntries = result.rows;
const factionTotals = factions.map((faction) => {
const factionEntries = gameEntries.filter(game => game.faction_id === faction.id)
faction.totalPoints = factionEntries.reduce((acc, curr) => {
return acc + curr.points;
}, 0)
return faction;
})
player.factionTotals = factionTotals;
})
player.factionTotals.forEach(element => console.log(element.totalPoints))
// total scores are recording correctly here 4,0,0 5,0,3 0,5,0 (from DB data)
return await player;
}))
await playerResults.forEach(player => player.factionTotals.forEach(element => console.log(element.totalPoints)));
// total scores are wrong here. Each set is 0,5,0. Seems to just replicate the last set from within the promise
res.send(await playerResults);
})
NOTE: the ONLY data that is incorrect is player.factionTotals the rest of the player data remains accurate.
I am sure there are a few more awaits than necessary in there. I have been looking at this problem for too long. I hope I can get some assistance on this one.
As identified by Jaromanda X the issue is that you modify the factions for every player, so only the last modification will "stick around".
In general i would recommend to not have side effects when using map and only return new objects or unmodified objects. To that end you should be able to modify your code to something like this:
router.get('/withFactions', async (req, res) =>
{
const { rows: players } = await db.query('SELECT * FROM players');
const { rows: factions } = await db.query('SELECT * FROM factions');
const playerResults = await Promise.all(players.map(async player =>
{
const { rows: gameEntries } = await db.query('SELECT * FROM gameentry WHERE player_id = $1', [player.id]);
const factionTotals = factions.map(faction =>
{
const factionEntries = gameEntries.filter(game => game.faction_id === faction.id)
var totalPoints = factionEntries.reduce((acc, curr) => acc + curr.points, 0);
return { ...faction, totalPoints }; // Copies faction properties instead of modifying
})
const extendedPlayer = { ...player, factionTotals };
extendedPlayer.factionTotals.forEach(element => console.log(element.totalPoints))
return extendedPlayer;
}))
playerResults.forEach(player => player.factionTotals.forEach(element => console.log(element.totalPoints)));
res.send(playerResults);
})
Using the spread (...) copies the properties, so the original factions list is the same for every player iteration. The second spread for creating extendedPlayer is probably not strictly necessary to prevent your issue, but keeps the outer map pure as well.
The console in the end returns empty array.
The console runs before ids.map function finishes
var ids = [];
var allLync = []
var user = await User.findOne(args.user)
ids.push(user._id)
user.following.map(x => {
ids.push(x)
})
ids.map(async x => {
var lync = await Lync.find({ "author": x })
lync.map(u => {
allLync.push[u]
})
})
console.log(allLync)
What am I doing wrong?
The .map code isn't awaited, so the console.log happens before the mapping happens.
If you want to wait for a map - you can use Promise.all with await:
var ids = [];
var allLync = []
var user = await User.findOne(args.user)
ids.push(user._id)
user.following.map(x => {
ids.push(x)
})
// note the await
await Promise.all(ids.map(async x => {
var lync = await Lync.find({ "author": x })
lync.map(u => {
allLync.push(u); // you had a typo there
})
}));
console.log(allLync)
Note though since you're using .map you can shorten the code significantly:
const user = await User.findOne(args.user)
const ids = users.following.concat(user._id);
const allLync = await Promise.all(ids.map(id => Lync.find({"author": x })));
console.log(allLync);
Promise.map() is now an option that would be a tiny bit more succinct option, if you don't mind using bluebird.
It could look something like:
const user = await User.findOne(args.user);
const ids = users.following.concat(user._id);
const allLync = await Promise.map(ids, (id => Lync.find({"author": x })));
console.log(allLync);
http://bluebirdjs.com/docs/api/promise.map.html. I have really enjoyed using it.