I got some data which I'm calling from API and I am using axios for that. When data is retrieved, I dump it inside of a function called "RefractorData()" just to organize it a bit, then I push it onto existing array. The problems is, my array gets populated inside forEach and I can console.log my data there, but once I exit the loop, my array is empty.
let matches: any = new Array();
const player = new Player();
data.forEach(
async (match: any) => {
try {
const result = await API.httpRequest(
`https://APILink.com/matches/${match.id}`,
false
);
if (!result) console.log("No match info");
const refractored = player.RefractorMatch(result.data);
matches.push({ match: refractored });
console.log(matches);
} catch (err) {
throw err;
}
}
);
console.log(matches);
Now the first console.log inside forEach is displaying data properly, second one after forEach shows empty array.
Managed to do it with Promise.all() and Array.prototype.map()
.
const player = new Player();
const matches = result.data;
const promises = matches.map(async (match: any) => {
const response: any = await API.httpRequest(
`https://API/matches/${match.id}`,
false
);
let data = response.data;
return {
data: player.RefractorMatch(data)
};
});
const response: any = await Promise.all(promises);
You must understand that async functions almost always run later, because they deppend on some external input like a http response, so, the second console.log is running before the first.
There a few ways to solve this. The ugliest but easiest to figure out is to create a external promise that you will resolve once all http requests are done.
let matches = [];
let promise = new Promise((resolve) => {
let complete = 0;
data.forEach((match: any) => {
API.httpRequest(...).then((result) => {
// Your logic here
matches.push(yourLogicResult);
complete++;
if (complete === data.length) {
resolve();
}
}
}
};
console.log(matches); // still logs empty array
promise.then(() => console.log(matches)); // now logs the right array
You can solve this using other methods, for example Promise.all().
One very helpful way to solve it is using RxJs Observables. See https://www.learnrxjs.io/
Hope I helped you!
Related
I have a start(node, array) function that should perform a DFS by traversing an object tree via recursive calls to an API through callMsGraph(token, end) until image properties are found at the end of the tree, at which point they are pushed to array. The function seems like it works, but I can't get the output unless I wrap it in a 2 second setTimeout which indicates the recursion is not being waited on to complete. I would want to play around with async/await more, but it's not at the top-level.
I'm not sure if the nextNode.then is doing anything or maybe callMsGraph() needs to be awaited on differently to how I know. A solution would be much appreciated.
shelfdb.data = async (accessToken) => {
const token = accessToken;
const endpoint = 'https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/01GNYB5KPQ57RHLPZCJFE2QMVKT5U3NYY3/children'
function start(node, array) {
if(node.value.length > 0) {
node.value.forEach(function(child) {
var end = 'https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/' + child.id + '/children';
var nextNode = callMsGraph(token, end);
nextNode.then(function(currResult) {
if (currResult.value.length > 0) {
if ('image' in currResult.value[0]) {
currResult.value.forEach(function(imgChild) {
let img = {
'name': imgChild.name,
'job': imgChild.parentReference.path.split("/")[6],
'path': imgChild.webUrl,
'id': imgChild.id
}
array.push(img);
})
// complete storing images at tail object, go one level up after loop
return;
}
// if no 'image' or value, go into child
start(currResult, array);
}
}).catch(function(e) {
console.error(e.message);
})
})
}
return array;
}
var res = await callMsGraph(token, endpoint); // start recursion
var output = start(res, []);
console.log(output); // only displays value if wrapped in setTimeout
return output; // empty []
}
Each query to the API via callMsGraph(), returns an object like this, where subsequent queries are made with the id of each object/folder (as new endpoint) in value until an object with image property is found. The MS Graph API requires that folders are expanded at each level to access their children.
{
id: '01GNYB5KPQ57RHLPZCJFE2QMVKT5U3NYY3'
value: [
{
id: '01GNYB5KJMH5T4GXADUVFZRSITWZWNQROS',
name: 'Folder1',
},
{
id: '01GNYB5KMJKILOFDZ6PZBZYMXY4BGOI463',
name: 'Folder2',
}
]
}
This is the callMsGraph() helper:
function callMsGraph(accessToken, graphEndpoint) {
const headers = new Headers();
const bearer = `Bearer ${accessToken}`;
headers.append("Authorization", bearer);
const options = {
method: "GET",
headers: headers
};
return fetch(graphEndpoint, options)
.then(response => response.json())
.catch(error => {
console.log(error);
throw error;
});
}
The rule with promises is that once you opt into one (more likely, are forced into it by a library), all code that needs to block for a result anywhere after it also has to await. You can't "go back" to sync and if even a single piece of the promise chain between where the promise starts and where you want its result isn't awaited, the result will be unreachable*.
Taking a snippet of the code:
function start(node, array) { // not async!
// ..
node.value.forEach(function(child) { // doesn't await!
// ..
nextNode.then(function(currResult) {
// this promise is not hooked up to anything!
start(...) // recurse without await!
There's no await in front of then, start doesn't return a promise and isn't awaited recursively, and forEach has no way to await its callback's asynchronous results, so each promise in the nextNode.then chain is orphaned into the void forever*.
The solution is a structure like this:
async function start(node, array) {
// ..
for (const child of node.value) {
// ..
const currResult = await callMsGraph(token, end);
// ..
await start(...);
array.push(currResult);
}
// returns a promise implicitly
}
// ..
await start(...);
// `array` is populated here
Or Promise.all, which runs in parallel and returns an array (which could replace the parameter array):
function start(node, array) {
return Promise.all(node.value.map(async child => {
const currResult = await callMsGraph(token, end);
// ..
await start(...);
return currResult;
}));
}
I'd be happy to provide a minimal, runnable example, but the code you've provided isn't runnable, so you'll have to massage this a bit to work for you. If you make sure to await everything, you're good to go (and generally avoid mixing .then and async/await--the latter seems easier for this use case).
* (for all practical intents and purposes)
There is a few places where you are not handling promises returned in you code. nextNode.then if your forEach loop is just "called", next line of the code will not wait for it to complete, forEach loop will complete execution before then callbacks are called.
I changed you code a bit, but I have no way to check if it works correctly due to i would need to populate dummy data for callMsGraph but if you encounter any - tell me and I'll modify the answer
shelfdb.data = async (accessToken) => {
const token = accessToken;
const endpoint = 'https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/01GNYB5KPQ57RHLPZCJFE2QMVKT5U3NYY3/children'
const images = [];
async function start(node, array) {
if (node.value.length <= 0) return array; // or === 0 or whatever
for (const child of node.value) {
const end = `https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/${child.id}/children`;
const nextNode = await callMsGraph(token, end);
if (nextNode.value.length > 0) {
if ('image' in nextNode.value[0]) {
const mapped = nextNode.value.map(imgChild => {
return {
'name': imgChild.name,
'job': imgChild.parentReference.path.split("/")[6],
'path': imgChild.webUrl,
'id': imgChild.id
}
});
array.push(...mapped);
}
// if no 'image' or value, go into child
await start(nextNode, array);
}
}
return array;
}
var res = await callMsGraph(token, endpoint);
var output = await start(res, []);
console.log(output);
return output;
}
Also, please, feel free to add a try{} catch{} blocks in any place you need them, I skipped them
I am using Node JS and Express JS, here is my controller code:
const UserComment = require("../model/UserComment");
router.post("/get/comments", async (request, response) =>{
try{
let currentUserID = request.body.userID;
let myUserComment = await UserComment.find({userID: currentUserID});
let friendsCommentsArray = [ ...myUserComment];
let friendsComments = await axios.post(`http://localhost:5000/router/accounts/account/following/list`, {userID: currentUserID})
.then((resp) => {
resp.data.message.map((parentArrayOfArray) =>{
parentArrayOfArray.map((friendID) =>{
let friendsCommentsToLookUp = UserComment.find({userID: friendID})
friendsCommentsToLookUp.then((commentsArray) =>{
commentsArray.map((comment) =>{
if(String(comment.userID) === friendID){
friendsCommentsArray.push(comment);
}else{
console.log("no")
}
})
});
});
});
}).catch((err) =>{
console.log("err: ", err);
throw err;
});
return response.status(200).json({message: friendsPostsArray});
}catch(err){
return response.status(400).json({message: `${err}`});
}
});
The friendsCommentsArray, when I console.log it I can see the data, but when I return it, it’s empty. What is the problem, why is it empty, even though i'm pushing every comment iterated over to the friendsCommentsArray.
However, the returned friendsCommentsArray is empty. how to solve this issue ?
Thanks.
To make await Promise.all() work you need to return the promise
return axios.get(`http://localhost:5000/comments/by/post/${post._id}`)
Generally when you use await, you don't need to use .then(). Your problem is that your inner .map() is using friendsCommentsToLookUp.then(), but nothing is waiting for these promises to resolve before you move on in your code. One might think that you can await the friendsCommentsToLookUp promise, but this won't work, as the calls to the map callback are not awaited.
Removing the .then()'s makes this easier to work with:
const resp = await axios.post(`http://localhost:5000/router/accounts/account/following/list`, {userID: currentUserID});
const message = resp.data.message;
for(const parentArrayOfArray of message) {
for(const friendID of parentArrayOfArray) {
const commentsArray = await UserComment.find({userID: friendID});
for(const comment of commentsArray) {
if(String(comment.userID) === friendID){
friendsCommentsArray.push(comment);
}
}
}
}
Above the for..of allows us to pause moving to the next iteration of the for loop until the Promises within the current iteration of the for loop have resolved. ie: it's sequential (note: if you tried to do this with .forEach() or .map(), your code would proceed directly to the portion after the loop before your Promises have resolved). Although, what you're after doesn't need to be sequential. We can create an array of Promises that we pass to Promise.all() which we can wait to resolve in parallel. Below I've shown a different approach of using .flatMap() to create an array of Promises that we can await in parallel with Promise.all():
const resp = await axios.post(`http://localhost:5000/router/accounts/account/following/list`, {userID: currentUserID});
const message = resp.data.message;
const promises = message.flatMap(parentArr => parentArr.map(async friendID => {
const commentsArray = await UserComment.find({userID: friendID});
return commentsArray.filter(comment => String(comment.userID) === friendID);
}));
const nestedComments = await Promise.all(promises);
const friendsCommentsArray = [...myUserComment, ...nestedComments.flat()];
instead of push try concatenation array and let me know if its work.
friendsCommentsArray = [...friendsCommentsArray , {...comment}];
// insted of
friendsCommentsArray.push(comment);
also try to use forEach instead of map() while you don't want to return a new array from your map statement.
The map method is very similar to the forEach method—it allows you to execute a function for each element of an array. But the difference is that the map method creates a new array using the return values of this function. map creates a new array by applying the callback function on each element of the source array. Since map doesn't change the source array, we can say that it’s an immutable method.
SCENARIO
So I am trying to do multiple API requests with different URLs, add all the responses to an array, then return the array and use it. My code:
const getData = (dataURLs) => {
let returnData = [];
for (let i = 0; i < dataURLs.length; i++) {
getFetch(dataURLs[i]).then((response) => {
returnData.push(response);
return response;
});
}
console.log(returnData[0]);
return returnData;
};
So this is the function that makes the requests, here is what the getFetch function is:
const getFetch = async (url) => {
return fetch(url)
.then((response) => {
if (!response.ok) {
// get error message from body or default to response status
const error = (response && response.message) || response.status;
return error;
}
return response.json();
})
.catch((error) => {
return error;
});
};
This just makes the request and returns the JSON which is what I want, and this function works as I use it in other places.
PROBLEM
My issue is, when i make the request using the getData function, it will return a blank array '[]', however when I click on this array in inspect element, it displays this.
[] ->
0: {nodes: Array(5), edges: Array(5), self: Array(1)}
length: 1
If I try to access anything in this array in js it just doesn't let me. But if I look at it in Inspect Element, it will be a blank array that I can expand and it displays the requested data inside it
Just wondering if anyone knew a fix to this?
Thanks :)
The issue you're running into here is that the function returns the array before the promises have resolved (and put the data that you want into the arrays). You will need to wait for the promises first. There are a few ways you can do this, but one way is to put the promises into the array and use a Promise.all() to get all of the values when they are available.
const getData = (dataURLs) => {
let returnPromises = [];
for (let i = 0; i < dataURLs.length; i++) {
returnPromises.push(getFetch(dataURLs[i]));
}
return Promise.all(returnPromises);
};
From here on you will continue to use this function's result as a promise.
getData([...]).then(([result1, result2, ...resultN]) => {...})
I'm trying to build a method which reads from firestore an array of elements (object):
I have a service which retrieves the data from firestore, first it gets an array of document references
var data = snapshot.get('elements');
and then it gets all the objects:
getElements(){
return new Promise(res =>{
this.AngularAuth.currentUser
.then( user => {
this.useruid = user.uid;
this.db.firestore.doc(`/users/${this.useruid}`).get().then(snapshot =>{
if(snapshot.exists){
var data = snapshot.get('elements'); //This gets the array of elements
data.forEach(element => {
this.db.firestore.doc(element).get().then(object =>{
if(object.exists){
var elem = object.data() as object;
this.array.push(elem);//I kind of push in the array instances of object
}
else{
console.log("Error. Doc doesn't exist")
}
}).catch(err =>{
console.log(err);
})
});
res(this.array);
}
else{
console.log("Error. Doc doesn't exist")
}
}).catch(function(error) {
// An error happened.
})
})
.catch(function(error) {
// An error happened.
})
});
}
Then in a component I have an async method which calls the service, and tries to push into another array all the names from each object in the first array:
async retrieveArray(){
this.array = await this.service.getElements();
this.array.forEach(element => {
this.names.push(element.name);
});
console.log(this.array);
console.log(this.names);
}
However when I look to the console, the first array (array) gives me indeed an array of objects, but the other array (names) is empty.
I used the method get to retrieve the data because I don't want to listen to it, I might need the value just once.
Personally I find the async/await syntax infinitely more elegant and easier to deal with than a good old .then() callback hell :
async getElements() {
let user;
try{
user = await this.AngularAuth.currentUser();
} catch(err) {
console.log(err);
return;
}
this.useruid = user.uid;
const snapshot = await this.db.firestore.doc(`/users/${this.useruid}`).get();
if (!snapshot.exists) {
console.log("Error. Doc doesn't exist")
return
}
const data = snapshot.get('elements'); //This gets the array of elements
let toReturn = [];
for(let element of data){ // can also use 'await Promise.all()' here instead of for...of
const object = await this.db.firestore.doc(element).get();
toReturn.push(elem);
}
return toReturn;
}
async retrieveArray(){
this.array = await this.service.getElements();
this.names = this.array.map( element => element.name ) // Also use .map() here
console.log(this.array);
console.log(this.names);
}
If you use for...of, all calls will be made one after the other, in order. If you use await Promise.all(), all calls will be made and awaited simultaneously, which is faster but recommended only if you have a small number of calls to make (otherwise this could overload the server you're calling, or even be considered as a DDoS attack.)
I think the issue is in this part of your code:
if(snapshot.exists){
var data = snapshot.get('elements'); //This gets the array of elements
data.forEach(element => {
this.db.firestore.doc(element).get().then(object =>{
if(object.exists){
var elem = object.data() as object;
this.array.push(elem);//I kind of push in the array instances of object
}
else{
console.log("Error. Doc doesn't exist")
}
}).catch(err =>{
console.log(err);
})
});
res(this.nombres);
}
You're looping through the elements and fetching the object from firebase for each one. Each time is an async call, but you're not waiting for each of these calls to finish before calling res(this.nombres).
As for why the console.log(this.array) shows a populated array is that the console can be misleading. It provides the data in a kind of 'live' way (it's a reference to the array), and sometimes by the time the data arrives on the console, it's different to what the data looked like when console.log was called.
To make sure you see the data precisely as it was when console.log was called, try this:
console.log(JSON.parse(JSON.stringify(this.array));
As for the issue with your code, you need to wait for all the elements to have been fetched before you call the resolve function of your promise. Because you don't necessarily know the order in which the responses will come back, one option is to simply have a counter of how many results are remaining (you know how many you are expecting), and once the last response has been received, call the resolve function. This is how I would do it, but obviously I can't test it so it might not work:
if(snapshot.exists){
var data = snapshot.get('elements'); //This gets the array of elements
// *** we remember the number of elements we're fetching ***
let count = data.length;
data.forEach(element => {
this.db.firestore.doc(element).get().then(object =>{
// *** decrement count ***
count--;
if(object.exists){
var elem = object.data() as object;
this.array.push(elem);//I kind of push in the array instances of object
// *** If count has reached zero, now it's time to call the response function
if (count === 0) {
res(this.nombres);
}
}
else{
console.log("Error. Doc doesn't exist")
}
}).catch(err =>{
console.log(err);
})
});
// *** remove this line because it's calling the resolve function before nombres is populated
//res(this.nombres);
}
You might want to add behaviour for when the result of snapshot.get('elements') is empty, but hopefully with this you'll be on your way to a solution.
** EDIT **
I'm keeping this up just because the console.log issue might well be useful for you to know about, but I highly recommend the async/await approach suggested by Jeremy. I agree that's it's much more readable and elegant
I receive an object bigListFromClient that includes an arbitrary number of objects each of which may have an arbitrary number of children. Every object needs to be entered into my database, but the DB needs to assign each of them a unique ID and child objects need to have the unique ID of their parents attached to them before they are sent off to the DB.
I want to create some sort of Promise or other calling structure that would call itself asynchronously until it reached the last object in bigListFromClient but I'm having trouble figuring out how to write it.
for(let i = 0; i < bigListFromClient.length; i++){
makeDbCallAsPromise(bigListFromClient[i].queryString, console.log); //I'm not just accepting anything from a user here, but how I get my queryString is kind of out of scope for this question
for(let j = 0; j < bigListFromClient[i].children.length; j++){
//the line below obviously doesn't work, I'm trying to figure out how to do this with something other than a for loop
makeDbCallAsPromise(bigListFromClient[i].children[j].queryString + [the uniqueID from the DB to insert this correctly as a child], console.log);
}
}
//this promise works great
makeDbCallAsPromise = function(queryString){
return new Promise((resolve, reject) => {
connection = mysql.createConnection(connectionCredentials);
connection.connect();
query = queryString;
connection.query(query, function (err, rows, fields) {
if (!err) {
resolve(rows);
} else {
console.log('Error while performing Query.');
console.log(err.code);
console.log(err.message);
reject(err);
}
});
connection.end();
})
};
My attempts at solving this on my own are so embarrassingly bad that even describing them to you would be awful.
While I could defer all the calls to creating children until the parents have been created in the DB, I wonder if the approach I've described is possible.
There are essentially two ways to do this. One is making the database calls sequential and the other one is making the calls parallel.
Javascript has a built-in function for parallel called Promise.all, you pass it an array of Promise instances and it returns a Promise instance containing the array.
In your case your code would look like this:
const result = Promise.all(
bigListFromClient.map(item =>
makeDbCallAsPromise(item.queryString).then(result =>
Promise.all(
item.children.map(item =>
makeDbCallAsPromise(item.queryString + [result.someId])
)
)
])
})
result will now contain a Promise that resolves to an array of arrays. These arrays contain the result of intserting children.
Using a more modern approach (with async await), sequential and with all results in a flat array:
const result = await bigListFromClient.reduce(
async (previous, item) => {
const previousResults = await previous
const result = await makeDbCallAsPromise(item.queryString)
const childResults = await item.children.reduce(
async (result, item) =>
[...(await result), await makeDbCallAsPromise(item.queryString + [result.someId])],
[]
)
return [...previousResults, result, ...childResults)
]),
[]
})
Depending on what you want to achieve and how you want to structure your code you can pick and choose from the different approaches.
For this sort of operation, try looking into bulk inserting. If you are intent on performing a single DB query/transaction per iteration, loop recursively over each parent and/or execute the same procedure for each child.
const dbCall = async (elm) => {
elm.id = Math.random().toString(36).substring(7)
if (elm.children) {
await Promise.all(elm.children.map(child => {
child.parentId = elm.id
return dbCall(child)
}))
}
return elm
}
const elms = [
{
queryString: '',
children: [
{
queryString: ''
}
]
}
]
Promise.all(elms.map(dbCall)).then(elm => /* ... */)