I have a fuction which includes the following:
const newThreads = newItems.reduce( (acc, item) => {
request(item.href, function(error2, response2, html2){
if(!error2) {
const thread = cheerio.load(html2)
const today = thread('div#ignmsgbttns1').parent().parent().find("b:contains('Today')")
if(today.text()) {
acc.push(item)
}
}
})
return acc
}, [])
console.log(newThreads)
Of course the log returns an empty array, because of the async stuff
(request) executed in the reduce loop.
So what I would like to do is:
const newThreads = await newItems.reduce( etc...
And wait for the requests in the reduce loop to be done.
But I don't get my head around how to do it properly.
So I know I have to use async, await or promises, but don't know how to do it.
I think the reduce callback also has to be async but absolutely not sure on this point.
The request method comes from the npm request package , they also provide some packages to use promises, but to be honest, I don't know how to apply it with reduce.
I'm pretty sure there is already a similar question somewhere but couldn't find it.
Any help would be greatly appreciated.
ps: for those wondering what cheerio is, here the link.
Final code after applying answer
I had to use the async-request package
const newThreads = newItems.reduce(async (acc, item) => {
const current = await acc;
const html2 = await requestAsync(item.href);
const thread = cheerio.load(html2.body);
const today = thread('div#ignmsgbttns1')
.parent()
.parent()
.find("b:contains('Today')");
if (today.text()) current.push(item);
return current;
}, []);
newThreads.then((res) => {
//..doing stuff with res
})
In order to make this work you'll need the Promise returning version.
const newThreads = newItems.reduce(async (acc, item) => { // note async
const current = await acc; // unwrap the previous Promise
try {
const html2 = await request(item.href); // unwrap request Promise
const thread = cheerio.load(html2);
const today = thread('div#ignmsgbttns1')
.parent()
.parent()
.find("b:contains('Today')");
if (today.text()) current.push(item);
} catch (error2) {
// do whatever
}
return current;
}, []);
The newThreads variable will be a Promise of an array of items that passed the conditional check.
Related
I have a start(node, array) function that should perform a DFS by traversing an object tree via recursive calls to an API through callMsGraph(token, end) until image properties are found at the end of the tree, at which point they are pushed to array. The function seems like it works, but I can't get the output unless I wrap it in a 2 second setTimeout which indicates the recursion is not being waited on to complete. I would want to play around with async/await more, but it's not at the top-level.
I'm not sure if the nextNode.then is doing anything or maybe callMsGraph() needs to be awaited on differently to how I know. A solution would be much appreciated.
shelfdb.data = async (accessToken) => {
const token = accessToken;
const endpoint = 'https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/01GNYB5KPQ57RHLPZCJFE2QMVKT5U3NYY3/children'
function start(node, array) {
if(node.value.length > 0) {
node.value.forEach(function(child) {
var end = 'https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/' + child.id + '/children';
var nextNode = callMsGraph(token, end);
nextNode.then(function(currResult) {
if (currResult.value.length > 0) {
if ('image' in currResult.value[0]) {
currResult.value.forEach(function(imgChild) {
let img = {
'name': imgChild.name,
'job': imgChild.parentReference.path.split("/")[6],
'path': imgChild.webUrl,
'id': imgChild.id
}
array.push(img);
})
// complete storing images at tail object, go one level up after loop
return;
}
// if no 'image' or value, go into child
start(currResult, array);
}
}).catch(function(e) {
console.error(e.message);
})
})
}
return array;
}
var res = await callMsGraph(token, endpoint); // start recursion
var output = start(res, []);
console.log(output); // only displays value if wrapped in setTimeout
return output; // empty []
}
Each query to the API via callMsGraph(), returns an object like this, where subsequent queries are made with the id of each object/folder (as new endpoint) in value until an object with image property is found. The MS Graph API requires that folders are expanded at each level to access their children.
{
id: '01GNYB5KPQ57RHLPZCJFE2QMVKT5U3NYY3'
value: [
{
id: '01GNYB5KJMH5T4GXADUVFZRSITWZWNQROS',
name: 'Folder1',
},
{
id: '01GNYB5KMJKILOFDZ6PZBZYMXY4BGOI463',
name: 'Folder2',
}
]
}
This is the callMsGraph() helper:
function callMsGraph(accessToken, graphEndpoint) {
const headers = new Headers();
const bearer = `Bearer ${accessToken}`;
headers.append("Authorization", bearer);
const options = {
method: "GET",
headers: headers
};
return fetch(graphEndpoint, options)
.then(response => response.json())
.catch(error => {
console.log(error);
throw error;
});
}
The rule with promises is that once you opt into one (more likely, are forced into it by a library), all code that needs to block for a result anywhere after it also has to await. You can't "go back" to sync and if even a single piece of the promise chain between where the promise starts and where you want its result isn't awaited, the result will be unreachable*.
Taking a snippet of the code:
function start(node, array) { // not async!
// ..
node.value.forEach(function(child) { // doesn't await!
// ..
nextNode.then(function(currResult) {
// this promise is not hooked up to anything!
start(...) // recurse without await!
There's no await in front of then, start doesn't return a promise and isn't awaited recursively, and forEach has no way to await its callback's asynchronous results, so each promise in the nextNode.then chain is orphaned into the void forever*.
The solution is a structure like this:
async function start(node, array) {
// ..
for (const child of node.value) {
// ..
const currResult = await callMsGraph(token, end);
// ..
await start(...);
array.push(currResult);
}
// returns a promise implicitly
}
// ..
await start(...);
// `array` is populated here
Or Promise.all, which runs in parallel and returns an array (which could replace the parameter array):
function start(node, array) {
return Promise.all(node.value.map(async child => {
const currResult = await callMsGraph(token, end);
// ..
await start(...);
return currResult;
}));
}
I'd be happy to provide a minimal, runnable example, but the code you've provided isn't runnable, so you'll have to massage this a bit to work for you. If you make sure to await everything, you're good to go (and generally avoid mixing .then and async/await--the latter seems easier for this use case).
* (for all practical intents and purposes)
There is a few places where you are not handling promises returned in you code. nextNode.then if your forEach loop is just "called", next line of the code will not wait for it to complete, forEach loop will complete execution before then callbacks are called.
I changed you code a bit, but I have no way to check if it works correctly due to i would need to populate dummy data for callMsGraph but if you encounter any - tell me and I'll modify the answer
shelfdb.data = async (accessToken) => {
const token = accessToken;
const endpoint = 'https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/01GNYB5KPQ57RHLPZCJFE2QMVKT5U3NYY3/children'
const images = [];
async function start(node, array) {
if (node.value.length <= 0) return array; // or === 0 or whatever
for (const child of node.value) {
const end = `https://graph.microsoft.com/v1.0/sites/webgroup.sharepoint.com,23e7ef7a-a529-4dde-81ba-67afb4f44401,0fa8e0f7-1c76-4ad0-9b6e-a485f9bfd63c/drive/items/${child.id}/children`;
const nextNode = await callMsGraph(token, end);
if (nextNode.value.length > 0) {
if ('image' in nextNode.value[0]) {
const mapped = nextNode.value.map(imgChild => {
return {
'name': imgChild.name,
'job': imgChild.parentReference.path.split("/")[6],
'path': imgChild.webUrl,
'id': imgChild.id
}
});
array.push(...mapped);
}
// if no 'image' or value, go into child
await start(nextNode, array);
}
}
return array;
}
var res = await callMsGraph(token, endpoint);
var output = await start(res, []);
console.log(output);
return output;
}
Also, please, feel free to add a try{} catch{} blocks in any place you need them, I skipped them
I am using Node JS and Express JS, here is my controller code:
const UserComment = require("../model/UserComment");
router.post("/get/comments", async (request, response) =>{
try{
let currentUserID = request.body.userID;
let myUserComment = await UserComment.find({userID: currentUserID});
let friendsCommentsArray = [ ...myUserComment];
let friendsComments = await axios.post(`http://localhost:5000/router/accounts/account/following/list`, {userID: currentUserID})
.then((resp) => {
resp.data.message.map((parentArrayOfArray) =>{
parentArrayOfArray.map((friendID) =>{
let friendsCommentsToLookUp = UserComment.find({userID: friendID})
friendsCommentsToLookUp.then((commentsArray) =>{
commentsArray.map((comment) =>{
if(String(comment.userID) === friendID){
friendsCommentsArray.push(comment);
}else{
console.log("no")
}
})
});
});
});
}).catch((err) =>{
console.log("err: ", err);
throw err;
});
return response.status(200).json({message: friendsPostsArray});
}catch(err){
return response.status(400).json({message: `${err}`});
}
});
The friendsCommentsArray, when I console.log it I can see the data, but when I return it, it’s empty. What is the problem, why is it empty, even though i'm pushing every comment iterated over to the friendsCommentsArray.
However, the returned friendsCommentsArray is empty. how to solve this issue ?
Thanks.
To make await Promise.all() work you need to return the promise
return axios.get(`http://localhost:5000/comments/by/post/${post._id}`)
Generally when you use await, you don't need to use .then(). Your problem is that your inner .map() is using friendsCommentsToLookUp.then(), but nothing is waiting for these promises to resolve before you move on in your code. One might think that you can await the friendsCommentsToLookUp promise, but this won't work, as the calls to the map callback are not awaited.
Removing the .then()'s makes this easier to work with:
const resp = await axios.post(`http://localhost:5000/router/accounts/account/following/list`, {userID: currentUserID});
const message = resp.data.message;
for(const parentArrayOfArray of message) {
for(const friendID of parentArrayOfArray) {
const commentsArray = await UserComment.find({userID: friendID});
for(const comment of commentsArray) {
if(String(comment.userID) === friendID){
friendsCommentsArray.push(comment);
}
}
}
}
Above the for..of allows us to pause moving to the next iteration of the for loop until the Promises within the current iteration of the for loop have resolved. ie: it's sequential (note: if you tried to do this with .forEach() or .map(), your code would proceed directly to the portion after the loop before your Promises have resolved). Although, what you're after doesn't need to be sequential. We can create an array of Promises that we pass to Promise.all() which we can wait to resolve in parallel. Below I've shown a different approach of using .flatMap() to create an array of Promises that we can await in parallel with Promise.all():
const resp = await axios.post(`http://localhost:5000/router/accounts/account/following/list`, {userID: currentUserID});
const message = resp.data.message;
const promises = message.flatMap(parentArr => parentArr.map(async friendID => {
const commentsArray = await UserComment.find({userID: friendID});
return commentsArray.filter(comment => String(comment.userID) === friendID);
}));
const nestedComments = await Promise.all(promises);
const friendsCommentsArray = [...myUserComment, ...nestedComments.flat()];
instead of push try concatenation array and let me know if its work.
friendsCommentsArray = [...friendsCommentsArray , {...comment}];
// insted of
friendsCommentsArray.push(comment);
also try to use forEach instead of map() while you don't want to return a new array from your map statement.
The map method is very similar to the forEach method—it allows you to execute a function for each element of an array. But the difference is that the map method creates a new array using the return values of this function. map creates a new array by applying the callback function on each element of the source array. Since map doesn't change the source array, we can say that it’s an immutable method.
I have this function that returns a list, but the list stays empty even after I add items to the list in the forEach. So I tried using Promises, but I need a little help. Right now the result is "undefined". How do I return the result list after the forEach loop is done?
async function return_list() {
var result = [];
var list_of_skus = [...];
var promise = new Promise(() => {
list_of_skus.forEach((number) => {
api.get(...)
//api request
result.push(api_data)
});
});
promise.then({
return result;
})
}
edit:
i changed the code a bit:
In the forEach loop im using an api request to get some data, and then add an item the result list
Here you can find a bit more information about promises: Promises MDN
As long as you do nothing async (like network requests etc) you probably don't need promises in your forEach.
This should work:
function return_list() {
const result = [];
const list_of_skus = [...];
list_of_skus.forEach((number) => {
result.push(1)
});
return result;
}
If you need promises, this would work:
async function return_list() {
const list_of_skus = [...];
const promises = list_of_skus.map(async (number) => {
return await somethingAsync(number)
});
return await Promise.all(promises)
}
As you transform every sku list item, you can use a map to transform all skus into a promise which does something async.
Using Promise.all() it is possible to execute the promises in parallel.
I'm trying to parse a specification website from saved HTML on my computer. I can post the file upon request.
I'm burnt out trying to figure out why it won't run synchronously. The comments should log the CCCC's first, then BBBB's, then finally one AAAA.
The code I'm running will not wait at the first hurdle (it prints AAAA... first). Am I using request-promise incorrectly? What is going on?
Is this due to the .each() method of cheerio (I'm assuming it's synchronous)?
const rp = require('request-promise');
const fs = require('fs');
const cheerio = require('cheerio');
async function parseAutodeskSpec(contentsHtmlFile) {
const topics = [];
const contentsPage = cheerio.load(fs.readFileSync(contentsHtmlFile).toString());
const contentsSelector = '.content_htmlbody table td div div#divtreed0e338374 nobr .toc_entry a.treeitem';
contentsPage(contentsSelector).each(async (idx, topicsAnchor) => {
const topicsHtml = await rp(topicsAnchor.attribs['href']);
console.log("topicsHtml.length: ", topicsHtml.length);
});
console.log("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA");
return topics;
}
Try it this way:
let hrefs = contentsPage(contentsSelector).map((idx, topicsAnchor) => {
return topicsAnchor.attribs['href']
}).get()
let topicsHtml
for(href of hrefs){
topicsHtml = await rp(href);
console.log("topicsHtml.length: ", topicsHtml.length);
}
Now the await is outside of map or each which doesn't quite work the way you think.
As #lumio stated in his comment, I also think that this is because of the each function being synchrone.
You should rather use the map method, and use the Promise.all() on the result to wait enough time:
const obj = contentsPage(contentsSelector).map(async (idx, topicsAnchor) => {
const topicsHtml = await rp(topicsAnchor.attribs['href']);
console.log("topicsHtml.length: ", topicsHtml.length);
const topicsFromPage = await parseAutodeskTopics(topicsHtml)
console.log("topicsFromPage.length: ", topicsFromPage.length);
topics.concat(topicsFromPage);
})
const filtered = Object.keys(obj).filter(key => !isNaN(key)).map(key => obj[key])
await Promise.all(filtered)
console.log("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA");
Based on the other answers here I came to a rather elegant conclusion. Note the avoidance of async/await in the .map() callback, as cheerio's callbacks (and from what I've learned about async/await, generally all callbacks) seem not to honour the synchronous nature of await well:
async function parseAutodeskSpec(contentsHtmlFile) {
const contentsPage = cheerio.load(fs.readFileSync(contentsHtmlFile).toString());
const contentsSelector = '.content_htmlbody table td div div#divtreed0e338374 nobr .toc_entry a.treeitem';
const contentsReqs = contentsPage(contentsSelector)
.map((idx, elem) => rp(contentsPage(elem).attr('href')))
.toArray();
const topicsReqs = await Promise.all(contentsReqs)
.map(req => parseAutodeskTopics(req));
return await Promise.all(topicsReqs);
}
By default the Promise.All([]) function returns a number based index array that contains the results of each promise.
var promises = [];
promises.push(myFuncAsync1()); //returns 1
promises.push(myFuncAsync1()); //returns 2
Promise.all(promises).then((results)=>{
//results = [0,1]
}
What is the best vanilla way to return a named index of results with Promise.all()?
I tried with a Map, but it returns results in an array this way:
[key1, value1, key2, value2]
UPDATE:
My questions seems unclear, here is why i don't like ordered based index:
it's crappy to maintain: if you add a promise in your code you may have to rewrite the whole results function because the index may have change.
it's awful to read: results[42] (can be fixed with jib's answer below)
Not really usable in a dynamic context:
var promises = [];
if(...)
promises.push(...);
else{
[...].forEach(... => {
if(...)
promises.push(...);
else
[...].forEach(... => {
promises.push(...);
});
});
}
Promise.all(promises).then((resultsArr)=>{
/*Here i am basically fucked without clear named results
that dont rely on promises' ordering in the array */
});
ES6 supports destructuring, so if you just want to name the results you can write:
var myFuncAsync1 = () => Promise.resolve(1);
var myFuncAsync2 = () => Promise.resolve(2);
Promise.all([myFuncAsync1(), myFuncAsync2()])
.then(([result1, result2]) => console.log(result1 +" and "+ result2)) //1 and 2
.catch(e => console.error(e));
Works in Firefox and Chrome now.
Is this the kind of thing?
var promises = [];
promises.push(myFuncAsync1().then(r => ({name : "func1", result : r})));
promises.push(myFuncAsync1().then(r => ({name : "func2", result : r})));
Promise.all(promises).then(results => {
var lookup = results.reduce((prev, curr) => {
prev[curr.name] = curr.result;
return prev;
}, {});
var firstResult = lookup["func1"];
var secondResult = lookup["func2"];
}
If you don't want to modify the format of result objects, here is a helper function that allows assigning a name to each entry to access it later.
const allNamed = (nameToPromise) => {
const entries = Object.entries(nameToPromise);
return Promise.all(entries.map(e => e[1]))
.then(results => {
const nameToResult = {};
for (let i = 0; i < results.length; ++i) {
const name = entries[i][0];
nameToResult[name] = results[i];
}
return nameToResult;
});
};
Usage:
var lookup = await allNamed({
rootStatus: fetch('https://stackoverflow.com/').then(rs => rs.status),
badRouteStatus: fetch('https://stackoverflow.com/badRoute').then(rs => rs.status),
});
var firstResult = lookup.rootStatus; // = 200
var secondResult = lookup.badRouteStatus; // = 404
If you are using typescript you can even specify relationship between input keys and results using keyof construct:
type ThenArg<T> = T extends PromiseLike<infer U> ? U : T;
export const allNamed = <
T extends Record<string, Promise<any>>,
TResolved extends {[P in keyof T]: ThenArg<T[P]>}
>(nameToPromise: T): Promise<TResolved> => {
const entries = Object.entries(nameToPromise);
return Promise.all(entries.map(e => e[1]))
.then(results => {
const nameToResult: TResolved = <any>{};
for (let i = 0; i < results.length; ++i) {
const name: keyof T = entries[i][0];
nameToResult[name] = results[i];
}
return nameToResult;
});
};
A great solution for this is to use async await. Not exactly ES6 like you asked, but ES8! But since Babel supports it fully, here we go:
You can avoid using only the array index by using async/await as follows.
This async function allows you to literally halt your code inside of it by allowing you to use the await keyword inside of the function, placing it before a promise. As as an async function encounters await on a promise that hasn't yet been resolved, the function immediately returns a pending promise. This returned promise resolves as soon as the function actually finishes later on. The function will only resume when the previously awaited promise is resolved, during which it will resolve the entire await Promise statement to the return value of that Promise, allowing you to put it inside of a variable. This effectively allows you to halt your code without blocking the thread. It's a great way to handle asynchronous stuff in JavaScript in general, because it makes your code more chronological and therefore easier to reason about:
async function resolvePromiseObject(promiseObject) {
await Promise.all(Object.values(promiseObject));
const ret = {};
for ([key, value] of Object.entries(promiseObject)) {
// All these resolve instantly due to the previous await
ret[key] = await value;
};
return ret;
}
As with anything above ES5: Please make sure that Babel is configured correctly so that users on older browsers can run your code without issue. You can make async await work flawlessly on even IE11, as long as your babel configuration is right.
in regards to #kragovip's answer, the reason you want to avoid that is shown here:
https://medium.com/front-end-weekly/async-await-is-not-about-making-asynchronous-code-synchronous-ba5937a0c11e
"...it’s really easy to get used to await all of your network and I/O calls.
However, you should be careful when using it multiple times in a row as the await keyword stops execution of all the code after it. (Exactly as it would be in synchronous code)"
Bad Example (DONT FOLLOW)
async function processData() {
const data1 = await downloadFromService1();
const data2 = await downloadFromService2();
const data3 = await downloadFromService3();
...
}
"There is also absolutely no need to wait for the completion of first request as none of other requests depend on its result.
We would like to have requests sent in parallel and wait for all of them to finish simultaneously. This is where the power of asynchronous event-driven programming lies.
To fix this we can use Promise.all() method. We save Promises from async function calls to variables, combine them to an array and await them all at once."
Instead
async function processData() {
const promise1 = downloadFromService1();
const promise2 = downloadFromService2();
const promise3 = downloadFromService3();
const allResults = await Promise.all([promise1, promise2, promise3]);