Develop Reference

Create an object or associative array with elements of an existing array and the result of a callout for each element

This is in the context of a node express route. I receive a get request with a query param that is a list of IDs. Now I need to make a call-out for each ID and store the result of the callout in an array or object. Each element of the first array (containing the IDs) need to be mapped to its corresponding result from the call-out. I don't have a way to modify the endpoint that I'm hitting from this route so I have to make single calls for each ID. I've done some research and so far I have a mixture of code and sudo code like this:
const ids = req.query.ids;
const idMembers = Promise.all(ids.map(async id => {
// here I'd like to create a single object or associative array
[ id: await callout(id); ]
}));
When all promises resolved I need the final result of idMembers to be like: (The response will be an object with nested arrays and objects I've just simplified it for this post but I need to grab that from the res.payload)
{
'211405': { name: 'name1', email: 'email1#test.co' },
'441120': { name: 'name2', email: 'email2#test.co' },
'105020': { name: 'name3', email: 'email4#test.co' }
}
Oh and of course I need to handle the callout and the promise failures and that's when my lack of experience with javascript becomes a real issue. I appreciate your help in advance!!
Some extra thought I'm having is that I'd have to map the results of the resolved promises to their id and then in a separate iteration I can then create my final array/object that maps the ids to the actual payloads that contain the object. This is still not answering any of my questions though. I'm just trying to provide as much information as I've gathered and thought of.

Promise.all returns an array of results (one item per each promise).
Having this temporary structure it is possible to build the needed object.
const arrayOfMembers = Promise.all(ids.map(async id => {
// ...
return { id, value: await callout(id) } // short syntax for { id: id, value: ... } (see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer)
}));
// arrayOfMembers = [
// { id: 211405, value: { name: 'name1', email: 'email1#test.co' } },
// ...
// ]
In pure JS it can be done with for loop or .forEach() call to iterate:
const res = {};
arrayOfMembers.forEach(el => {
const { id, value } = el;
res[el] = value;
});
or by using a single reduce() call
const res = arrayOfMembers.reduce((accumulator, el) => {
const { id, value } = el;
return { ...accumulator, [id]: value };
}, {});
in both cases res will be:
// res = {
// '211405': { name: 'name1', email: 'email1#test.co' },
// ...
// }
P.S.
There is a handy library called lodash. It has tons of small methods for data manipulation.
For example, _.fromPairs() can build an object from [[key1, value1], [key2, value2]] pairs.
As you mentioned you have lodash, so I think the following should work:
const arrayOfKeyValuePairs = Promise.all(ids.map(async id => {
// ...
return [ id, await callout(id) ] // array here so it matches what fromPairs needs
}));
const res = _.fromPairs(arrayOfKeyValuePairs);

Possible race condition with cursor when using Promise.all

In the project that I am working on, built using nodejs & mongo, there is a function that takes in a query and returns set of data based on limit & offset provided to it. Along with this data the function returns a total count stating all the matched objects present in the database. Below is the function:
// options carry the limit & offset values
// mongoQuery carries a mongo matching query
function findMany(query, options, collectionId) {
const cursor = getCursorForCollection(collectionId).find(query, options);
return Promise.all([findManyQuery(cursor), countMany(cursor)]);
}
Now the problem with this is sometime when I give a large limit size I get an error saying:
Uncaught exception: TypeError: Cannot read property '_killCursor' of undefined
At first I thought I might have to increase the pool size in order to fix this issue but after digging around a little bit more I was able to find out that the above code is resulting in a race condition. When I changed the code to:
function findMany(query, options, collectionId) {
const cursor = getCursorForCollection(collectionId).find(query, options);
return findManyQuery(cursor).then((dataSet) => {
return countMany(cursor).then((count)=> {
return Promise.resolve([dataSet, count]);
});
);
}
Everything started working perfectly fine. Now, from what I understand with regard to Promise.all was that it takes an array of promises and resolves them one after the other. If the promises are executed one after the other how can the Promise.all code result in race condition and the chaining of the promises don't result in that.
I am not able to wrap my head around it. Why is this happening?

Since I have very little information to work with, I made an assumption of what you want to achieve and came up with the following using Promise.all() just to demonstrate how you should use Promise.all (which will resolve the array of promises passed to it in no particular order. For this reason, there must be no dependency in any Promise on the order of execution of the Promises. Read more about it here).
// A simple function to sumulate findManyQuery for demo purposes
function findManyQuery(cursors) {
return new Promise((resolve, reject) => {
// Do your checks and run your code (for example)
if (cursors) {
resolve({ dataset: cursors });
} else {
reject({ error: 'No cursor in findManyQuery function' });
}
});
}
// A simple function to sumulate countMany for demo purposes
function countMany(cursors) {
return new Promise((resolve, reject) => {
// Do your checks and run your code (for example)
if (cursors) {
resolve({ count: cursors.length });
} else {
reject({ error: 'No cursor in countMany' });
}
});
}
// A simple function to sumulate getCursorForCollection for demo purposes
function getCursorForCollection(collectionId) {
/*
Simulating the returned cursor using an array of objects
and the Array filter function
*/
return [{
id: 1,
language: 'Javascript',
collectionId: 99
}, {
id: 2,
language: 'Dart',
collectionId: 100
},
{
id: 3,
language: 'Go',
collectionId: 100
}, {
id: 4,
language: 'Swift',
collectionId: 99
}, {
id: 5,
language: 'Kotlin',
collectionId: 101
},
{
id: 6,
language: 'Python',
collectionId: 100
}].filter((row) => row.collectionId === collectionId)
}
function findMany(query = { id: 1 }, options = [], collectionId = 0) {
/*
First I create a function to simulate the assumed use of
query and options parameters just for demo purposes
*/
const filterFunction = function (collectionDocument) {
return collectionDocument.collectionId === query.id && options.indexOf(collectionDocument.language) !== -1;
};
/*
Since I am working with arrays, I replaced find function
with filter function just for demo purposes
*/
const cursors = getCursorForCollection(collectionId).filter(filterFunction);
/*
Using Promise.all([]). NOTE: You should pass the result of the
findManyQuery() to countMany() if you want to get the total
count of the resulting dataset
*/
return Promise.all([findManyQuery(cursors), countMany(cursors)]);
}
// Consuming the findMany function with test parameters
const query = { id: 100 };
const collectionId = 100;
const options = ['Javascript', 'Python', 'Go'];
findMany(query, options, collectionId).then(result => {
console.log(result); // Result would be [ { dataset: [ [Object], [Object] ] }, { count: 2 } ]
}).catch((error) => {
console.log(error);
});

There are ways to write this function in a "pure" way for scalability and testing.
So here's your concern:
In the project that I am working on, built using nodejs & mongo, there is a function that takes in a query and returns set of data based on limit & offset provided to it. Along with this data the function returns a total count stating all the matched objects present in the database.
Note: You'll need to take care of edge case.
const Model = require('path/to/model');
function findManyUsingPromise(model, query = {}, offset = 0, limit = 10) {
return new Promise((resolve, reject) => {
model.find(query, (error, data) => {
if(error) {
reject(error);
}
resolve({
data,
total: data.length || 0
});
}).skip(offset).limit(limit);
});
}
// Call function
findManyUsingPromise(Model, {}, 0, 40).then((result) => {
// Do something with result {data: [object array], total: value }
}).catch((err) => {
// Do something with the error
});

Child functions and async await

In one of my API endpoints I fetch a json resource (1) from the web and edit it to fit my needs. In the "lowest" or "deepest" part of the tree I'm trying to fetch another resource and add it to the final json object. I'm relatively new to async/await but am trying to move away from the "old" Promises since I see the advantage (or the gain) of using async/await.
The object from (1) looks like;
const json = {
date,
time,
trips: [{
name,
legs: [{
id
},
{
id
}
]
}]
};
Here's how I "reformat" and change the json object;
{
date,
time,
trips: json.trips.map(trip => formatTrip(trip))
};
function formatTrip(trip) {
return {
name,
legs: trip.legs.map(leg => formatLeg(leg))
};
};
async function formatLeg(leg) {
const data = await fetch();
return {
id,
data
};
};
The problem with this is that after I've "reformatted/edited" the original json to look how I want it (and ran through all format... functions) the legs objects are empty {}.
I figured this might be due to the async/await promises not finishing. I've also read that if a child-function uses async/await all the higher functions has to use async/await as well.
Why? How can I rewrite my code to work and look good? Thanks!
EDIT:
I updated my code according to Randy's answer. getLegStops(leg) is still undefined/empty.
function formatLeg(leg) {
return {
other,
stops: getLegStops(leg)
};
};
function getLegStops(leg) {
Promise.all(getLegStopRequests(leg)).then(([r1, r2]) => {
/* do stuff here */
return [ /* with data */ ];
});
};
function getLegStopRequests(leg) {
return [ url1, url2 ].map(async url => await axios.request({ url }));
};

Two things lead you to want to nest these Promises:
The old way of thinking about callbacks and then Promises
Believing the software process must match the data structure
It appears you only need to deal with the Promises once if I understand correctly.
Like this:
async function getLegs(){
return trip.legs.map(async leg => await fetch(...)); // produces an array of Promises
}
const legs = Promise.all(getLegs());
function formatLegs(legs) {
// do something with array of legs
};
function formatTrip(){
//format final output
}
EDIT: per your comment below, this snippet represents what I've demonstrated and what your goal should look like. Please review your code carefully.
const arr = [1, 2, 3, ];
const arrPromises = arr.map(async v => await new Promise((res) => res(v)));
const finalPromise = Promise.all(arrPromises);
console.log(finalPromise.then(console.log));

mapping a Promise function to an array does not persist results

I have an object with two arrays as properties:
I want to populate the arrays by running promises in series.
I fetch the result of the promises, and map a function to decorate all the items in my arrays.
While one array get populated and persist, the other get populated only while in the map function, but at the end the array is returned still empty.
Can you help to understand why?
I check the Promise is actually returned, and indeed in one case it works, not in the other.
this is my pseudo-code:
function formatMyObject( arrayOfIds ) {
// initialize the objet
var myObj = {
decorators = [],
nodes = []
...
}
// I map the Promise reconciliate() and push the results in the array:
return reconciliateNode(arrayOfIds)
.then( data => {
data.map( node => {
// I fetch results, and myObj.nodes
myObj.nodes.push( { ... })
})
})
return myObj
})
.then( myObj => {
// myObj.nodes is now a NON empty array
// I want to the same with myObj.decorators:
var data = myObj.nodes
// I think I am doing just as above:
data.map( node =>
decorateNode(node.source)
.then( decoration => {
decoration = decoration[node.source]
myObj['decorators'].push( {
...
} )
// I check: the array is NOT empty and getting populated:
console.log('myObj.decorators', myObj)
debugger
})
)
// instead now, just after the map() function, myObj.decorators is EMPTY!
console.log('myObj.decorators', myObj);
debugger
return myObj
)
... // other stuff
}

As in the second case the map callback returns a promise, that case is quite different from the first case.
In the second case, you would need to await all those promises, for which you can use Promise.all.
The code for the second part could look like this:
.then( myObj => {
return Promise.all(myObj.nodes.map(node => decorateNode(node.source)));
}).then(decorations => {
myObj.decorators = decorations.map(decoration => {
decoration = decoration[node.source];
return ({
...
});
})
console.log('myObj.decorators', myObj);
return myObj;
})

Holding a future value of a promise in a variable

I have a database that's calling for a list of recent messages. Each message is an object and is stored as an Array of these message objects in chatListNew.
Each message object has a property "from", which is the ID of the user who posted it. What I want to do, is loop through this Array and append the actual profile information of the "From" user into the object itself. That way when the Frontend receives the information, it has access to one specific message's sender's profile in that respective message's fromProfile property.
I thought about looping through each one and doing a Promise.All for every one, however, that's hugely expensive if only a handful over users posted hundreds of messages. It would make more sense to only run the mongoose query once for each user. So I invented a caching system.
However, I'm confused as to how to store the promise of a future value inside of an array element. I thought setting the "fromProfile" to the previously called promise would magically hold this promise until the value was resolved. So I used Promise.all to make sure all the promises were completed and then returned by results, but the promises I had stored in the arrays were not the values I had hoped for.
Here is my code:
//chatListNew = an array of objects, each object is a message that has a "from" property indicating the person-who-sent-the-message's user ID
let cacheProfilesPromises = []; // this will my basic array of the promises called in the upcoming foreach loop, made for Promise.all
let cacheProfilesKey = {}; // this will be a Key => Value pair, where the key is the message's "From" Id, and the value is the promise retrieving that profile
let cacheProfileIDs = []; // this another Key => Value pair, which basically stores to see if a certain "From" Id has already been called, so that we can not call another expensive mongoose query
chatListNew.forEach((message, index) => {
if(!cacheProfileIDs[message.from]) { // test to see if this user has already been iterated, if not
let thisSearch = User.findOne({_id : message.from}).select('name nickname phone avatar').exec().then(results => {return results}).catch(err => { console.log(err); return '???' ; }); // Profile retrieving promise
cacheProfilesKey[message.from] = thisSearch;
cacheProfilesPromises.push(thisSearch); // creating the Array of promises
cacheProfileIDs[message.from] = true;
}
chatListNew[index]["fromProfile"] = cacheProfilesKey[message.from]; // Attaching this promise (hoping it will become a value once promise is resolved) to the new property "fromProfile"
});
Promise.all(cacheProfilesPromises).then(_=>{ // Are all promises done?
console.log('Chat List New: ', chatListNew);
res.send(chatListNew);
});
And this is my console output:
Chat List New: [ { _id: '5b76337ceccfa2bdb7ff35b5',
updatedAt: '2018-08-18T19:50:53.105Z',
createdAt: '2018-08-18T19:50:53.105Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Hey everyone!',
fromProfile:
Promise { emitter: [EventEmitter], emitted: [Object], ended: true } },
{ _id: '5b78712deccfa2bdb7009d1d',
updatedAt: '2018-08-18T19:41:29.763Z',
createdAt: '2018-08-18T19:41:29.763Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Yo!',
fromProfile:
Promise { emitter: [EventEmitter], emitted: [Object], ended: true } } ]
Whereas I was hoping for something like:
Chat List New: [ { _id: '5b76337ceccfa2bdb7ff35b5',
updatedAt: '2018-08-18T19:50:53.105Z',
createdAt: '2018-08-18T19:50:53.105Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Hey everyone!',
fromProfile:
Promise {name: xxx, nickname: abc... etc} },
{ _id: '5b78712deccfa2bdb7009d1d',
updatedAt: '2018-08-18T19:41:29.763Z',
createdAt: '2018-08-18T19:41:29.763Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Yo!',
fromProfile:
{name: xxx, nickname: abc... etc} } ]
Thank you guys! Open to other ways of accomplishing this :)
Pete

When a Promise is assigned to a variable, that variable will always be a Promise, unless the variable is reassigned. You need to get the results of your Promises from your Promise.all call.
There's also no point to a .then that simply returns its argument, as with your .then(results => {return results}) - you can leave that off entirely, it doesn't do anything.
Construct the array of Promises, and also construct an array of from properties, such that each Promise's from corresponds to the item in the other array at the same index. That way, once the Promise.all completes, you can transform the array of resolved values into an object indexed by from, after which you can iterate over the chatListNew and assign the resolved value to the fromProfile property of each message:
const cacheProfilesPromises = [];
const messagesFrom = [];
chatListNew.forEach((message, index) => {
const { from } = message;
if(messagesFrom.includes(from)) return;
messagesFrom.push(from);
const thisSearch = User.findOne({_id : from})
.select('name nickname phone avatar')
.exec()
.catch(err => { console.log(err); return '???' ; });
cacheProfilesPromises.push(thisSearch);
});
Promise.all(cacheProfilesPromises)
.then((newInfoArr) => {
// Transform the array of Promises into an object indexed by `from`:
const newInfoByFrom = newInfoArr.reduce((a, newInfo, i) => {
a[messagesFrom[i]] = newInfo;
return a;
}, {});
// Iterate over `chatListNew` and assign the *resolved* values:
chatListNew.forEach((message) => {
message.fromProfile = newInfoByFrom[message.from];
});
});

A Promise is an object container, like a Array. The difference being that a Promise holds a value that will sometimes be.
So, since you do not know when the value will be resolved in Promise jargon, generally you tell the promise what to do with the value, when it is resolved.
So for example,
function (id) {
const cache = {}
const promise = expensiveQuery(id)
// promise will always be a promise no matter what
promise.then(value => cache[id] = value)
// After the callback inside then is executed,
// cache has the value you are looking for,
// But the following line will not give you the value
return cache[params.id]
}
Now, what you might do to fix that code is, return the promise when the query is run for the first time, or return the cached value.
// I moved this out of the function scope to make it a closure
// so the cache is the same across function calls
const cache = {}
function (id) {
if(cache[id]) return cache[id]
const promise = expensiveQuery(id)
// promise will always be a promise no matter what
promise.then(value => cache[id] = value)
// now we just return the promise, because the query
// has already run
return promise
}
Now you'll have a value or a promise depending on whether the function has already been called once before for that id, and the previous call has been resolved.
But that's a problem, because you want to have a consistent API, so lets tweak it a little.
// I moved this out of the function scope to make it a closure
// so the cache is the same across function calls
const cache = {}
function cachingQuery (id) {
if(cache[id]) return cache[id]
const promise = expensiveQuery(id)
// Now cache will hold promises and guarantees that
// the expensive query is called once per id
cache[id] = promise
return promise
}
Ok, now you always have a promise, and you only call the query once. Remember that doing promise.then doesn't perform another query, it simply uses the last result.
And now that we have a caching query function, we can solve the other problem. That is adding the result to the message list.
And also, we dont' want to have a cache that survives for too long, so the cache can't be right on the top scope. Let's wrap all this inside a cacheMaker function, it will take an expensive operation to run, and it will return a function that will cache the results of that function, based on its only argument.
function makeCacher(query) {
const cache = {}
return function (id) {
if(cache[id]) return cache[id]
const promise = query(id)
cache[id] = promise
return promise
}
}
Now we can try to solve the other problem, which is, assign the user to each message.
const queryUser = makeCacher((id) => User.findOne({_id : id})
.select('name nickname phone avatar')
.exec())
const fromUsers = chatListNew.map((message) => queryUser(message.from))
Promise.all(fromUsers)
.then(users =>
chatListNew.map(message =>
Object.assign(
{},
message,
{ fromProfile: users.find(x => x._id === message.from)})))
.then(messagesWitUser => res.json(messagesWitUser) )
.catch(next) // send to error handler in express