Develop Reference

(Mongo/Mongoose) How to handle waiting for the result of multiple queries

I'm writing a Discord bot that generates weekly Guild stats for text and voice channel usage. My code divides several Mongo queries up into separate methods:
function getTopActiveTextChannels() {
let topTextChannels = []
ChannelModel.find({}).sort({"messageCountThisWeek": -1}).limit(topLimit)
.exec(channels => {
channels.forEach(c => {
topTextChannels.push({"name": c.name, "messageCount": c.messageCount})
})
console.log(topTextChannels)
return topTextChannels
})
}
function getTopActiveVoiceMembers() {
let topVoiceMembers = []
UserModel.find({}).sort({"timeSpentInVoice": -1}).limit(topLimit)
.exec(users => {
users.forEach(u => {
topVoiceMembers.push({"username": u.username, "timeSpentInVoice": u.timeSpentInVoice})
})
console.log(topVoiceMembers)
return topVoiceMembers
})
}
I then have one method that calls both those and (for now) prints the values to console:
function getWeeklyGuildStats(client) {
let topActiveTextChannels = getTopActiveTextChannels()
let topVoiceMembers = getTopActiveVoiceMembers()
let promisesArray = [topActiveTextChannels, topVoiceMembers]
Promise.all(promisesArray).then(values => {console.log(values)})
}
Executing getWeeklyGuildStats(client) outputs: [ undefined, undefined ]. I am sure I'm not using promises correctly, but when I follow Mongoose's documentation, it tells me to use exec() instead of then(), but I get a channels = null error with that.
Does anything jump out to anyone? This seems like a fairly common pattern. Does anyone have a solution for how to resolve multiple Mongoose queries in a single method?

Promise.all should take an array of promises, while your functions are returning normal array, so you need to return the whole query in the helper method which getting the users and channels, then do your logic after the promise.all
your functions may look something like that
function getTopActiveTextChannels() {
return ChannelModel.find({}).sort({"messageCountThisWeek": -1}).limit(topLimit).exec();
}
function getTopActiveVoiceMembers() {
return UserModel.find({}).sort({"timeSpentInVoice": -1}).limit(topLimit).exec();
}
then the function that calls these two methods will be something like
function getWeeklyGuildStats(client) {
let topActiveTextChannels = getTopActiveTextChannels()
let topVoiceMembers = getTopActiveVoiceMembers()
let promisesArray = [topActiveTextChannels, topVoiceMembers]
Promise.all(promisesArray).then(values => {
console.log(values);
// here you could do your own logic, the for loops you did in the helper methods before
});
}

You do not have any return statements in the root level of your functions, so they are always synchronously returning undefined. I'm not familiar with the library you're using, but if for example ChannelModel.find({}).exec(callback) returns a promise with the return value of callback as your code implies, then you just need to add a return statement to your functions.
For example:
function getTopActiveTextChannels() {
let topTextChannels = []
// Return this! (Assuming it returns a promise.) Otherwise you're always returning `undefined`.
return ChannelModel.find({}).sort({"messageCountThisWeek": -1}).limit(topLimit)
.exec(channels => {
channels.forEach(c => {
topTextChannels.push({"name": c.name, "messageCount": c.messageCount})
})
console.log(topTextChannels)
return topTextChannels
})
}

Add an undefined method promise to an array to be resolved later in Promise.all()

I want to queue up DB calls that will be executed once it's connected. The DB object is created and stored as a member of a module when it's connected.
DB Module:
var db = {
localDb: null,
connectLocal: (dbName) => {
// Do stuff
this.localDb = new PouchDB(dbName) // has a allDocs() method
}
}
Adding calls to queue:
var dbQueue = []
function getDocs () {
dbQueue.push (
db.localDb.allDocs () // allDocs() not yet defined; returns promise
)
}
// Called when connected and queue is not empty:
function processQueue () {
Promise.all (dbQueue)
.then(...)
}
If getDocs() is called before db.connectLocal() sets db.localDb, then I get the following error (or similar) because db.localDb is not yet defined:
TypeError: Cannot read property 'then' of undefined
Is it possible to add an undefined method, that returns a promise, to an array to be resolved later in Promise.all()? Any other ideas as to how I can solve this issue?
Also, I'm using Vue.js and PouchDB.

You can make a promise in your db module instead of just localDb property:
let localDb = null;
let resolveLocalDb = null;
let localDbPromise = new Promise(function(resolve, reject) {
resolveLocalDb = resolve;
});
var db = {
getLocalDb: () {
return localDbPromise;
}
connectLocal: (dbName) => {
// Do stuff
localDb = new PouchDB(dbName) // has a allDocs() method
resolveLocalDb(localDb);
}
}
Then, exchange .localDb to getLocalDb(), which returns a promise.
dbQueue.push(
db.getLocalDb().then(db => db.allDocs())
)

I solved my queue issue, but it wasn't at all how I was trying to go about it.
My first problem was thinking that Promise.all() deferred calling my methods until it was called, but they are called when added to the array. This caused the error I mentioned in my question. So I needed to rethink how to populate the queue with methods that may not yet exist.
The solution was to add the calls to an array (the queue) as strings (e.g. "getDocs"), then loop through the array calling the methods using bracket notation (e.g. db["getDocs"]()).
My app is written in Vue.js so it's obviously different, but here's a simplified, working example:
// Dummy DB object
var db = {
docs: [1, 2, 3]
};
// Queue were the DB ops are stored
var dbQueue = [];
// Process the queue - called elsewhere once the DB is connected
// The processed array and Promise.all() aren't necessary as you could just call
// the method outright, but I want to log the results in order
async function processQueue() {
var processed = []; // Called queue methods
// Add valid methods to
dbQueue.forEach(method => {
if (typeof db[method] === "function") {
return processed.push(db[method]());
} else {
console.error(`"${method}" is not a name of a valid method.`);
}
});
// Log promise results
await Promise.all(processed).then(res => {
console.log("Processed:", res);
});
// Empty the queue
dbQueue = [];
}
// Add some calls to the queue of methods that don't yet exist
dbQueue.push("getDocs");
dbQueue.push("getDocs");
// Simulate adding the method
db.getDocs = function() {
return new Promise(resolve => {
resolve(this.docs);
});
};
// Process queue once conditions are met (e.g. db is connected); called elsewhere
processQueue();
And here's a fiddle with an example that allows arguments for the methods: https://jsfiddle.net/rjbv0284/1/

How to use Promises in a loop and ensure all have fulfilled before continuing?

I'm using the localforage library to access localStorage/IndexedDB. To retrieve an item, the localforage.getItem() function is called, which returns a Promise that fulfills when the data is retrieved.
I need to iterate through the localforage keys, calling 'getItem' on any key that matches my criteria, and placing the value of that key in the 'matches' array. However, I don't want to continue the function until I'm sure all the values have been successfully added to 'matches'.
I am quite new to Promises, and I can't figure out how to wait until all the getItem() Promises have fulfilled before I move on.
I realize localforage has an 'iterate' function, but I'm really interested in becoming more proficient in the use of Promises, and would really like to know how this should work.
Here's what I'm doing:
var matches = []; // Array to store matched items
localforage.keys() // Get all keys in localforage
.then(function(keys) // When all keys are retrieved, iterate:
{
for(var i in keys)
{
// If the current key matches what I am looking for, add it to the 'matches' array.
if (keys[i].indexOf('something i am looking for') > -1)
{
// Here I need to add this value to my array 'matches', but this requires using the getItem method which returns a Promise and doesn't necessarily fulfill immediately.
localforage.getItem(keys[i])
.then(function(value)
{
matches.push(value);
});
}
}
});
// At this point, I want to proceed only after *all* matches have been added to the 'matches' array (i.e. the getItem() Promises are fulfilled on all items in the loop).
How do I do this? Is this where the 'await' expression is applied? For example, should I do
await localforage.getItem(keys[i])
.then(function(value)
... etc
Would this make the getItem function synchronous?
Thanks for any suggestions/pointers.

You can use Promise.all() in this situation. The basic idea is that you push a bunch of promises into and array then pass that array to Promise.all() when all the promises in the array resolve, Promise.all() resolves to the an array of the values:
localforage.keys()
.then(function(keys){
var matches = []
for(let i in keys) {
if (keys[i].indexOf('something i am looking for') > -1) {
// matches will be an array of promises
matches.push(localforage.getItem(keys[i]))
}
}
// Promise.all returns a promise that resolves to an array
// of the values they resolve to
return Promise.all(matches)
})
.then(values => {
// values is an array of your items
})
You can use async/await for this too with something like this with mocked out keys and getItems to make the snippet run:
let localforage = {
keys() {
return Promise.resolve([1, 2, 3, 4, 5])
},
getItem(k) {
return Promise.resolve("found: " + k)
}
}
async function getStuff() {
let matches = []
let keys = await localforage.keys()
for (let key of keys) {
matches.push(await localforage.getItem(key))
}
return matches
}
getStuff()
.then(values => {
console.log(values)
})

What am I doing wrong in this recursive function, when returning promise

I need to get the initial timestamps of all znodes in the zookeeper. I am using the getChildren method of node-zookeeper-client to do so. I am calling my getInitialTimeStamp recursively to traverse along the path. My
function looks something like this
function getInitialTimeStamp(client,path){
return new Promise((resolve,reject) => {
client.getChildren(
path,
function(error,children,stats){
//if children empty, return
if (typeof children === 'undefined' || children.length <= 0) {resolve();}
timeStamp[path]= {ctime: stats.ctime, mtime: stats.mtime};
children.map(child => {
getInitialTimeStamp(client,path+'/'+child);
});
});
});
}
it is being called like this
getInitialTimeStamp(client,path)
.then(() => {
console.log(timeStamp);
console.log("finished");
});
The problem is I can not get my .then() part to run. I know this is related to returning promise but I do not know what is being done wrong here. Consider my lack of knowledge in promises and async programming and provide me with a solution.

there are two things wrong .... if children is not empty, you never resolve ... and your children.map may as well be a forEach the way you're using it
So, firstly, you want to resolve something if children has a length, and sescondly, you only want to do so once ALL the getInitialTimeStamp of the children is finished, by use of Promise.all
function getInitialTimeStamp(client,path){
return new Promise((resolve,reject) => {
client.getChildren(
path,
function(error,children,stats){
//if children empty, return
if (typeof children === 'undefined' || children.length <= 0) {
resolve();
}
timeStamp[path]= {ctime: stats.ctime, mtime: stats.mtime};
// use promise.all to wait for all child timestamps
Promise.all(children.map(child => getInitialTimeStamp(client,path+'/'+child)))
// and then resolve this path
.then(resolve);
});
});
}
Although that can be cleaned up a bit
function getInitialTimeStamp(client, path) {
return new Promise((resolve, reject) => {
client.getChildren(path, (error, children, stats) => {
timeStamp[path]= {ctime: stats.ctime, mtime: stats.mtime};
resolve(Promise.all((children||[]).map(child => getInitialTimeStamp(client, path + '/' + child))));
});
});
}
but still no error checking is done ... i.e. test if error is truthy

I would suggest this type of implementation that promisifies at a lower level by promisifying client.getChildren(). That makes it a lot easier to write all your logic using promises and avoids common pitfalls of JaramandaX's implementation such as completely missing error handling and error propagation.
Since promises only resolve to a single value, when promisifying something that passes multiple values to its callback, you have to shoehorn each of the values into an object and resolve with that object.
Also, your implementation seems to be modifying some timeStamp global or higher scope variable which seems less than desirable. So, I've made it so you can optionally pass in an object to start with, but if you don't it will default to an empty object and, in either case, the function will return a promise that will resolve to the object filled with the desired properties, including a cnt property so you can more easily see how many are there.
getInitialTimeStamp() returns a promise that resolves to an object that contains your desired path properties.
// make promisified version that resolves to an object with properties on it
// Depending upon the situation, you might add this to the prototype rather than
// to an instance
client.getChildrenP = function(path) {
return new Promise((resolve, reject) => {
this.getChildren(path, (error, children, stats) => {
if (error) return reject(error);
resolve({children, stats});
});
});
}
// Returns a promise that resolves to a timeStamp object
// You can optionally pass in an object to be modified or that will default
// to an empty object. In either case, the returned promise resolves to
// the finished object.
function getInitialTimeStamp(client, path, resultsObj){
let obj = resultsObj || {cnt: 0};
obj.cnt = obj.cnt || 0;
return client.getChildrenP(path).then(results => {
if (typeof results.children === 'undefined' || children.length <= 0) {
// return results so far
return obj;
}
++obj.cnt;
obj[path]= {ctime: results.stats.ctime, mtime: results.stats.mtime};
return Promise.all(children.map(child => {
getInitialTimeStamp(client,path+'/'+child, obj);
})).then(results => {
return obj;
});
});
}
Usage:
getInitialTimeStamp(client, somePath).then(resultsObj => {
// process resultsObj here
}).catch(err => {
// process error here
});

RxJS: Cache Several XHR Calls without Mutability and Side Effects

I’m looking for an RxJS example how to cache a series of XHR calls (or other async operations), so the same call does not have to be repeated, while respecting immutability and with no side effects.
Here's a bare-bones, mutable example:
var dictionary = {}; // mutable
var click$ = Rx.Observable.fromEvent(document.querySelector('button'), 'click', function (evt) {
return Math.floor(Math.random() * 6) + 1; // click -> random number 1-6 (key)
})
.flatMap(getDefinition);
var clicksub = click$.subscribe(function (key) {
console.log(key);
});
function getDefinition (key) {
if ( dictionary[key] ) { // check dict. for key
console.log('from dictionary');
return Rx.Observable.return(dictionary[key]);
}
// key not found, mock up async operation, to be replaced with XHR
return Rx.Observable.fromCallback(function (key, cb) {
dictionary[key] = key; // side effect
cb(dictionary[key); // return definition
})(key);
}
JSBin Demo
Question: Is there a way to accomplish caching several similar async operations without resorting to using the dictionary variable, due to mutability and side effect?
I’ve looked at scan as a means to “collect” the XHR call results, but I don’t see how to handle an async operation within scan.
I think I’m dealing with two issues here: one is state management maintained by the event stream rather than kept in a variable, and the second is incorporating a conditional operation that may depend on an async operation, in the event stream flow.

Using the same technique than in a previous question (RxJS wait until promise resolved), you could use scan and add the http call as a part of the state you keep track of. This is untested but you should be able to easily adapt it for tests :
restCalls$ = click$
.scan(function (state, request){
var cache = state.cache;
if (cache.has(request)) {
return {cache : cache, restCallOrCachedValue$ : Rx.Observable.return(cache.get(request))}
}
else {
return {
cache : cache,
restCallOrCachedValue$ : Rx.Observable
.defer(function(){
return Rx.Observable
.fromPromise(executeRestCall(request))
.do(function(value){cache.add(request,value)})
})
}
}
}, {cache : new Cache(), restCallOrCachedValue$ : undefined})
.pluck('restCallOrCachedValue$')
.concatAll()
So basically you pass an observable which does the call down the stream or you directly return the value in the cache wrapped in an observable. In both cases, the relevant observables will be unwrapped in order by concatAll. Note how a cold observable is used to start the http call only at the time of the subscription (supposing that executeRestCall executes the call and returns immediately a promise).

Working from #user3743222's response, this code does cache the XHR calls:
// set up Dictionary object
function Dictionary () { this.dictionary = {} }
Dictionary.prototype.has = function (key) { return this.dictionary[key] }
Dictionary.prototype.add = function (key, value) { this.dictionary[key] = value }
Dictionary.prototype.get = function (key) { return this.dictionary[key] }
var definitions$ = click$
.scan(function (state, key) {
var dictionary = state.dictionary
// check for key in dict.
if ( dictionary.has(key) ) {
console.log('from dictionary')
return {
dictionary: dictionary,
def$: Rx.Observable.return(dictionary.get(key))
}
}
// key not found
else {
return {
dictionary: dictionary,
def$: Rx.Observable.fromPromise(XHRPromise(key))
.do(function (definition) { dictionary.add(key, definition) })
}
}
}, {dictionary : new Dictionary(), def$ : undefined})
.pluck('def$') // pull out definition stream
.concatAll() // flatten
Updated: In the // key not found block, executing the function XHRPromise (that returns a promise) passing in the key and the result passed to the RxJS method fromPromise. Next, chain a do method in which the definition is grabbed and added to the dictionary cache.
Follow up question: It appears we've removed the side effect issue, but is this still considered mutating when the definition and key are added to the dictionary?
Putting this here for archival purposes, the first iteration had this code, accomplishing the same caching procedure:
// ALTERNATIVE key not found
else {
var retdef = Rx.Observable.fromPromise(function () {
var prom = XHRPromise(key).then(function (dd) {
dictionary.add(key, dd) // add key/def to dict.
return dd
})
return prom
}()) // immediately invoked anon. func.
return { dictionary: dictionary, def$: retdef }
}
Here, using the function XHRPromise that returns a promise. Before returning that promise to the event stream, though, pull out a then method in which the promised definition is grabbed and added to the dictionary cache. After that the promise object is returned to the event stream through the RxJS method fromPromise.
To get access to the definition returned from the XHR call, so it can be added to the dictionary cache, an anonymous, immediately invoked function is used.