Versions: Keystone v4
I have a Mongo database with >20k items. What I want is a paginator that would allow the user to quickly scroll through the Mongo database 25 elements at a time. Currently, this feature is implemented, but the server takes >40 seconds to return the results because it queries the entire (20k item) database. However, only 25 elements are displayed on a single page, so I feel like if it just fetches 25 results instead of 20k, it should be quicker. How could I implement this? I know about the .limit() function, but I can't seem to figure out pagination in keystone while using that.
Current Code:
var q = Items.model.find();
q.exec(function(err, newss) {
console.log('There are %d', newss.length); // Prints out 20k number
...//skip
locals.cnts = newss;
// console.log(newss[0])
locals.pagerr = pager({
page: parseInt(req.query.page, 10) || 1,
perPage: 25,
total: newss.length
});
locals.itemsss = locals.cnts.slice(
locals.pagerr.first - 1,
locals.pagerr.last
);
next();
})
In it's current implmentation, it takes >40 seconds to return the paginated results. How can I fix this?
The model.find() function you're using here is equivalent to the Mongoose find() function. As you're calling it without any filters, this code is retrieving all 25k items from the database each time it runs. This data is being transferred to the web server/node process where the body of your function(err, newss) {...} function is run. Only then are the 25 items you're after being extracted from the set.
Instead, if you want to use offset-based pagination like this, you should be using the query.limit() and query.skip() functions. If you need to count the total items first, do so in a separate query using query.count().
I haven't tested this code (and it's been a while since I used Mongoose), but I think you want something like this:
// Warning! Untested example code
Items.model.find().count(function (err, count) {
console.log('There are %d', count);
locals.pager = pager({
page: parseInt(req.query.page, 10) || 1,
perPage: 25,
total: count
});
Items.model.find()
.skip(locals.pager.first)
.limit(25)
.exec(function(err, results) {
locals.results = results;
next();
});
});
On a more general note – if you like Keystone and want to use Mongo, keep an eye on the Keystone 6 updates. Keystone 6 uses Prisma 2 as it's ORM layer and they recently released support for Mongo. As soon as that functionality production ready, we'll be supporting it in Keystone too.
Related
I have an angular app where i am querying my firebase database as below:
constructor() {
this.getData();
}
getData() {
this.projectSubscription$ = this.dataService.getAllProjects()
.pipe(
map((projects: any) =>
projects.map(sc=> ({ key: sc.key, ...sc.payload.val() }))
),
switchMap(appUsers => this.dataService.getAllAppUsers()
.pipe(
map((admins: any) =>
appUsers.map(proj =>{
const match: any = admins.find(admin => admin.key === proj.admin);
return {...proj, imgArr: this.mapObjectToArray(proj.images), adminUser: match.payload.val()}
})
)
)
)
).subscribe(res => {
this.loadingState = false;
this.projects = res.reverse();
});
}
mapObjectToArray = (obj: any) => {
const mappedDatas = [];
for (const key in obj) {
if (Object.prototype.hasOwnProperty.call(obj, key)) {
mappedDatas.push({ ...obj[key], id: key });
}
}
return mappedDatas;
};
And here is what I am querying inside dataService:
getAllProjects() {
return this.afDatabase.list('/projects/', ref=>ref.orderByChild('createdAt')).snapshotChanges();
}
getAllAppUsers() {
return this.afDatabase.list('/appUsers/', ref=>ref.orderByChild('name')).snapshotChanges();
}
The problem I am facing with this is I have 400 rows of data which I am trying to load and it is taking around 30seconds to load which is insanely high. Any idea how can I query this in a faster time?
We have no way to know whether the 30s is reasonable, as that depends on the amount of data loaded, the connection latency and bandwidth of the client, and more factors we can't know/control.
But one thing to keep in mind is that you're performing 400 queries to get the users of each individual app, which is likely not great for performance.
Things you could consider:
Pre-load all the users once, and then use that list for each project.
Duplicate the name of each user into each project, so that you don't need to join any data at all.
If you come from a background in relational databases the latter may be counterintuitive, but it is actually very common in NoSQL data modeling and is one of the reasons NoSQL databases scale so well.
I propose 3 solutions.
1. Pagination
Instead of returning all those documents on app load, limit them to just 10 and keep record of the last one. Then display the 10 (or any arbitrary base number)
Then make the UI in such a way that the user has to click next or when the user scrolls, you fetch the next set based on the previous last document's field's info.
I'm supposing you need to display all the fetched data in some table or list so having the UI paginate the data should make sense.
2. Loader
Show some loader UI on website load. Then when all the documents have fetched, you hide the loader and show the data as you want. You can use some custom stuff for loader, or choose from any of the abundant libraries out there, or use mat-progress-spinner from Angular Material
3. onCall Cloud Function
What if you try getting them through an onCall cloud function? It night be faster because it's just one request that the app will make and Firebase's Cloud Functions are very fast within Google's data centers.
Given that the user's network might be slow to iterate the documents but the cloud function will return all at once and that might give you what you want.
I guess you could go for this option only if you really really need to display all that data at once on website load.
... Note on cost
Fetching 400 or more documents every time a given website loads might be expensive. It'll be expensive if the website is visited very frequently by very many users. Firebase cost will increase as you are charged per document read too.
Check to see if you could optimise the data structure to avoid fetching this much.
This doesn't apply to you if this some admin dashboard or if fetching all users like this is done rarely making cost to not be high in that case.
I am building a web application, and I am spending so long time to take care of updates between related documents.
For example, I have 'Task' document and 'User' document. When task is made, multiple users will be assigned to it. Thus,
taskA.assigned = ["1321231fsdfsdf"(userA's _id), "12312313asdasdasd"(userB's _id)]
userA.tasks = [..., "1231321"(taskA's _id),...]
userB.tasks = [..., "12313211"(taskB's _id),...]
I could handle it well when it comes to just creating tasks. However, it becomes too tricky when I am going to edit tasks. If user B is deleted from taskA, I have to delete userB's id and go to the userB's tasks property and delete taskA's id too.
Is there any shortcut and automatic way to deal with it? Thank you for your time to read it. Let me know if I was too vague, I will add more detail.
In a relational database like MySQL using foreign keys and cascade updates could be done automatically, but in MongoDB that's not possible.
But I see in the tags you are using Moongose, so using a post save hook could do the trick. You can set a hook that updates automatically the user collection each time a task is updated, or viceversa.
Other option would be changing your data estructure, but this depends on your case, there are some facts to take into account. I think we don't have enough information to judge, but there are many resources speaking about data normalization in MongoDB, you can check for example the official MongoDB manual.
I found a way to do this easily and with less lines of codes. Around 100 lines reduced to around 30 lines by doing this.
Long story short, I used 'update()' method and various mongo operators, such as $in, $push, or $pull.
Here is my final codes that are optimized with use of update method.
var edit = req.body;
edit.assignedTo = edit.assignedTo.split(',');
var old = req.task;
var idQueries = edit.assignedTo.map(function (x) {
return mongoose.Types.ObjectId(x);
});
User.update({tasks: old._id}, {$pull: {tasks: old._id}}, {multi: true}, function () {
// Update to remove original task's id from users assigned to it.
User.update({_id: {$in: idQueries}}, {$push: {tasks: old._id}}, {multi: true}, function () {
// Update to add edited tasks'id to new users assigned to it.
old.lastAction = 'edited';
old.edited = true;
old.editedAt = Date.now();
old.titke = edit.title;
old.desc = edit.desc;
old.dueBy = edit.dueBy;
old.assignedTo = edit.assignedTo;
old.save(function (err, task) {
if (err) return next(err);
User.populate(task, 'assignedTo', function (err, task) {
res.json(task);
});
});
});
});
Wish this help some people!
I'm looking to implement a solution where I can query the Mongoose Database on a regular interval and then store the results to serve to my clients.
I'm assuming this will reduce my response time when my users pull the collection.
I attempted to implement this plan by creating an empty global object and then writing a function that queries the db and then stores the results as the global object mentioned previously. At the end of the function I setTimeout for 60 seconds and then ran the function again. I call this function the first time the server controller gets called when the app is first run.
I then set my clients up so that when they requested the collection, it would first look to see if the global object exists, and if so return that as the response. I figured this would cut my 7-10 second queries down to < 1 sec.
In my novice thinking I assumed that Nodejs being 'single-threaded' something like this could work quite well - but it just seemed to eat up all my RAM and cause fatal errors.
Am I on the right track with my thinking or is it better to query the db every time people pull the collection?
Here is the code in question:
var allLeads = {};
var getAllLeads = function(){
allLeads = {};
console.log('Getting All Leads...');
Lead.find().sort('-lastCalled').exec(function(err, leads) {
if (err) {
console.log('Error getting leads');
} else {
allLeads = leads;
}
});
setTimeout(function(){
getAllLeads();
}, 60000);
};
getAllLeads();
Thanks in advance for your assistance.
I was searching for an easy and simple database for a little highscore system for a some games I'm developing in javascript.
I saw Orchestrate.io in github's student developer pack. I found a suitable drivermodule nodejs orchestrate and have integrated them.
The problem comes with querying orchestrate for my data. I have managed saving scores and querying them with db.list('collection'), but this seems to not responding with all data. It appered to me that some values are not returned.
I read about the db.search('collection','query') function. But I don't really understand how I could return all data because I don't want to query in a specific way.
My objects are as simple as follows:
{"name":"Jack","score":1337}
As I understand, one has to send a key, when putting such values to an orchestrate-collection. But I'd like to query the whole collection and get the values in a descendant order in regard to the score.
As for now I end up sorting the result on the client-side.
I hope you guys can give me some hints for a query that can sort for specific values!
You have the option to use a SearchBuilder
db.newSearchBuilder() //Build a search object
.collection('collection') //Set the collection to be searched
.sort(score, 'desc') //Set the order of the results
.query("*") //Empty search
.then(function (res) { //Callback function for results
//Do something with the results
})
Source
By default, .list uses a pagination limit of 10. You can either increase that, e.g.:
db.list('collection', { limit: 100 })
Or use .links, .links.next (from the docs):
db.list('collection', { limit: 10 })
.then(function (page1) {
// Got First Page
if (page1.links && page1.links.next) {
page1.links.next.get().then(function (page2) {
// Got Second Page
})
}
})
I have a set of records that I would like to update sequentially in perpetuity. Basically:
Get least recently updated record
Update record
Set date of record to now (aka. send it to the back of the list)
Back to step 1
Here is what I was thinking using Firebase:
// update record function
var updateRecord = function() {
// get least recently updated record
firebaseOOO.limit(1).once('value', function(snapshot) {
key = _.keys(snapshot.val())[0];
/*
* do 1-5 seconds of non-Firebase processing here
*/
snapshot.ref().child(key).transaction(
// update record
function(data) {
return updatedData;
},
// update priority after commit (would like to do it in transaction)
function(error, committed, snap2) {
snap2.ref().setPriority(snap2.dateUpdated);
}
);
});
};
// listen whenever priority changes (aka. new item needs processing)
firebaseOOO.on('child_moved', function(snapshot) {
updateRecord();
});
// kick off the whole thing
updateRecord();
Is this a reasonable thing to do?
In general, this type of daemon is precisely what was envisioned for use with the Firebase NodeJS client. So, the approach looks good.
However, in the on() call it looks like you're dropping the snapshot that's being passed in on the floor. This might be application specific to what you're doing, but it would be more efficient to consume that snapshot in relation to the once() that happens in the updateRecord().