Develop Reference

The right way to do pagination in mongo db

I am trying to create a blog website with pagination using MongoDB and express
where users will be able to get pages with limit on how many posts on every page sorted by newest to oldest
My implementation of the pagination is as follows...
For example, if a client want to make request to get posts for the third page, with page size of 5 posts per page, he will make the following request to the server
GET// {serverUrl}/posts?pagesize=5&pagenumber=3
this is the code in node/express
app.get('/posts', async function(req, res, next) {
const pagesize = req.query.pagesize || 0; //5
const pagenumber = req.query.pagenumber || 0; //3
const postsToSkip = (pagenumber -1) * pagesize; //10
try {
// skip 10 and limit 5
const posts = await posts.find().sort({date: -1}).skip(postsToSkip).limit(pagesize)
return res.status(200).send(posts.toArray());
} catch(error) {
return res.status(500).send('something went wrong');
}
});
The code works perfectly fine, the problem is as follows
for simplicity peruse lat's say that there are 100 posts in the database, and they all have an ID 1-100, so the oldest post has an ID of 1 and the newest post has an ID of 100
let's say that a user is going to the blog posts website (build by React, for example)
the React make a request to the server
GET// {serverUrl}/posts?pagesize=5&pagenumber=1
the server then makes a request to MongoDB to get the newest 5 posts (ID's 96-100)
posts.find().skip(0).limit(5)
and then he sends it back to React, and react render them on to the page,
now if for example before the user goes to the next page, 5 new posts are added to the database by a different user (they all get ID's of 101-105), now when the user goes to the next page, React will make a request to the server get the second page
GET// {serverUrl}/posts?pagesize=5&pagenumber=2
the server will now make a request to MongoDB to get page number 2
posts.find().skip(5).limit(5)
but now the second page will be identical to first page, since when the user first went to the website and was on the first page the 5 newest posts were with ID of 96-100, but when he went to the second page after the addition of the 5 posts, the 5 newest posts were ID 101-105 and the second 5 newest posts will be the posts with ID's of 96-100 same as were on the first page when the user first went to the website, thus causing the second page to be identical to the first page
I would like to know if there are any implementation to overcome this weird behavior
This is my first question I am posting to stackoverflow.com, I would like to hear your feedback...
thank you very much 😊😊😊

try with this example.
app.get('/posts', async function (req, res, next) {
try {
const pagesize = 10; // 10 records per page
const pageNumber = req.query.pageNumber; // pageNumber
const posts = await posts.find().skip((pageNumber - 1) *
pagesize).limit(pagesize)
return res.status(200).send(posts.toArray());
} catch (error) {
return res.status(500).send('something went wrong');
}
});

Get latest data after delete/insert operation in couchbase Node.js

I have difficult time to get latest bucket documents after any operation (insert/delete). I use node.js + express + couchbase, and here's my code, derived from its official website. it basically just a simple address book web-app and has three function: insert, show or delete contact.
After delete or insert operation, I use res.redirect("\") to reload my page. The problem is: it just show old data, only after I reload it for second time, it shows up. I just don't know what's wrong here. Can anybody help?
/* GET home page. */
router.get('/', function(req, res) {
var query = ViewQuery.from('view_people', 'by_id');
bucket.query(query, function(err, results){
return res.render('index', {people: results});
});
});
/* INSERT Contact here */
router.post('/insert', function(req, res){
var name = req.body.name;
var address = req.body.address;
var job = req.body.job;
var age = req.body.age;
bucket.insert(name, {
name : name,
address : address,
job : job,
age : age
}, function(err, reply){
if(err) return res.send(err);
return res.redirect('/');
})
});
/* REMOVE the contact */
router.get('/remove', function(req, res){
var id = req.query.id;
bucket.remove(id, function(err, response){
if (err) {return res.send(err);}
return res.redirect('/');
})
});
module.exports = router;

In Couchbase, views are eventually consistent and it looks like you are using views, thus the lag you are seeing. You'd be better off getting the document by key if you can, then Couchbase is strongly consistent. You'd get the most performance if you had a meta-data object that is a manual index of all of that user's contacts. Then when you want their contacts, you pull that meta-data object, then do a parallelized bulk get of the objects you want by key. That should perform VERY well and do the exact same thing. On top of that, you have a key pattern that uniquely identifies the object to the user.
Another option is to have a counter object per user that is a key value with just the counter. So For example have a key pattern
username::contacts::counter
Filled out, the object's key might look like
hernandez94::contacts::139
the counter would be the upper bound of the array of contacts for that user. to get the most recent contact, you just get the counter object, take that integer, construct the key and get exactly that object. It'd be very fast. To get all contacts, you get the counter object and then generate the keys to do a parallelized bulk get. Again, that'd be very fast. No querying, just raw speed that scales consistently.

Looks like you need to use stale=false in your view query . See : https://forums.couchbase.com/t/how-does-stale-query-work/870

Adding a filter inside a beforeRemote remote hook

I have a problem I can't find an answer to in Loopback's docs.
Say I have a model Company and a modelEmployee. There is an 1Xn relation between the Company and its Employees. When /api/Employees is called, server returns all the employees.
I only want to return the list of employees who are in the same company with the user requesting the list.
For this, I created a remote hook
Employee.beforeRemote('find', function(context, modelInstance, next) {
var reject = function() {
process.nextTick(function() {
next(null, false);
});
};
// do not allow anonymous users
var userId = context.req.accessToken.userId;
if (!userId) {
return reject();
}
//I get the details of the user who sent the request
//to learn which company does he belong to
Employee.findById(userId, function(err, user) {
if(!context.req.query.filter) context.req.query.filter={};
context.req.query.filter.where = {brandId:user.companyId};
console.log(context.req.query);
next();
});
});
I thought this should work every time, but appearantly it only works when find already has some query filters like include - although the console.log prints a correct context.req.query object.
What am I missing? Any help would be greatly appreciated!

context.args.filter seems to work for this purpose.
As a side note, instead of replacing where, you might want to merge it with something provided by client. For implementation idea you can refer to: https://github.com/strongloop/loopback-datasource-juggler/blob/master/lib/utils.js#L56-L122

Setup page on first login, express.js

I'm using Passport.js to authorize user into my express.js app.
I want to show setup page when user logins first time.
Can I do this with Passport?

This Code MUST be modified mor your db, etc.
Init The User:
var user = new User({ ... });
user.newUser = true;
And
app.get("/account",requireLogin, function(req,res){
if(req.user.newUser){
req.user.newUser = false;
req.user.save(function(err){
if(err) return handleError(err);
res.redirect("/setup");
})
}
})

You need to place newuser attribute within the database as answered by "Ari Porad" but the better approach would be to change it to false when you are done with account first time setup because we dont really know if the user has completed his first time setup or not (browser may crash and he might not be able to complete it :D ) , so its better not to modify the "newuser" attribute as false on first visit but rather you should be modifying it after he has submitted the data that we require on the first visit.

Is there any better way to handle the nested callbacks in Node.js/Expressjs?

I am using Node.js and Expressjs for my server side coding and MongoDB as back end. I am new to all these technologies.
I need to do a list of actions based on the request.
For example in user management
Check the user already registered
If registered resend the activation email
If not registered then get the userId from another table that will maintain the ids for user,assets etc..[ I know MongoDB provide unique _id; but I need to have a unique integer id as userId ]
Create the user
Send the success or failure response.
To implement this i wrote the following code :
exports.register = function(req,res,state,_this,callback) {
switch(state) {
case 1: //check user already registered or not
_this.checkUser(req, res, ( state + 1 ), _this, _this.register, callback);
break;
case 2: //Already registered user so resend the activation email
_this.resendEmail(req, res, 200, _this, _this.register, callback);
break;
case 3: //not registered user so get the userId from another table that will maintain the ids for user,assets etc
_this.getSysIds(req, res, ( state + 2 ), _this, _this.register, callback);
break;
case 4: //create the entry in user table
_this.createUser(req, res, ( state + 1 ), _this, _this.register, callback);
break;
case 200: //Create Success Response
callback(true);
break;
case 101://Error
callback(false);
break;
default:
callback(false);
break;
}
};
The check user code is something like this
exports.checkUser = function(req,res,state,_this,next,callback) {
//check user already registered or not
if(user) {//Already registered user so resend the activation email
next(req,res,state,_this,callback);
}
else {//not registered user so get the userId
next(req,res,(state + 1),_this,callback);
}
}
And similarly the other functions.
The first call to the register function will do from the app.get as
user.register(req,res,1,this,function(status) {
//do somthing
});
Is there any better way to do this? The problem I am facing is based on some condition I have to follow a sequence of action. I can write all these in a nested callback structure but in that case I cannot reuse my code.
One of the problem that my boss told me was that in the code I am calling the function register and put it in a stack of callbacks as
state 1: chekuser
state 3: getIds
state 4: create user
And finally in state 200 and here i am just exiting from the stack? It may cause a stack overflow!
Is there any better way to handle the callbacks in Node.js/Expressjs ??
NOTE: The above is a sample code.I have many different situations like this.

Maybe a more elegant way would just be to put all of your various possible functions into a single object and call whichever one you need based on the state you're in. Kind of like
var States = {
1: checkuser,
2: resendEmail,
3: getSysIds,
4: createUser,
5: whateverIsNext
}
Then you just have to have one function that does everything
function exec(req,res,state,_this,callback){
States[state].apply(this, arguments)
}
And each of your functions would just set the state argument to whatever it needed to be, then recall exec, for example
function checkUser(req,res,state,_this,callback){
var doneRight = do_whatever_you_need;
state = doneRight? state++: state; //just set state to whatever number it should be
exec(req,res,state,_this,callback);
}
This way you've got your list of steps nice and readable/changeable. You've got your one function that executes everything, and each step just has to manage what the next step should be. If you wanted, you could also replace the numbered steps with named, which would make it more readable, but you'd lose the ability to increment/decrement your step variable.

You are overengineering here mixing middlewares, database layer and route handlers. Let's make it simpler and more reusable.
First of all, lets make sure, that our middlewares and route handlers can communicate with each other. That could be accomplished with session middleware:
// Somewhere inside configuration:
app.use(express.cookieParser()
app.use(express.session({secret: 'cookie monster'));
// If you don't want sessions, change with this:
app.use(function (req, res, next) {
req.session = {};
next();
});
What you want now is to have user info everywhere during request handling. For now, it's just email and whether it registered or not. That would be first stand-alone middleware. Let's assume for brevity that you detect whether user registred or not by querying db with email from query string.
// Fills `req.session.user` with user info (email, registration).
function userInfo (req, res, next) {
var user = req.session.user = req.session.user || {};
user.email = req.query.email; // I'll skip validation.
// I'm not sure which mongo driver you are using, so this is more of a pseudo-code.
db.users.findOne({email: user.email}, function(err, bson) {
if (err) {
return next(err); // Express 3 will figure that error occured
// and will pass control to error handler:
// http://expressjs.com/guide.html#error-handling
}
// Could remember information from DB, but we'll just set `registred` field.
user.registred = !!bson;
next();
});
}
After determining registration status you should either send email or create new user, that will be request handler (last param for app.get). Prior to executing handler we want our userInfo to run (can pass just userInfo (not array), but I personally like this way more):
app.get('/register', [userInfo], function (req, res) {
var user = req.session.user;
var fn = user.registred ? sendActivationEmail
: registerUser;
// If signatures differs or you need different reply for cases...
// Well, you'll figure.
fn(user.email, function (err) {
return res.send(err ? 500 : 200);
});
});
Request handler should not be bothered with what's going on inside this two functions (how many db queries, who is sending emails and how). All it knows is that first argument of provided callback is not null if error happens.
Now about creating new user. Write registerUser function which will query table with IDs, prepare and save user info in DB. For how to synchronize this two operations, see async and other SO questions. When creation is done or errored, call callback with results. This function should be part of DB layer. At least, create db.js module and export important stuff like registerUser; querying user information by email (inside first middleware) should also live inside DB layer.
Similar applies to sendActivationEmail.
Side Note: you can put whatever you want inside _id in mongo documet, not just ObjectId.