Say I have this endpoint on an express server:
app.get('/', async (req, res) => {
var foo = await databaseGetFoo();
if (foo == true) {
foo = false;
somethingThatShouldOnlyBeDoneOnce();
await databaseSetFoo(foo);
}
})
I think this creates a race condition if the endpoint is called twice simultaneously?
If so how can I prevent this race condition from happening?
OK, so based on the comments, I've got a little better understanding of what you want here.
Assuming that somethingThatShouldOnlyBeDoneOnce is doing something asynchronous (like writing to a database), you are correct that a user (or users) making multiple calls to that endpoint will potentially cause that operation to happen repeatedly.
Using your comment about allowing a single comment per user, and assuming you've got middleware earlier in the middleware stack that can uniquely identify a user by session or something, you could naively implement something like this that should keep you out of trouble (usual disclosures that this is untested, etc.):
let processingMap = {};
app.get('/', async (req, res, next) => {
if (!processingMap[req.user.userId]) {
// add the user to the processing map
processingMap = {
...processingMap,
[req.user.userId]: true
};
const hasUserAlreadySubmittedComment = await queryDBForCommentByUser(req.user.userId);
if (!hasUserAlreadySubmittedComment) {
// we now know we're the only comment in process
// and the user hasn't previously submitted a comment,
// so submit it now:
await writeCommentToDB();
delete processingMap[req.user.userId];
res.send('Nice, comment submitted');
} else {
delete processingMap[req.user.userId];
const err = new Error('Sorry, only one comment per user');
err.statusCode = 400;
next(err)
}
} else {
delete processingMap[req.user.userId];
const err = new Error('Request already in process for this user');
err.statusCode = 400;
next(err);
}
})
Since insertion into the processingMap is all synchronous, and Node can only be doing one thing at a time, the first request for a user to hit this route handler will essentially lock for that user until the lock is removed when we're finished handling the request.
BUT... this is a naive solution and it breaks the rules for a 12 factor app. Specifically, rule 6, which is that your applications should be stateless processes. We've now introduced state into your application.
If you're sure you'll only ever run this as a single process, you're fine. However, the second you go to scale horizontally by deploying multiple nodes (via whatever method--PM2, Node's process.cluster, Docker, K8s, etc.), you're hosed with the above solution. Node Server 1 has no idea about the local state of Node Server 2 and so multiple requests hitting different instances of your multi-node application can't co-manage the state of the processing map.
The more robust solution would be to implement some kind of queue system, likely leveraging a separate piece of infrastructure like Redis. That way all of your nodes could use the same Redis instance to share state and now you can scale up to many, many instances of your application and all of them can share info.
I don't really have all the details on exactly how to go about building that out and it seems out of scope for this question anyway, but hopefully I've given you at least one solution and some idea of what to think about at a broader level.
Related
This code works like it should work, but after fifth GET request it does what it should do on the backend(stores the data in db) but it's not logging anything on the server and no changes on frontend(reactjs)
const express = require('express');
const router = express.Router();
const mongoose = require('mongoose');
const User = require('./login').User;
mongoose.connect('mongodb://localhost:27017/animationsdb');
router.get('/', async(req, res) => {
await User.findOne({ username: req.query.username }, (err, result) => {
if (result) {
// when user goes to his profile we send him the list of animations he liked
// list is stored in array at db, field likedAnimations
res.send({ animationList: result.likedAnimations });
console.log("Lajkovane animacije:", result.likedAnimations);
} else {
console.log("no result found");
res.sendStatus(404)
}
});
});
router.put('/', async(req, res) => {
console.log("username:", req.body.username);
console.log("link:", req.body.link);
// if animation is already liked, then dislike it
// if it's not liked, then store it in db
const user = await User.findOne({ username: req.body.username });
if (user.likedAnimations.indexOf(req.body.link) === -1) {
user.likedAnimations.push(req.body.link);
} else {
user.likedAnimations = arrayRemove(user.likedAnimations, user.likedAnimations[user.likedAnimations.indexOf(req.body.link)]);
}
user.save();
});
function arrayRemove(arr, value) {
return arr.filter((item) => {
return item != value;
});
}
module.exports = router;
For first five requests I get this output:
Liked animations: ["/animations/animated-button.html"]
GET /animation-list/?username=marko 200 5.152 ms - 54
Liked animations: ["/animations/animated-button.html"]
GET /animation-list/?username=marko 304 3.915 ms - -
After that I don't get any output on server console and no changes on front end untill I refresh the page, even though db operations still work and data is saved.
It appears you have a couple issues going on. First, this request handler is not properly coded to handle errors and thus it leaves requests as pending and does not send a response and the connection will stay as pending until the client eventually times it out. Second, you likely have some sort of database concurrency usage error that is the root issue here. Third, you're not using await properly with your database. You either use await or you pass a callback to your database, not both. You need to fix all three of these.
To address the first and third issues:
router.get('/', async(req, res) => {
try {
let result = await User.findOne({ username: req.query.username };
if (result) {
console.log("Liked animations:", result.likedAnimations);
res.send({ animationList: result.likedAnimations });
} else {
console.log("no database result found");
res.sendStatus(404);
}
} catch(e) {
console.log(e);
res.sendStatus(500);
}
});
For the second issue, the particular database error you mention appears to be some sort of concurrency/locking issue internal to the database and is triggered by the sequence of database operations your code executes. You can read more about that error in the discussion here. Since the code you show us only shows a single read operation, we would need to see a much larger context of relevant code including the code related to this operation that writes to the database in order to be able to offer any ideas on how to fix the root cause of this issue.
We can't see the whole flow here, but you need to use atomic update operations in your database. Your PUT handler you show is an immediate race condition. In multi-client databases, you don't get a value, modify it and then write it back. That's an opportunity for a race condition because someone else could modify the value while you're sitting their holding it. When you then modify your held value, you overwrite the change that the other client just made. That's a race condition. Instead, you use an atomic operation that updates the operation directly in one database call or you use transactions to make a multi-step operation into a safe operation.
I'd suggest you read this article on atomic operations in mongodb. And, probably you want to use something like .findAndModify() so you can find and change an item in the database in one atomic operation. If you search for "atomic operations in mongodb", there are many other articles on the topic.
Can someone explain me how to prevent race conditions in node.js with Express?
If have for example this two methods:
router.get('/addUser/:department', function(req, res) { ...})
router.get('/deleteUser/:department', function(req, res) { ...})
Both functions are using a non blocking I/O Operation ( like writing to a file or a database).
Now someone calls 'addUser' with Department 'A' and someone tries to delete all users with department 'A'. How can I solve this (or other similar) race conditions?
How can I solve the problem if every user has its own file/database-record?
How can I solve the problem if I have a single user (filesystem) file that I have to read alter and write again?
Note: This is just an example for understanding. No optimization tipps needed here.
To archive this goal, you need to implement a communication within the two services.
This can be done with a simple queue of operations to process each request in order.
The counter effect is that the request waiting for the queue will have a delayed response (and may occur timeout).
A simple "meta" implementation is:
const operationQueue = new Map();
const eventEmitter = new events.EventEmitter();
router.get('/addUser/:department', function(req, res) {
const customEvent = `addUser-${new Date().getTime()}`;
const done = () => {
res.send('done');
operationQueue.delete(customEvent);
};
eventEmitter.once(customEvent, done);
operationQueue.set(customEvent, () => addUser(customEvent, req));
})
router.get('/deleteUser/:department', function(req, res) {
const customEvent = `deleteUser-${new Date().getTime()}`;
const done = () => {
res.send('done');
operationQueue.delete(customEvent);
};
eventEmitter.once(customEvent, done);
operationQueue.set(customEvent, () => deleteUser(customEvent, req));
})
function addUser(customEvent, req){
// do the logic
eventEmitter.emit(customEvent, {done: true});
}
function deleteUser(customEvent, req){
// do the logic
eventEmitter.emit(customEvent, {done: true});
}
// not the best performance
setInterval(()=>{
const process = operationQueue.shift();
if(process) {
process();
}
}, 1);
Of course, if you'll use tools like a DB or a Redis queue it could fit better than this solution in terms of robustness and failover.
(This is a very broad question.)
Typically, one would use a database (instead of regular text files) and make use its in-built locking mechanisms.
Example of locking mechanisms in the Postgres database management system: https://www.postgresql.org/docs/10/static/explicit-locking.html
I'm trying to use node.js+express.js to build web server.And their are two type of users in my site: normal user and manager. Manager have the right to write data in some field of my database, but normal user can only read them.
So my question is : should I using separated instantiation of express server for manager and normal user ? So they using different port and won't disturb each other?
There are many approaches to setting up user permissions for your backend Node.js server, but setting up separate instances of the server based on the role should not be one of them. There are many packages already out in the NPM ecosystem that could assist you in solving your problem, but depending it may be something as simple as a middleware you can write yourself for Express.js to solve your issue. Example:
export default function permit(...allowed) {
const isAllowed = role => allowed.indexOf(role) > -1;
// return a middleware
return (req, res, next) => {
if (req.user && isAllowed(req.user.role))
next(); // role is allowed, so continue on the next middleware
else {
response.status(403).json({message: "Forbidden"}); // user is forbidden
}
}
}
// Use as so
app.use("/api/private", permit("admin"));
app.use(["/api/foo", "/api/bar"], permit("owner", "employee"));
See middleware.
I am using NodeJs on server and I need to add some basic auth function, which asks the user for the credentials before any file (like html or js files) is downloaded to the user's browser. When I tried to view the cource code when I am not log in, I can't be able to access it like in this.
This works fine with basic-auth module, but only for one username and password.
app.use(basicAuth('username', 'password'));
And there is the problem - I need to load this credentions from Database and check if the user is admin and compare his encrypted password.
I tryed this:
// ****
var basicAuth = require('basic-auth');
// ****
app.use(function (req, res, next) {
var user = basicAuth(req);
if (user) {
var c = mysql.createConnection(db);
var q = "SELECT * FROM user WHERE email = '" + user.name + "' AND admin = 1;";
c.query(q, function (error, result) {
c.end();
if (error) {
console.log(error);
res.set('WWW-Authenticate', 'Basic realm="example"');
return res.status(401).send();
} else {
if (result.length === 1 &&
bcrypt.compareSync(user.pass, result[0].password)) {
return next();
} else {
res.set('WWW-Authenticate', 'Basic realm="example"');
return res.status(401).send();
}
}
});
} else {
res.set('WWW-Authenticate', 'Basic realm="example"');
return res.status(401).send();
}
});
This works fine, but this function is called every time, when the browser communicates with the server, which is about 70 times on loading simple page (It is not asking the user for the credentials again, "only" run this function). This function lasts about 200ms on every request (because of the query and password comparing), in sum the loading lasts about 15 seconds, which is not acceptable.
Can you someone please help me, how to figured out this problem or recommend me some other NodeJs library, which can help me with that?
Thank you.
So if you store you hash in the DB and the Admin Flag in your DB, why don't you do the following Select request :
var q = "SELECT * FROM user WHERE email = '" + user.name + "' AND password = '" + bcrypt.hashSync(user.pass) + "' AND admin = '1';";
If you get only one result it is OK, otherwise it is refused...
If I understand your question, every request to the server is running your authentication, causing page loads to be slow. I don't understand how someone can log in if all the routes are behind the basic auth check you are doing, but that is besides the point.
The way I set my node apps up are to create an array of functions for auth and pass these in to the specific routes where I require authentication, this way it is being run for each resource I may be serving. For example(psuedocode-ish):
// this is the authentication function array, each function in the array
// will be run when this array is called on a route, require next()
// after successful auth to continue on through the array/route handlers
// for your example, adminAuth would be the anonymous
// function in your app.use method
var auth = [adminAuth];
// some random route examples
// this is an unprotected route
app.get('/login', loginHandler);
// these are protected routes and will have auth called
app.get('/api/users', auth, getUserHandler);
app.post('/api/user', auth, postUserHandler);
So in conclusion, you can put the auth call only on protected resources, this way images, html, js, css, etc can be served without running the auth check everytime.
There are a few issues with your code. I recognize that you need to do the password check every request, but you don't necessarily need to hit the database every time. Assuming that you must do HTTP Basic (not recommended anymore, and I hope if you are, that you're doing it over HTTPS), then one major improvement you can make is if you create some kind of fast data cache. The easiest form of this would be in-memory (assuming you're OK with each instance of your app keeping its own copy), but you could also use something like memcached or redis for it instead depending on your use case.
A really trivial implementation would be just keeping an in-memory JS object with a key equal to the username and the value equal to the data returned from the db. Something like:
var userCache = {}; // something local to the module is the easiest way
Then, when you are going to query the db, you just check the userCache first:
if(userCache[user.name]) // check password
else
// do your query and insert a successful result into the userCache for later.
Just doing that should significantly reduce your response times per resource, though again for a production solution you'll probably want multi-tier caching (e.g. an in-memory cache and a redis or memcached cache to help performance across multiple instances of your app).
Second, you should use the callback version of the bcrypt check. It won't be faster on a given request, but it may give your application more overall request capacity, which should help.
Third, your actual SQL query is a bad idea for three reasons. First, in most databases, you'll have worse performance for using a SELECT * vs selecting the specific fields you want. This is both on the DB side (query cache optimization) as well as on the Node side (time to deal with fields getting converted / cast between a string response and a JS object for example). Second, the string concatenation you're doing is an invitation for SQL Injection attacks. Look into using parameterized queries instead. Lastly, if you know your users will be unique, you should also add a LIMIT 1 to the query, since you're only expecting one result and again the DB can do some optimization around that.
I want to completely dissociate my client app from Parse server, to ease the switch to other Baas/custom backend in the future. As such, all client request will point to a node.js server who will make the request to Parse on behalf of the user.
Client <--> Node.js Server <--> Parse Server
As such, I need the node.js server to be able to switch between users so I can keep the context of their authentification.
I know how to authentificate, then keep the sessionToken of the user, and I ve seen during my research than the "accepted" solution to this problem was to call Parse.User.disableUnsafeCurrentUser, then using Parse.User.become() to switch the current user to the one making a request.
But that feels hackish, and I m pretty sure it will, sooner or later, lead to a race condition where the current user is switched before the request is made to Parse.
Another solution I found was to not care about Parse.User, and use the masterKey to save everything by the server, but that would make the server responsible of the ACL.
Is there a way to make request from different user other than thoses two?
Any request to the backend (query.find(), object.save(), etc) takes an optional options parameter as the final argument. This lets you specify extra permissions levels, such as forcing the master key or using a specific session token.
If you have the session token, your server code can make a request on behalf of that user, preserving ACL permissions.
Let's assume you have a table of Item objects, where we rely on ACLs to ensure that a user can only retrieve his own Items. The following code would use an explicit session token and only return the Items the user can see:
// fetch items visible to the user associate with `token`
fetchItems(token) {
new Parse.Query('Item')
.find({ sessionToken: token })
.then((results) => {
// do something with the items
});
}
become() was really designed for the Parse Cloud Code environment, where each request lives in a sandbox, and you can rely on a global current user for each request. It doesn't really make sense in a Node.js app, and we'll probably deprecate it.
I recently wrote a NodeJS application and had the same problem. I found that the combination of Parse.User.disableUnsafeCurrentUser and Parse.User.become() was not only hackish, but also caused several other problems I wasn't able to anticipate.
So here's what I did: I used
Parse.Cloud.useMasterKey(); and then loaded the current user by session ID as if it was a regular user object. It looked something like this:
module.exports = function(req, res, next) {
var Parse = req.app.locals.parse, query;
res.locals.parse = Parse;
if (req.session.userid === undefined) {
res.locals.user = undefined;
return next();
}
Parse.Cloud.useMasterKey();
query = new Parse.Query(Parse.User);
query.equalTo("objectId", req.session.userid);
query.first().then(function(result) {
res.locals.user = result;
return next();
}, function(err) {
res.locals.user = undefined;
console.error("error recovering user " + req.session.userid);
return next();
});
};
This code can obviously be optimized, but you can see the general idea. Upside: It works! Downside: No more use of Parse.User.current(), and the need to take special care in the backend that no conditions occur where someone overwrites data without permission.