Here is the problem I am working on.
The client needs to poll the node server for some data using an API. For the node server to respond, it needs a data set (to be read from DB). I want to avoid reading the database in every poll. How do I maintain the data set across multiple polls?
And if this is possible, would it have any impact on server performance.
db query reduction is always a complicated question. my solution is making db query a promise and cache it till it is expired.
For example:
var cache = {};
var cachedQuery = function(id) {
if(id in cache) return cache[id];
return cache[id] = new Promise(function(resolve, reject) {
db.query('select * from test where id=?', id, function(err, rows) {
delete cache[id];
if(err) reject(err);
else resolve(rows);
}
});
}
Assume that you have 100+ queries at the same time that shares the same request id, those queries will share the same db request. The request is established when the first query comes, and all queries will return the same result when the query completes.
For a more generic usage, you can use a lru-cache to store the promises created, giving it a expriation time.
You can use an in-memory cache. The idea is that before making a trip to the database, you check if the requested value exists in the cache. If it is - serve it. If not - fetch from database, save to cache and return it.
Serving from memory is the fastest you can get. There are existing solutions for node out there, like node-cache.
Related
I have a Node.js server which queries MySQL database. It serves as an api end point where it returns JSON and also backend server for my Express application where it returns the retrieved list as an object to the view.
I am looking into implementing flat-cache for increasing the response time. Below is the code snippet.
const flatCache = require('flat-cache');
var cache = flatCache.load('productsCache');
//get all products for the given customer id
router.get('/all/:customer_id', flatCacheMiddleware, function(req, res){
var customerId = req.params.customer_id;
//implemented custom handler for querying
queryHandler.queryRecordsWithParam('select * from products where idCustomers = ? order by CreatedDateTime DESC', customerId, function(err, rows){
if(err) {
res.status(500).send(err.message);
return;
}
res.status(200).send(rows);
});
});
//caching middleware
function flatCacheMiddleware(req, res, next) {
var key = '__express__' + req.originalUrl || req.url;
var cacheContent = cache.getKey(key);
if(cacheContent){
res.send(cacheContent);
} else{
res.sendResponse = res.send;
res.send = (body) => {
cache.setKey(key,body);
cache.save();
res.sendResponse(body)
}
next();
}
}
I ran the node.js server locally and the caching has indeed greatly reduced the response time.
However there are two issues I am facing that I need your help with.
Before putting that flatCacheMiddleware middleware, I received the response in JSON, now when I test, it sends me an HTML. I am not too well versed with JS strict mode (planning to learn it soon), but I am sure the answer lies in the flatCacheMiddleware function.
So what do I modify in the flatCacheMiddleware function so it would send me JSON?
I manually added a new row to the products table for that customer and when I called the end point, it still showed me the old rows. So at what point do I clear the cache?
In a web app it would ideally be when the user logs out, but if I am using this as an api endpoint (or even on webapp there is no guarantee that the user will log out the traditional way), how do I determine if new records have been added and the cache needs to be cleared.
Appreciate the help. If there are any other node.js caching related suggestions you all can give, it would be truly helpful.
I found a solution to the issue by parsing the content to JSON format.
Change line:
res.send(cacheContent);
To:
res.send(JSON.parse(cacheContent));
I created cache 'brute force' invalidation method. Calling clear method will clear both cache file and data stored in memory. You have to call it after db change. You can also try delete specified key using cache.removeKey('key');.
function clear(req, res, next) {
try {
cache.destroy()
} catch (err) {
logger.error(`cache invalidation error ${JSON.stringify(err)}`);
res.status(500).json({
'message' : 'cache invalidation error',
'error' : JSON.stringify(err)
});
} finally {
res.status(200).json({'message' : 'cache invalidated'})
}
}
Notice, that calling the cache.save() function will remove other cached API function. Change it into cache.save(true) will 'prevent the removal of non visited keys' (like mentioned in comment in the flat-cache documentation.
I am using NodeJs on server and I need to add some basic auth function, which asks the user for the credentials before any file (like html or js files) is downloaded to the user's browser. When I tried to view the cource code when I am not log in, I can't be able to access it like in this.
This works fine with basic-auth module, but only for one username and password.
app.use(basicAuth('username', 'password'));
And there is the problem - I need to load this credentions from Database and check if the user is admin and compare his encrypted password.
I tryed this:
// ****
var basicAuth = require('basic-auth');
// ****
app.use(function (req, res, next) {
var user = basicAuth(req);
if (user) {
var c = mysql.createConnection(db);
var q = "SELECT * FROM user WHERE email = '" + user.name + "' AND admin = 1;";
c.query(q, function (error, result) {
c.end();
if (error) {
console.log(error);
res.set('WWW-Authenticate', 'Basic realm="example"');
return res.status(401).send();
} else {
if (result.length === 1 &&
bcrypt.compareSync(user.pass, result[0].password)) {
return next();
} else {
res.set('WWW-Authenticate', 'Basic realm="example"');
return res.status(401).send();
}
}
});
} else {
res.set('WWW-Authenticate', 'Basic realm="example"');
return res.status(401).send();
}
});
This works fine, but this function is called every time, when the browser communicates with the server, which is about 70 times on loading simple page (It is not asking the user for the credentials again, "only" run this function). This function lasts about 200ms on every request (because of the query and password comparing), in sum the loading lasts about 15 seconds, which is not acceptable.
Can you someone please help me, how to figured out this problem or recommend me some other NodeJs library, which can help me with that?
Thank you.
So if you store you hash in the DB and the Admin Flag in your DB, why don't you do the following Select request :
var q = "SELECT * FROM user WHERE email = '" + user.name + "' AND password = '" + bcrypt.hashSync(user.pass) + "' AND admin = '1';";
If you get only one result it is OK, otherwise it is refused...
If I understand your question, every request to the server is running your authentication, causing page loads to be slow. I don't understand how someone can log in if all the routes are behind the basic auth check you are doing, but that is besides the point.
The way I set my node apps up are to create an array of functions for auth and pass these in to the specific routes where I require authentication, this way it is being run for each resource I may be serving. For example(psuedocode-ish):
// this is the authentication function array, each function in the array
// will be run when this array is called on a route, require next()
// after successful auth to continue on through the array/route handlers
// for your example, adminAuth would be the anonymous
// function in your app.use method
var auth = [adminAuth];
// some random route examples
// this is an unprotected route
app.get('/login', loginHandler);
// these are protected routes and will have auth called
app.get('/api/users', auth, getUserHandler);
app.post('/api/user', auth, postUserHandler);
So in conclusion, you can put the auth call only on protected resources, this way images, html, js, css, etc can be served without running the auth check everytime.
There are a few issues with your code. I recognize that you need to do the password check every request, but you don't necessarily need to hit the database every time. Assuming that you must do HTTP Basic (not recommended anymore, and I hope if you are, that you're doing it over HTTPS), then one major improvement you can make is if you create some kind of fast data cache. The easiest form of this would be in-memory (assuming you're OK with each instance of your app keeping its own copy), but you could also use something like memcached or redis for it instead depending on your use case.
A really trivial implementation would be just keeping an in-memory JS object with a key equal to the username and the value equal to the data returned from the db. Something like:
var userCache = {}; // something local to the module is the easiest way
Then, when you are going to query the db, you just check the userCache first:
if(userCache[user.name]) // check password
else
// do your query and insert a successful result into the userCache for later.
Just doing that should significantly reduce your response times per resource, though again for a production solution you'll probably want multi-tier caching (e.g. an in-memory cache and a redis or memcached cache to help performance across multiple instances of your app).
Second, you should use the callback version of the bcrypt check. It won't be faster on a given request, but it may give your application more overall request capacity, which should help.
Third, your actual SQL query is a bad idea for three reasons. First, in most databases, you'll have worse performance for using a SELECT * vs selecting the specific fields you want. This is both on the DB side (query cache optimization) as well as on the Node side (time to deal with fields getting converted / cast between a string response and a JS object for example). Second, the string concatenation you're doing is an invitation for SQL Injection attacks. Look into using parameterized queries instead. Lastly, if you know your users will be unique, you should also add a LIMIT 1 to the query, since you're only expecting one result and again the DB can do some optimization around that.
Right now I send my requests via ajax to the backend server, which does some operations, and returns a response:
function getData() {
new Ajax().getResponse()
.then(function (response) {
// handle response
})
.catch(function (error) {
// handle error
});
}
The thing is that each time a user refreshes the website, every request is sent again. I've been thinking about caching them inside the local storage:
function getData() {
if (Cache.get('getResponse')) {
let response = Cache.get('getResponse');
// handle response
return true;
}
new Ajax().getResponse()
.then(function (response) {
// handle response
})
.catch(function (error) {
// handle error
});
}
This way if a user already made a request, and the response is cached inside the localStorage, I don't have to fetch data from the server. If a user changes values from the getResponse, I would just clear the cache.
Is this a good approach? If it is, is there a better way to do this? Also, should I cache backend responses the same way? What's the difference between frontend and backend caching?
Is this a good approach? It depends on what kind of data you are storing
Be aware that everything stored on frontend can be changed by the user so this is potential security vulnerability.
This is the main difference between backend and frontend caching, backend caching can't be edited by the user.
If you decide to do frontend caching here is a code how to do it:
localStorage.setItem('getResponse', JSON.stringify(response));
For retrieving stored data from local storage
var retrievedObject = localStorage.getItem('getResponse');
NOTE:
I assume that you are storing object not a string or integer . If you are storing a string, integer, float... Just remove JSON.stringify
The best practice is to use The Cache API "a system for storing and retrieving network requests and their corresponding responses".
The Cache API is available in all modern browsers. It is exposed via the global caches property, so you can test for the presence of the API with a simple feature detection:
I want to completely dissociate my client app from Parse server, to ease the switch to other Baas/custom backend in the future. As such, all client request will point to a node.js server who will make the request to Parse on behalf of the user.
Client <--> Node.js Server <--> Parse Server
As such, I need the node.js server to be able to switch between users so I can keep the context of their authentification.
I know how to authentificate, then keep the sessionToken of the user, and I ve seen during my research than the "accepted" solution to this problem was to call Parse.User.disableUnsafeCurrentUser, then using Parse.User.become() to switch the current user to the one making a request.
But that feels hackish, and I m pretty sure it will, sooner or later, lead to a race condition where the current user is switched before the request is made to Parse.
Another solution I found was to not care about Parse.User, and use the masterKey to save everything by the server, but that would make the server responsible of the ACL.
Is there a way to make request from different user other than thoses two?
Any request to the backend (query.find(), object.save(), etc) takes an optional options parameter as the final argument. This lets you specify extra permissions levels, such as forcing the master key or using a specific session token.
If you have the session token, your server code can make a request on behalf of that user, preserving ACL permissions.
Let's assume you have a table of Item objects, where we rely on ACLs to ensure that a user can only retrieve his own Items. The following code would use an explicit session token and only return the Items the user can see:
// fetch items visible to the user associate with `token`
fetchItems(token) {
new Parse.Query('Item')
.find({ sessionToken: token })
.then((results) => {
// do something with the items
});
}
become() was really designed for the Parse Cloud Code environment, where each request lives in a sandbox, and you can rely on a global current user for each request. It doesn't really make sense in a Node.js app, and we'll probably deprecate it.
I recently wrote a NodeJS application and had the same problem. I found that the combination of Parse.User.disableUnsafeCurrentUser and Parse.User.become() was not only hackish, but also caused several other problems I wasn't able to anticipate.
So here's what I did: I used
Parse.Cloud.useMasterKey(); and then loaded the current user by session ID as if it was a regular user object. It looked something like this:
module.exports = function(req, res, next) {
var Parse = req.app.locals.parse, query;
res.locals.parse = Parse;
if (req.session.userid === undefined) {
res.locals.user = undefined;
return next();
}
Parse.Cloud.useMasterKey();
query = new Parse.Query(Parse.User);
query.equalTo("objectId", req.session.userid);
query.first().then(function(result) {
res.locals.user = result;
return next();
}, function(err) {
res.locals.user = undefined;
console.error("error recovering user " + req.session.userid);
return next();
});
};
This code can obviously be optimized, but you can see the general idea. Upside: It works! Downside: No more use of Parse.User.current(), and the need to take special care in the backend that no conditions occur where someone overwrites data without permission.
I have an ElasticSearch instance running in AWS which I was able to connect to via the JavaScript client in a MeteorJS application. There was no issue creating mappings(indices and analyzers) or updating mappings.
The problem arises whenever there is an index, update or delete request to the instance. After serving above 200 request, the ElasticSearch instance starts throwing request timeout error with code 408. Initially, I thought making multiple single request is the casue, so I decided to do bulk push. Below is the snippet for the bulk push request.
var bulk = SearchService.ElasticQueue.splice(0, 1000);
console.log('Size: ', bulk.length);
if (bulk.length > 0) {
EsClient.bulk({
body: bulk
}, function (error, response) {
if (!error) {
console.log(response);
} else {
console.log(error);
}
});
}
The SearchService.ElasticQueue is a form of queue and a cron job runs frequently to fetch data from it and run bulk requests. I also tried reducing the number of documents in the bulk request and also increased request Timeout in the connection config, but it doesn't seem to help. I would appreciate any suggestion made.
Thanks.
There is only one way that you can use:
wait_for_completion=false
Which will return a Task ID and then you can pull the data using this Task ID