Develop Reference

Node.js flat-cache, when to clear caches

I have a Node.js server which queries MySQL database. It serves as an api end point where it returns JSON and also backend server for my Express application where it returns the retrieved list as an object to the view.
I am looking into implementing flat-cache for increasing the response time. Below is the code snippet.
const flatCache = require('flat-cache');
var cache = flatCache.load('productsCache');
//get all products for the given customer id
router.get('/all/:customer_id', flatCacheMiddleware, function(req, res){
var customerId = req.params.customer_id;
//implemented custom handler for querying
queryHandler.queryRecordsWithParam('select * from products where idCustomers = ? order by CreatedDateTime DESC', customerId, function(err, rows){
if(err) {
res.status(500).send(err.message);
return;
}
res.status(200).send(rows);
});
});
//caching middleware
function flatCacheMiddleware(req, res, next) {
var key = '__express__' + req.originalUrl || req.url;
var cacheContent = cache.getKey(key);
if(cacheContent){
res.send(cacheContent);
} else{
res.sendResponse = res.send;
res.send = (body) => {
cache.setKey(key,body);
cache.save();
res.sendResponse(body)
}
next();
}
}
I ran the node.js server locally and the caching has indeed greatly reduced the response time.
However there are two issues I am facing that I need your help with.
Before putting that flatCacheMiddleware middleware, I received the response in JSON, now when I test, it sends me an HTML. I am not too well versed with JS strict mode (planning to learn it soon), but I am sure the answer lies in the flatCacheMiddleware function.
So what do I modify in the flatCacheMiddleware function so it would send me JSON?
I manually added a new row to the products table for that customer and when I called the end point, it still showed me the old rows. So at what point do I clear the cache?
In a web app it would ideally be when the user logs out, but if I am using this as an api endpoint (or even on webapp there is no guarantee that the user will log out the traditional way), how do I determine if new records have been added and the cache needs to be cleared.
Appreciate the help. If there are any other node.js caching related suggestions you all can give, it would be truly helpful.

I found a solution to the issue by parsing the content to JSON format.
Change line:
res.send(cacheContent);
To:
res.send(JSON.parse(cacheContent));
I created cache 'brute force' invalidation method. Calling clear method will clear both cache file and data stored in memory. You have to call it after db change. You can also try delete specified key using cache.removeKey('key');.
function clear(req, res, next) {
try {
cache.destroy()
} catch (err) {
logger.error(`cache invalidation error ${JSON.stringify(err)}`);
res.status(500).json({
'message' : 'cache invalidation error',
'error' : JSON.stringify(err)
});
} finally {
res.status(200).json({'message' : 'cache invalidated'})
}
}
Notice, that calling the cache.save() function will remove other cached API function. Change it into cache.save(true) will 'prevent the removal of non visited keys' (like mentioned in comment in the flat-cache documentation.

Error creating json file in node js

I have followed many solutions provided in the previous questions but mine is not working. The problem is in .json extension. Whenever I use filename.json, the app will crash with ERR_CONNECTION_RESET but successfully created an empty .json file. However, if I change the extension to filename.txt, the fs.writeFile will successfully create the filename.txt with the data inside and the app will work as expected. Did I miss any configuration here to create the JSON file?
Here is the example code I used.
var jsonData = '{"persons":[{"name":"John","city":"New York"},{"name":"Phil","city":"Ohio"}]}';
// parse json
var jsonObj = JSON.parse(jsonData);
console.log(jsonObj);
// stringify JSON Object
var jsonContent = JSON.stringify(jsonObj);
console.log(jsonContent);
fs.writeFile("./public/output.json", jsonContent, 'utf8', function(err) {
if (err) {
console.log("An error occured while writing JSON Object to File.");
return console.log(err);
}
console.log("JSON file has been saved.");
});

So, ERR_CONNECTION_RESET means that the connection was closed midway. My guess, as in the comments, would be that it's a reloading server.
Try using --ignore public/**/*.json and it should work.

Update a JSON file in AngularJS

I've got some data from a JSON file, which I use in my HTML getting it first from AngularJS like this:
$http.get('js/data.json').success(function(data) {
$scope.data = data;
});
And I want to update this JSON file after clicking a button in the HTML:
<button ng-click="postData(id)">Post</button>

You cannot write on files via JavaScript only (AngularJS).
You are to go via server side and point your "post" request to a server side script (i.e: PHP) and make that script do the job.

This sort of thing won't work. The file you are trying to write to would be on a server; and as it is right now, it would be a static resource. I'd suggest reading up on Angular resources, here. You can set up your server-side code to perform CRUD operations on the json file, but an actually database would be best. If you prefer to use a json format, Mongodb is your best choice; here is a link to Mongodb University, which offers free courses. I've done it in the past, and it's been great.
Now, for some actually help in your situation:
You can perform a GET request on your json file because it's seen as a static resource. The POST request, however, needs server-side scripting to do anything.
$http.get('api/YOUR_RESOURCE').success(function(data) {
$scope.database = data;
});
$http.post('api/YOUR_RESOURCE', {
data_key: data_value,
data_key2: data_value2
}).success(function(data) {
data[id].available = false;
});
This may be further ahead on your path to learning Angular, but here is a snippet of Node.js server code, with a Mongo database and Mongoose to handle the 'Schema', to help you get an idea of how this works:
var mongoose = require('mongoose'),
YOUR_RESOURCE = mongoose.model('YOUR_RESOURCE');
app.route('/api/YOUR_RESOURCE')
// This should be your GET request; 'api/
.get(
// Get all docs in resource
YOUR_RESOURCE.find().exec(function (err, data) {
if (err) {
return res.status(400).send({
message: SOME_ERROR_HANDLER
});
} else {
res.json(data); // return list of all docs found
}
});)
// Add new doc to database
.post(function (req, res) {
// The keys of the object sent from your Angular app should match
// those of the model
var your_resource = new YOUR_RESOURCE(req.body);
your_resource.save(function (err) {
if (err) {
return res.status(400).send({
message: SOME_ERROR_HANDLER
});
} else {
// returns newly created doc to Angular after successful save
res.json(your_resource);
}
});
);
Here is an SO page with a list of resources on getting started with Node; I recommend Node because of it's ease of use and the fact that it is written in JS. The Mongo University lessons also go through setting up you server for use with the database; you can choose between several flavors, such as Java, .NET, Python or Node.
There is a bit left out in the examples above, such as the Mongoose model and Node setup, but those will be covered in the resources I've linked to on the page, if you choose to read them. Hope this helps :)

How to get documents from CouchDB database in Node.js

I'm developing a simple application using CouchDB and Node.js. I have few documents in my database and created a view to retrieve the documents. How can I get the view from CouchDB in my javascript file?
app.get('/find', function(req, res){
var cradle = require('cradle');
var connection = new(cradle.Connection)('http://localhost:5984/', 443);
var db = connection.database('db');
db.view('_design/usersView/users', function (err, response) {
console.log(response);
});
The response says: undefined

Try querying the view like this
db.view('usersView/users', function (err, response) {
console.log(response);
});
You don't need a _design/ with cradle (although you need that if you query it with default couchdb api)
Also I think it would be better if you moved the connection and the require statements out of the the controller.

I solved the problem. Just sent http get request to couchDB api to retrieve the view with this command:
db.get('_design/usersView/_view/users', function (err, doc) {
res.end(JSON.stringify(doc));
});

Updating document within find

I'm having issues updating a document from within a find using Mongoose. The issue is only when I attempt to overwrite the document with an object (e.g doc = req.body). I am however able to directly set properties of the original document to a specific string (e.g. doc.name = 'jason borne';).
I've verified that res.body is an object, so I don't see why I'm unable to set it this way.
Client.findById(req.params.client_id, function (err, client) {
if (err)
return next(new restify.InternalError(err));
// client.name = 'jason borne';
client = req.body;
client.save(function(err) {
if (err)
return next(new restify.InternalError(err));
res.send(client);
});
});
When attempting to set the doc to an object, I receive the error:
TypeError: Object # has no method 'save'
I'm aware that I can do an update with a simple Client.update(...) command, however this method does not allow my schema middleware or validation to run (which is notated in the Mongoose documentation).
Any thoughts? I'm new to Node, and Mongoose.

You need to use something like underscore's extend method to copy the properties of req.body into the client object instead of just re-pointing client to req.body as you are now.
var _ = require('underscore');
Client.findById(req.params.client_id, function (err, client) {
if (err)
return next(new restify.InternalError(err));
_.extend(client, req.body);
client.save(function(err) {
if (err)
return next(new restify.InternalError(err));
res.send(client);
});
});

The symptoms you're now getting are caused by the fact that you're replacing a mogoose model object (with methods like save, find, etc), by a simple json object parsed from your body, which is missing save.
Try doing an update instead of find/save.
Client.update({_id: req.params.client_id}, {$set : req.body}, function(err) {...});
Or try to merge your req.body to the client object.