I am writing my first very simple express server for data a collection purpose. This seems like a beginner question but I failed to find an answer so far. The data is very small (less than 500 integers) and will never grow, but it should be able to be changed through POST requests.
I essentially (slightly simplified) want to:
Have the data in a .json file that is loaded when the server starts.
On a POST request, modify the data and update the .json file.
On a GET request, simply send the .json containing the data.
I don't want to use a database for this as the data is just a single small array that will never grow in size. My unclarities are mainly how to handle modifying the global data and file reading / writing safely, i.e. concurrency and how exactly does Node run the code.
I have the following
const express = require('express');
const fs = require('fs');
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
app.post("/", (req, res) => {
const client_data = req.body;
// modify global data
fs.writeFileSync("./data.json", JSON.stringify(data), "utf8");
});
Now I have no idea if or why this is safe to do. For example, modifying the global data variable and writing to file. I first assumed that requests cannot run concurrently without explicitly using async functions, but that seems to not be the case: I inserted this:
const t = new Date(new Date().getTime() + 5000);
while(t > new Date()){}
into the app.post(.. call to try and understand how this works. I then made simultaneous POST requests and they finished at the same time, which I did not expect.
Clearly, the callback I pass to app.post(.. is not executed all at once before other POST requests are handled. But then I have a callback running concurrently for all POST requests, and modifying the global data and writing to file is unsafe / a race condition. Yet all code I could find online did it in this manner.
Am I correct here? If so, how do I safely modify the data and write it to file? If not, I don't understand how this code is safe at all?
Code like that actually opens up your system to race conditions. Node actually runs that code in a single-threaded kind of way, but when you start opening files and all that stuff, it gets processed by multiple threads (opening files are not Node processes, they are delegated to the OS).
If you really, really want to use files as your global data, then I guess you can use an operating system concept called Mutual Exclusions. Basically, its a 'lock' used to prevent race conditions by forcing processes to wait while something is currently accessing the shared resource (or if the shared resource is busy). In Node, this can be implemented in many ways, but one recommendation is to use async-mutex library to handle concurrent connections and concurrent data modifications. You can do something like:
const express = require('express');
const fs = require('fs');
const Mutex = require('async-mutex').Mutex;
// Initializes shared mutual exclusion instance.
const mutex = new Mutex()
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
// Turn this into asynchronous function.
app.post("/", async (req, res) => {
const client_data = req.body;
const release = await mutex.acquire();
try {
fs.writeFileSync('./data.json', JSON.stringify(data), 'utf8');
res.status(200).json({ status: 'success' });
} catch (err) {
res.status(500).json({ err });
finally {
release();
}
});
You can also use Promise.resolve() in order to achieve similar results with the async-mutex library.
Note that I recommend you to use a database instead, as it is much better and abstracts a lot of things for you.
References:
Node.js Race Conditions
Related
I am using Expressjs and the Auth0 API for authentication and ReactJs for client side.
Because of the limitations of the Auth0 API (spoke with their team) I am sending updated user details to my backend and then using app.set() to be able to use the req.body in another route.
I need to call the app.patch() route automatically after the app.post() route has been hit.
The end goal is that the users data will be updated and shown client side.
const express = require('express');
const cors = require('cors');
const path = require('path');
const app = express();
require('dotenv').config()
const { auth } = require("express-openid-connect");
app.use(express.json());
app.use(cors());
app.use(express.static(path.join(__dirname, 'build')));
app.use(
auth({
issuerBaseURL: process.env.AUTH0_ISSUER_BASE_URL,
baseURL: process.env.BASE_URL,
clientID: process.env.AUTH0_CLIENT_ID,
secret: process.env.SESSION_SECRET,
authRequired: false,
auth0Logout: true,
})
);
app.get('/', async (req, res) => {
res.sendFile(path.join(__dirname, 'build', 'index.html'));
});
app.get('/api', async (req, res) => {
const stripe = require('stripe')(`${process.env.REACT_APP_Stripe_Live}`);
const invoice = await stripe.invoices.list({
limit: 3,
});
res.json(invoice);
});
app.post('/updateuser', (req, ) => {
app.set('data', req.body);
})
app.patch(`https://${process.env.AUTH0_ISSUER_BASE_URL}/api/v2/users/:id`,(req,res) => {
let val = app.get('data');
req.params = {id: val.id};
console.log(req.params);
})
app.listen(process.env.PORT || 8080, () => {
console.log(`Server listening on 8080`);
});
I'd suggest you just take the code from inside of app.patch() and make it into a reusable function. Then it can be called from either the app.patch() route directly or from your other route that wants to do the same funtionality. Just decide what interface for that function will work for both, make it a separate function and then you can call it from both places.
For some reason (which I don't really understand, but seems to happen to lots of people), people forget that the code inside of routes can also be put into functions and shared just like any other Javascript code. I guess people seems to think of a route as a fixed unit by itself and forget that it can still be broken down into components and those components shared with other code.
Warning. On another point. This comment of yours sounds very wrong:
and then using app.set() to be able to use the req.body in another route
req.body belongs to one particular user. app.set() is global to your server (all user's requests access it). So, you're trying to store temporary state for one single user in essentially a global. That means that multiple user's request that happen to be in the process of doing something similar will trounce/overwrite each other's data. Or worse, one user's data will accidentally become some other user's data. You cannot program a multi-user server this way at all.
The usual way around this is to either 1) redesign the process so you don't have to save state on the server (stateless operations are generally better, if possible) or 2) Use a user-specific session (like with express-session) and save the temporary state in the user's session. Then, it is saved separately for each user and one user's state won't overwrite anothers.
If this usage of app.set() was to solve the original problem of executing a .patch() route, then the problem is solved by just calling a shared function and passing the req.body data directly to that shared function. Then, you don't have to stuff it away somewhere so a later route can use it. You just execute the functionality you want and pass it the desired data.
my scenario id a nodejs + express app connected to a mysql database.
There are some context configurations, valid for all the website and all users, that I need to load from the database at every page request (not at app init because I don't want to reload the entire server when the configuration changes).
My idea was to use a simple middleware to inject the context into the request, like that:
const express = require('express');
const contextSetup = require('./contextSetup');
const app = express();
...
app.use(contextSetup.middleWare);
...
and in contextSetup.js:
const DB = require('./DB');
module.exports.middleWare = (req, res, next) => {
DB.query('select `code`, `data` from `setting` order by `sort`')
.then((rows) => {
const keyVals = {};
rows.forEach((row) => {
keyVals[row.code] = row.data;
});
req.appContext = keyVals;
next();
})
.catch((err) => next(err));
};
The only warning is to use the middleware after app.use(express.static()); otherwise my middleware would be called even for static files request (images, css, js, ...).
Is this approach correct?
Thank you
this works in my book! I might suggest that you put this on a router, so this works only for logged in users, for example. You might also further isolate your routers so that if there are any ajax routes, that these values are not requeried only to be ignored. So the code seems fine, and if performance becomes an issue, just implement a cache which, say, refreshes every half minute or ten requests. But its only one query, so that seems fine for now.
I can use koajs middleware with app.use(function *() { ... }) But, how can I make koajs when the app is launched?
It is trivial to simply code anything in js before all actions, but what if I would like it to perform some async stuff before and outside the middlewares? For example, I might want to obtain a certain key with an API call to an external server, to store it into a variable and return it when I get any request.
As you said, you can simply put it outside any middleware and call app.listen() only when your task is done:
var koa = require('koa');
var app = koa();
// add all your middlewares
loadKeyOrSomethingAsync().then(function() {
app.listen(3000);
});
This way your server will wait your async task to complete before start listening for requests.
I'm trying to extract data from a request (in this case a POST) and am having trouble. I'm doing so using the body-parser module. Below is a portion of my code (note I am using ES6 syntax):
let bodyParser = require('body-parser')
var urlEncodedParser = bodyParser.urlEncoded({extended: true})
app.post('*', setFileMeta, setDirDetails, urlEncodedParser, (req, res, next) => {
async ()=> {
if (!req.stat) return res.send(405, 'File does not exist')
if (req.isDir) return res.send(405, 'Path is a directory') // This is an advanced case
await fs.promise.truncate(req.filePath, 0)
req.pipe(fs.createWriteStream(req.filePath)) // Filepath is a file
// This below line is where I need the body
sendToClients('update', req.url, 'file', req.body, Date.now())
res.end()
}().catch(next)
})
For the actual extraction of the data using body-parser, urlEncoded is the only way I was able to successfully do it (the data is just a string for now), and it's giving me in the format {content: ''} where content is the actual string I'm using. This isn't ideal but it works in this simple. However, this is breaking the createWriteStream(req.filePath) as seen above - the file is created, but there is no content.
There must be something obvious that I'm doing incorrectly, as I'm new to Node and Express. Since I wrote the majority of this with the help of an instructional video, my gut tells me it's the body extraction part since I'm doing that on my own.
body-parser exhausts (fully reads) the request stream in order to parse the incoming parameters, so there's no data left in the request stream to write to your file.
It seems to me that you're trying to implement file uploads. In that case, you probably want to use a module like multer instead of body-parser.
I'm creating a simple testing platform for an app and have the following code setup as my server.js file in the root of my app:
var restify = require('restify'),
nstatic = require('node-static'),
fs = require('fs'),
data = __dirname + '/data.json',
server = restify.createServer();
// Serve static files
var file = new nstatic.Server('');
server.get(/^\/.*/, function(req, res, next) {
file.serve(req, res, next);
});
// Process GET
server.get('/api/:id', function(req, res) {
// NEVER FIRES
});
It serves static files perfectly, however, when I try to make a call to the /api it just hangs and times out. Imagine I'm missing something stupid here, any help would be greatly appreciated.
node-static is calling next with an error, which means it's never yielding to other handlers.
You can move your other handlers above node-static or ignore it's errors by intercepting it's callback.
I made a working version here: http://runnable.com/UWXHRONG7r1zAADe
You may make yourself sure the api get call is caught by moving the second get before the first. The reason is your api calls routes are already matched by the first pattern.