my scenario id a nodejs + express app connected to a mysql database.
There are some context configurations, valid for all the website and all users, that I need to load from the database at every page request (not at app init because I don't want to reload the entire server when the configuration changes).
My idea was to use a simple middleware to inject the context into the request, like that:
const express = require('express');
const contextSetup = require('./contextSetup');
const app = express();
...
app.use(contextSetup.middleWare);
...
and in contextSetup.js:
const DB = require('./DB');
module.exports.middleWare = (req, res, next) => {
DB.query('select `code`, `data` from `setting` order by `sort`')
.then((rows) => {
const keyVals = {};
rows.forEach((row) => {
keyVals[row.code] = row.data;
});
req.appContext = keyVals;
next();
})
.catch((err) => next(err));
};
The only warning is to use the middleware after app.use(express.static()); otherwise my middleware would be called even for static files request (images, css, js, ...).
Is this approach correct?
Thank you
this works in my book! I might suggest that you put this on a router, so this works only for logged in users, for example. You might also further isolate your routers so that if there are any ajax routes, that these values are not requeried only to be ignored. So the code seems fine, and if performance becomes an issue, just implement a cache which, say, refreshes every half minute or ten requests. But its only one query, so that seems fine for now.
Related
I am writing my first very simple express server for data a collection purpose. This seems like a beginner question but I failed to find an answer so far. The data is very small (less than 500 integers) and will never grow, but it should be able to be changed through POST requests.
I essentially (slightly simplified) want to:
Have the data in a .json file that is loaded when the server starts.
On a POST request, modify the data and update the .json file.
On a GET request, simply send the .json containing the data.
I don't want to use a database for this as the data is just a single small array that will never grow in size. My unclarities are mainly how to handle modifying the global data and file reading / writing safely, i.e. concurrency and how exactly does Node run the code.
I have the following
const express = require('express');
const fs = require('fs');
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
app.post("/", (req, res) => {
const client_data = req.body;
// modify global data
fs.writeFileSync("./data.json", JSON.stringify(data), "utf8");
});
Now I have no idea if or why this is safe to do. For example, modifying the global data variable and writing to file. I first assumed that requests cannot run concurrently without explicitly using async functions, but that seems to not be the case: I inserted this:
const t = new Date(new Date().getTime() + 5000);
while(t > new Date()){}
into the app.post(.. call to try and understand how this works. I then made simultaneous POST requests and they finished at the same time, which I did not expect.
Clearly, the callback I pass to app.post(.. is not executed all at once before other POST requests are handled. But then I have a callback running concurrently for all POST requests, and modifying the global data and writing to file is unsafe / a race condition. Yet all code I could find online did it in this manner.
Am I correct here? If so, how do I safely modify the data and write it to file? If not, I don't understand how this code is safe at all?
Code like that actually opens up your system to race conditions. Node actually runs that code in a single-threaded kind of way, but when you start opening files and all that stuff, it gets processed by multiple threads (opening files are not Node processes, they are delegated to the OS).
If you really, really want to use files as your global data, then I guess you can use an operating system concept called Mutual Exclusions. Basically, its a 'lock' used to prevent race conditions by forcing processes to wait while something is currently accessing the shared resource (or if the shared resource is busy). In Node, this can be implemented in many ways, but one recommendation is to use async-mutex library to handle concurrent connections and concurrent data modifications. You can do something like:
const express = require('express');
const fs = require('fs');
const Mutex = require('async-mutex').Mutex;
// Initializes shared mutual exclusion instance.
const mutex = new Mutex()
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
// Turn this into asynchronous function.
app.post("/", async (req, res) => {
const client_data = req.body;
const release = await mutex.acquire();
try {
fs.writeFileSync('./data.json', JSON.stringify(data), 'utf8');
res.status(200).json({ status: 'success' });
} catch (err) {
res.status(500).json({ err });
finally {
release();
}
});
You can also use Promise.resolve() in order to achieve similar results with the async-mutex library.
Note that I recommend you to use a database instead, as it is much better and abstracts a lot of things for you.
References:
Node.js Race Conditions
I am using Expressjs and the Auth0 API for authentication and ReactJs for client side.
Because of the limitations of the Auth0 API (spoke with their team) I am sending updated user details to my backend and then using app.set() to be able to use the req.body in another route.
I need to call the app.patch() route automatically after the app.post() route has been hit.
The end goal is that the users data will be updated and shown client side.
const express = require('express');
const cors = require('cors');
const path = require('path');
const app = express();
require('dotenv').config()
const { auth } = require("express-openid-connect");
app.use(express.json());
app.use(cors());
app.use(express.static(path.join(__dirname, 'build')));
app.use(
auth({
issuerBaseURL: process.env.AUTH0_ISSUER_BASE_URL,
baseURL: process.env.BASE_URL,
clientID: process.env.AUTH0_CLIENT_ID,
secret: process.env.SESSION_SECRET,
authRequired: false,
auth0Logout: true,
})
);
app.get('/', async (req, res) => {
res.sendFile(path.join(__dirname, 'build', 'index.html'));
});
app.get('/api', async (req, res) => {
const stripe = require('stripe')(`${process.env.REACT_APP_Stripe_Live}`);
const invoice = await stripe.invoices.list({
limit: 3,
});
res.json(invoice);
});
app.post('/updateuser', (req, ) => {
app.set('data', req.body);
})
app.patch(`https://${process.env.AUTH0_ISSUER_BASE_URL}/api/v2/users/:id`,(req,res) => {
let val = app.get('data');
req.params = {id: val.id};
console.log(req.params);
})
app.listen(process.env.PORT || 8080, () => {
console.log(`Server listening on 8080`);
});
I'd suggest you just take the code from inside of app.patch() and make it into a reusable function. Then it can be called from either the app.patch() route directly or from your other route that wants to do the same funtionality. Just decide what interface for that function will work for both, make it a separate function and then you can call it from both places.
For some reason (which I don't really understand, but seems to happen to lots of people), people forget that the code inside of routes can also be put into functions and shared just like any other Javascript code. I guess people seems to think of a route as a fixed unit by itself and forget that it can still be broken down into components and those components shared with other code.
Warning. On another point. This comment of yours sounds very wrong:
and then using app.set() to be able to use the req.body in another route
req.body belongs to one particular user. app.set() is global to your server (all user's requests access it). So, you're trying to store temporary state for one single user in essentially a global. That means that multiple user's request that happen to be in the process of doing something similar will trounce/overwrite each other's data. Or worse, one user's data will accidentally become some other user's data. You cannot program a multi-user server this way at all.
The usual way around this is to either 1) redesign the process so you don't have to save state on the server (stateless operations are generally better, if possible) or 2) Use a user-specific session (like with express-session) and save the temporary state in the user's session. Then, it is saved separately for each user and one user's state won't overwrite anothers.
If this usage of app.set() was to solve the original problem of executing a .patch() route, then the problem is solved by just calling a shared function and passing the req.body data directly to that shared function. Then, you don't have to stuff it away somewhere so a later route can use it. You just execute the functionality you want and pass it the desired data.
I need strategy on organizing the urls.
Our application has around 150 urls. Some are rather simple without any Route parameters. While others have often more than 1 route parameter in them. For example it could be like this
api/School/Class
api/School/1/Class/2/Student
api/School/Class/revaluate
So in the first one it has no parameter , while second has two parameters and finally third one has 1 but last part is not a resource but an action.
I don't want to store the url just where we would consume it, since maintaining the urls would be technical nightmare. I was hoping if we could have single file api.js or multiple files like api/School.js , api/Teacher.js for storing the files.
In express, you call this a router.
const express = require('express');
const app = express();
const schoolRouter = require('./routers/school');
//...
app.use('/api/school', schoolRouter); // Forwards any requests to the schoolrouter
//...
School.js:
// The router works just like express app (which is also a router)
const schools = require('express').Router();
// we used example.com/api/schools/something to get here
// but the router only cares about what comes next
schools.get('/', function(req, res, next) {
// res.send()
});
// Get a single school (etc, you get the point)
schools.get('/:schoolId', function(req, res, next) {
let id = req.params.schoolId;
// get data from server and res.send()
});
//...
module.exports = schools;
And you can chain routers, but because you only have a partial route, parameters might get lost. So the normal thing is to store parameters to the req object.
schools.use('/:schoolId/classes', function(req, res, next) {
req.schoolId = req.params.schoolId;
next()
}, classesRouter);
This way we can access req.schoolId at any time further down the chain.
So I'm not really sure if the title is descriptive enough, but here is a super simple example.
My site has a public area and a restricted admin area.
example.com/admin (admin home page)
example.com/admin/news (news page)
example.com/admin/posts (posts page)
And because I don't want people who aren't administrators or logged in to be able to access it, I have a simple middleware function to check for cookies.
app.js
const express = require('express');
const app = express();
const authMiddleWere = async (req, res, next) => {
// pseudo-code, do some cookie validity check here
console.log(`Route: ${req.url}`)
if (cookie) {
next();
}
};
const adminRouter = require('./routes/private/home');
const newsRouter = require('./routes/private/news');
const postsRouter = require('./routes/private/posts');
app.use('/admin/', authMiddleWere, adminRouter);
app.use('/admin/news', authMiddleWere, newsRouter);
app.use('/admin/posts', authMiddleWere, postsRouter);
/routes/private/home.js
const express = require('express');
const router = express.Router();
router.get('/', async (req, res, err) => {
res.render('private/home');
});
module.exports = router;
The problem here is that this authMiddleWere function gets called twice when I visit nested paths such as example.com/admin/news which shares the same pattern - it's starting with /admin/......
I can tell that for sure because we are logging the req.url in our middleware function so if I go to example.com/admin it will log out:
Route: /
But if I go to example.com/admin/news it will log out both:
Route: /
Route: /news
So what is causing this and how do I work my way around it? I'm assuming that what I described is the intended behavior of Express.js so I am looking for a way to get around this or (re)structure my code better.
Cheers!
You can use a regex for your route.
app.use(/\/admin$/, authMiddlewear, authRouter);
This will match only routes that end in admin. You may need to handle cases where the route is /admin/ instead of /admin, but iirc, express handles that intelligently.
Well one way you can fix this is by creating a separate route file and splitting everything into a MVC manner. For example:
Inside your main app.js just create a route pointing to the /admin like so:
app.use('/admin', authMiddleWere, require('./src/your-route-to-the-file/admin.route'));
Inside the admin.route file, call your controller like this:
const express = require("express");
const router = express.Router();
const mainAdminCtrl = require("../controllers/admin.controller");
router.get("/news", mainAdminCtrl.adminAuthDisplay);
module.exports = router;
Where the const mainAdminCtrl is your controller and the function adminAuthDisplay is your service.
Essentially, you are splitting your functionality in to a dedicated router, controller and service file. So when you try to access the route /admin, it will look for any suffix inside the router file.
In a case where you want to access the /news endpoint, your API will only make the call once.
If this helps, I can expand my explanation further.
I'm trying to set up a specific configuration with Express and I can't, for the life of me, figure out how to make everything work. I've looked at tons of resources and questions on SO but no dice. I'm hoping if I explain all I'm trying to do someone will be able to point me in the right direction.
Here's what I want:
For sub.domain.com, do some logic and serve static files from /public/sub/;
For domain.com/sub/, rewrite the URL to sub.domain.com then proceed with the same logic as in 1;
For domain.com, serve static files from /public/.
Whatever I do, I run into issues with one of these things...
Here's my current code:
(update based on Stock Overflaw's answer)
const baseUrl = '/bundle/cloud/public/', // This is needed as I'm hosting on NodeChef and that's their folder structure
vhost = require('vhost'), // Also tried with express-subdomain
express = require('express'),
routerMain = express.Router(),
routerSub = express.Router(),
staticOpts = { extensions: ['html', 'htm'] };
//sub.domain.com
routerSub.use((req, res, next) => {
console.log('routerSub called');
next();
});
routerSub.use(express.static(baseUrl + 'sub/', staticOpts));
// domain.com
routerMain.get('/sub', (req, res) => {
res.redirect('https://sub.domain.com/');
});
routerMain.use(express.static(baseUrl, staticOpts));
// Connect routers
app.use(vhost('sub.domain.com', routerSub));
app.use(routerMain);
Here are the results:
✅ domain.com/ 🠞 /public/index.html
✅ domain.com/file 🠞 /public/file.html
✅ domain.com/sub/ 🠞 redirect to sub.domain.com/ (but then, see below)
✅ sub.domain.com/ 🠞 /public/sub/index.html
❌ domain.com/sub/file 🠞 /public/sub/file.html (should redirect to sub.domain.com/file)
When calling sub.domain.com/, I get 4 "routerSub called" logs, which makes sense since there's 1 for the html, and then 1 for css and 2 for js, despite the css and js files not making being read.
The css tag in /public/sub/index.html is <link rel="stylesheet" href="style.min.css">, and /public/sub/style.min.css exists, so I don't understand why it can't find it. Plus, I'm specifying the extension so I shouldn't need to add 'css' to staticOpts.
Update: css and js files weren't served when reaching sub.domain.com/ simply because of my browser's cache. /facepalm
The issue that remains is that when you try to access a file by entering its full path like domain.com/sub/file.html it does serve it, when it should first redirect you to sub.domain.com/file.
That's it...
Anyone know how to help with that? 🙏
Cheers.
while sub.domain.com does get me /public/sub/index.html, I can't get the css and js files that are required for that page
This is probably due to staticOpts = { extensions: ['html', 'htm'] };, you might want to try adding 'css', 'js' to that list.
I feel like it's trying to get static files from /public/
Use some console.log to see which handler is called, if any - and if the previous point I made didn't fix this issue.
domain.com/sub/ does not redirect at all
I think you want to use res.redirect instead of res.location (which only sets the response header, not the HTTP code). And apparently you don't need next().
Also, I couldn't make this setup work with router.use(subdomain('sub', routerSub)), I think it's because router is not itself wrapped in a subdomain call*. However, declaring app.use instad of router.use did the job.
*: pure hypothesis there, but the only way they use "imbricated" routers is when they're doing multi-level subdomains, and they show no root level router at the end (i.e. no router that is not wrapped in a subdomain call).
Here is a working and simplified mock of your domain redirection:
// declare stuff
const express = require('express');
const subdomain = require('express-subdomain');
const router = express.Router();
const routerSub = express.Router();
const app = express();
// make handlers to be tested
routerSub.get('/', (req, res) => {
console.log('routerSub called');
res.send('ok');
});
router.get('/', (req, res, next) => {
console.log('router called');
res.redirect('http://api.myhost.localdomain:3000');
});
// register everything properly
app.use(subdomain('api', routerSub));
app.use(router);
// start server
app.listen(3000);
I set up my /etc/hosts file with myhost.localdomain and api.myhost.localdomain pointing to localhost. (Obviously.)
Hope this helps!
EDIT
Indeed I considered only the subdomain thing, forgot about the routes themselves. This should help:
routerMain.get(['/sub', '/sub/*'], (req, res) => {
res.redirect('https://sub.domain.com/' + (req.params[0] ? req.params[0] : ''));
});
// or, another approach easing a pass-through for the query string as well:
routerMain.get(/^(?:(\/sub\/).+|(\/sub\/?))$/, (req, res) => {
const substr = req.params[0] ? req.params[0].length : req.params[1].length;
res.redirect('https://sub.domain.com/' + req.originalUrl.substr(substr));
});