Develop Reference

How to safely modify globals / write files in express POST request?

I am writing my first very simple express server for data a collection purpose. This seems like a beginner question but I failed to find an answer so far. The data is very small (less than 500 integers) and will never grow, but it should be able to be changed through POST requests.
I essentially (slightly simplified) want to:
Have the data in a .json file that is loaded when the server starts.
On a POST request, modify the data and update the .json file.
On a GET request, simply send the .json containing the data.
I don't want to use a database for this as the data is just a single small array that will never grow in size. My unclarities are mainly how to handle modifying the global data and file reading / writing safely, i.e. concurrency and how exactly does Node run the code.
I have the following
const express = require('express');
const fs = require('fs');
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
app.post("/", (req, res) => {
const client_data = req.body;
// modify global data
fs.writeFileSync("./data.json", JSON.stringify(data), "utf8");
});
Now I have no idea if or why this is safe to do. For example, modifying the global data variable and writing to file. I first assumed that requests cannot run concurrently without explicitly using async functions, but that seems to not be the case: I inserted this:
const t = new Date(new Date().getTime() + 5000);
while(t > new Date()){}
into the app.post(.. call to try and understand how this works. I then made simultaneous POST requests and they finished at the same time, which I did not expect.
Clearly, the callback I pass to app.post(.. is not executed all at once before other POST requests are handled. But then I have a callback running concurrently for all POST requests, and modifying the global data and writing to file is unsafe / a race condition. Yet all code I could find online did it in this manner.
Am I correct here? If so, how do I safely modify the data and write it to file? If not, I don't understand how this code is safe at all?

Code like that actually opens up your system to race conditions. Node actually runs that code in a single-threaded kind of way, but when you start opening files and all that stuff, it gets processed by multiple threads (opening files are not Node processes, they are delegated to the OS).
If you really, really want to use files as your global data, then I guess you can use an operating system concept called Mutual Exclusions. Basically, its a 'lock' used to prevent race conditions by forcing processes to wait while something is currently accessing the shared resource (or if the shared resource is busy). In Node, this can be implemented in many ways, but one recommendation is to use async-mutex library to handle concurrent connections and concurrent data modifications. You can do something like:
const express = require('express');
const fs = require('fs');
const Mutex = require('async-mutex').Mutex;
// Initializes shared mutual exclusion instance.
const mutex = new Mutex()
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
// Turn this into asynchronous function.
app.post("/", async (req, res) => {
const client_data = req.body;
const release = await mutex.acquire();
try {
fs.writeFileSync('./data.json', JSON.stringify(data), 'utf8');
res.status(200).json({ status: 'success' });
} catch (err) {
res.status(500).json({ err });
finally {
release();
}
});
You can also use Promise.resolve() in order to achieve similar results with the async-mutex library.
Note that I recommend you to use a database instead, as it is much better and abstracts a lot of things for you.
References:
Node.js Race Conditions

Subdomains, redirects and static files with Express

I'm trying to set up a specific configuration with Express and I can't, for the life of me, figure out how to make everything work. I've looked at tons of resources and questions on SO but no dice. I'm hoping if I explain all I'm trying to do someone will be able to point me in the right direction.
Here's what I want:
For sub.domain.com, do some logic and serve static files from /public/sub/;
For domain.com/sub/, rewrite the URL to sub.domain.com then proceed with the same logic as in 1;
For domain.com, serve static files from /public/.
Whatever I do, I run into issues with one of these things...
Here's my current code:
(update based on Stock Overflaw's answer)
const baseUrl = '/bundle/cloud/public/', // This is needed as I'm hosting on NodeChef and that's their folder structure
vhost = require('vhost'), // Also tried with express-subdomain
express = require('express'),
routerMain = express.Router(),
routerSub = express.Router(),
staticOpts = { extensions: ['html', 'htm'] };
//sub.domain.com
routerSub.use((req, res, next) => {
console.log('routerSub called');
next();
});
routerSub.use(express.static(baseUrl + 'sub/', staticOpts));
// domain.com
routerMain.get('/sub', (req, res) => {
res.redirect('https://sub.domain.com/');
});
routerMain.use(express.static(baseUrl, staticOpts));
// Connect routers
app.use(vhost('sub.domain.com', routerSub));
app.use(routerMain);
Here are the results:
✅ domain.com/ 🠞 /public/index.html
✅ domain.com/file 🠞 /public/file.html
✅ domain.com/sub/ 🠞 redirect to sub.domain.com/ (but then, see below)
✅ sub.domain.com/ 🠞 /public/sub/index.html
❌ domain.com/sub/file 🠞 /public/sub/file.html (should redirect to sub.domain.com/file)
When calling sub.domain.com/, I get 4 "routerSub called" logs, which makes sense since there's 1 for the html, and then 1 for css and 2 for js, despite the css and js files not making being read.
The css tag in /public/sub/index.html is <link rel="stylesheet" href="style.min.css">, and /public/sub/style.min.css exists, so I don't understand why it can't find it. Plus, I'm specifying the extension so I shouldn't need to add 'css' to staticOpts.
Update: css and js files weren't served when reaching sub.domain.com/ simply because of my browser's cache. /facepalm
The issue that remains is that when you try to access a file by entering its full path like domain.com/sub/file.html it does serve it, when it should first redirect you to sub.domain.com/file.
That's it...
Anyone know how to help with that? 🙏
Cheers.

while sub.domain.com does get me /public/sub/index.html, I can't get the css and js files that are required for that page
This is probably due to staticOpts = { extensions: ['html', 'htm'] };, you might want to try adding 'css', 'js' to that list.
I feel like it's trying to get static files from /public/
Use some console.log to see which handler is called, if any - and if the previous point I made didn't fix this issue.
domain.com/sub/ does not redirect at all
I think you want to use res.redirect instead of res.location (which only sets the response header, not the HTTP code). And apparently you don't need next().
Also, I couldn't make this setup work with router.use(subdomain('sub', routerSub)), I think it's because router is not itself wrapped in a subdomain call*. However, declaring app.use instad of router.use did the job.
*: pure hypothesis there, but the only way they use "imbricated" routers is when they're doing multi-level subdomains, and they show no root level router at the end (i.e. no router that is not wrapped in a subdomain call).
Here is a working and simplified mock of your domain redirection:
// declare stuff
const express = require('express');
const subdomain = require('express-subdomain');
const router = express.Router();
const routerSub = express.Router();
const app = express();
// make handlers to be tested
routerSub.get('/', (req, res) => {
console.log('routerSub called');
res.send('ok');
});
router.get('/', (req, res, next) => {
console.log('router called');
res.redirect('http://api.myhost.localdomain:3000');
});
// register everything properly
app.use(subdomain('api', routerSub));
app.use(router);
// start server
app.listen(3000);
I set up my /etc/hosts file with myhost.localdomain and api.myhost.localdomain pointing to localhost. (Obviously.)
Hope this helps!
EDIT
Indeed I considered only the subdomain thing, forgot about the routes themselves. This should help:
routerMain.get(['/sub', '/sub/*'], (req, res) => {
res.redirect('https://sub.domain.com/' + (req.params[0] ? req.params[0] : ''));
});
// or, another approach easing a pass-through for the query string as well:
routerMain.get(/^(?:(\/sub\/).+|(\/sub\/?))$/, (req, res) => {
const substr = req.params[0] ? req.params[0].length : req.params[1].length;
res.redirect('https://sub.domain.com/' + req.originalUrl.substr(substr));
});

Connecting Node.js script to a front-end project

As a javascript newbie, I want to create a front-end project with a very little backend solution using Node.js.
I have a user inteface with some variables and a button. These variables have to be passed to a simple .txt file(on my filesystem) and overwrite its content. For this, I'm using a nodejs script:
var fs = require('fs');
fs.writeFile('log.txt', 'This is my text', function (err) {
if (err) throw err;
console.log('Replaced!');
});
But I want this to work onClick, whereas this only works from the command-line.
So I have the two following problems:
I want to update the .txt file with the button click. So basically the above written code has to be executed, when I click the button. But I don't understand, how to connect the front-end with this nodejs file.
I want the content to be dynamical. I'm sure, once the first problem is solved, this will be straightforward, but right now I don't know this either.
I'm 100% sure I'm missing something(maybe a web-server, but even then, how?). I did my research but all I found was either an overkill or I didn't even understand it. Any help would be appreciated, a tutorial, an advice or just a keyword, that I can use for my research.

Have a look at express. It's a "Fast, unopinionated, minimalist web framework for node". With this you can build a small webserver:
var express = require('express');
var app = express();
app.get('/', function (req, res) {
res.send('Hello World');
});
app.listen(3000); // Webserver is running on port 3000
If you run it and got to your browser on http://localhost:3000, it would print Hello World.
Now what you would do is calling your logging function every time a specific URL is requested.
var fs = require('fs');
function log(text, callback) {
fs.writeFile('log.txt', text, callback);
}
app.get('/', function (req, res) {
log('This is my text', function (err) {
if (err) throw err;
res.send('Replaced!');
});
});
Now when you click the button you need to make an AJAX request to the server (maybe using jQuery).

Node.js doesnt have a built in front-library like some other scripting languages such as Python or VB. To create a node.js your intuition was correct in that you will need your app to expose itself as a web-server.
The popular library to do this is called express and it is very easy to get started.
I suggest that you follow the express quickstart tutorial to get into it.
From here you can wrap your code inside a route of the webserver (say /save) for example
var fs = require('fs');
var express = require('express');
var app = express();
app.get('/save', function (req, res) {
fs.writeFile('log.txt', 'This is my text', function (err) {
if (err) throw err;
console.log('Replaced!');
res.send('Replaced!')
});
})
app.listen(3000, () => console.log('Example app listening on port 3000!'))
With this example with the dependencies installed opening localhost:3000/save in your browser would cause your code to be run.

Express app.use refresh behavior

I'm using Express with Node.js and am quite confused about refreshing behavior. Upon refresh of /test, it seems like something is cached server-side when it hits app.use because the array length is nonzero (see sample code below). I would expect the array length to reset to zero since I'm hitting /testagain when I'm refreshing the browser.
Does app.use cache things by default? How does Express middleware work in terms of refresh? I couldn't find anything that explained this clearly in the Express 4.14 docs.
==================
Browser Endpoint: localhost:8000/test
Client:
$.get('/data').done(function(response){...}
Route:
module.exports = function(app) {
app.get('/test', function(req,res) {
var arr =[];
app.use('/data', function(req,res, next) {
console.log(arr.length); // this is nonzero on refresh
arr.push(/* some stuff*/);
res.send({"arr": arr});
}
res.render('test.html')
}
}
Server:
var express = require('express');
var app = express();
require('./routes/route')(app);
app.set('views',__dirname + '/views');
app.set('view engine', 'ejs');
app.engine('html', require('ejs').renderFile);
var server = app.listen(8000, function() {
console.log("server started 8000");
});
app.use(express.static(__dirname + '/public'));

It's not really server caching. It's because you are registering middleware inside a closure and thus those closure variables (like arr) are retained for the next invocation of the middleware. In addition, you're registering the middleware over and over (a new one each time /test is hit).
When you register app.use() inside an app.get() that means that every time you hit the /test route, you will add another duplicate app.use() handler. They will accumulate over time and the same middleware will get run multiple times for for the same request, retaining the previous value for arr from when it was originally registered, but each with their own value for that array.
The general solution here is to NOT register app.use() inside of app.get() because you only want one handler - you don't want them to accumulate.
It's unclear what you're trying to accomplish with your app.use('/data/, ...) middleware. It is clear that your current structure is wrong, but without understanding what you were trying to do with that, it's not clear exactly how it should be written. The usual function of middleware is to be registered once during the initialization of the server and never inside a request handler.
If you're trying to respond to your ajax call:
$.get('/data').done(function(response){...}
Then, you would want an app.get('/data', ...) at the top level of your app module to make that work.
Please explain what the arr.push() is supposed to accomplish for us to help in any more detail.

Nodejs, expressjs - how to serve delayed response

I am building a webservice, for which i am using nodejs, phantomjs and expressjs. I am learning all the three.
I want to serve a delayed response to the clients after processing their query. Like for example,
I am processing certain inputs from my client, then, i want to process the data at the backend which will take approx 10 sec on an avg. Then i wanted to serve this page to the client.
Is it possible in node to send multiple responses to the same request or delayed responses so that the template will automatically update the contents.
Or , should i use the same method , like store the json in a file in the server , then serve the page with ajax which will query the page.
please help me. here is the code which i wrote ,
app-server.js(the main file):
// import express module
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
// define all required template files to be served and also define the template engine
app.engine('.html', require('ejs').__express);
app.set('views', __dirname + '/views');
app.set('view engine', 'html');
// Useful modules
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({extended: true}));
// import the routes
require('./router')(app);
app.listen(8080);
router.js:
var crypto = require('crypto');
var express = require('express');
module.exports = function (app) {
// define the static routes.
app.use('/static', express.static('./static'));
app.use('/media', express.static('./media'));
//defining the controller.
var parserlib = require('./controller.js')
// Define the home root path
app.get('/', function (req, res) {
// shows the home search page.
res.render('index', {content:'template success'});
});
app.get('/search', function(req, res){
res.redirect('/');
});
app.post('/search', parserlib.parserlib);
}
controller.js:
var crypto = require('crypto');
var path = require('path')
var childProcess = require('child_process')
exports.parserlib= function(req, res){
var output = '';
var url = req.body.search_url;
var childArgs = [
path.join(__dirname, 'external-script.js'),
url,
]
// execute the script in a separate thread.
childProcess.execFile(binPath, childArgs, function(err, stdout, stderr) {
// handle results
console.log(stdout);
output = stdout;
//console.log(err);
//res.send(output);
});
//res.send(output);
};
so , what i want to see is, first send a response to client stating that its loading, then i want to update the with processed data. In other languages its not possible to send multiple responses. Not sure about nodejs.
Also, do i have to store the json output from the processed lib to a file and then use ajax to query ? or is it possible to directly update the json object to the client ?
Thanks

This is just not how HTTP works. The clients won't expect it. This has nothing to do with Node or any other framework. The way to do what you're attempting is to actually send a response that the thing is loading, and then have some other mechanism for reporting state.
As an example, you might design a RESTful API. In that RESTful API you might define a endpoint for creating new things:
POST /api/things
The client would post data to that to create a new thing. The response should be something that provides a location of the newly created resource, for example an HTTP 301 to /api/things/1.
If the user goes to /api/things/1 and the thing isn't done getting made yet, then you can either do a temporary redirect (303) to /api/things/1/status which provides some helpful status information, or just issue a 404.
If you actually want to send back server-side pushes of status information, then you should be looking at WebSockets or a pure Socket API of some kind, neither of which is provided by Express, but both of which are available in Node (checkout the socket.io library and the net core library)