I'm fairly new to Node.js and am having trouble understanding the way to go about loading libraries or files, in runtime.
Apparently, it is a bad idea to load files in runtime using Node.js's native "require" function because it is blocking i/o, and therefore should not be used from within request handlers. So, I'm assuming something like this is to be avoided:
var http = require('http').createServer(function(req, res) {
var file = require('./pages/'+req.url);
res.end();
}).listen(8080);
So then is there a way to require files in runtime, in a non-blocking/asynchronous way?
I don't think it would always be possible to load files in "boot time" rather than runtime because like in the example above, the only way to know what file to load/require is by getting the name through the req.url property.
So that seems like the only option. Unless, all the files in the folder are preloaded and then called upon by name, in the callback (By using fs.readdirSync or something to iterate through all the files in the folder and compare the gotten files' names to the req.url property), but that seems wasteful. "Preloading" all the files in the folder (maybe around 50 files) and then only using 1 of them, doesn't seem like a good idea. Am I wrong?
Either way, I would just like to know if there is a way to require files in runtime in a better, non-blocking/asynchronous way.
Thank you!
The function require() is generally used for caching modules or configuration files before most of your application runs. You can think of using require() somewhat like this:
var file = fs.readFileSync('/path');
// depending on the file type
eval(file);
JSON.parse(file);
The reason it is done this way is so that dependencies are loaded in order. If you want to read a file after initializing the application, you should use a asynchronous read, and respond in the callback like this:
var http = require('http');
http.createServer(function(req, res) {
fs.readFile('./pages/' + req.url, function(err, data) {
res.end(data);
});
}).listen(8080);
If you needed to check if a file existed, then you could use fs.stat() to check the existence of a file, rather than querying the directory.
var http = require('http');
http.createServer(function(req, res) {
var file = './pages/' + req.url;
fs.stat(file, (err, stats) {
if (err || !stats.isFile()) {
res.writeHead(404);
res.send();
return;
}
fs.readFile(file, function(err, data) {
res.writeHead(200);
res.end(data);
});
});
}).listen(8080);
Related
I am writing my first very simple express server for data a collection purpose. This seems like a beginner question but I failed to find an answer so far. The data is very small (less than 500 integers) and will never grow, but it should be able to be changed through POST requests.
I essentially (slightly simplified) want to:
Have the data in a .json file that is loaded when the server starts.
On a POST request, modify the data and update the .json file.
On a GET request, simply send the .json containing the data.
I don't want to use a database for this as the data is just a single small array that will never grow in size. My unclarities are mainly how to handle modifying the global data and file reading / writing safely, i.e. concurrency and how exactly does Node run the code.
I have the following
const express = require('express');
const fs = require('fs');
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
app.post("/", (req, res) => {
const client_data = req.body;
// modify global data
fs.writeFileSync("./data.json", JSON.stringify(data), "utf8");
});
Now I have no idea if or why this is safe to do. For example, modifying the global data variable and writing to file. I first assumed that requests cannot run concurrently without explicitly using async functions, but that seems to not be the case: I inserted this:
const t = new Date(new Date().getTime() + 5000);
while(t > new Date()){}
into the app.post(.. call to try and understand how this works. I then made simultaneous POST requests and they finished at the same time, which I did not expect.
Clearly, the callback I pass to app.post(.. is not executed all at once before other POST requests are handled. But then I have a callback running concurrently for all POST requests, and modifying the global data and writing to file is unsafe / a race condition. Yet all code I could find online did it in this manner.
Am I correct here? If so, how do I safely modify the data and write it to file? If not, I don't understand how this code is safe at all?
Code like that actually opens up your system to race conditions. Node actually runs that code in a single-threaded kind of way, but when you start opening files and all that stuff, it gets processed by multiple threads (opening files are not Node processes, they are delegated to the OS).
If you really, really want to use files as your global data, then I guess you can use an operating system concept called Mutual Exclusions. Basically, its a 'lock' used to prevent race conditions by forcing processes to wait while something is currently accessing the shared resource (or if the shared resource is busy). In Node, this can be implemented in many ways, but one recommendation is to use async-mutex library to handle concurrent connections and concurrent data modifications. You can do something like:
const express = require('express');
const fs = require('fs');
const Mutex = require('async-mutex').Mutex;
// Initializes shared mutual exclusion instance.
const mutex = new Mutex()
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
// Turn this into asynchronous function.
app.post("/", async (req, res) => {
const client_data = req.body;
const release = await mutex.acquire();
try {
fs.writeFileSync('./data.json', JSON.stringify(data), 'utf8');
res.status(200).json({ status: 'success' });
} catch (err) {
res.status(500).json({ err });
finally {
release();
}
});
You can also use Promise.resolve() in order to achieve similar results with the async-mutex library.
Note that I recommend you to use a database instead, as it is much better and abstracts a lot of things for you.
References:
Node.js Race Conditions
So I'm very new to node.js and javascript, and i made a server that works great by loading up an html file on request. This html file does not contain any of it's own data, it simply sources from the internet and displays some images and text i wrote. I've decided to make the site play an audio file when it is opened. I know this is done easily with the <audio> tag in html5 but the src="" lets me take a file from the computer and place it there, of course when i open the site from another computer the file obviously isn't found and thus isn't played. I figure the audio file must be kept as a variable on the server and passed into the html file's <audio src= > tag. How do i do this? It is an .mp3(but i can get it to any other audio format) file about 30 seconds long. I just want it to play when the site is loaded from another computer(over the internet). Also how would i go about doing the same with pictures or any other data that i don't want to source from the internet but rather keep as data in my server?
var http = require('http');
var fs = require('fs');
var simpleServer = http.createServer(function(request, response){
response.writeHead(200, {"Content-Type":"text/html"});
fs.readFile('./Picture.html', null, function(error, data){
if(error){
response.writeHead(404);
} else{
response.write(data);
}
response.end();
})
});
simpleServer.listen(80, '0.0.0.0', function() {
console.log('Listening to port: ' + 80);
});
console.log("Server running...");
Short Answer
Bypassing using HTML altogether, you can also simply serve the audio file instead of Picture.html:
fs.readFile("./audiofile.mp3", function(error, data) {
if (error) {
response.writeHead(404);
} else {
response.writeHead(200, { "Content-Type": "audio/mpeg"});
response.end(data, 'utf-8');
}
});
Note:
You will have to replace the filename audiofile.mp3 and the content type audio/mpeg to their appropriate values for the file you want to send.
Check Mozilla's Complete List of MIME Types for a full list of file extensions and their associated content types.
Better Answer:
The http module is fairly low-level and is unnecessarily complicated if you're learning.
You can install express.js to your project using the command npm install express --save.
With express your code simplifies to:
const express = require('express');
const app = express();
const port = 80;
app.get('/', (request, response) => {
response.sendFile(__dirname + '/Picture.html');
});
// Anything put in the public folder is available to the world!
app.use(express.static(__dirname + '/public'));
app.listen(port, () => {
console.log(`Listening on port: ${port}`)
});
Then you just have to place all your files into a folder called "public" under your project directory and you can call them from HTML!
So I am just learning and I am trying to build a web app using node, I know I could use express but I am trying to build it all using Node to get a better understanding.
The problem is I cant get the main.js page to load on the app.
I have a router module that builds the page eg
function home(req, res) {
res.writeHead(200, {'Content-Type': 'text/html'});
res.write(fs.readFileSync('./views/header.html'));
res.write('<style>' + fs.readFileSync('./css/styles.css') + '</style>');
res.write(fs.readFileSync('./views/startpage.html'));
res.end(fs.readFileSync('./views/footer.html'));
};
and the script tag is in the footer partial just before the closing body tag, would i need to load it a different way ?
Here is the full app if this is not enough information, thanks
https://github.com/naassi/taxi-log
You probably want a static file server for all your css and javascript files. Instead of sending the CSS file content in home, include
<style>./css/styles.css</style>, and answer the request for *.css files with the appropriate file:
function staticServer(req, res) {
fs.readFile(__dirname + req.url, function (err,data) {
if (err) {
res.writeHead(404);
res.end(JSON.stringify(err));
return;
}
res.writeHead(200);
res.end(data);
});
})
In footer, you want to load a .js file - your Node server is responsible for serving that, just like it handles a request for e.g., home.
If you're learning Node, try to stay away fron Sync functions, they block the event loop, and defeat Node's good parts :)
I would like to do the following inside a client side java script from a file hosted using node and express
var rootURL = <%= "someurlfromserverconfig" %>;
I simply host a web directory from my node app. I have no need for template engines. I just want to access some simple server properties for examples an API URL. ASP and PHP have a similar feature.
Simple things as that are easy to handle with toString and replace:
var url = 'example.com'
app.get('/', function(req, res, next) {
fs.readFile('index.html', function(err, data) {
if (err) return res.sendStatus(500)
res.set('Content-Type', 'text/html')
res.send(data.toString().replace('<%= "someurlfromserverconfig" %>', '"' + url + '"'))
})
})
This would yield: var rootUrl = "example.com";
For caching purposes you might want to read the file into memory and run your replace beforehand instead of on each request, but that's your choice.
To elaborate on the workflow; fs.readFile returns a Buffer that you can run toString() on which then allows you to run replace().
If you are intent on not having to process a template on every request, and if the data you want to include are not going to change on the fly, you might consider ES6 template strings. You could host your code in a file like this:
'use strict';
const config = require('./server-config');
module.exports = `
var rootURL = "${config.rootURL}";
// ...
`;
And then you would require the file in whatever file is handling the routing. The template will only be processed once, even if it is required by multiple files.
Alternatively, you could just use a lightweight template engine, render it once, and then serve it whenever it is requested. If you want to use exactly that format, I would recommend EJS.
'use strict';
const ejs = require('ejs');
const config = require('./server-config');
let template = fs.readFileSync('./some-template.js', 'utf8');
let rendered = ejs.render(template, config);
app.get('/', (req, res) => {
res.send(rendered);
});
If the data you are sending are constantly changing, you will have to render the template every time. Even ASP and PHP have to do that under the hood.
I'm creating a simple testing platform for an app and have the following code setup as my server.js file in the root of my app:
var restify = require('restify'),
nstatic = require('node-static'),
fs = require('fs'),
data = __dirname + '/data.json',
server = restify.createServer();
// Serve static files
var file = new nstatic.Server('');
server.get(/^\/.*/, function(req, res, next) {
file.serve(req, res, next);
});
// Process GET
server.get('/api/:id', function(req, res) {
// NEVER FIRES
});
It serves static files perfectly, however, when I try to make a call to the /api it just hangs and times out. Imagine I'm missing something stupid here, any help would be greatly appreciated.
node-static is calling next with an error, which means it's never yielding to other handlers.
You can move your other handlers above node-static or ignore it's errors by intercepting it's callback.
I made a working version here: http://runnable.com/UWXHRONG7r1zAADe
You may make yourself sure the api get call is caught by moving the second get before the first. The reason is your api calls routes are already matched by the first pattern.