I am doing node.js exercises from nodeschool.io (learnyounode). One of the exercises involves creating a http server which serves a text file from a readable file stream. I'm very new to asynchronous programming. The solution I came up with is:
var http = require('http');
var fs = require('fs');
var readable = fs.createReadStream(process.argv[3]);
var server = http.createServer(function(request, response) {
readable.on('data', function(chunk) {
response.write(chunk);
})
});
server.listen(process.argv[2]);
This works, however the official solution uses a pipe instead of on-data event:
var http = require('http')
var fs = require('fs')
var server = http.createServer(function (req, res) {
res.writeHead(200, { 'content-type': 'text/plain' })
fs.createReadStream(process.argv[3]).pipe(res);
})
server.listen(Number(process.argv[2]))
What are the (potential) differences and/or benefits of doing it either way?
Well, there's more code in your version, and that usually means you have more options to make mistakes. Take into account some edge cases, like what happens when the stream throws an error?
I'm not exactly sure what the behavior would be (you can check yourself by e.g. inserting some non-existing filename) but chances are that in your version the error handling is not working very well, potentially ignoring errors (because you're not listening for error events).
Related
I am writing my first very simple express server for data a collection purpose. This seems like a beginner question but I failed to find an answer so far. The data is very small (less than 500 integers) and will never grow, but it should be able to be changed through POST requests.
I essentially (slightly simplified) want to:
Have the data in a .json file that is loaded when the server starts.
On a POST request, modify the data and update the .json file.
On a GET request, simply send the .json containing the data.
I don't want to use a database for this as the data is just a single small array that will never grow in size. My unclarities are mainly how to handle modifying the global data and file reading / writing safely, i.e. concurrency and how exactly does Node run the code.
I have the following
const express = require('express');
const fs = require('fs');
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
app.post("/", (req, res) => {
const client_data = req.body;
// modify global data
fs.writeFileSync("./data.json", JSON.stringify(data), "utf8");
});
Now I have no idea if or why this is safe to do. For example, modifying the global data variable and writing to file. I first assumed that requests cannot run concurrently without explicitly using async functions, but that seems to not be the case: I inserted this:
const t = new Date(new Date().getTime() + 5000);
while(t > new Date()){}
into the app.post(.. call to try and understand how this works. I then made simultaneous POST requests and they finished at the same time, which I did not expect.
Clearly, the callback I pass to app.post(.. is not executed all at once before other POST requests are handled. But then I have a callback running concurrently for all POST requests, and modifying the global data and writing to file is unsafe / a race condition. Yet all code I could find online did it in this manner.
Am I correct here? If so, how do I safely modify the data and write it to file? If not, I don't understand how this code is safe at all?
Code like that actually opens up your system to race conditions. Node actually runs that code in a single-threaded kind of way, but when you start opening files and all that stuff, it gets processed by multiple threads (opening files are not Node processes, they are delegated to the OS).
If you really, really want to use files as your global data, then I guess you can use an operating system concept called Mutual Exclusions. Basically, its a 'lock' used to prevent race conditions by forcing processes to wait while something is currently accessing the shared resource (or if the shared resource is busy). In Node, this can be implemented in many ways, but one recommendation is to use async-mutex library to handle concurrent connections and concurrent data modifications. You can do something like:
const express = require('express');
const fs = require('fs');
const Mutex = require('async-mutex').Mutex;
// Initializes shared mutual exclusion instance.
const mutex = new Mutex()
let data = JSON.parse(fs.readFileSync('./data.json'));
const app = express();
app.listen(3000);
app.use(express.json());
app.get("/", (req, res) => {
res.sendFile('./data.json', { root: __dirname });
});
// Turn this into asynchronous function.
app.post("/", async (req, res) => {
const client_data = req.body;
const release = await mutex.acquire();
try {
fs.writeFileSync('./data.json', JSON.stringify(data), 'utf8');
res.status(200).json({ status: 'success' });
} catch (err) {
res.status(500).json({ err });
finally {
release();
}
});
You can also use Promise.resolve() in order to achieve similar results with the async-mutex library.
Note that I recommend you to use a database instead, as it is much better and abstracts a lot of things for you.
References:
Node.js Race Conditions
I am currently reading through "Node.js in action" as well as following through a number of online learning resources. One of the first examples in the book is showing how to pipe a stream through to a response. Like so:
var http = require('http'),
fs = require('fs');
http.createServer(function(req, res){
res.writeHead(200, {"Content-Type": "image/png"});
fs.createReadStream("./image.png").pipe(res);
}).listen(xxxx)
My question is how valid is this code? I was under the impression that when ever using the http you should always end with:
res.end();
Is this not necessary as piping it implies an end? When ever writing a response should I always end it?
When your readable stream finishes reading (the image.png file), by default, it emits and end() event, which will call the end() event on the writable stream (the res stream). You don't need to worry about calling end() in this case.
It's worth point out that, in this scenario, your res will no longer be writable after the end() event is called. So, if you want to keep it writable, just pass the end: false option to pipe(), like:
fs.createReadStream("./image.png").pipe(res, { end: false });
, and then call the end() event sometime in the future.
end() is not necessary for streams. There is a great set of tutorials here. One of the exercises (#11 http file server with streams) is to create a static file server. here is what the code looks like:
var fs = require('fs'),
http = require('http'),
port = parseInt( process.argv[2] ),
file = process.argv[3],
opts = { encoding:'utf8' },
server;
server = http.createServer(function(req, res) {
console.log( 'request url: ', req.url );
res.writeHead(200, { 'Content-type':'text/plain' });
var stream = fs.createReadStream( file );
stream.pipe( res );
stream.on('end', function() {
console.log('stream ended...');
});
});
server.listen( port, function() {
console.log('server listening on port: ', port );
});
Lots of other good tutorials and examples as well. Hope this helps.
I saw a similar question, but I'm looking for a way to do it manually. I don't want to use express or another library to do it.
var http = require('http');
var server = http.createServer(function(req, res) {
res.end('<h1 >Hi!</h1>'); //I want to to fetch a file ex: index.html
});
server.listen(9334);
How would i do that? Also as a sub-question, just because I'm curious. Is it possible to use jQuery ajax to fetch this file?
Here is one way to do is using 'fs'.
var http = require('http');
var fs = require('fs');
var server = http.createServer(function(req, res) {
fs.createReadStream("filename.ext").pipe(res);
});
server.listen(9334);
This is also good because if the file is big the data is streamed to the response instead of waiting for the read operation to be completed. Although you might have to set 'Content-Type' header in some cases.
I'm trying to get back from the remote site(URL) in the following node.js codes.
Case 1: Using basic HTTP module + request module
var http = require('http');
var fs = require('fs');
var request = require('request');
var server = http.createServer(function(req, res) {
// Can't get back any response from remote URL
// Can't use 'res' or nothing have in res
request('http://remote.foo.com/app/lists/dosomething.json',
function(err, response, body) {
console.log(body);
}
);
var host = 'http://remote.foo.com';
var app = '/app/lists/dosomething.json?';
var url = 'param1=a¶m2=b';
// I can get something to print out back
request.get(host+app+url).pipe(res);
});`
Case 2: Using Express + request module
var express = require('express')
, http = require('http')
, path = require('path')
, net = require('net')
, url = require('url')
, qs = require('qs');
var request = require('request');
//.....
app.get('/test/foo.json', function(req, res) {
var url_parts = url.parse(req.url, true);
var query = qs.parse(url_parts.query);
var host = 'http://remote.foo.com';
var app = '/path/dosomething.json?';
var url = 'param1=a¶m2=b';
//the following line, the same code above Case 1
//can't get any result, just blank..
request.get(host+app+url).pipe(res);
res.end();
});
//....
Can anyone explain it?
Thanks
Expanding on my comment a little bit:
When you pipe() a readable stream into a writable stream, by default writable.end() is called when the readable stream emits the end event (see documentation). This means that when you pipe streams, you don't have to explicitly close the destination stream by yourself: it is automatically closed when the source closes. This should make sense: when the source stream ends, the destination stream doesn't have anything to do, and should indeed close.
In your second example, you first pipe a readable stream (request) into a writable stream (res), but the very next instruction closes the res stream, without even reading anything, since the first read() call is executed on the next tick (see source). The solution is then to remove res.end(), since it will be handled automatically.
As a side note: if you need to keep the destination stream open for some reason, you can pass {end: false} to pipe.
I just setup a basic node.js server with socket.io on my local machine. Is there a way to set a document root so that you can include other files. Ie. Below I have a DIV with a a background image. The path the image is relative to the location of the server, however this is not working. Any ideas? Thanks!
var http = require('http'),
io = require('socket.io'), // for npm, otherwise use require('./path/to/socket.io')
server = http.createServer(function(req, res){
// your normal server code
res.writeHead(200, {'Content-Type': 'text/html'});
res.end('<div style="background-image:url(img/carbon_fibre.gif);"><h1>Hello world</h1></div>');
});
server.listen(8080);
// socket.io
var socket = io.listen(server);
Use Express or Connect. Examples: https://github.com/spadin/simple-express-static-server, http://senchalabs.github.com/connect/middleware-static.html
For the background-image style, browser will create a entirely new HTTP Request to your server with path *img/carbon_fibre.gif*, and this request will certainly hit your anonymous function, but your response function only write back a div with ContentType: text/html regardless the req.pathname so that the image cannot be properly displayed.
You may add some code to your function like:
var http = require('http'),
io = require('socket.io'),
fs = require('fs'),
server = http.createServer(function(req, res){
// find static image file
if (/\.gif$/.test(req.pathname)) {
fs.read(req.pathname, function(err, data) {
res.writeHead(200, { 'Content-Type': 'image/gif' });
res.end(data);
});
}
else {
// write your div
}
});
server.listen(8080);
I'm not very familiar with nodejs, so the code above only demonstrates a logic but not the actual runnable code block.