I'm new to node.js although I'm pretty familiar with JavaScript in general. My question is regarding "best practices" on how to handle errors in node.js.
Normally when programming web servers, FastCGI servers or web pages in various languages I'm using Exceptions with blocking handlers in a multi-threading environment. When a request comes in I usually do something like this:
function handleRequest(request, response) {
try {
if (request.url=="whatever")
handleWhateverRequest(request, response);
else
throw new Error("404 not found");
} catch (e) {
response.writeHead(500, {'Content-Type': 'text/plain'});
response.end("Server error: "+e.message);
}
}
function handleWhateverRequest(request, response) {
if (something)
throw new Error("something bad happened");
Response.end("OK");
}
This way I can always handle internal errors and send a valid response to the user.
I understand that with node.js one is supposed to do non-blocking calls which obviously leads to various number of callbacks, like in this example:
var sys = require('sys'),
fs = require('fs');
require("http").createServer(handleRequest).listen(8124);
function handleRequest(request, response) {
fs.open("/proc/cpuinfo", "r",
function(error, fd) {
if (error)
throw new Error("fs.open error: "+error.message);
console.log("File open.");
var buffer = new require('buffer').Buffer(10);
fs.read(fd, buffer, 0, 10, null,
function(error, bytesRead, buffer) {
buffer.dontTryThisAtHome(); // causes exception
response.end(buffer);
}); //fs.read
}); //fs.open
}
This example will kill the server completely because exceptions aren't being catched.
My problem is here that I can't use a single try/catch anymore and thus can't generally catch any error that may be raised during the handling of the request.
Of course I could add a try/catch in each callback but I don't like that approach because then it's up to the programmer that he doesn't forget a try/catch. For a complex server with lots of different and complex handlers this isn't acceptable.
I could use a global exception handler (preventing the complete server crash) but then I can't send a response to the user since I don't know which request lead to the exception. This also means that the request remains unhandled/open and the browser is waiting forever for a response.
Does someone have a good, rock solid solution?
Node 0.8 introduces a new concept called "Domains". They are very roughly analogousness to AppDomains in .net and provide a way of encapsulating a group of IO operations. They basically allow you to wrap your request processing calls in a context specific group. If this group throws any uncaught exceptions then they can be handled and dealt with in a manner which gives you access to all the scope and context specific information you require in order to successfully recover from the error (if possible).
This feature is new and has only just been introduced, so use with caution, but from what I can tell it has been specifically introduced to deal with the problem which the OP is trying to tackle.
Documentation can be found at: http://nodejs.org/api/domain.html
Checkout the uncaughtException handler in node.js. It captures the thrown errors that bubble up to the event loop.
http://nodejs.org/docs/v0.4.7/api/process.html#event_uncaughtException_
But not throwing errors is always a better solution. You could just do a return res.end('Unabled to load file xxx');
This is one of the problems with Node right now. It's practically impossible to track down which request caused an error to be thrown inside a callback.
You're going to have to handle your errors within the callbacks themselves (where you still have a reference to the request and response objects), if possible. The uncaughtException handler will stop the node process from exiting, but the request that caused the exception in the first place will just hang there from the user point of view.
Very good question. I'm dealing with the same problem now. Probably the best way, would be to use uncaughtException. The reference to respone and request objects is not the problem, because you can wrap them into your exception object, that is passed to uncaughtException event. Something like this:
var HttpException = function (request, response, message, code) {
this.request = request;
this.response = response;
this.message = message;
this.code = code || 500;
}
Throw it:
throw new HttpException(request, response, 'File not found', 404);
And handle the response:
process.on('uncaughtException', function (exception) {
exception.response.writeHead(exception.code, {'Content-Type': 'text/html'});
exception.response.end('Error ' + exception.code + ' - ' + exception.message);
});
I haven't test this solution yet, but I don't see the reason why this couldn't work.
I give an answer to my own question... :)
As it seems there is no way around to manually catch errors. I now use a helper function that itself returns a function containing a try/catch block. Additionally, my own web server class checks if either the request handling function calls response.end() or the try/catch helper function waitfor() (raising an exception otherwise). This avoids to a great extent that request are mistakenly left unprotected by the developer. It isn't a 100% error-prone solution but works well enough for me.
handler.waitfor = function(callback) {
var me=this;
// avoid exception because response.end() won't be called immediately:
this.waiting=true;
return function() {
me.waiting=false;
try {
callback.apply(this, arguments);
if (!me.waiting && !me.finished)
throw new Error("Response handler returned and did neither send a "+
"response nor did it call waitfor()");
} catch (e) {
me.handleException(e);
}
}
}
This way I just have to add a inline waitfor() call to be on the safe side.
function handleRequest(request, response, handler) {
fs.read(fd, buffer, 0, 10, null, handler.waitfor(
function(error, bytesRead, buffer) {
buffer.unknownFunction(); // causes exception
response.end(buffer);
}
)); //fs.read
}
The actual checking mechanism is a little more complex, but it should be clear how it works. If someone is interested I can post the full code here.
One idea: You could just use a helper method to create your call backs and make it your standard practice to use it. This does put the burden on the developer still, but at least you can have a "standard" way of handling your callbacks such that the chance of forgetting one is low:
var callWithHttpCatch = function(response, fn) {
try {
fn && fn();
}
catch {
response.writeHead(500, {'Content-Type': 'text/plain'}); //No
}
}
<snipped>
var buffer = new require('buffer').Buffer(10);
fs.read(fd, buffer, 0, 10, null,
function(error, bytesRead, buffer) {
callWithHttpCatch(response, buffer.dontTryThisAtHome()); // causes exception
response.end(buffer);
}); //fs.read
}); //fs.open
I know that probably isn't the answer you were looking for, but one of the nice things about ECMAScript (or functional programming in general) is how easily you can roll your own tooling for things like this.
At the time of this writing, the approach I am seeing is to use "Promises".
http://howtonode.org/promises
https://www.promisejs.org/
These allow code and callbacks to be structured well for error management and also makes it more readable.
It primarily uses the .then() function.
someFunction().then(success_callback_func, failed_callback_func);
Here's a basic example:
var SomeModule = require('someModule');
var success = function (ret) {
console.log('>>>>>>>> Success!');
}
var failed = function (err) {
if (err instanceof SomeModule.errorName) {
// Note: I've often seen the error definitions in SomeModule.errors.ErrorName
console.log("FOUND SPECIFIC ERROR");
}
console.log('>>>>>>>> FAILED!');
}
someFunction().then(success, failed);
console.log("This line with appear instantly, since the last function was asynchronous.");
Two things have really helped me solve this problem in my code.
The 'longjohn' module, which lets you see the full stack trace (across multiple asyncronous callbacks).
A simple closure technique to keep exceptions within the standard callback(err, data) idiom (shown here in CoffeeScript).
ferry_errors = (callback, f) ->
return (a...) ->
try f(a...)
catch err
callback(err)
Now you can wrap unsafe code, and your callbacks all handle errors the same way: by checking the error argument.
I've recently created a simple abstraction named WaitFor to call async functions in sync mode (based on Fibers): https://github.com/luciotato/waitfor
It's too new to be "rock solid".
using wait.for you can use async function as if they were sync, without blocking node's event loop. It's almost the same you're used to:
var wait=require('wait.for');
function handleRequest(request, response) {
//launch fiber, keep node spinning
wait.launchFiber(handleinFiber,request, response);
}
function handleInFiber(request, response) {
try {
if (request.url=="whatever")
handleWhateverRequest(request, response);
else
throw new Error("404 not found");
} catch (e) {
response.writeHead(500, {'Content-Type': 'text/plain'});
response.end("Server error: "+e.message);
}
}
function handleWhateverRequest(request, response, callback) {
if (something)
throw new Error("something bad happened");
Response.end("OK");
}
Since you're in a fiber, you can program sequentially, "blocking the fiber", but not node's event loop.
The other example:
var sys = require('sys'),
fs = require('fs'),
wait = require('wait.for');
require("http").createServer( function(req,res){
wait.launchFiber(handleRequest,req,res) //handle in a fiber
).listen(8124);
function handleRequest(request, response) {
try {
var fd=wait.for(fs.open,"/proc/cpuinfo", "r");
console.log("File open.");
var buffer = new require('buffer').Buffer(10);
var bytesRead=wait.for(fs.read,fd, buffer, 0, 10, null);
buffer.dontTryThisAtHome(); // causes exception
response.end(buffer);
}
catch(err) {
response.end('ERROR: '+err.message);
}
}
As you can see, I used wait.for to call node's async functions in sync mode,
without (visible) callbacks, so I can have all the code inside one try-catch block.
wait.for will throw an exception if any of the async functions returns err!==null
more info at https://github.com/luciotato/waitfor
Also in synchronous multi-threaded programming (e.g. .NET, Java, PHP) you can't return any meaningful information to the client when a custom unkown Exception is caught. You may just return HTTP 500 when you have no info regarding the Exception.
Thus, the 'secret' lies in filling a descriptive Error object, this way your error handler can map from the meaningful error to the right HTTP status + optionally a descriptive result. However you must also catch the exception before it arrives to process.on('uncaughtException'):
Step1: Define a meaningful error object
function appError(errorCode, description, isOperational) {
Error.call(this);
Error.captureStackTrace(this);
this.errorCode = errorCode;
//...other properties assigned here
};
appError.prototype.__proto__ = Error.prototype;
module.exports.appError = appError;
Step2: When throwing an Exception, fill it with properties (see step 1) that allows the handler to convert it to meannigul HTTP result:
throw new appError(errorManagement.commonErrors.resourceNotFound, "further explanation", true)
Step3: When invoking some potentially dangerous code, catch errors and re-throw that error while filling additional contextual properties within the Error object
Step4: You must catch the exception during the request handling. This is easier if you use some leading promises library (BlueBird is great) which allows you to catch async errors. If you can't use promises than any built-in NODE library will return errors in callback.
Step5: Now that your error is caught and contains descriptive information about what happens, you only need to map it to meaningful HTTP response. The nice part here is that you may have a centralized, single error handler that gets all the errors and map these to HTTP response:
//this specific example is using Express framework
res.status(getErrorHTTPCode(error))
function getErrorHTTPCode(error)
{
if(error.errorCode == commonErrors.InvalidInput)
return 400;
else if...
}
You may other related best practices here
Related
fs.readFileSync(process.env.id_to_name, 'utf-8', function(err, data) {
if (err) throw err;
/*
a lot of stuff
*/
fs.mkdirSync(`clips`);
fs.writeFileSync(`clips/recap.json`, '{"players":[]}', 'utf8');
});
fs.readFileSync(`clips/recap.json`, 'utf-8', function(err, data) {
var info = JSON.parse(data);
info.players.push(/* stuff */);
fs.writeFileSync(`clips/recap.json`, JSON.stringify(info), 'utf8', function (err) { });
});
I don't know what I'm doing wrong here.
The second fs.readFileSync just doesn't wait for the first one to end so it doesn't find the file he should read.
You're using fs.readFileSync() incorrectly. It does not accept a callback as an argument and does not call a callback. See doc here.
I don't know if you meant to show us fs.readFile() that does accept a callback or not.
fs.readFileSync() returns its result directly (in a synchronous fashion) from the function call as in:
let data = fs.readFileSync(someFileName, someEncoding);
It does not use a callback and it throws an exception if there's an error reading the file.
If you meant to show us an example using fs.readFile(), then it's a non-blocking, asynchronous call. If you want your second file read to wait until the first one is done, you would have to put the second file read INSIDE the completion callback of the first.
Also, please never write code like if (err) throw err; inside an asynchronous callback. That's a pointless piece of code that does nothing useful as nothing can catch that throw and you have no way to communicate back errors. It is most unfortunate that nodejs documentation shows that regularly in its examples as real world code should never do that. You will need to write real error handling where you either handle the error in some way and continue or you abort the process or you communicate back the error (probably with a callback) so the calling code can handle and see the error. Exceptions throw in asynchronous callbacks do NOT propagate back to the caller. They end up going back to the bowels of the file system code where the callback was triggered from where you cannot catch or handle them.
If you really mean to be using all synchronous calls, then you would write your code like this:
try {
let data1 = fs.readFileSync(process.env.id_to_name, 'utf-8');
// other code here
fs.mkdirSync(`clips`);
fs.writeFileSync(`clips/recap.json`, '{"players":[]}', 'utf8');
let data2 = fs.readFileSync(`clips/recap.json`, 'utf-8');
var info = JSON.parse(data2);
info.players.push(/* stuff */);
fs.writeFileSync(`clips/recap.json`, JSON.stringify(info));
} catch(err) {
// handle errors here
console.log(err);
}
Note that this code can likely only be run once without error because fs.mkdirSync('clips') will fail unless you set the recursive flag.
Hint, you can use require() to load and parse a JSON file in one step so you don't have to read it and then parse it into a Javascript object.
A short preface: I am currently learning Socket.io, coming from Asyncio in Python. I am making multiclient socket servers. In Asyncio with Python, I am used to making a separate class file for my client. So in the event of any errors, the client's specific class only has the error, and it doesn't affect the rest of the server.
Now, I understand classes are sort of "looked down" upon in Node, so I am unsure how to structure this Socket.io chat program. So far, I have the following code:
server.on("connection", (socket) => {
clientLog(socket.handshake.address, "Client connected.");
socket.on("auth", (auth) => {
if(valid(auth['token']))
{
var clientBase = {
"index" : clientsIndex++,
"addr" : socket.handshake.address,
"token" : auth['token'],
}
clients[socket] = clientBase;
clientLog(clients[socket]['addr'], "Client verified.");
}
else
{
log(socket.handshake.address, "Client failed verification!");
socket.disconnect();
}
});
socket.on("message", (message) => {
handleMessage(socket, message);
});
socket.on("disconnect", () => {
if(socket in clients)
{
delete clients[socket];
}
console.log("Client Disconnected!");
});
});
There is more at the top of the file, but it is not very significant. Basically, this is the start of my simple chat messenger program. Now, there is an issue: when errors come up, the entire program dies. This is different from Python/Asyncio, because in Python, when errors show up, unless they are fatal, the program continues to run, and only the client is disconnected. In Socket.io, the entire program seems to die as soon as the smallest error comes up.
Errors are inevitable, and this chat messenger is planned to be used by a significant amount of people. Therefore, I was wondering if I can restructure this program in a way where any unhandled errors are just logged or something, and the program continues to run. I understand I can do this with unhandledException, but I also understand unhandledException is unethical, and should be used as a last-resort method.
With that, I was wondering if my program has some fundamental problem in its structure. If so, how should I restructure this program to better handle multiple clients? This is just the base of it, and I can restructure if necessary.
It is just a simple Socket.io program, which passes around messages (and will eventually use MySQL to authenticate users).
What do I do?
I don't think your issue is directly related to Node.js or Socket.io. I'd say you should start using exception handling with try..catch blocks (optionally with finally statement if necessary). This is a generic technique present in every modern programming language.
See: https://nodejs.org/en/knowledge/errors/what-is-try-catch/
An alternative to try/catch suggested by the same page above:
But wait, isn't it Node.js convention to not use try-catch?
In the core Node.js libraries, the only place that one really needs to use a try-catch is around JSON.parse(). All of the other methods use either the standard Error object through the first parameter of the callback or emit an error event. Because of this, it is generally considered standard to return errors through the callback rather than to use the throw statement.
According to this alternative technique you should start your callbacks with:
if (error){
//error handling code goes here
}
Which is often seen in Node.js codes.
I found a good example here:
https://stackify.com/node-js-error-handling/ at the Handling async (callback) errors section.
var request = require('request'); //http wrapped module
function requestWrapper(url, callback) {
request.get(url, function (err, response) {
if (err) {
callback(err);
} else {
callback(null, response);
}
})
}
Can u check on nodemon? Its a package which is used in dev environments to keep the server running in case of run time errors. Use it along with pm2 to run the server in the background.
Im' trying to understand using promises with Google Cloud Functions a bit better. I just learned about the 'finally' method on promises, which is called after all promises in the chain are fully resolved or rejected. In a http function is it good practice to put response.send() inside of the finally method?
The below code uses request-promise-native for the http request. In the first .then() I call parseSchedule, which uses the cheerio web scraping api to loop through some data and on a website, and add it to the scheduledGames array (synchronously, I think).
I return from that and the then log that data to the console in writeDB, but one thing I noticed is that I see response.send() log 'execution finished' before I see the data from scheduleGames in the log. Is that correct?
Should I be using the 'finally' block like this?
Thanks,
const options = {
uri: 'https://www.cbssports.com/nba/schedule/' + urlDate,
Connection: 'keep-alive',
transform: function (body) {
return cheerio.load(body);
}
};
return request(options)
.then(parseSchedule)
.then(writeSchedule)
.catch((err) => console.log("there was an error: " + err))
.finally(res.send("execution finished"));
function parseSchedule($){
const scheduledGames = [];
$('tbody').children('tr').each((i, element) => {
const gameTime = $(element).children('td').eq(2).find('a').text()
const scheduledGame = { gameTime: gameTime};
scheduledGames.push(scheduledGame);
});
return scheduledGames;
}
function writeDB(scheduledGames){
console.log(scheduledGames);
}
}
It typically makes more sense to send a success response at the time in the promise chain when everything is successful, or send an error response in a catch handler. If you do these two things, it doesn't make sense to use finally at all, since success and error are the only two cases you really need to handle. Unless you have some special case, stick to just success and error.
I have the following code
try {
for (const customer of customers)
{
followUser(ig, customer.trim());
}
} catch (err){
console.log('ERROR: ' + " " + err);
}
followUser is an API call.. which sometimes out of the blue can return 400 or 404 errors when server has issues.. how do I break out of the for loop when one of the followUser generates an error?
The problem with procedural loops and async calls is that you'll often try to access the data you're supposed to get before it's even fetched.
You might want to look at Promises, which are objects storing your request and that automatically change when the request ends. You can use multiple functions including Promise.all() which calls a function for each element you supply. You can then catch the error if one of them fails and handle the error the way you want.
More info here : Promise, Promise.all()
Since followUser is an API call (so an asynchronous operation), I believe you should be using promises to make sure the flow goes alright. The Promise library should have something you can use.
This question on StackOverflow might be helpful for you. :)
Try this code:
for (const customer of customers) {
try {
followUser(ig, customer.trim());
} catch (err) {
console.log('ERROR: ' + err);
break;
}
}
I am having an issue where Intellij is warning me about 'throw' of exception caught locally. After doing some digging on why this is not ok it makes sense, errors should not be used for flow control. The problem I am facing though is in async I cannot reject the promise without throw something locally and I get the warning.
Some of my example code.
Top level takes in the request and awaits for the response from the controller:
Router.post("/", async (req, res) => {
try {
let itemController = new ItemController(req.body);
let response = await itemController.request();
res.send(response);
} catch (error) {
res.status(500).send({ error: error});
}
});
The controller takes in the request and awaits on other functions to get some data.
async request() {
try {
await isValidItem();
return await this.initialize();
} catch(error) {
throw error;
}
}
Then I have a function which gets manufacturer ID for this item and I run into my problem. If the SQL query doesn't fail and nothing is in the response I need to throw a local error so that the request can fail gracefully. and send a proper 500 error to the client.
async queryManufacturerID() {
try {
let result = await this.queryManufacturerID(this.itemID, this.brand);
if (result === false) {
throw new Error("this item has no manufacturer ID");
} else {
this.manufacturerID = result["manufacturerItemID"];
}
} catch (error) {
throw error;
}
}
My problem is I know I can adjust this so other functions that get a reply from this can know that this function failed without a error but that would have to happen in this entire chain to prevent throwing locally. Seems like a lot of bloat.
The only thing that makes this code bloaty and the IDE complain is not throwing errors, but this:
try {
//...
} catch (error) {
throw error;
}
It's a no op. You can safely remove it without changing the logic.
The only case where you should use catch is when you actually plan to handle the error and get the execution back on track.
After doing some digging on why this is not ok it makes sense, errors should not be used for flow control
I disagree. Errors are a part of proper flow control, they allow you to handle unplanned things. Throwing an error if something unexpected occured makes sense, also in this case.