Node.js socket.send( ) functions failing to complete before exit - javascript

In some Node.js scripts that I have written, I notice that even if the last line is a synchronous call, sometimes it doesn't complete before Node.js exits.
I have never seen a console.log statement fail to run/complete before exiting, but I have seen some other statements fail to complete before exiting, and I believe they are all synchronous. I could see why the callback of an async function would fail to fire of course in this case.
The code in question is a ZeroMQ .send() call like so:
var zmq = require('zmq');
var pub = zmq.socket('pub');
pub.bindSync('tcp://127.0.0.1:5555');
setInterval(function(){
pub.send('polyglot');
},500);
The above code works as expected...but if I remove setInterval() and just call it like this:
var zmq = require('zmq');
var pub = zmq.socket('pub');
pub.bindSync('tcp://127.0.0.1:5555');
pub.send('polyglot'); //this message does not get delivered before exit
process.exit(0);
...Then the message will not get delivered - the program will apparently exit before the pub.send() call completes.
What is the best way to ensure a statement completes before exiting in Node.js ? Shutdown hooks would work here, but I am afraid that would just be masking the problem since you can't put everything that you need to ensure runs in a shutdown hook.
This problem can also be demonstrated this way:
if (typeof messageHandler[nameOfHandlerFunction] == 'function') {
reply.send('Success');
messageHandler[nameOfHandlerFunction](null, args);
} else {
reply.send('Failure'); //***this call might not complete before the error is thrown below.***
throw new Error('SmartConnect error: no handler for ZMQ message sent from Redis CSV uploader.');
}
I believe this is a legit/serious problem because a lot of programs just need to publish messages and then die, but how can we effectively ensure all messages get sent (though not necessarily received)?
EDIT:
One (potential) way to fix this is to do:
socket.send('xyz');
socket.close(); // supposedly this will block until the above message is sent
process.exit(0);

Diving into zeromq.node, you can see what Socket.send just pushes your data to _outgoing:
this._outgoing.push([msg, flags]);
... and then calls _flush iff zmq.ZMQ_SNDMORE is unset:
this._flush();
Looks like _flush is actually doing the socket write. If _flush() fails, it emits an error.
Edit:
I'm guessing calling pub.unbind() before exiting, will force the _flush() to be called:
pub.unbind('tcp://127.0.0.1:5555', function(err) {
if (err) console.log(err);
process.exit(0); // Probably not even needed
});

I think the simple answer is the the socket.send() method is in fact asynchronous and that is why we see the behavior I described in the OP.
The question then is - why does socket.send() have to be asynchronous - could there not be a blocking/synchronous version that we could use instead for the purpose intended in the OP? Could we please have socket.sendSync()?

Related

PM2 delete function problem in Node.js API

I got a very annoying problem with PM2's (version 5.1.0) pm2.delete(process, errback) function. After two days of debugging, trying to find the root of the problem, it starts to look like an issue which is related to the PM2 library itself.
Here a simplified code snippet of what I am doing. I will explain the issue afterwards.
const debug = require("debug")("...");
const pm2 = require("pm2");
const Database = require("./Database");
...
class Backend {
static cleanup() {
return new Promise((resolve, reject) => {
pm2.connect((error) => {
if (error) {
debug("...");
reject();
return;
}
// MongoDB query and async forEach iteration
Database.getCollection("...").find({
...
})
.forEach((document) => {
pm2.delete("service_...", (error, process) => {
if (!error && process[0].pm2_env.status === "stopped") {
debug("...");
} else {
debug("...");
}
});
})
.catch(...)
.then(...);
}
});
}
...
}
Now to the problem: my processes do not get terminated and errback of pm2.delete(process, errback) is not executed AT ALL.
I printed the error parameter of the connect callback and it is always null, hence a connection to PM2 is established successfully
I placed debug text directly at the beginning of the delete callback and it is not printed
I wrapped the delete function in a while loop which only stops if the delete callback is executed at least once and the loop runs forever
I started a debug session and noticed that PM2's delete function in node_modules/pm2/lib/API.js (line 533) gets called, but for some reason the process does not get terminated and my provided callback function is not executed at all (I went through the commands step by step in debug mode but still can not tell where it fails to execute the callback (it seems to happen in PM2's API.js though))
I also noticed that when running the code step by step in debug mode with breakpoints that sometimes my process gets terminated with the API call if I cancle the execution at a certain point in between (however, the callback was still not executed)
I use PM2's delete function at another place of my software as well and there it is working like a charm
So for some reason the pm2.delete(process, errback) is not executed correctly and I don't know what to do at this point. Is someone experienced with PM2's source code or had a similar issue at some point? Any advice would be helpful.
It looks like I found the root of the problem:
At a later point in the promise chain after the forEach call, I use pm2.disconnect();. After further investigation I noticed that it is not perfectly synchronized which means in my case that PM2 gets disconnected before the delete function is completely finished. This gives the described results and the weird debugging behaviour.
All in all, the API works therefore perfectly fine but one has to pay very close attention to asynchronous code as it can cause really complicated behaviour which is also hard to debug.
One has to make sure that the delete functions are really finished before pm2.disconnect(); is called.

NodeJS & express catch write function of HTTP module using a Promise

I am writing an express application using
NodeJS v8
express (latest version)
After looking at the onHeaders module and finding out how the module rewrites the HTTP head, I wanted to make use of that function of JavaScript.
I wanted to write a small session system using my SQL server. I am aware of the session module from express, but this module is not able to handle the specific tables and customization, I need.
For convenience reasons I wanted the session to be inserted into the request before the controllers and saved after all controllers finished. (e.g. the writeHead method has been called)
My code in the session looks like:
core = async function(req, res, next) {
res.writeHead = hijackHead(res.writeHead); // Hijack the writeHead method to apply our header at the last
}
//[...](Omitted code)
hijackHead = function(writeFunction) {
let fired = false;
return function hackedHead(statusCode) {
if ( fired ) {
return;
}
//[...](Omitted code)
debug("Session data has changed. Writing");
sessionManager.storeSessionData(session.identifier, session).then(() => { // Promise taking ~60ms to resolve
debug("Finished writing...");
callMe(); // Current idea of calling the writeHead of the original res
});
let that = this, // Arguments to apply to the original writeHead
args = arguments
function callMe() {
debug("Finished! Give control to http, statuscode: %i", statusCode);
writeFunction.apply(that, args); // Call the original writeHead from the response
debug("Sent!")
}
} // End of hackedHead
} // End of hijackHead
The core function is being passed to express as a middleware.
Additionally sessionManager.storeSessionData is a Promise storing data and fulfilling after that, taking ~60ms. The Promise has been testes and works perfectly.
When I now make a request using this Controller, the Node net Module returns the error:
TypeError: Invalid data, chunk must be a string or buffer, not object
at Socket.write (net.js:704:11)
at ServerResponse._flushOutput (_http_outgoing.js:842:18)
at ServerResponse._writeRaw (_http_outgoing.js:258:12)
at ServerResponse._send (_http_outgoing.js:237:15)
at write_ (_http_outgoing.js:667:15)
at ServerResponse.end (_http_outgoing.js:751:5)
at Array.write (/usr/lib/node_modules/express/node_modules/finalhandler/index.js:297:9)
at listener (/usr/lib/node_modules/express/node_modules/on-finished/index.js:169:15)
at onFinish (/usr/lib/node_modules/express/node_modules/on-finished/index.js:100:5)
at callback (/usr/lib/node_modules/express/node_modules/ee-first/index.js:55:10)
Since the new function needs about 30ms to react and return the Promise, the function finishes earlier causing Node to crash.
I already tried blocking the Node loop with a while, timeout or even a recursive function. Neither of them worked.
I tries to simplfy the code as much as possible and I hope that I didn't simplify it too much.
Now I am asking if anybody can help me, how to call the writeHead function properly after the Promise has resolved?
The issue with this is, that net.js directly responds to those methods when writeHead has finished. Even though the head has not been written, it tries to write the body.
Instead it is possible to catch the end()method which will await everything and then close the connection.

Why is it not possible to try/catch some things?

For example I just tried to catch ENOENT by naively doing this:
try {
res.sendFile(path);
} catch (e) {
if (e.code === 'ENOENT') {
res.send('placeholder');
} else { throw e; }
}
This doesn't work!
I know that the right way is to use the error callback of sendFile, but it's really surprising and bad for me that the fundamental language feature of exceptions doesn't work here.
I guess maybe express itself is doing the catching. And they want to not have the errors kill the server so quickly. It's understandable.
But I just get this lame message:
Error: ENOENT: no such file or directory, stat '<file>'
at Error (native)
Not great.
Due to documentation res.sendFile is async function, so try/catch won't work in this case. If you want to handle res.sendFile result you must pass callback as last argument.
res.sendFile(path, function (e) {
if (e) {
if (e.code === 'ENOENT') {
res.send('placeholder');
} else {
throw e;
}
}
});
Its because of the Asynchronous nature of Javascript that the code will not catch the exception you are throwing. The res.sendFile is executed outside the scope of try block and the try block execution will be over after the call to res.sendFile method.
That is the reason why it is always advised to use the callback mechanism with err object as the first argument to the callback and you can check that first before proceeding
res.sendFile(path, function (e) {
// check the error first
// then procedd with the execution
});
Generally speaking, you want to avoid using exceptions to control program flow.
Also, since you're apparently programming this in Node.js, you pass a callback function to keep Node running asynchronously. Node.js is not multithreaded, because JavaScript is not multithreaded, so it cannot have more than one thing going on at a time. That means that if your code is hung up handling an exception, nothing else is happening. And Exceptions are expensive. Doing expensive cleanup in a single-threaded server-side application will harm performance and scalability. Nor is JavaScript itself magically asynchronous. It's asynchronous only if you use callback functions to loosen it up.
So when you pass a callback function to res.send, that function gets called asynchronously when the res.send function finishes (or exits without finishing due to an error), without the overhead of a thrown exception. Thus, if you want to handle errors, passing a callback method is the way to do it.

Node.js Callback Function Error Parameter Explanation?

I have a basic understanding of javascript and have been learning how asynchronous functions work in node.js. I've been very confused by callback functions with the parameter error. For example, here's some code:
contact.saveContacts = function(contactArray, done) {
var jsonfile = require('jsonfile')
jsonfile.writeFile('data.json', contactArray, done)
}
Contact.saveContacts(contacts, function(err) {
console.log('success')
}
My question is, why does the callback function contain the parameter error? I'm confused to why it's there because it seems as if it serves no purpose in the function it calls.
This is a pattern called an error first callback and is used a lot in javascript.
See this article for reference.
Typically synchronous functions either return successfully, possibly with a value, or throw an exception if there is a problem. The calling code can choose what to do if an exception is thrown, by either catching and inspecting the error or by letting it fall through to other code that may handle the error.
Asynchronous callback functions are called after the calling code has already executed. This means there's no opportunity to catch thrown exceptions. So instead of throwing, errors are passed through to the callback function so the calling code can handle both success and error states.
in case there is a problem with write operation such as permissions error object is invoked and the reason in this to prevent unexpected errors.
Imagine we are giving a order to do computer starts doing it but on the way theres a block about permissions that computer cannot write into that dir in this case computer doesnt know what to do and our program crashes to prevent this inside callback we specify what to do in such cases for example if permission is denied and the reason is write permission prompt user for password and force writing or open a box to user that user must run this as user
The error parameter has no usage if everything goes fine. But, its too useful when there comes an error. Any type of error e.g. runtime error, of file has been deleted or anything, if happens, the details of the error will be present in the error parameter of the callback. So, its better to use that parameter as following:
Contact.saveContacts(contacts, function(err) {
if(err){
console.log(err);
}
else{
console.log('success');
}
}
In this way, you will get to know any error, if that happens with the function.

WebSQL transaction works in the console, not in code

I'm trying to write a Sencha Touch 2.0 WebSql proxy that supports tree-data. I started from tomalex0's WebSql/Sqlite proxy. https://github.com/tomalex0
When modifying the script I ran into a strange debugging issue:
(I'm using Chrome 17.0.963.78 m)
The following snipped just got jumped over. The transaction never takes place! But when I set a breakpoint above or below and I run the same code in the console, it does work!
dbConn.transaction(function(tx){
console.log(tx);
if (typeof callback == 'function') {
callback.call(scope || me, results, me);
}
tx.executeSql(sql, params, successcallback, errorcallback);
});
The blue log you can see, the green log is from the success handler. When the query would be performed there would be exactly the same log above (it's a SELECT * FROM ...; so when performing multiple times without changing data I would expect the same result)
I found out that when I add the code block to the watch expressions it also runs.
It isn't being skipped over. It is being scheduled, but not being executed till much later due to the asynchronous nature of the request:
http://ejohn.org/blog/how-javascript-timers-work/
Since the code is being executed synchronously to make the asynchronous call it will delay the call till after the synchronous code has been executed, due to the single threadedness of javascript.

Categories

Resources