Why would a NodeJS/restify server *rarely* report EPERM on accept? - javascript

I'm running a restify server in NodeJS. On very rare occasions, on the order of 0.05% of HTTPS requests cause net.js to report the following error:
Error: accept EPERM
at exports._errnoException (util.js:742:11)
at TCP.onconnection (net.js:1280:24)
There is nothing special about the HTTP requests. Before this error is reported, the server may have serviced thousands of requests and even responded to dozens of identical requests. I have not been able to find any information about why a server might generate an EPERM error for a socket that has been successfully accepting connections for several hours.
By the way, this error occurs outside of any execution context of our source code. So it's not as if the EPERM is about our code accessing a file or performing some other system call. The EPERM is happening deep within the NodeJS TCP code when the new request arrives and before our code is invoked.
At first, when the error occurred it would cause NodeJS to terminate. So then I added code to catch application-level exceptions:
process.on("uncaughtException", onUncaughtException );
But since I don't know why this error is happening, it's not at all clear what is the recovery process.
Not sure if it will matter, but here is most of the code relevant to starting up the restify service:
var restify = require("restify");
// skipping some other init code
// configuration data is read from a JSON file
var serverOptions = {
name: configuration.server.name,
version: configuration.server.version,
formatters: {
"application/json": jsonResponseFormatter,
"text/html": textResponseFormatter
},
serverOptions.key: fs.readFileSync(configuration.server.sslKey),
serverOptions.cert: fs.readFileSync(configuration.server.sslCert)
}
var server = restify.createServer( serverOptions );
// skipping middleware inits and URL registrations
server.listen(
configuration.server.port, // using HTTPS 443
configuration.server.serverip );
By the way, we are running an old version of NodeJS: v0.11.13. My long-term plan is to upgrade to the latest stable version, but we may not be able to update for a few months.

Let me leave my solution here in case anyone else stumbles across this same problem in the future.
Technically, I did not discover why this error was occurring, but I did find out how to handle the error condition successfully: trap and release. The error must be trapped at the application level because it is being generated deep within net.js outside any try-catch context of my source code. So if I don't trap it, then it will crash my application. But the error is non-fatal and it appears that it can be safely ignored. In testing, the socket continued to receive new connections even after this error occurred.
process.on("uncaughtException", onUncaughtException );
function onUncaughtException(error) {
// put some code here to log the error occurrence, then ...
if( error.code==="EPERM" && error.syscall==="accept" ) {
// non-fatal error: do nothing; just ignore this error
}
else {
// handle other application errors here
}
}
So while it might still be interesting to know why a server socket can occasionally have an EPERM error, for now I'm satisfied knowing the proper way to handle the error when it occurs.

$ man 2 accept
...
In addition, Linux accept() may fail if:
EPERM Firewall rules forbid connection.
To be honest, I'm not entirely sure what type of firewall rule would cause this error, all I can think of is that you may have a rule that allows incoming connections from a particular client but disallows outgoing data to that client's IP/network/port/...

Related

hudson.AbortException: Ansible playbook execution failed jenkins

I have triggered a build for my app in jenkins and it got failed by returning the following error
hudson.AbortException: Ansible playbook execution failed
Then I have reverted my changes and triggered it again even then the same error appeared with status fail.
Then I have triggered a build for other branch of the same project but it got success. I am new to Jenkins. Can anyone please help me understand the situation?
First of all, Ansible playbooks can be very resource intensive. Especially when running against many hosts and/or using process forks, caching, etc.
It's common to see Ansible process allocating a lot of system memory. This can lead to an out-of-memory situation. Then the operating system picks and kills a running process in order to free the memory. This might affect your running Jenkins or Ansible.
Check your system logs for these out-of-memory exceptions.
For Linux use dmesg -T | grep "Out of memory" to filter out the relevant exceptions.

What's the cause of the error 'getaddrinfo EAI_AGAIN'?

My server threw this today, which is a Node.js error I've never seen before:
Error: getaddrinfo EAI_AGAIN my-store.myshopify.com:443
at Object.exports._errnoException (util.js:870:11)
at errnoException (dns.js:32:15)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:78:26)
I'm wondering if this is related to the DynDns DDOS attack which affected Shopify and many other services today. Here's an article about that.
My main question is what does dns.js do? What part of node is it a part of? How can I recreate this error with a different domain?
If you get this error with Firebase Cloud Functions, this is due to the limitations of the free tier (outbound networking only allowed to Google services).
Upgrade to the Flame or Blaze plans for it to work.
EAI_AGAIN is a DNS lookup timed out error, means it is a network connectivity error or proxy related error.
My main question is what does dns.js do?
The dns.js is there for node to get ip address of the domain(in brief).
Some more info:
http://www.codingdefined.com/2015/06/nodejs-error-errno-eaiagain.html
If you get this error from within a docker container, e.g. when running npm install inside of an alpine container, the cause could be that the network changed since the container was started.
To solve this, just stop and restart the container
docker-compose down
docker-compose up
Source: https://github.com/moby/moby/issues/32106#issuecomment-578725551
As xerq's excellent answer explains, this is a DNS timeout issue.
I wanted to contribute another possible answer for those of you using Windows Subsystem for Linux - there are some cases where something seems to be askew in the client OS after Windows resumes from sleep. Restarting the host OS will fix these issues (it's also likely restarting the WSL service will do the same).
For those who perform thousand or millions of requests per day, and need a solution to this issue:
It's quite normal to get getaddrinfo EAI_AGAIN errors when performing a lot of requests on your server. Node.js itself doesn't perform any DNS caching, it delegates everything DNS related to the OS.
You need to have in mind that every http/https request performs a DNS lookup, this can become quite expensive, to avoid this bottleneck and getaddrinfo errors, you can implement a DNS cache.
http.request (and https) accepts a lookup property which defaults to dns.lookup()
http.get('http://example.com', { lookup: yourLookupImplementation }, response => {
// do something here with response
});
I strongly recommend to use an already tested module, instead of writing a DNS cache yourself, since you'll have to handle TTL correctly, among other things to avoid hard to track bugs.
I personally use cacheable-lookup which is the one that got uses (see dnsCache option).
You can use it on specific requests
const http = require('http');
const CacheableLookup = require('cacheable-lookup');
const cacheable = new CacheableLookup();
http.get('http://example.com', {lookup: cacheable.lookup}, response => {
// Handle the response here
});
or globally
const http = require('http');
const https = require('https');
const CacheableLookup = require('cacheable-lookup');
const cacheable = new CacheableLookup();
cacheable.install(http.globalAgent);
cacheable.install(https.globalAgent);
NOTE: have in mind that if a request is not performed through Node.js http/https module, using .install on the global agent won't have any effect on said request, for example requests made using undici
The OP's error specifies a host (my-store.myshopify.com).
The error I encountered is the same in all respects except that no domain is specified.
My solution may help others who are drawn here by the title "Error: getaddrinfo EAI_AGAIN"
I encountered the error when trying to serve a NodeJs & VueJs app from a different VM from where the code was developed originally.
The file vue.config.js read :
module.exports = {
devServer: {
host: 'tstvm01',
port: 3030,
},
};
When served on the original machine the start up output is :
App running at:
- Local: http://tstvm01:3030/
- Network: http://tstvm01:3030/
Using the same settings on a VM tstvm07 got me a very similar error to the one the OP describes:
INFO Starting development server...
10% building modules 1/1 modules 0 activeevents.js:183
throw er; // Unhandled 'error' event
^
Error: getaddrinfo EAI_AGAIN
at Object._errnoException (util.js:1022:11)
at errnoException (dns.js:55:15)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)
If it ain't already obvious, changing vue.config.js to read ...
module.exports = {
devServer: {
host: 'tstvm07',
port: 3030,
},
};
... solved the problem.
I started getting this error (different stack trace though) after making a trivial update to my GraphQL API application that is operated inside a docker container. For whatever reason, the container was having difficulty resolving a back-end service being used by the API.
After poking around to see if some change had been made in the docker base image I was building from (node:13-alpine, incidentally), I decided to try the oldest computer science trick of rebooting... I stopped and started the docker container and all went back to normal.
Clearly, this isn't a meaningful solution to the underlying problem - I am merely posting this since it did clear up the issue for me without going too deep down rabbit holes.
I was having this issue on docker-compose. Turns out I forgot to add my custom isolated named network to my service which couldn't be found.
TLDR; Make sure, in your compose file, you have your custom-networks defined on both services that need to talk to each other.
My error looked like this: Error: getaddrinfo EAI_AGAIN minio-service. The error was coming from my server's backend when making a call to the minio-service using the minio-service hostname. This tells me that minio-service's running service, was not reachable by my server's running service. The way I was able to fix this issue is I changed the minio-service in my docker-compose from this:
docker-compose.yml
version: "3.8"
# ...
services:
server:
# ...
networks:
my-network:
# ...
minio-service:
# ... (missing networks: section)
# ...
networks:
my-network:
To include my custom isolated named network, like this:
docker-compose.yml
version: "3.8"
# ...
services:
server:
# ...
networks:
my-network:
# ...
minio-service:
# ...
networks:
my-network:
# ...
# ...
networks:
my-network:
More details on docker-compose networking can be found here.
This is the issue related to hosts file setup.
Add the following line to your hosts file
In Ubuntu: /etc/hosts
127.0.0.1 localhost
In windows: c:\windows\System32\drivers\etc\hosts
127.0.0.1 localhost
In my case the problem was the docker networks ip allocation range, see this post for details
#xerq pointed correctly, here's some more reference
http://www.codingdefined.com/2015/06/nodejs-error-errno-eaiagain.html
i got the same error, i solved it by updating "hosts" file present under this location in windows os
C:\Windows\System32\drivers\etc
Hope it helps!!
In my case, connected to VPN, the error happens when running Ubuntu from inside Windows Terminal but doesn't happen when opening Ubuntu directly from Windows (not from inside the Windows Terminal)
I had a same problem with AWS and Serverless. I tried with eu-central-1 region and it didn't work so I had to change it to us-east-2 for the example.
I was getting this error after I recently added a new network to my docker-compose file.
I initially had these services:
services:
frontend:
depends_on:
- backend
ports:
- 3005:3000
backend:
ports:
- 8005:8000
I decided to add a new network which hosts other services I wanted my frontend service to have access to, so I did this:
networks:
moar:
name: moar-network
attachable: true
services:
frontend:
networks:
- moar
depends_on:
- backend
ports:
- 3005:3000
backend:
ports:
- 8005:8000
Unfortunately, the above made it so that my frontend service was no longer visible on the default network, and only visible in the moar network. This meant that the frontend service could no longer proxy requests to backend, therefore I was getting errors like:
Error occured while trying to proxy to: localhost:3005/graphql/
The solution is to add the default network to the frontend service's network list, like so:
networks:
moar:
name: moar-network
attachable: true
services:
frontend:
networks:
- moar
- default # here
depends_on:
- backend
ports:
- 3005:3000
backend:
ports:
- 8005:8000
Now we're peachy!
One last thing, if you want to see which services are running within a given network, you can use the docker network inspect <network_name> command to do so. This is what helped me discover that the frontend service was not part of the default network anymore.
Enabled Blaze and it still doesn't work?
Most probably you need to set .env from the right path, require('dotenv').config({ path: __dirname + './../.env' }); won't work (or any other path). Simply put the .env file in the functions directory, from which you deploy to Firebase.

How can I catch an EHOSTDOWN socket error when creating an http request with iojs?

I have a simple iojs http server which communicates with another http backend on my development machine. Now my ip has changed and the backend request won't work due to a wrong ip. I have (or at least thought so) error management in place however the server crashes in some situations due to an unhandled exception:
When doing 2 subsequent requests the first one "hangs" and then the second request crashes the server:
events.js:141
throw er; // Unhandled 'error' event
^
Error: connect EHOSTDOWN 192.168.1.11:80 - Local (192.168.1.10:54125)
at Object.exports._errnoException (util.js:846:11)
at exports._exceptionWithHostPort (util.js:869:20)
at connect (net.js:840:14)
at lookupAndConnect (net.js:933:5)
at Socket.connect (net.js:902:5)
at Agent.exports.connect.exports.createConnection (net.js:61:35)
at Agent.createSocket (_http_agent.js:177:16)
at Agent.addRequest (_http_agent.js:147:23)
at new ClientRequest (_http_client.js:132:16)
at Object.exports.request (http.js:30:10)
The error in question won't trigger backRequest.on("error", errFn) and it is not a standard error as in function(err, response, body). It can't be catched using try...catch.
How can i catch and gracefully handle this error?
I have dugged a bit into net.js/http.js. When emitting a socket error manually:
backendRequest.on("socket", function(socket) {
socket.emit("error", "DIE!");
});
it is handled by the normal error handler. The EHOSTDOWN error however is not. The reason for this seems to be that the socket error handler is only installed on the next tick after Socket.connect() (see https://github.com/joyent/node/blob/master/lib/_http_client.js#L498).
"My" error however is triggered directly upon connect() (must have something to do with the first connection still trying to find the host).
I can gracefully catch that error using a custom Agent.createConnection:
var socket = net.createConnection(options);
// install error handler immediately
socket.on("error", function(err) {
console.log("AHA", err);
});
return socket;
however that seems very verbose and bulky. Is that truly the best way to do that? The error seems to be not that special and seems to be unhandled in the core libs. Why is that so and why is the socket error handler only installed upon nextTick?
How would you properly catch that error? Is the agent the correct way?
A "full" example to play with can be found here: http://pastebin.com/jaCUPaHX
As pointed out by the commenters: It was a bug https://github.com/nodejs/io.js/pull/2054

Websocket Handshake error on Ionic app (Status line does not end with CRLF)

I am trying to connect my Ionic App to a Meteor App, using DDP (i tried using both Mondora's DDP.JS and Mondora's Asteroid solutions, but they both give me the same error)
// Connect to a Meteor backend
var options = {
endpoint: "ws://my.meteor.app/websocket",
SocketConstructor: WebSocket
};
var ddp = new DDP(options);
ddp.on("connected", function () {
alert("Connected");
});
ddp.on("disconnected", function () {
alert("Disconnected");
});
The thing is, when i run it on the chrome emulator (Ionic Serve) it works flawlessly, but when i run on the android phone (Ionic Run), it always gives me this error.
WebSocket connection to 'ws://my.meteor.app/websocket' failed: Error during WebSocket handshake: Status line does not end with CRLF
I have seen a question similar to this here on Stack, but i can't find the error on the DDP.JS file, it doesn't explicitly show me the Headers
I think the error is a bit misleading. Every header should end with CRLF and not only the first one (the 'status line'), it seems. At least according to the test case. (PhistucK on Websocket communication between chrome(client) and hotspot(server) (Status line does not end with CRLF)).
I have tried adding this '\r' thing on almost every variable on the DDP.JS file, with no success. Does anyone know what's happenning?
Thanks in advance,
Bruno

node.js - PM2 log uncaught exceptions to third-party service (as Logentries)

I'm using pm2 (https://github.com/Unitech/pm2) in my node.js project. Also I'm sending logs of errors of the app in Logentries (https://logentries.com).
I wonder is it possible to log uncaught exceptions from the app (when something fails badly and pm2 restarts the app for example)? I know that using process.on('uncaughtException') is bad practice so would like to hear some suggestions.
Thanks!
Where did you read that process.on('uncaughtException') is a bad practice?
As long as you exit the process after logging the exception I don't see what's bad, here is an example:
process.on('uncaughtException', function(e) {
console.error('Ouch, an unhandled exception');
//I like using new Error() for my errors (1)
console.error(e instanceof Error ? e.message : e);
process.exit(1);
});
(1): Javascript Error reference
Edit
pm2-interface is now deprecated, use require('pm2') instead. You will be able to do exactly the same as below by using bus system events.
An alternative with pm2 is to use pm2-interface and listening to the process:exit or process:exception events:
var ipm2 = require('pm2-interface')();
ipm2.on('ready', function() {
console.log('Connected to pm2');
ipm2.bus.on('process:exception', function(data){
console.log(data.pm2_env.name + 'had an exception');
});
});
This is really usefull when managing more than one process through a monitoring process.
You might want to check the blog post on how to build a custom pm2 logger. It can give you some ideas about monitoring processes through pm2-interface.

Categories

Resources