When I connect and send some sockets to my Linux, node.js server inside a VPC and behind a load balancer I get a unusually long delay, followed by WebSocket connection to [address] failed: Connection closed before receiving a handshake response
And then a few seconds later I get responses for all the sockets I sent, and then everything works fine. No long delays.
But just on this initial connect, there's a horrible wait followed by this error message. Everything still works, it just takes a bit.
I'm using Amazon Web Service EC2 load-balancers and AWS VPCs
When I'm accessing the same server directly, I get no delays.
I was unable to connect to my server when having just a load-balancer.
I was unable to connect to my server when having just a VPC, so I can't isolate the problem to just my load-balancer or the VPC.
What's going on?
The correct answer was Michael's comment that I marked as helpful.
The first person who puts this into an answer format gets points.
The health of connection from the Load Balancer to the server is determined by the way in which your Health Check is set up.
Try and set it up differently.
eg
Use a TCP based Health Check rather than a HTTP based one, and change the thresholds.
If you see some different behaviour, you'll know that the Health Check is the issue.
It is hard to know exactly without debugging, but note that there are issues on using Elastic Load Balancer for Web Sockets. They parse HTTP requests (unless in TCP mode) and they have a 60 seconds idle connection timeout.
Related
I have not been able to get an answer to this anywhere online. I want to remove possible jitter from my nodejs server. I am using socket.io to create connections to node.
If a user goes to a specific part of my website, a connection is started. However, if the user refreshes the site too quickly and often, the connection is created very frequently, and issues arise with my server.
While I realized it's possible this could be solved a couple different ways, I am hoping a server solution is out there. Meaning, whenever a user connects, make sure the user is connected for at least 5 seconds. Then move on. Otherwise, disconnect the user. Thanks for any insight!
First off a little background. With a default configuration, when a socket.io connection starts, it first does 2-5 http connections and then once it has established the "logical" connection, it tries to establish a connection using the webSocket transport. If that is successful, then it keeps that webSocket connection as a long lasting connection and sends socket.io packets over it.
If the client refreshes in the middle of the transition to a webSocket connection, it creates a period of unknown state on the server where the server isn't sure if the user is just still in the middle of the transition to a lasting webSocket connection, if the user is gone entirely already, if the user is having some sort of connection issues or if the user is doing some refresh thing. You can easily end up with a situation where the server thinks there are multiple connections all from the same user in the process of being confirmed. It can be a bit messy if your server is sensitive to that kind of thing.
The quickest thing you can do is to force the connection process to go immediately to the webSocket transport. You can do that in the client by adding an options to your connection code:
let socket = io(yourURL, {transports: ["websocket"]});
You can also configure the server to only accept webSocket connections if you're try to protect against any other types of connections besides just from your own web pages.
This will then go through the usual webSocket connection which starts with a single http request that is then "upgraded" to the webSocket protocol. Once connection, one socket. The server will know right away, either the user is or isn't connected. And, once they've switched over to the webSocket protocol, the server will known immediately if the user hits refresh because the browser will close the webSocket immediately.
The "start with http first" feature in socket.io is largely present because in the early days of webSockets, there were some browsers that didn't yet support them and some network infrastructure (like corporate proxies) that didn't always support webSocket connections. The browser issue is completely gone now. All browsers in use support webSocket connections. I don't personally have any data on the corporate proxies issues, but I don't ever hear about any issues with people using webSockets these days so I don't think that's much of an issue any more either.
So, the above change will get you a quick, confirmed connection and get rid of the confusion around whether a user is or isn't connected early in the connection process.
Now, if you still have users who are messing things up by rapid refresh, you probably need to just implement some protection on your server for that. If you cookie each user that arrives on your server, you could create some middleware that would keep track of how many page requests in some recent time interval have come from the browser with this cookie and just return them an error page that explains they can't make requests that quickly. I would probably implement this at the web page level, not the webSocket level as that will give users better feedback to stop hitting refresh. If it's really a refresh you're trying to protect against and not general navigation on your site, then you can keep a record of a combination cookie and URL and if you see even two of those within a few seconds, then return the error page instead of the expected content. If you redirect to an error page, it forces a more conscious action to go back to the right page again before they can get to the content.
So I have server receiving wss connections in Java hosted on google kubernetes engine.
My website connects to the websocket connection in javascript using the default WebSocket library that comes with javascript.
The client connects to the server and works fine sometimes, but sometimes it fails to connect at all, doesn't get to the on open event or anything. To see if its server-side or client-side issue I made a small java app that connects to the websocket and had it run overnight doing 30 connections at once and it didn't have any problems so I think its something client side, or related to sessions keeping something cached.
Sometimes its very "sticky" once you don't connect once, even if you try to reconnect a million times it won't work, but if you use a different browser, or reset your wifi it will start working again? Also soon as the server restarts it usually works okay, but after a little while (hours or next day) it won't work.
One of the strangest behaviors is that if I inspect the request in firefox's network tab and do "edit and send" on the headers it will ALWAYS work even if the page "stuck" not working no matter how much I refresh. Here is a pic of what I mean:
this always works
The java websocket library I'm using is this one: https://github.com/TooTallNate/Java-WebSocket
I'm just using built in javascript connection library:
var wsocket = new WebSocket(address);
All I want is a consistent connection. I even tried to right code to retry the connection if it doesn't happen after a certain time, but because of the "sticky" behavior that doesn't help much.
Here is a screenshot of wireshark info, I don't really know what any of it means but maybe it can help you guys: wireshark
I feel like it has something to do with the browser/server keeping a session history or something?
On the server I get the following error at least some of the time when this occurs:
The connection was closed because the other endpoint did not respond with a pong in time.
I am working on multiple apps that communicate between each other. I am using Chrome and Firefox both to test my apps on. The problem seems to be persistent in both browsers.
The problem:
I am sending a PUT request from app nr.1 to the Express Node server that essentially sends an update to my mongo database server. Once updated app nr.2 will retrieve the updated value with a GET request. Websockets are being used to notify apps on changes.
The problem however is that the HTTP GET requests on the receiving app nr.2 is taking multiple seconds for it to complete (after a few of them have been done).
To explain the written lines above look at the screenshot below:
the first few GET request take 3-5ms to complete, then the upcoming GET requests will take up to 95634ms to complete....
What could be the cause of this and how could this be fixed?
It is difficult to tell without seeing your whole stack.
Sometimes a reverse-proxy that sits in front of you applications can cause issues like this
They could be trying to route to ipv6 instead of ipv4 especially if you are using localhost to point your GET requests. The fix is to use 127.0.0.1 instead of localhost
Also, a high keepalive timeout setting on a proxy can cause this
Good first places to look in a situation like this are
Proxy logs
Node logs
Server logs (ie firewall or throttling)
I have an application written in node.js with a timer function. Whenever a second has passed, the server sends the new time value to every connected client. While this works perfectly fine on localhost, it's very choppy when hosted online. Clients won't update immediately and the value will sometimes jump two or three seconds at a time.
I discovered, however, if I repeatedly send the timer data to the clients (using setInterval), it runs perfectly without any delay from anywhere.
Does anyone have any idea why this might be the case? It doesn't make sense to me why sending the same data more often would fix the issue. If anything, shouldn't this be more slow? I was thinking I could use this approach and have the client notify the server when it has updated but this seems unnecessary and inefficient.
I'm very new to node.js but this has got me stumped. Any insight would be greatly appreciated.
Where are you hosting it? Does it support websockets? Some hosts do not support/allow them. My guess is that your host is not allowing websockets and socket.io is falling back to the polling transport.
In your browser, you can find the websocket connection and inspect it in developer tools:
How do you inspect websocket traffic with Chrome Developer Tools?
If it does not undergo the 101 Switching Protocols http status to successfully upgrade the first request to a websocket, you'll see the polling requests recur in the developer tools.
I've a web sockets based chat application (HTML5).
Browser opens a socket connection to a java based web sockets server over wss.
When browser connects to server directly (without any proxy) everything works well.
But when the browser is behind an enterprise proxy, browser socket connection closes automatically after approx 2 minutes of no-activity.
Browser console shows "Socket closed".
In my test environment I have a Squid-Dansguardian proxy server.
IMP: this behaviour is not observed if the browser is connected without any proxy.
To keep some activity going, I embedded a simple jquery script which will make an http GET request to another server every 60 sec. But it did not help. I still get "socket closed" in my browser console after about 2 minutes of no action.
Any help or pointers are welcome.
Thanks
This seems to me to be a feature, not a bug.
In production applications there is an issue related with what is known as "half-open" sockets - see this great blog post about it.
It happens that connections are lost abruptly, causing the TCP/IP connection to drop without informing the other party to the connection. This can happen for many different reasons - wifi signals or cellular signals are lost, routers crash, modems disconnect, batteries die, power outages...
The only way to detect if the socket is actually open is to try and send data... BUT, your proxy might not be able to safely send data without interfering with your application's logic*.
After two minutes, your Proxy assume that the connection was lost and closes the socket on it's end to save resources and allow new connections to be established.
If your proxy didn't take this precaution, on a long enough timeline all your available resources would be taken by dropped connections that would never close, preventing access to your application.
Two minutes is a lot. On Heroku they set the proxy for 50 seconds (more reasonable). For Http connections, these timeouts are often much shorter.
The best option for you is to keep sending websocket data within the 2 minute timeframe.
The Websocket protocol resolves this issue by implementing an internal ping mechanism - use it. These pings should be sent by the server and the browser responds to them with a pong directly (without involving the javascript application).
The Javascript API (at least on the browser) doesn't let you send ping frames (it's a security thing I guess, that prevents people from using browsers for DoS attacks).
A common practice by some developers (which I think to be misconstructed) is to implement a JSON ping message that is either ignored by the server or results in a JSON pong.
Since you are using Java on the server, you have access to the Ping mechanism and I suggest you implement it.
I would also recommend (if you have control of the Proxy) that you lower the timeout to a more reasonable 50 seconds limit.
* The situation during production is actually even worse...
Because there is a long chain of intermediaries (home router/modem, NAT, ISP, Gateways, Routers, Load Balancers, Proxies...) it's very likely that your application can send data successfully because it's still "connected" to one of the intermediaries.
This should start a chain reaction that will only reach the application after a while, and again ONLY if it attempts to send data.
This is why Ping frames expect Pong frames to be returned (meaning the chain of connection is intact.
P.S.
You should probably also complain about the Java application not closing the connection after a certain timeout. During production, this oversight might force you to restart your server every so often or experience a DoS situation (all available file handles will be used for the inactive old connections and you won't have room for new connections).
check the squid.conf for a request_timeout value. You can change this via the request_timeout. This will affect more than just web sockets. For instance, in an environment I frequently work in, a perl script is hit to generate various configurations. Execution can take upwards of 5-10 minutes to complete. The timeout value on both our httpd and the squid server had to be raised to compensate for this.
Also, look at the connect_timeout value as well. That's defaulted to one minute..