So we have a signalr application that when the server has a hickup, the clients call "reconnect" every 2 seconds indefinitely (even though the server is back up). When the page refreshes it connects just fine. couples things that I noticed:
The reconnect request in the Network dev tools shows "Status Code: 101 Switching Protocols"
We're using ms-signalr-client for the wrapper around the javascript client, it runs "reconnect", then immediately is successful, then immediately closes the connection
This is causing our prod server to go down 3 or 4 times a day as soon as the clients lose connection for a second
Figured out the issue. there was an exception being thrown SILENTLY in the Reconnected Stack. not sure why considering the same code is called in the Connect Stack. anyway, reworked it that we don't have to make the same call, and not need it. works fine now
Related
So I have server receiving wss connections in Java hosted on google kubernetes engine.
My website connects to the websocket connection in javascript using the default WebSocket library that comes with javascript.
The client connects to the server and works fine sometimes, but sometimes it fails to connect at all, doesn't get to the on open event or anything. To see if its server-side or client-side issue I made a small java app that connects to the websocket and had it run overnight doing 30 connections at once and it didn't have any problems so I think its something client side, or related to sessions keeping something cached.
Sometimes its very "sticky" once you don't connect once, even if you try to reconnect a million times it won't work, but if you use a different browser, or reset your wifi it will start working again? Also soon as the server restarts it usually works okay, but after a little while (hours or next day) it won't work.
One of the strangest behaviors is that if I inspect the request in firefox's network tab and do "edit and send" on the headers it will ALWAYS work even if the page "stuck" not working no matter how much I refresh. Here is a pic of what I mean:
this always works
The java websocket library I'm using is this one: https://github.com/TooTallNate/Java-WebSocket
I'm just using built in javascript connection library:
var wsocket = new WebSocket(address);
All I want is a consistent connection. I even tried to right code to retry the connection if it doesn't happen after a certain time, but because of the "sticky" behavior that doesn't help much.
Here is a screenshot of wireshark info, I don't really know what any of it means but maybe it can help you guys: wireshark
I feel like it has something to do with the browser/server keeping a session history or something?
On the server I get the following error at least some of the time when this occurs:
The connection was closed because the other endpoint did not respond with a pong in time.
I have an application written in node.js with a timer function. Whenever a second has passed, the server sends the new time value to every connected client. While this works perfectly fine on localhost, it's very choppy when hosted online. Clients won't update immediately and the value will sometimes jump two or three seconds at a time.
I discovered, however, if I repeatedly send the timer data to the clients (using setInterval), it runs perfectly without any delay from anywhere.
Does anyone have any idea why this might be the case? It doesn't make sense to me why sending the same data more often would fix the issue. If anything, shouldn't this be more slow? I was thinking I could use this approach and have the client notify the server when it has updated but this seems unnecessary and inefficient.
I'm very new to node.js but this has got me stumped. Any insight would be greatly appreciated.
Where are you hosting it? Does it support websockets? Some hosts do not support/allow them. My guess is that your host is not allowing websockets and socket.io is falling back to the polling transport.
In your browser, you can find the websocket connection and inspect it in developer tools:
How do you inspect websocket traffic with Chrome Developer Tools?
If it does not undergo the 101 Switching Protocols http status to successfully upgrade the first request to a websocket, you'll see the polling requests recur in the developer tools.
I have googled many things related to this issue. I am doing signalr connection using WebSocket and longpooling, I am pinging the server every 30 sec from the connection.
When I load the first time, I don't doing anything on the page, no refresh, after some time connection is disconnected due to inactivity(he client has been inactive since 26/03/2018 30000ms and it has exceeded the inactivity timeout), and reconnected again as I have written reconnect function,
though if I calling ping method to server every 20 sec after some time server is disconnected and I don't know why.
I have enabled logging=true on client side,
my client is HTML page.
If the server is not reconnected, an error like
longpooling request is failed
occurs. On the server keepalive is null.
The first time the client is able to reconnect, but the second time not.
I don't know why the server disconnects, as I am using longpooling.
The ping method is an empty method only for the heartbeat I used.
The application is hosted using Amazon ELB.
Please help, I have tried for 3-4 days, but I could not find the issue.
This is a problem, i have discovered in a system i am working on, i just want to know if chrome closes connections that does not send messages within a certain period of time. If so then i can fix it, i would just like to know how long does chrome give before it closes such a connection. I have look through chrome settings to see if there is anything it could tell me.
this is what i see on the python server there is no error it just states that the websocket has closed.
Yes, chrome will timeout a WebSocket connection if no data is passed between the client and server, generally, the common approach here is to have a ping-pong type system in place or a heartbeat from the server side that just sends an empty message every x number of seconds.
It is based on an interval since the last message was sent/received, which after some research appears to be approximately 10 seconds in chrome, but may vary for different browsers and/or server configuration.
I've a web sockets based chat application (HTML5).
Browser opens a socket connection to a java based web sockets server over wss.
When browser connects to server directly (without any proxy) everything works well.
But when the browser is behind an enterprise proxy, browser socket connection closes automatically after approx 2 minutes of no-activity.
Browser console shows "Socket closed".
In my test environment I have a Squid-Dansguardian proxy server.
IMP: this behaviour is not observed if the browser is connected without any proxy.
To keep some activity going, I embedded a simple jquery script which will make an http GET request to another server every 60 sec. But it did not help. I still get "socket closed" in my browser console after about 2 minutes of no action.
Any help or pointers are welcome.
Thanks
This seems to me to be a feature, not a bug.
In production applications there is an issue related with what is known as "half-open" sockets - see this great blog post about it.
It happens that connections are lost abruptly, causing the TCP/IP connection to drop without informing the other party to the connection. This can happen for many different reasons - wifi signals or cellular signals are lost, routers crash, modems disconnect, batteries die, power outages...
The only way to detect if the socket is actually open is to try and send data... BUT, your proxy might not be able to safely send data without interfering with your application's logic*.
After two minutes, your Proxy assume that the connection was lost and closes the socket on it's end to save resources and allow new connections to be established.
If your proxy didn't take this precaution, on a long enough timeline all your available resources would be taken by dropped connections that would never close, preventing access to your application.
Two minutes is a lot. On Heroku they set the proxy for 50 seconds (more reasonable). For Http connections, these timeouts are often much shorter.
The best option for you is to keep sending websocket data within the 2 minute timeframe.
The Websocket protocol resolves this issue by implementing an internal ping mechanism - use it. These pings should be sent by the server and the browser responds to them with a pong directly (without involving the javascript application).
The Javascript API (at least on the browser) doesn't let you send ping frames (it's a security thing I guess, that prevents people from using browsers for DoS attacks).
A common practice by some developers (which I think to be misconstructed) is to implement a JSON ping message that is either ignored by the server or results in a JSON pong.
Since you are using Java on the server, you have access to the Ping mechanism and I suggest you implement it.
I would also recommend (if you have control of the Proxy) that you lower the timeout to a more reasonable 50 seconds limit.
* The situation during production is actually even worse...
Because there is a long chain of intermediaries (home router/modem, NAT, ISP, Gateways, Routers, Load Balancers, Proxies...) it's very likely that your application can send data successfully because it's still "connected" to one of the intermediaries.
This should start a chain reaction that will only reach the application after a while, and again ONLY if it attempts to send data.
This is why Ping frames expect Pong frames to be returned (meaning the chain of connection is intact.
P.S.
You should probably also complain about the Java application not closing the connection after a certain timeout. During production, this oversight might force you to restart your server every so often or experience a DoS situation (all available file handles will be used for the inactive old connections and you won't have room for new connections).
check the squid.conf for a request_timeout value. You can change this via the request_timeout. This will affect more than just web sockets. For instance, in an environment I frequently work in, a perl script is hit to generate various configurations. Execution can take upwards of 5-10 minutes to complete. The timeout value on both our httpd and the squid server had to be raised to compensate for this.
Also, look at the connect_timeout value as well. That's defaulted to one minute..