I create 60 client connections to socket.io server in google chrome browser.
Server at specific time send screenshot to the clients. And some websocket connections, that are subprotocol of socket.io are broken, so connection at about 1-4 chrome tabs are closed. I tried to increase pingTimeout, it helped to overcome tcp transport close problem only (this problem I have as well), but this solution doesn't help to fix sending screenshot problem.
In my opinion google chrome can't support about 50-60 tabs at one time, because CPU and RAM are increased to the max values because of sending screenshots to 60 clients (each client has 2 websocket connection: the first for simple messages, the second for graphics (to send screenshots)), so chrome closes some websocket connections.
Part of code for the server socket io here:
// server
this.http = this._createHttpServer(sslCert, sslKey);
this.io = socketIo(this.http, {
'pingTimeout': 180000,
'pingInterval': 60000
});
const jwtAuth = socketioJwt.authorize({
secret: jwtSecret,
timeout: 15000
});
this.io.on('connection', (socket) => {
socket.once('authenticate', (data) => {
socket.rawAuthData = data;
});
jwtAuth(socket);
});
// client
var connOptions = {
"reconnectionAttempts": 2
};
var socket = io(options.url, connOptions);
socket.on('connect', function() {
if (options.token) {
socket.emit('authenticate', {token: options['token'], tag : tag});
socket.on('authenticated', function() {
ctx.printLog('Authorized. Waiting for handshake');
socket.once('tunnel-handshake', function() {
ctx.printLog('handshake received! connection is ready');
processConnected();
});
}).on('unauthorized', function(msg) {
ctx.printLog("Authorization failed: " + JSON.stringify(msg.data));
eventHandlers.onerror({ code: ctx.ERROR_CODE.INVALID_TOKEN});
});
} else {
processConnected();
}
});
socket.on('reconnect_failed', eventHandlers.onerror.bind(this, {code: 1, reason: "Reconnection failed"}));
socket.on('disconnect', eventHandlers.onclose);
socket.on('error', eventHandlers.onerror);
Does exist any ideas, what the cause could be? Does exist any solution of this problem?
Is it google chrome problem or socket.io options problem?
Thanks
Changing socket.io to 3.0 version can't resolve the issue. socket.io v3.0 has engine.io v4.0. Next information in release notes of engine.io v4.0 describes the problem ("Heartbeat mechanism reversal" title):
We have received a lot of reports from users that experience random disconnects due to ping timeout, even though their Internet connection is up and the remote server is reachable. It should be noted that in that case the client reconnects right away, but still it was an annoying issue.
After analysis, it seems to be caused by delayed timers on the
client-side. Those timers are used in the ping-pong mechanism which
helps to ensure the connection between the server and the client is
still healthy. A delay on the client-side meant the client sent the
ping packet too late, and the server considered that the connection
was closed.
That’s why the ping packets will now be sent by the server, and the
client will respond with a pong packet.
But increasing pingTimeout and pingInterval to 1073741823 value resolves the issue.
Related
I am attempting to write an web application with a persistent echo connection to a laravel-echo-server instance, which needs to detect disconnections and attempt to reconnect gracefully. The scenario I am attempting to overcome now is a user's machine has gone to sleep / reawoke and their session key has been invalidated (echo server requires an active session in our app). Detecting this situation from an HTTP perspective is solved - I setup a regular keepAlive, and if that keepAlive detects a 400-level error, it reconnects and updates the session auth_token.
When my Laravel session dies, I cannot tell that has happened from an echo perspective. The best I've found is I can attach to the 'disconnect' event, but that only gets triggered if the server-side laravel-echo-server process dies, rather than the session is invalid:
this.echoConnection.connector.socket.on('connect', function() {
logger.log('info', `Echo server running`);
})
this.echoConnection.connector.socket.on('disconnect', function() {
logger.log('warn', `Echo server disconnected`);
});
On the laravel-echo-server side, I can tell that the connection is dead - it will show this error:
⚠ [7:03:30 PM] - 5TwHN2qUys5VEFP5AAAG could not be authenticated to private.1
I cannot figure out how to catch this failure event programmatically from the client. Is there a way to capture it? Again, I can tell the session is dead eventually because I poll the server regularly via a http keepAlive function, but I would definitely also like to tell directly from the echo connection if possible, as it polls at a much higher natural rate.
As a second (more important) question, if I detect that my session has died, what should I do to recycle the echo connection (after I have logged in again via HTTP and gotten a new auth_token)? Is there anything specific I should call / etc? I've had some success calling disconnect() then setting up the connection again from scratch, but I do see errors such as:
websocket.js:201 WebSocket is already in CLOSING or CLOSED state.
Here is my current (naive) reconnection code, which is my initial connection code with an attempt to disconnect first stapled onto it:
async attemptEchoReconnect() {
if (this.echoConnection !== null) {
this.echoConnection.disconnect();
this.echoConnection = null;
}
const thisConnectionParams = this.props.connections[this.connectionName];
const curThis = this;
this.echoConnection = new Echo({
broadcaster: 'socket.io',
host: thisConnectionParams.echoHost,
authEndpoint: 'api/broadcasting/auth',
auth: {
headers: {
Authorization: `Bearer ` + thisConnectionParams.authToken
}
}
});
this.echoConnection.connector.socket.on('connect', function() {
logger.log('info', `Echo server running`);
})
this.echoConnection.connector.socket.on('disconnect', function() {
logger.log('warn', `Echo server disconnected`);
});
this.echoConnection.join('everywhere')
.here(users => {
logger.log('info', `Rejoined presence channel`);
});
this.echoConnection.private(`private.${this.props.id}`)
.listen(...);
setTimeout(() => { this.keepAlive() }, 120 * 1000);
}
Any help would be so great - these APIs are not well documented to the end that I really want, and I am hoping I can get some stability with this connection rather than having to do something ugly like force restart.
For anyone who needs help with this problem, my above echo reconnection code seems to be pretty stable, along with a keepAlive function to determine the state of the HTTP connection. I am still a bit uncertain of the origin of the console errors I am seeing, but I suspect they have to do with connection loss during a sleep cycle, which is not something I am particularly worried about.
I'd still be interested in hearing other thoughts if anyone has any. I am somewhat inclined to believe long-term stability of an echo connection is possible, though it does appear you have to proactively monitor it with what tools you have available.
I have found that WebSockets in Chrome and Firefox disconnect after exactly one minute of inactivity. Based on stuff I've seen online, I was all set to blame proxies or some server settings or something, but this does not happen in IE or Edge. It seems like if sockets are disconnected by the server after one minute of inactivity that would apply to IE and Edge just as much as Chrome and Firefox.
Does anyone know why this is? Is it documented anywhere? I know a possible way to stop it by pinging, but I'm more interested in why it's happening. The reason code given on disconnect is 1006, indicating that the browser closed the connection. No errors are thrown and the onerror event for the socket is not triggered.
This project was built at https://glitch.com/edit/#!/noiseless-helmet where you can see and run everything. The client page is served here: https://noiseless-helmet.glitch.me/
Here is my client page:
<div id="div">
</div>
<script>
let socket = new WebSocket("wss://noiseless-helmet.glitch.me/");
socket.onmessage = function(event) {
div.innerHTML += "<br>message " + new Date().toLocaleString() + " " + event.data;
};
socket.onopen = function (event) {
div.innerHTML += "<br>opened " + new Date().toLocaleString();
socket.send("Hey socket! " + new Date().toLocaleString());
};
socket.onclose = function(event) {
div.innerHTML += "<br>socket closed " + new Date().toLocaleString();
div.innerHTML += "<br>code: " + event.code;
div.innerHTML += "<br>reason: " + event.reason;
div.innerHTML += "<br>clean: " + event.wasClean;
};
socket.onerror = function(event) {
div.innerHTML += "<br>error: " + event.error;
};
</script>
And here is my Node.js server code:
var express = require('express');
var app = express();
app.use(express.static('public'));
let server = require('http').createServer(),
WebSocketServer = require('ws').Server,
wss = new WebSocketServer({ server: server });
app.get("/", function (request, response) {
response.sendFile(__dirname + '/views/index.html');
});
let webSockets = [];
wss.on('connection', function connection(socket) {
webSockets.push(socket);
webSockets.forEach((w) => { w.send("A new socket connected"); });
socket.on('close', (code, reason) => {
console.log('closing socket');
console.log(code);
console.log(reason);
let i = webSockets.indexOf(socket);
webSockets.splice(i, 1);
});
});
server.on('request', app);
server.listen(process.env.PORT, function () {
console.log('Your app is listening on port ' + server.address().port);
});
It seems like if sockets are disconnected by the server after one minute of inactivity that would apply to IE and Edge just as much as Chrome and Firefox.
Hmmm, no, it doesn't. IE and Edge might be implementing a ping packet as part of the WebSocket protocol.
The WebSocket protocol includes support for a protocol level ping that the JavaScript API doesn't expose. It's a bit lower-level than the user level pinging that is often implemented.
This ping-pong traffic resets the timers in any network intermediaries (proxies, load balancers, etc') - and they all time connections to mark stale connections for closure (for example, the Heroku setup times connections at 55 seconds).
Most browsers trust the server to implement the ping, which is polite (since servers need to manage their load and their timeout for pinging...
...however it's also slightly frustrating, since browsers have no idea if a connection was abnormally lost and JavaScript doesn't emit an event for the WebSocket protocol ping. This is why many JavaScript clients implement a user level ping (i.e., a JSON {event: "ping", data: {...}} or another "empty" event message).
Anyway, I just wanted to point out that your assumption was incorrect, this is still a timeout occurring and the difference in browser behavior is probably related to the browsers themselves.
For a few specifics regarding nginx default timeouts (when proxying WebSocket connections) you can read #Hendry's answer.
As much as i understood from researching this, this is caused by websocket timing out over a period of time when no data is sent. This is probably per browser.
You could use pings to resolve this or just reconnect when you need to use the socket again.
It makes sense to not keep sockets open when they are not used from server side as from browser side. For example, Chrome has a limit how many connections can be open, if the limit would be 64 connections and you have open 64 tabs (which is very likely for me as i always have loads of tabs open) and each tab is connected to a server, no more connections could be done (Actually similar thing happened to me once, when i ran out of available sockets in Chrome, funny).
There is proxy_read_timeout (http://nginx.org/r/proxy_read_timeout)
which as well applies to WebSocket connections. You have to bump
it if your backend do not send anything for a long time.
Alternatively, you may configure your backend to send websocket
ping frames periodically to reset the timeout (and check if the
connection is still alive).
https://forum.nginx.org/read.php?2,236382,236383#msg-236383
Web Sockets have an idle timeout of 60 seconds: if you do not use a heartbeat or similar via ping and pong frames then the socket assumes that the user has closed the page and closes the socket to save resources.
https://www.codeproject.com/Questions/1205863/Websocket-is-closed-after-min
https://github.com/tornadoweb/tornado/issues/1070
The WebSocket protocol specification defines Ping and Pong frames that can be used for keep-alive, heart-beats, network status probing. Ping means client/server is sending an iq to tell the other side server/client that to keep the connection alive and also the other side will send an acknowledgement with pong having same payload data.
You can also define a timeout when the browser stops respond or be considered dead.
read more: http://vunse.blogspot.in/2014/04/websocket-ping-pong.html
Maybe not a clean solution but this is how I implemented websocket in JS to automatically reconnect when disconnected
var socket_main
const mainSocketMessageListener = (event) => {
//retreive the data here
console.log(event.data)
}
const mainSocketOpenListener = (event) => {
console.log("Websocket opened")
//Example of sending message to websocket here
socket_main.send(JSON.stringify({
event: "subscribe",
data: ["all"]
}))
}
const mainSocketCloseListener = (event) => {
if (socket_main) {
console.error('Websocket disconnected.')
}
socket_main = new WebSocket('wss://ws.example.com')
socket_main.addEventListener('open', mainSocketOpenListener)
socket_main.addEventListener('message', mainSocketMessageListener)
socket_main.addEventListener('close', mainSocketCloseListener)
}
//connect the first time
mainSocketCloseListener()
I think the issue is related to policy changes on chrome and other browsers.
Please see discussions at
"Chrome terminates WebSocket connection": https://github.com/SignalR/SignalR/issues/4536
"How to detect when browser throttles timers and websockets disconnection": How to detect when browser throttles timers and websockets disconnection after a user leaves a tab or turns off the screen? (javascript)
I have a browser game using ws module. But sometimes a client on a distant computer will close connection for unknown reason when playing the game for a while. The screen just freezes when the connection is closed. I don't have this problem testing it on local server computer.
I have this ws listner on my server:
const WSServer = WebSocket.Server;
const server = require('http').createServer(app);
let wss = new WSServer({
server,
});
wss.on('connection', function(ws, req) {
const ip = req.connection.remoteAddress;
console.log(ip + ' connected');
if (ws.readyState === WebSocket.OPEN) {
const id = req.headers['sec-websocket-key'];
gameServer.socketActions(ip, ws, wss.clients);
}
ws.on('error', function(err) {
logger.debug('Found error: ' + err);
});
ws.on('close', function() {
logger.debug(ip + ' disconnected.');
});
});
So when the client connection is closed unexpectedly it just tells me some ip disconnected.
But sometimes a client on a distant computer will close connection for unknown reason when playing the game for a while.
Welcome to The Internet™, a notoriously hostile environment where anything that can go wrong, will.
Seriously, people cut fiber with backhoes all the time. Sometimes floods cause power outages. Other times, someone's laptop battery died. Lots of people use wireless access of some sort or another which is subject to all sorts of interference... by design! You won't ever really know why this happens. Eventually, a bunch of packets get lost, some threshold is exceeded, and the problem works its way through the layers.
A properly built application will deal with this gracefully. You can try to reconnect if you want.
I'm using socket.io-client to create a socket connection to my locally-running server. See my code below:
// Working example of connecting to a local server that is not SSL protected
var io = require('socket.io-client')
var socket = io.connect('http://localhost:3000', {reconnect: true});
socket.on('connect', function(){ console.log("inside 'connect'") } );
socket.on('connection', function(){ console.log("inside 'connection'") } );
socket.on('event', function(data){ console.log("inside 'event'") } );
socket.on('disconnect', function(){ console.log("inside 'disconnect'") } );
var payload = {email: 'fake#gmail.com', password: 'tester'};
var tokens = {browserId: 'b965e554-b4d2-5d53-fd69-b2ca5483537a'};
socket.emit("publish", {logic:"user", method:"signIn"}, payload, tokens, function(err, creds) {
console.log("inside the socket client emit callback. err: " + err);
console.log("creds: " + creds);
});
Now for my problem. As I stated in the comment at the top of that code, I can connect to my local nodejs server and get the response I expect when I turn off SSL encryption on my server. As soon as I turn SSL on, I stop getting any response at all from the code above. I don't see any message in my server logs or from the command line, where I'm running the code above with node.
My goal is to be able to run the code above, with SSL turned on in my server, and get the same response that I get when SSL is turned off. I've tried a bunch of variations on the code I included above, such as:
connecting to "https://localhost:3000"
connecting to "//localhost:3000"
connecting to "https://localhost:3443" (this is the port I have to connect to when I have the nodejs server running with SSL)
changing {reconnect:true} to {reconnect:true,secure:true}
I'm truly stumped, and I've been doing a bunch of research on the web and on my node server. It's my company's code and I didn't originally implement the SSL components, so I've spent a few hours looking at our code and trying to understand how adding SSL changes everything. I'm also a student and have about 2 years of experience behind me, so I'm good but I'm no expert. Have I said anything above that indicates if my task is impossible to achieve, or if maybe I have just overlooked something? Any leads on things to check out would be appreciated :)
I am working on Backbone based application (It is also an tablet application), which uses websockets, .
Websockets are working well, but Application doesn't have any logic for error/network handing.
App should show message to user that he is disconnected, and app should retry to connect again, and once its connected back, things starts working again, like gmail.
I haven't written any server-side websocket code, and I am new to Websockets.
Is there any good article on how to handle network disconnection, reconnection for Websockets?
I am unable to find how to set timeout on Websockets, Or how to reconnect again etc.
As its an tablet app, so there will be frequent network disconnections, also App will be getting into sleep mode. Is there any special considerations or practices ?
According to this article, you can use try/catch to treat erroneous cases, like not being able to connect:
function connect(){
try{
var socket;
var host = "ws://localhost:8000/socket/server/startDaemon.php";
var socket = new WebSocket(host);
message('<p class="event">Socket Status: '+socket.readyState);
socket.onopen = function(){
message('<p class="event">Socket Status: '+socket.readyState+' (open)');
}
socket.onmessage = function(msg){
message('<p class="message">Received: '+msg.data);
}
socket.onclose = function(){
message('<p class="event">Socket Status: '+socket.readyState+' (Closed)');
}
} catch(exception){
message('<p>Error'+exception);
}
}
I couldn't find any mention of setting timeout duration even in the WebSocket spec, that might not be possible.