BACKGROUND:
I'm implementing a client that communicates to a server using a WebSocket. The core of the client application is built ontop of LibGDX and it is deployed solely in browser form using GWT (HTML5/JavaScript). This is for a side-project - a fast-paced (high network traffic), online, browser-based game.
Without providing any source - essentially the client code is an empty LibGDX application that provides a correct implementation of gwt-websockets (a commonly recommended GWT wrapper for JavaScript websockets).
SYMPTOMS OF THE PROBLEM:
(pre-note: This appears to be a purely client-side issue) I start the application and connect to the server. On 1000 ms intervals the client application sends a packet to the server and receives a packet back - no problems. There are also no problems when I send and/or receive two, three, or even five packets per second.
Speeding up that process faster however ... approximately sending more than 10 packets per second (100 ms intervals) from the client to the server OR receiving more than 10 packets per second (100 ms intervals) OR both sending and receiving 10 packets per second (100 ms intervals) causes the client to slowly drop in FPS while the packets are pouring in, out, or both. The more network communication, the lower the FPS floor becomes (slowly drains from 60...55..50..45..all the way down to 1 if you keep sending the packets). Meanwhile, the server is completely healthy.
Here's the weird thing which makes me suspect there is some sort of buffer overflow on the client - after a packet has been "handled" (note: my Websocket.onHandle() method is empty), the FPS jumps back up to ~60 as if nothing ever happened. From this point, if a packet is sent or received, the FPS drops right back down to the floor value (except a tad bit worse each time this occurs).
FURTHER DEBUGGING:
I've looked into the possibility of memory leaks on my end, but after
going through a 15 hour debugging session I doubt my code is at fault
here (not to mention it is essentially empty). My network class that
communicates via Websocket contains literally a bare-bones
implementation and the symptoms only occur upon network activity.
I've read about potential garbage collection causing undesirable
performance hits in a GWT deployment. I don't know much about this
other than I cannot rule it out.
I doubt this matters but my server uses Java-WebSocket to
communicate with the client's gwt-websocket
MY BEST GUESS:
I'm really stumped here. My leading suspicion is that there exists some sort of bug in gwt-websockets with buffers and the handling of frequent network traffic or possibly there is a memory leak.
Related
We have a real time application where AppServer( written in c/c++) used to broadcast network event details
(typically few hundred to around thousand rows with around 40-50 columns per second) to Client GUI app written in gtk
over network using XML/RPC, as well as writing to DB(running on same machine or LAN). 95% communication is from server to client.
It was working fine with no issues.
We have to port the application to web. In initial attempt we kept C/C++ app server as it is. Now we are sending the data to java web
application server through xml/rpc. Java server keeps a circular queue (initially kept small size of 2000) and pushes data
to client through websockets. At client we are using angular ui-grid to display the data.
Currently the problem is browser can not handle this amount of data at this frequency and
after some time (few hours) become unresponsive. This is a network monitoring app supposed to be run 24/7 and though we have very few
users, we don't have much control over their machine's configuration (mostly low/medium end). Server (tomacat 8) was running on 2*6 core, 16 GB RAM.
Can you give suggestions for improving browser performance? We are even ready to implement new solution from scratch.
But I think browser performance will be always the bottlneck.
if the pushed data is persisted somewhere else, you should implement a worker to clean the browser data periodically.
For an example say you have been pushing 1000 records per minute, and the browser crashes around one hour period, you could implement a worker where every half an hour, it cleans the browser DOM in order to keep the memory footprint at its lowest.
To access previous data, you should have an API implemented where a user could fetch data for a given time/data period.
it's hard to identify what is causing the crashes, but IMO it should be a heavy DOM which the browser is unable to handle after a certain time. It's better if you could provide an details report of the browser when it crashed.
I play with Javascript AJAX and long-polling.
Try to find best value for server response timeout.
I read many docs but couldn't find a detailed explanation for timeout.
Someone choose 20 secs, other 30 secs...
I use logic like on diagram
How can I choose better value for timeout?
Can I use 5 minutes?
Is it normal practice?
PS: Possible Ajax client internet connections: Ethernet RJ-45, WiFi, 3G, 4G, also, with NAT, Proxy.
I worry about connection can be dropped by third party in some cases by long timeout.
Maybe its your grasp of English which is the problem, but its the lifetime of the connection (time between connection opening and closing) you need to worry about more than the timeout (length of time with no activity after which the connection will be terminated).
Despite the existence of websockets, there is still a lot of deployed hardware which will drop connections regardless of activity (and some which will look for inactivity) where it thinks the traffic is HTTP or HTTPS - sometimes as a design fault, sometimes as a home-grown mitigation to sloloris attacks. That you have 3G and 4G clients means you can probably expect problems with a 5 minute lifespan.
Unfortunately there's no magic solution to knowing what will work universally. The key thing is to know how widely distributed your users are. If they're all on your LAN and connecting directly to the server, then you should be able to use a relatively large value, however setting the duration to unlimited will reveal any memory leaks in your app - sometimes its better to do refresh every now and again anyway.
Taking the case where there is infrastructure other than hubs and switches between your server and the clients, you need to provide a mechanism for detecting and re-establishing a dropped connection regardless of the length of time. When you have worked out how to do this, then:
dropped connections are only a minor performance glitch and do not have a significant effect on the functionality
it's trivial to then add the capability to log dropped connections and thereby determine the optimal connection time to eliminate the small problem described in (1)
Your English is fine.
TL;DR - 5-30s depending on user experience.
I suggest long poll timeouts be 100x the server's "request" time. This makes a strong argument for 5-20s timeouts, depending on your urgency to detect dropped connections and disappeared clients.
Here's why:
Most examples use 20-30 seconds.
Most routers will silently drop connections that stay open too long.
Clients may "disappear" for reasons like network errors or going into low power state.
Servers cannot detect dropped connections. This makes 5 min timeouts a bad practice as they will tie up sockets and resources. This would be an easy DOS attack on your server.
So, < 30 seconds would be "normal". How should you choose?
What is the cost-benefit of making the long-poll connections?
Let's say a regular request takes 100ms of server "request" time to open/close the connection, run a database query, and compute/send a response.
A 10 second timeout would be 10,000 ms, and your request time is 1% of the long-polling time. 100 / 10,000 = .01 = 1%
A 20 second timeout would be 100/20000 = 0.5%
A 30 second timeout = 0.33%, etc.
After 30 seconds, the practical benefit of the longer timeout will always be less than: 0.33% performance improvement. There is little reason for > 30s
Conclusion
I suggest long poll timeouts be 100x the server's "request" time. This makes a strong argument for 5-20s timeouts, depending on your urgency to detect dropped connections and disappeared clients.
Best practice: Configure your client and server to abandon requests at the same timeout. Give the client extra network ping time for safety. E.g. server = 100x request time, client = 102x request time.
Best practice: Long polling is superior to websockets for many/most use cases because of the lack of complexity, more scalable architecture, and HTTP's well-known security attack surface area.
I've a web sockets based chat application (HTML5).
Browser opens a socket connection to a java based web sockets server over wss.
When browser connects to server directly (without any proxy) everything works well.
But when the browser is behind an enterprise proxy, browser socket connection closes automatically after approx 2 minutes of no-activity.
Browser console shows "Socket closed".
In my test environment I have a Squid-Dansguardian proxy server.
IMP: this behaviour is not observed if the browser is connected without any proxy.
To keep some activity going, I embedded a simple jquery script which will make an http GET request to another server every 60 sec. But it did not help. I still get "socket closed" in my browser console after about 2 minutes of no action.
Any help or pointers are welcome.
Thanks
This seems to me to be a feature, not a bug.
In production applications there is an issue related with what is known as "half-open" sockets - see this great blog post about it.
It happens that connections are lost abruptly, causing the TCP/IP connection to drop without informing the other party to the connection. This can happen for many different reasons - wifi signals or cellular signals are lost, routers crash, modems disconnect, batteries die, power outages...
The only way to detect if the socket is actually open is to try and send data... BUT, your proxy might not be able to safely send data without interfering with your application's logic*.
After two minutes, your Proxy assume that the connection was lost and closes the socket on it's end to save resources and allow new connections to be established.
If your proxy didn't take this precaution, on a long enough timeline all your available resources would be taken by dropped connections that would never close, preventing access to your application.
Two minutes is a lot. On Heroku they set the proxy for 50 seconds (more reasonable). For Http connections, these timeouts are often much shorter.
The best option for you is to keep sending websocket data within the 2 minute timeframe.
The Websocket protocol resolves this issue by implementing an internal ping mechanism - use it. These pings should be sent by the server and the browser responds to them with a pong directly (without involving the javascript application).
The Javascript API (at least on the browser) doesn't let you send ping frames (it's a security thing I guess, that prevents people from using browsers for DoS attacks).
A common practice by some developers (which I think to be misconstructed) is to implement a JSON ping message that is either ignored by the server or results in a JSON pong.
Since you are using Java on the server, you have access to the Ping mechanism and I suggest you implement it.
I would also recommend (if you have control of the Proxy) that you lower the timeout to a more reasonable 50 seconds limit.
* The situation during production is actually even worse...
Because there is a long chain of intermediaries (home router/modem, NAT, ISP, Gateways, Routers, Load Balancers, Proxies...) it's very likely that your application can send data successfully because it's still "connected" to one of the intermediaries.
This should start a chain reaction that will only reach the application after a while, and again ONLY if it attempts to send data.
This is why Ping frames expect Pong frames to be returned (meaning the chain of connection is intact.
P.S.
You should probably also complain about the Java application not closing the connection after a certain timeout. During production, this oversight might force you to restart your server every so often or experience a DoS situation (all available file handles will be used for the inactive old connections and you won't have room for new connections).
check the squid.conf for a request_timeout value. You can change this via the request_timeout. This will affect more than just web sockets. For instance, in an environment I frequently work in, a perl script is hit to generate various configurations. Execution can take upwards of 5-10 minutes to complete. The timeout value on both our httpd and the squid server had to be raised to compensate for this.
Also, look at the connect_timeout value as well. That's defaulted to one minute..
I have Node.js(v0.10.26) + Express.js as my web server, which hosts a website and it serves:
Hosting for all resources (both static and non-static ones)
Responding for HTTP requests to get a dynamic page
Restful APIs
I am using jade as template.
I am using node-mysql to read/write db data.
I am running nodejs on a Windows Server 2008 R2 machine.
When I started to run performance test with Apache JMeter with 5 concurrent clients (100 seconds to head-up), I found that:
The one CPU core for the nodejs instance goes up to almost 100%, and always keeps 100%.
The memory usage for nodejs instance started with about 90MB, and goes up slowly with about 10MB per day.
When I stopped the JMeter from sending requests, the memory usage went down quickly to normal status: about 90MB.
The requests count per seconds continues to go down, at the beginning I have about 12 requests per second, but it falls to 8 requests per second after 2 days running.
and during the performance test, there is no error occurs.
My question is:
Why the node.js instance memory goes up slowly? I am supposing it to goes up for a while and keep stable at a level.
If there is any memory leak, why the memory goes down after shutting down the JMeter?
Why the requests count per second falls?
I made a game with node.js and uploaded it.. it should be 30 frames per second..
The game is hosted in AppFog.
Now I testing it and it runs slow.. I mean in local network it runs a lot smoother
I do understand the problem -
The server is sending a message to the client for every frame, it's basically impossible --
network latencies below 16 ms (60 fps) or 33 ms (30 fps) over the internet are impossible to reliably guarantee. It might work over a local network, and almost certainly can work on a local machine, but not over the internet. If I need to get information from the client to the server and back in the next frame, my round-trip latency needs to be that low, including the processing on both sides. Since both xhr-polling and websockets use TCP, it gets worse because one slow/lost packet will pause everything after it until it finally shows up.
So I am in a pretty bad situation..
The game is set with xhr-pooling like this -
io.set('transports', ['xhr-polling']);
which use TCP and it's very problematic because one slow/lost packet will pause the game..
So how can I make the game smoother? any ideas? I really want to make it work out!
Thanks in advance