Develop Reference

How to send and receive data for JS games?

When I'm on an html based website I can send POST and GET requests how do I do the equivalent in a JS game? One of the most popular games may be agar.io. So if we use that as an example - How are details about my game state (position, size, etc) being sent to the server and how is my browser receiving information about other player's game states? Where can I see this information? When I inspect the page under the network tab in google chrome as I normally do to view GET and POST requests there's no activity for agar.io game play.

For GET and POST requests:
It could only get info at the moment from the server, and could never be real-time. Thus it would nearly be impossible to view your friends' game status unless you are sending ajax requests with an interval in a number of seconds.
HOWEVER, loading ajax every 2 seconds, let's say, will result in a big load on the server. Unless you have a server network which could handle hundreds of requests simultaneously, otherwise you are strongly adviced not to use this method.
Well, then, how to have real-time communications with your friends?
WebRTC
It is a technology developed by Google in 2013. Search on the web for more information on how to make good use of it to develop real-time communications.
Sadly this one is only supported by mainly mozilla and webkit browsers.
Push APIs
There are a lot of APIs regarding push notifications. A typical example using this will be Facebook, which receive messages real-time, or immediately after message being sent.

Service workers and page performance

I'm stuck at a wedding reception that I really don't want to be at and I'm driving, so obviously I'm reading about service workers. I'm on my phone so can't play about with anything but was thinking if they're a viable option for improving page performance?
Images are the biggest killer on my site and I'm half thinking we could use a service worker to cache them to help get page load times down. From what I can tell, the browser still makes the http request, it's just the response is from the SW cache, not the file location. Am I missing something here? Is there therefore any actual benefit to doing this?

While the regular http cache has a lot of overlap with ServiceWorker cache, one thing that the former can't handle very well is the dynamically generated html used in many client-side javascript applications.
Even when all the resources of the app are cache hits, there is still the delay as the javascript is compiled and executed before the app is usable.
Addy Osmani has demonstrated how ServiceWorker can be used to cache the Shell of an app. When the DOM is modified on the client, it is updated in the cache. The next time that URL is requested, the ServiceWorker replies with html that is ready for use before the app has booted.
The other advantage regards lie-fi: when it seems the network is available, but not enough packets are getting through. ServiceWorkers can afford to have a near-imperceptible timeout, because they can serve immediately from cache and wait for the response to load (if ever).

Your consideration is invalid.
Service worker is designed to work like a proxy server that can especially handle some off-page operations like offline ability, push notification, background synchronization, etc. So in your case, you will gain no performance benefits by caching images with service worker over the traditional browser's cache approach.

Real-time pubsub chat with history via websockets

I'm interested in creating what Disqus have done with their commenting system: http://highscalability.com/blog/2014/5/7/update-on-disqus-its-still-about-realtime-but-go-demolishes.html
The most impressive part of infrastructure is Nginx push stream module:
Still runs on 5 machines Nginx machines.
Uses NginxPushStream, which supprts EventSource, WebSocket, Long
Polling, and Forever Iframe.
All users are connected to these machines.
On a normal day each machine sees 3200 connections/s, 1 million
connections, 150K packets/s TX and 130K packets/s RX, 150 mbits/s TX
and 80 mbits/s RC, with <15ms delay end-to-end (which is faster than
Javascript can render a comment)
Had many issues with resource exhaustion at first. The configuration
for Nginx and the OS are given that help alleviate the problems,
tuning them to handle a scenario with many connections moving little
data.
Obviously this Nginx module doesn't support data storage. Only in-memory mechanism supported by push_stream_store_messages directive, but as author said:
The main target to stored messages is to deliver the message to a
subscriber that was offline when the message was published.
It is clear that Disqus don't publish messages to Nginx directly but rather though the Go backend that manages to store messages in Redis followed by publishing via internal POST to Nginx which keeps subscribers.
Does anyone have an experience with fetching messages history through Redis on a page load before even we use push stream module for newer messages? Do you have to push old messages history in one go or render as plain HTML followed by pubsub for new messages that appear after page is loaded?
The logic needs to be decoupled as much as possible. I don't intend to introduce blocking mechanism between user and Nginx for real time message communication. Would that be a good solution below?
Client pushes the message from the web page (via websockets)
Ajax request goes straight to push stream location and on Ajax complete callback it requests backend to store message in Redis (in the other direction it gets locked with backend)
Once user refreshes the page backend fetches Redis list and display the history
User can see the history and can post new messages
It requires only 2 backend requests to be developed: accept message and store in Redis, fetch the data and display on page load. Preferably light non-blocking backend required, like Lua module or even HTTP interface to Redis called Webdis.
I'd like to know smart people opinion on that mechanism from architecture point of view, no code example expected.

Server sent events and browser limits

I have a web application that listens for Server Sent Events. While I was working and testing with multiple windows open, things were not working and I banged my head for several times looking in the wrong direction: eventually, I realized that the problem was concurrent connections.
However I was testing a very limited number and even if I am running the test on Apache (I know, I should use node).
I then, switched browser and noticed something really interesting: apparently Chrome limits Server Sent Events connections to 4-5, while Opera doesn't. Firefox, on the other hand, after 4-5 simultaneous connections, refuses to load any other page.
What is the reason behind this? Does the limit only apply to SSE connections from the same source, or would it be the same if I were to test open them from a different domain? Is there any chance that I am misusing SSE and this is actually blocking the browsers, or this is a known behaviour? Is there any way around it?

The way this works in all browsers are that each domain gets a limited amount of connections and the limits are global for your whole application. That means if you have one connection open for realtime communication you have one less for loading images, CSS and other pages. On top of that you don't get new connections for new tabs or windows, all of them needs to share the same amount of connections. This is very frustrating but there are good reasons for limiting the connections. A few years back, this limit was 2 in all browsers (based on the rules in (http://www.ietf.org/rfc/rfc2616.txt) HTTP1.1 spec) but now most browsers use 4-10 connections in general. Mobile browsers on the other hand still needs to limit the amount of connections for battery saving purposes.
These tricks are available:
Use more host names. By assigning ex. www1.example.com, www2.example.com you get new connections for each host name. This trick works in all browsers. Don't forget to change the cookie domain to include the whole domain (example.com, not www.example.com)
Use web sockets. Web sockets are not limited by these restrictions and more importantly they are not competing with the rest of your websites content.
Reuse the same connection when you open new tabs/windows. If you have gathered all realtime communication logic to an object call Hub you can recall that object on all opened windows like this:
window.hub = window.opener ? window.opener.hub || new Hub()
4. or use flash - not quite the best advice these days but it might still be an option if websockets aren't an option.
5. Remember to add a few seconds of time between each SSE request to let queued requests to be cleared before starting a new one. Also add a little more waiting time for each second the user is inactive, that way you can concentrate your server resources on those users that are active. Also add a random number of delay to avoid the Thundering Herd Problem
Another thing to remember when using a multithreaded and blocking language such as Java or C# you risk using resources in your long polling request that are needed for the rest of your application. For example in C# each request locks the Session object which means that the whole application is unresponsive during the time a SSE request is active.
NodeJs is great for these things for many reasons as you have already figured out and if you were using NodeJS you would have used socket.io or engine.io that takes care of all these problems for you by using websockets, flashsockets and XHR-polling and also because it is non blocking and single threaded which means it will consume very little resources on the server when it is waiting for things to send. A C# application consumes one thread per waiting request which takes at least 2MB of memory just for the thread.

One way to get around this issue is to shut down the connections on all the hidden tabs, and reconnect when the user visits a hidden tab.
I'm working with an application that uniquely identifies users which allowed me to implement this simple work-around:
When users connect to sse, store their identifier, along with a timestamp of when their tab loaded. If you are not currently identifying users in your app, consider using sessions & cookies.
When a new tab opens and connects to sse, in your server-side code, send a message to all other connections associated with that identifier (that do not have the current timestamp) telling the front-end to close down the EventSource. The front-end handler would look something like this:
myEventSourceObject.addEventListener('close', () => {
myEventSourceObject.close();
myEventSourceObject = null;
});
Use the javascript page visibility api to check to see if an old tab is visible again, and re-connect that tab to the sse if it is.
document.addEventListener('visibilitychange', () => {
if (!document.hidden && myEventSourceObject === null) {
// reconnect your eventsource here
}
});
If you set up your server code like step 2 describes, on re-connect, the server-side code will remove all the other connections to the sse. Hence, you can click between your tabs and the EventSource for each tab will only be connected when you are viewing the page.
Note that the page visibility api isn't available on some legacy browsers:
https://caniuse.com/#feat=pagevisibility

2022 Update
This problem has been fixed in HTTP/2.
According to mozilla docs:-
When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6).
The issue has been marked as "Won't fix" in Chrome and Firefox.
This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.1.example and another 6 SSE connections to www.2.example (per Stackoverflow).
When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100).
Spring Boot 2.1+ ships by default with Tomcat 9.0.x which supports HTTP/2 out of the box when using JDK 9 or later.
If you are using any other backend, please enable http/2 to fix this issue.

You are right about the number of simultaneous connections.
You can check this list for max values: http://www.browserscope.org/?category=network
And unfortunately, I never found any work around, except multiplexing and/or using different hostnames.

Prevent recursive calls of XmlHttpRequest to server

I've been googling for hours for this issue, but did not find any solution.
I am currently working on this app, built on Meteor.
Now the scenario is, after the website is opened and all the assets have been loaded in browser, the browser constantly makes recursive xhr calls to server. These calls are made at the regular interval of 25 seconds.
This can be seen in the Network tab of browser console. See the Pending request of the last row in image.
I can't figure out from where it originates, and why it is invoked automatically even when the user is idle.
Now the question is, How can I disable these automatic requests? I want to invoke the requests manually, i.e. when the menu item is selected, etc.
Any help will be appriciated.
[UPDATE]
In response to the Jan Dvorak's comment:
When I type "e" in the search box, the the list of events which has name starting with letter "e" will be displayed.
The request goes with all valid parameters and the Payload like this:
["{\"msg\":\"sub\",\"id\":\"8ef5e419-c422-429a-907e-38b6e669a493\",\"name\":\"event_Coll_Search_by_PromoterName\",\"params\":[\"e\"]}"]
And this is the response, which is valid.
a["{\"msg\":\"data\",\"subs\":[\"8ef5e419-c422-429a-907e-38b6e669a493\"]}"]
The code for this action is posted here
But in the case of automatic recursive requests, the request goes without the payload and the response is just a letter "h", which is strange. Isn't it? How can I get rid of this.?

Meteor has a feature called
Live page updates.
Just write your templates. They automatically update when data in the database changes. No more boilerplate redraw code to write. Supports any templating language.
To support this feature, Meteor needs to do some server-client communication behind the scenes.
Traditionally, HTTP was created to fetch dead data. The client tells the server it needs something, and it gets something. There is no way for the server to tell the client it needs something. Later, it became needed to push some data to the client. Several alternatives came to existence:
polling:
The client makes periodic requests to the server. The server responds with new data or says "no data" immediately. It's easy to implement and doesn't use much resources. However, it's not exactly live. It can be used for a news ticker but it's not exactly good for a chat application.
If you increase the polling frequency, you improve the update rate, but the resource usage grows with the polling frequency, not with the data transfer rate. HTTP requests are not exactly cheap. One request per second from multiple clients at the same time could really hurt the server.
hanging requests:
The client makes a request to the server. If the server has data, it sends them. If the server doesn't have data, it doesn't respond until it does. The changes are picked up immediately, no data is transferred when it doesn't need to be. It does have a few drawbacks, though:
If a web proxy sees that the server is silent, it eventually cuts off the connection. This means that even if there is no data to send, the server needs to send a keep-alive response anyways to make the proxies (and the web browser) happy.
Hanging requests don't use up (much) bandwidth, but they do take up memory. Nowadays' servers can handle multiple concurrent TCP connections, so it's less of an issue than it was before. What does need to be considered is the amount of memory associated with the threads holding on to these requests - especially when the connections are tied to specific threads serving them.
Browsers have hard limits on the number of concurrent requests per domain and in total. Again, this is less of a concern now than it was before. Thus, it seems like a good idea to have one hanging request per session only.
Managing hanging requests feels kinda manual as you have to make a new request after each response. A TCP handshake takes some time as well, but we can live with a 300ms (at worst) refractory period.
Chunked response:
The client creates a hidden iFrame with a source corresponding to the data stream. The server responds with an HTTP response header immediately and leaves the connection open. To send a message, the server wraps it in a pair of <script></script> tags that the browser executes when it receives the closing tag. The upside is that there's no connection reopening but there is more overhead with each message. Moreover, this requires a callback in the global scope that the response calls.
Also, this cannot be used with cross-domain requests as cross-domain iFrame communication presents its own set of problems. The need to trust the server is also a challenge here.
Web Sockets:
These start as a normal HTTP connection but they don't actually follow the HTTP protocol later on. From the programming point of view, things are as simple as they can be. The API is a classic open/callback style on the client side and the server just pushes messages into an open socket. No need to reopen anything after each message.
There still needs to be an open connection, but it's not really an issue here with the browser limits out of the way. The browser knows the connection is going to be open for a while, so it doesn't need to apply the same limits as to normal requests.
These seem like the ideal solution, but there is one major issue: IE<10 doesn't know them. As long as IE8 is alive, web sockets cannot be relied upon. Also, the native Android browser and Opera mini are out as well (ref.).
Still, web sockets seem to be the way to go once IE8 (and IE9) finally dies.
What you see are hanging requests with the timeout of 25 seconds that are used to implement the live update feature. As I already said, the keep-alive message ("h") is used so that the browser doesn't think it's not going to get a response. "h" simply means "nothing happens".
Chrome supports web sockets, so Meteor could have used them with a fallback to long requests, but, frankly, hanging requests are not at all bad once you've got them implemented (sure, the browser connection limit still applies).